pax_global_header00006660000000000000000000000064145345310240014513gustar00rootroot0000000000000052 comment=b51db34ab61e52a8256d085212de68d10e55e6f1 optuna-3.5.0/000077500000000000000000000000001453453102400130265ustar00rootroot00000000000000optuna-3.5.0/.coveragerc000066400000000000000000000000741453453102400151500ustar00rootroot00000000000000[run] concurrency = multiprocessing,thread source = optuna/ optuna-3.5.0/.dockerignore000066400000000000000000000001131453453102400154750ustar00rootroot00000000000000# Ignore everything ** !pyproject.toml !README.md !optuna **/__pycache__ optuna-3.5.0/.github/000077500000000000000000000000001453453102400143665ustar00rootroot00000000000000optuna-3.5.0/.github/FUNDING.yml000066400000000000000000000000171453453102400162010ustar00rootroot00000000000000github: optuna optuna-3.5.0/.github/ISSUE_TEMPLATE/000077500000000000000000000000001453453102400165515ustar00rootroot00000000000000optuna-3.5.0/.github/ISSUE_TEMPLATE/bug-report.yml000066400000000000000000000041461453453102400213670ustar00rootroot00000000000000name: "\U0001F41BBug report" description: Create a report to help us improve Optuna. labels: ["bug"] body: - type: markdown attributes: value: | Thanks for taking the time to fill out this bug report! Please write a clear and concise description of what the bug is. - type: textarea id: expected-behavior attributes: label: Expected behavior description: Please write a clear and concise description of what you expected to happen. validations: required: true - type: textarea id: environment attributes: label: Environment description: | Please give us your environment information. You can get this information by typing the following. ``` python3 -c 'import optuna; print(f"- Optuna version:{optuna.__version__}")' python3 -c 'import platform; print(f"- Python version:{platform.python_version()}")' python3 -c 'import platform; print(f"- OS:{platform.platform()}")' ``` value: | - Optuna version: - Python version: - OS: - (Optional) Other libraries and their versions: validations: required: true - type: textarea id: logs attributes: label: Error messages, stack traces, or logs description: Please copy and paste any relevant error messages, stack traces, or log output. render: shell validations: required: true - type: textarea id: steps-to-reproduce attributes: label: Steps to reproduce description: Please provide how we reproduce your reported bugs. If possible, it is highly recommended to provide the reproducible example codes. Note that pickle files will not be accepted due to possible [security issues](https://docs.python.org/3/library/pickle.html). value: | 1. 2. 3. ```python # python code ``` validations: required: true - type: textarea id: additional-context attributes: label: Additional context (optional) description: Please provide additional contexts if you have. validations: required: false optuna-3.5.0/.github/ISSUE_TEMPLATE/code-fix.yml000066400000000000000000000020161453453102400207710ustar00rootroot00000000000000name: "\U0001F6A7Code fix" description: Suggest a code fix that does not change the behaviors of Optuna, such as code refactoring. labels: ["code-fix"] body: - type: markdown attributes: value: | Thanks for taking the time to raise a new issue! Please write a clear and concise description of the code fix. - type: textarea id: motivation attributes: label: Motivation description: | Please write the motivation for the proposal. If your code fix is related to a problem, please describe a clear and concise description of what the problem is. validations: required: true - type: textarea id: suggestion attributes: label: Suggestion description: Please explain your suggestion for the code change. validations: required: true - type: textarea id: additional-context attributes: label: Additional context (optional) description: Please provide additional contexts if you have. validations: required: false optuna-3.5.0/.github/ISSUE_TEMPLATE/config.yml000066400000000000000000000000341453453102400205360ustar00rootroot00000000000000blank_issues_enabled: false optuna-3.5.0/.github/ISSUE_TEMPLATE/documentation.yml000066400000000000000000000006511453453102400221470ustar00rootroot00000000000000name: "\U0001f4d6Documentation" description: Report an issue related to https://optuna.readthedocs.io/. labels: ["document"] body: - type: textarea id: explanation attributes: label: What is an issue? description: Thanks for taking time to raise an issue! Please write a clear and concise description of what content in https://optuna.readthedocs.io/ is an issue. validations: required: true optuna-3.5.0/.github/ISSUE_TEMPLATE/feature-request.yml000066400000000000000000000024141453453102400224160ustar00rootroot00000000000000name: "\U0001F4A1Feature request" description: Suggest an idea for Optuna. labels: ["feature"] body: - type: markdown attributes: value: | Thanks for taking the time for your feature requests! Please write a clear and concise description of the feature proposal. - type: textarea id: motivation attributes: label: Motivation description: | Please write the motivation for the proposal. If your feature request is related to a problem, please describe a clear and concise description of what the problem is. validations: required: true - type: textarea id: description attributes: label: Description description: Please write a detailed description of the new feature. validations: required: true - type: textarea id: alternatives attributes: label: Alternatives (optional) description: Please write a clear and concise description of any alternative solutions or features you've considered. validations: required: false - type: textarea id: additional-context attributes: label: Additional context (optional) description: Please add any other context or screenshots about the feature request here. validations: required: false optuna-3.5.0/.github/ISSUE_TEMPLATE/questions-help-support.yml000066400000000000000000000006751453453102400237760ustar00rootroot00000000000000name: "\U00002753Questions" description: Don't use GitHub Issues to ask support questions. body: - type: markdown attributes: value: | # PLEASE USE GITHUB DISCUSSIONS Don't use GitHub Issues to ask support questions. Use the [GitHub Discussions](https://github.com/optuna/optuna/discussions/categories/q-a) for that. - type: textarea attributes: label: "Don't use GitHub Issues to ask support questions." optuna-3.5.0/.github/PULL_REQUEST_TEMPLATE.md000066400000000000000000000012211453453102400201630ustar00rootroot00000000000000 ## Motivation ## Description of the changes optuna-3.5.0/.github/labeler.yml000066400000000000000000000006041453453102400165170ustar00rootroot00000000000000optuna.importance: - optuna/importance/**/* optuna.integration: - optuna/integration/**/* optuna.pruners: - optuna/pruners/**/* optuna.samplers: - optuna/samplers/**/* optuna.storages: - optuna/storages/**/* optuna.testing: - optuna/testing/**/* optuna.visualization: - optuna/visualization/**/* optuna.study: - optuna/study.py optuna.trial: - optuna/trial/**/* optuna-3.5.0/.github/workflows/000077500000000000000000000000001453453102400164235ustar00rootroot00000000000000optuna-3.5.0/.github/workflows/checks-integration.yml000066400000000000000000000035201453453102400227270ustar00rootroot00000000000000name: Checks (Integration) on: push: branches: - master schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: checks: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: 3.8 - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev libopenblas-dev # TODO(Shinichi): Remove the version constraint on mypy. - name: Install run: | python -m pip install -U pip pip install --progress-bar off -U .[benchmark] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off -U .[checking] pip install --progress-bar off -U .[integration] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off -U .[optional] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off -U .[test] pip install --progress-bar off -U bayesmark pip install --progress-bar off -U kurobako pip install --progress-bar off -U "mypy<1.7.0" - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: black run: black . --check --diff - name: flake8 run: flake8 . - name: isort run: isort . --check --diff - name: mypy run: mypy . --warn-unused-ignores - name: blackdoc run: blackdoc . --check --diff optuna-3.5.0/.github/workflows/checks.yml000066400000000000000000000020771453453102400204140ustar00rootroot00000000000000name: Checks on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: checks: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: 3.8 - name: Install run: | python -m pip install -U pip pip install --progress-bar off -U .[checking] - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: black run: black . --check --diff - name: flake8 run: flake8 . - name: isort run: isort . --check --diff - name: mypy run: mypy . - name: blackdoc run: blackdoc . --check --diff optuna-3.5.0/.github/workflows/coverage.yml000066400000000000000000000053541453453102400207500ustar00rootroot00000000000000name: Coverage on: push: branches: - master pull_request: {} concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: coverage: runs-on: ubuntu-latest # Not intended for forks. if: github.repository == 'optuna/optuna' steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.8 uses: actions/setup-python@v4 with: python-version: 3.8 - name: Setup cache uses: actions/cache@v3 env: cache-name: coverage with: path: ~/.cache/pip key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev libopenblas-dev - name: Install run: | python -m pip install --upgrade pip # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[integration] --extra-index-url https://download.pytorch.org/whl/cpu echo 'import coverage; coverage.process_startup()' > sitecustomize.py - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests env: OMP_NUM_THREADS: 1 PYTHONPATH: . # To invoke sitecutomize.py COVERAGE_PROCESS_START: .coveragerc # https://coverage.readthedocs.io/en/6.4.1/subprocess.html COVERAGE_COVERAGE: yes # https://github.com/nedbat/coveragepy/blob/65bf33fc03209ffb01bbbc0d900017614645ee7a/coverage/control.py#L255-L261 run: | coverage run --source=optuna -m pytest tests -m "not skip_coverage and not slow" coverage combine coverage xml - name: Multi-node tests env: OMP_NUM_THREADS: 1 PYTHONPATH: . COVERAGE_PROCESS_START: .coveragerc COVERAGE_COVERAGE: yes run: | export OMPI_MCA_rmaps_base_oversubscribe=yes mpirun -n 2 -- coverage run -m pytest tests/integration_tests/test_pytorch_distributed.py coverage combine --append coverage xml - name: Upload coverage to Codecov uses: codecov/codecov-action@v3 with: file: ./coverage.xml optuna-3.5.0/.github/workflows/dockerimage.yml000066400000000000000000000050721453453102400214240ustar00rootroot00000000000000name: Build Docker Image on: push: branches: - master release: types: [published] pull_request: paths: - .github/workflows/dockerimage.yml - Dockerfile - .dockerignore - pyproject.toml - .github/workflows/dockerimage.yml concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true env: DOCKER_HUB_BASE_NAME: optuna/optuna jobs: dockerimage: runs-on: ubuntu-latest # TODO (nzw0301): Remove `exclude` once integration works with Python 3.11. strategy: matrix: python_version: ['3.7', '3.8', '3.9', '3.10', '3.11'] build_type: ['', 'dev'] # "dev" installs all the dependencies including pytest. exclude: - python_version: '3.7' build_type: 'dev' - python_version: '3.11' build_type: 'dev' # This action cannot be executed in the forked repository. if: github.repository == 'optuna/optuna' steps: - uses: actions/checkout@v3 - name: Set ENV run: | export TAG_NAME="py${{ matrix.python_version }}" if [ "${{ github.event_name }}" = 'release' ]; then export TAG_NAME="${{ github.event.release.tag_name }}-${TAG_NAME}" fi if [ "${{matrix.build_type}}" = 'dev' ]; then export TAG_NAME="${TAG_NAME}-dev" fi echo "HUB_TAG=${DOCKER_HUB_BASE_NAME}:${TAG_NAME}" >> $GITHUB_ENV - name: Build the Docker image run: | if [ "${{ github.event_name }}" = 'release' ]; then # Cache is not available because the image tag includes the Optuna version. CACHE_FROM="" else CACHE_FROM="--cache-from=${HUB_TAG}" fi docker build ${CACHE_FROM} . --build-arg PYTHON_VERSION=${{ matrix.python_version }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --file Dockerfile --tag "${HUB_TAG}" env: DOCKER_BUILDKIT: 1 - name: Output installed packages run: | docker run "${HUB_TAG}" sh -c "pip freeze --all" - name: Output dependency tree run: | docker run "${HUB_TAG}" sh -c "pip install pipdeptree && pipdeptree" - name: Verify the built image run: | docker run "${HUB_TAG}" optuna --version - name: Login & Push to Docker Hub if: ${{ github.event_name != 'pull_request' }} env: DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }} run: | echo "${DOCKER_HUB_TOKEN}" | docker login -u optunabot --password-stdin docker push "${HUB_TAG}" optuna-3.5.0/.github/workflows/labeler.yml000066400000000000000000000003341453453102400205540ustar00rootroot00000000000000name: Labeler on: pull_request_target: types: [opened, reopened] jobs: triage: runs-on: ubuntu-latest steps: - uses: actions/labeler@v4 with: repo-token: "${{ secrets.GITHUB_TOKEN }}" optuna-3.5.0/.github/workflows/mac-tests.yml000066400000000000000000000075521453453102400210570ustar00rootroot00000000000000# Run tests and integration tests on Mac, which are triggered by each master push. # Currently, Python3.8 is only used as an environment. # This is mainly for the sake of speed. name: mac-tests on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-mac: runs-on: macos-latest # Scheduled Tests are disabled for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.8 uses: actions/setup-python@v4 with: python-version: 3.8 - name: Setup cache uses: actions/cache@v3 env: cache-name: test with: path: ~/Library/Caches/pip key: ${{ runner.os }}-3.8-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.8-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest tests -m "not integration" - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest tests -m "not integration and not slow" tests-integration-mac: runs-on: macos-latest # Scheduled Tests are disabled for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.8 uses: actions/setup-python@v4 with: python-version: 3.8 - name: Setup cache uses: actions/cache@v3 env: cache-name: test-integration with: path: ~/Library/Caches/pip key: ${{ runner.os }}-3.8-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.8-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup mac environment run: | brew install libomp brew install open-mpi brew install openblas - name: Install run: | python -m pip install --upgrade pip # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] pip install --progress-bar off .[integration] - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests run: | pytest tests -m "integration" env: OMP_NUM_THREADS: 1 - name: Tests MPI run: | export OMPI_MCA_rmaps_base_oversubscribe=yes mpirun -n 2 -- pytest tests/integration_tests/test_pytorch_distributed.py env: OMP_NUM_THREADS: 1 optuna-3.5.0/.github/workflows/performance-benchmarks-bayesmark.yml000066400000000000000000000110101453453102400255270ustar00rootroot00000000000000name: Performance benchmarks with bayesmark on: workflow_dispatch: inputs: sampler-list: description: 'Sampler List: A list of samplers to check the performance. Should be a whitespace-separated list of Optuna samplers. Each sampler must exist under `optuna.samplers` or `optuna.integration`.' required: false default: 'TPESampler CmaEsSampler' sampler-kwargs-list: description: 'Sampler Arguments List: A list of sampler keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{"multivariate":true,"constant_liar":true} {}' pruner-list: description: 'Pruner List: A list of pruners to check the performance. Should be a whitespace-separated list of Optuna pruners. Each pruner must exist under `optuna.pruners`.' required: false default: 'NopPruner' pruner-kwargs-list: description: 'Pruner Arguments List: A list of pruner keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{}' budget: description: 'Number of Trials if the pruning is not enabled. If the pruning is enabled, the total number of steps is equal to `budget * (steps per trial)`.' required: false default: '80' n-runs: description: 'Number of Studies' required: false default: '10' plot-warmup: description: Include warm-up steps in plots. type: boolean default: true jobs: benchmarks: runs-on: ubuntu-latest strategy: matrix: dataset: [breast, digits, iris, wine, diabetes] model: [kNN, SVM, DT, RF, MLP-sgd, ada, linear] steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.9 uses: actions/setup-python@v4 with: python-version: 3.9 - name: Install Python libraries run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools wheel pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[benchmark] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off bayesmark matplotlib pandas - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Run performance benchmark run: | python benchmarks/run_bayesmark.py \ --dataset ${{ matrix.dataset }} \ --model ${{ matrix.model }} \ --name-prefix "" \ --budget ${{ github.event.inputs.budget }} \ --n-runs ${{ github.event.inputs.n-runs }} \ --sampler-list '${{ github.event.inputs.sampler-list }}' \ --sampler-kwargs-list '${{ github.event.inputs.sampler-kwargs-list }}' \ --pruner-list '${{ github.event.inputs.pruner-list }}' \ --pruner-kwargs-list '${{ github.event.inputs.pruner-kwargs-list }}' \ --plot-warmup '${{ github.event.inputs.plot-warmup }}' - name: Upload plot uses: actions/upload-artifact@v3 with: name: benchmark-plots path: plots - name: Upload partial report uses: actions/upload-artifact@v3 with: name: partial-reports path: partial report: needs: benchmarks runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@master - name: Download partial reports uses: actions/download-artifact@v2 with: name: partial-reports path: partial - name: Setup Python3.9 uses: actions/setup-python@v4 with: python-version: 3.9 - name: Install Python libraries run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools wheel pip install --progress-bar off numpy scipy pandas Jinja2 - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Run benchmark report builder run: | python benchmarks/bayesmark/report_bayesmark.py - name: Upload report uses: actions/upload-artifact@v3 with: name: benchmark-report path: report - name: Cleanup partial reports uses: geekyeggo/delete-artifact@v1 with: name: partial-reports optuna-3.5.0/.github/workflows/performance-benchmarks-kurobako.yml000066400000000000000000000125051453453102400254000ustar00rootroot00000000000000name: Performance Benchmarks with kurobako on: workflow_dispatch: inputs: sampler-list: description: 'Sampler List: A list of samplers to check the performance. Should be a whitespace-separated list of Optuna samplers. Each sampler must exist under `optuna.samplers` or `optuna.integration`.' required: false default: 'RandomSampler TPESampler' sampler-kwargs-list: description: 'Sampler Arguments List: A list of sampler keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{} {\"multivariate\":true,\"constant_liar\":true}' pruner-list: description: 'Pruner List: A list of pruners to check the performance. Should be a whitespace-separated list of Optuna pruners. Each pruner must exist under `optuna.pruners`.' required: false default: 'NopPruner' pruner-kwargs-list: description: 'Pruner Arguments List: A list of pruner keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{}' budget: description: 'Number of Trials if the pruning is not enabled. If the pruning is enabled, the total number of steps is equal to `budget * (steps per trial)`.' required: false default: '80' n-runs: description: 'Number of Studies' required: false default: '10' n-concurrency: description: 'Number of Concurrent Trials' required: false default: '1' jobs: performance-benchmarks-kurobako: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.9 uses: actions/setup-python@v4 with: python-version: 3.9 - name: Install gnuplot run: | sudo apt update sudo apt -y install gnuplot - name: Setup cache uses: actions/cache@v3 env: # Caches them under a common name so that they can be used by other performance benchmark. cache-name: performance-benchmarks with: path: ~/.cache/pip key: ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install Python libraries run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[benchmark] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off kurobako - name: Cache kurobako CLI id: cache-kurobako uses: actions/cache@v3 with: path: ./kurobako key: kurobako-0-2-9 - name: Install kurobako CLI if: steps.cache-kurobako.outputs.cache-hit != 'true' run: | curl -L https://github.com/optuna/kurobako/releases/download/0.2.9/kurobako-0.2.9.linux-amd64 -o kurobako chmod +x kurobako ./kurobako -h - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Cache hpobench dataset id: cache-hpobench-dataset uses: actions/cache@v3 with: path: ./fcnet_tabular_benchmarks key: hpobench-dataset - name: Download hpobench dataset if: steps.cache-hpobench-dataset.outputs.cache-hit != 'true' run: | wget http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz tar xf fcnet_tabular_benchmarks.tar.gz - name: Cache nasbench dataset id: cache-nasbench-dataset uses: actions/cache@v3 with: path: ./nasbench_full.bin key: nasbench-dataset # Ref: https://github.com/optuna/kurobako/wiki/NASBench - name: Download nasbench dataset if: steps.cache-nasbench-dataset.outputs.cache-hit != 'true' run: | curl -L $(./kurobako dataset nasbench url) -o nasbench_full.tfrecord ./kurobako dataset nasbench convert nasbench_full.tfrecord nasbench_full.bin - name: Run performance benchmark run: | python benchmarks/run_kurobako.py \ --path-to-kurobako "." \ --name-prefix "" \ --budget ${{ github.event.inputs.budget }} \ --n-runs ${{ github.event.inputs.n-runs }} \ --n-jobs 10 \ --n-concurrency ${{ github.event.inputs.n-concurrency }} \ --sampler-list '${{ github.event.inputs.sampler-list }}' \ --sampler-kwargs-list '${{ github.event.inputs.sampler-kwargs-list }}' \ --pruner-list '${{ github.event.inputs.pruner-list }}' \ --pruner-kwargs-list '${{ github.event.inputs.pruner-kwargs-list }}' \ --seed 0 \ --data-dir "." \ --out-dir "out" - uses: actions/upload-artifact@v3 with: name: benchmark-report path: | out/report.md out/**/*.png - uses: actions/download-artifact@v2 with: name: benchmark-report path: | out/report.md out/**/*.png optuna-3.5.0/.github/workflows/performance-benchmarks-mo-kurobako.yml000066400000000000000000000105571453453102400260160ustar00rootroot00000000000000name: Performance Benchmarks with kurobako for multi-objective optimization on: workflow_dispatch: inputs: sampler-list: description: 'Sampler List: A list of samplers to check the performance. Should be a whitespace-separated list of Optuna samplers. Each sampler must exist under `optuna.samplers` or `optuna.integration`.' required: false default: 'RandomSampler TPESampler NSGAIISampler' sampler-kwargs-list: description: 'Sampler Arguments List: A list of sampler keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{} {\"multivariate\":true,\"constant_liar\":true} {\"population_size\":20}' budget: description: 'Number of Trials if the pruning is not enabled. If the pruning is enabled, the total number of steps is equal to `budget * (steps per trial)`.' required: false default: '120' n-runs: description: 'Number of Studies' required: false default: '10' n-concurrency: description: 'Number of Concurrent Trials' required: false default: '1' jobs: performance-benchmarks-mo-kurobako: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.9 uses: actions/setup-python@v4 with: python-version: 3.9 - name: Install gnuplot run: | sudo apt update sudo apt -y install gnuplot - name: Setup cache uses: actions/cache@v3 env: # Caches them under a common name so that they can be used by other performance benchmark. cache-name: performance-benchmarks with: path: ~/.cache/pip key: ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install Python libraries run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[benchmark] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off kurobako - name: Cache kurobako CLI id: cache-kurobako uses: actions/cache@v3 with: path: ./kurobako key: kurobako-0-2-9 - name: Install kurobako CLI if: steps.cache-kurobako.outputs.cache-hit != 'true' run: | curl -L https://github.com/optuna/kurobako/releases/download/0.2.9/kurobako-0.2.9.linux-amd64 -o kurobako chmod +x kurobako ./kurobako -h - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Cache nasbench dataset id: cache-nasbench-dataset uses: actions/cache@v3 with: path: ./nasbench_full.bin key: nasbench-dataset # Ref: https://github.com/optuna/kurobako/wiki/NASBench - name: Download nasbench dataset if: steps.cache-nasbench-dataset.outputs.cache-hit != 'true' run: | curl -L $(./kurobako dataset nasbench url) -o nasbench_full.tfrecord ./kurobako dataset nasbench convert nasbench_full.tfrecord nasbench_full.bin - name: Run performance benchmark run: | python benchmarks/run_mo_kurobako.py \ --path-to-kurobako "." \ --name-prefix "" \ --budget ${{ github.event.inputs.budget }} \ --n-runs ${{ github.event.inputs.n-runs }} \ --n-jobs 10 \ --n-concurrency ${{ github.event.inputs.n-concurrency }} \ --sampler-list '${{ github.event.inputs.sampler-list }}' \ --sampler-kwargs-list '${{ github.event.inputs.sampler-kwargs-list }}' \ --seed 0 \ --data-dir "." \ --out-dir "out" - uses: actions/upload-artifact@v3 with: name: benchmark-report path: | out/report.md out/**/*.png - uses: actions/download-artifact@v2 with: name: benchmark-report path: | out/report.md out/**/*.png optuna-3.5.0/.github/workflows/performance-benchmarks-naslib.yml000066400000000000000000000167041453453102400250400ustar00rootroot00000000000000name: Performance Benchmarks with NASLib on: workflow_dispatch: inputs: sampler-list: description: 'Sampler List: A list of samplers to check the performance. Should be a whitespace-separated list of Optuna samplers. Each sampler must exist under `optuna.samplers` or `optuna.integration`.' required: false default: 'RandomSampler TPESampler' sampler-kwargs-list: description: 'Sampler Arguments List: A list of sampler keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{} {\"multivariate\":true,\"constant_liar\":true}' pruner-list: description: 'Pruner List: A list of pruners to check the performance. Should be a whitespace-separated list of Optuna pruners. Each pruner must exist under `optuna.pruners`.' required: false default: 'NopPruner' pruner-kwargs-list: description: 'Pruner Arguments List: A list of pruner keyword arguments. Should be a whitespace-separated list of json format dictionaries.' required: false default: '{}' budget: description: 'Number of Trials if the pruning is not enabled. If the pruning is enabled, the total number of steps is equal to `budget * (steps per trial)`.' required: false default: '100' n-runs: description: 'Number of Studies' required: false default: '10' n-concurrency: description: 'Number of Concurrent Trials' required: false default: '1' jobs: performance-benchmarks-naslib: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.8 uses: actions/setup-python@v4 with: python-version: 3.8 - name: Install gnuplot run: | sudo apt update sudo apt -y install gnuplot - name: Setup cache uses: actions/cache@v3 env: # Caches them under a common name so that they can be used by other performance benchmark. cache-name: performance-benchmarks with: path: ~/.cache/pip key: ${{ runner.os }}-3.8-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.8-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install Python libralies run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off kurobako - name: Cache kurobako CLI id: cache-kurobako uses: actions/cache@v3 with: path: ./kurobako key: kurobako-0-2-10 - name: Install kurobako CLI if: steps.cache-kurobako.outputs.cache-hit != 'true' run: | curl -L https://github.com/optuna/kurobako/releases/download/0.2.10/kurobako-0.2.10.linux-amd64 -o kurobako chmod +x kurobako ./kurobako -h - name: Install naslib run: | curl -L https://github.com/automl/NASLib/archive/refs/heads/Develop.zip -o NASLib-Develop.zip unzip NASLib-Develop.zip mv NASLib-Develop NASLib cd NASLib pip install --upgrade pip setuptools wheel pip install -e . cd .. - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Cache nasbench201-cifar10 dataset id: cache-nb201-cifar10-dataset uses: actions/cache@v3 with: path: NASLib/naslib/data/nb201_cifar10_full_training.pickle key: cache-nb201-cifar10 - name: Download nasbench201-cifar10 dataset if: steps.cache-nb201-cifar10-dataset.outputs.cache-hit != 'true' run: | cd NASLib/naslib/data function wget_gdrive { GDRIVE_FILE_ID=$1 DEST_PATH=$2 wget --save-cookies cookies.txt 'https://docs.google.com/uc?export=download&id='$GDRIVE_FILE_ID -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1/p' > confirm.txt wget --load-cookies cookies.txt -O $DEST_PATH 'https://docs.google.com/uc?export=download&id='$GDRIVE_FILE_ID'&confirm='$( confirm.txt wget --load-cookies cookies.txt -O $DEST_PATH 'https://docs.google.com/uc?export=download&id='$GDRIVE_FILE_ID'&confirm='$( confirm.txt wget --load-cookies cookies.txt -O $DEST_PATH 'https://docs.google.com/uc?export=download&id='$GDRIVE_FILE_ID'&confirm='$( optuna/version.py - name: Build a tar ball run: | python -m build --sdist --wheel - name: Verify the distributions run: twine check dist/* - name: Publish distribution to TestPyPI # The following upload action cannot be executed in the forked repository. if: (github.event_name == 'schedule') || (github.event_name == 'release') uses: pypa/gh-action-pypi-publish@release/v1 with: repository-url: https://test.pypi.org/legacy/ - name: Publish distribution to PyPI # The following upload action cannot be executed in the forked repository. if: github.event_name == 'release' uses: pypa/gh-action-pypi-publish@release/v1 optuna-3.5.0/.github/workflows/reviewdog.yml000066400000000000000000000015321453453102400211420ustar00rootroot00000000000000name: Reviewdog on: pull_request: types: [opened, review_requested] jobs: reviewdog: runs-on: ubuntu-latest if: github.event.action == 'opened' || github.event.requested_team.name == 'reviewdog' steps: - name: Checkout uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: 3.8 - name: Install run: | python -m pip install -U pip pip install --progress-bar off -U .[checking] - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Apply formatters run: | black . blackdoc . isort . - name: Reviewdog uses: reviewdog/action-suggester@v1 with: tool_name: formatters optuna-3.5.0/.github/workflows/speed-benchmarks.yml000066400000000000000000000026071453453102400223660ustar00rootroot00000000000000name: Speed benchmarks on: schedule: - cron: '0 23 * * SUN-THU' jobs: speed-benchmarks: runs-on: ubuntu-latest # Not intended for forks. if: github.repository == 'optuna/optuna' steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.11 uses: actions/setup-python@v4 with: python-version: 3.11 - name: Setup cache uses: actions/cache@v3 env: cache-name: speed-benchmarks with: path: ~/.cache/pip key: ${{ runner.os }}-3.11-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.11-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[benchmark] --extra-index-url https://download.pytorch.org/whl/cpu asv machine --yes - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Speed benchmarks run: | asv run optuna-3.5.0/.github/workflows/sphinx-build.yml000066400000000000000000000046021453453102400215560ustar00rootroot00000000000000name: Sphinx on: push: branches: - master pull_request: {} concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: documentation: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: 3.8 # note (crcrpar): We've not updated tutorial frequently enough so far thus # it'd be okay to discard cache by any small changes including typo fix under tutorial directory. - name: Sphinx Gallery Cache uses: actions/cache@v3 env: cache-name: sphx-glry-documentation with: path: | tutorial/MNIST docs/source/tutorial key: py3.8-${{ env.cache-name }}-${{ hashFiles('tutorial/**/*') }} - name: Install Dependencies run: | python -m pip install -U pip pip install --progress-bar off -U .[document] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Build Document run: | cd docs make html cd ../ - uses: actions/upload-artifact@v3 with: name: built-html path: | docs/build/html - uses: actions/upload-artifact@v3 with: name: tutorial path: | docs/source/tutorial doctest: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 with: python-version: 3.8 - name: Sphinx Gallery Cache uses: actions/cache@v3 env: cache-name: sphx-glry-doctest with: path: | tutorial/MNIST docs/source/tutorial key: py3.8-${{ env.cache-name }}-${{ hashFiles('tutorial/**/*') }} - name: Install Dependencies run: | python -m pip install -U pip pip install --progress-bar off -U .[document] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Run Doctest run: | cd docs make doctest optuna-3.5.0/.github/workflows/stale.yml000066400000000000000000000020361453453102400202570ustar00rootroot00000000000000name: stale on: schedule: - cron: '0 23 * * SUN-THU' jobs: stale: runs-on: ubuntu-latest if: github.repository == 'optuna/optuna' steps: - uses: actions/stale@v6 with: repo-token: ${{ secrets.GITHUB_TOKEN }} stale-issue-message: 'This issue has not seen any recent activity.' stale-pr-message: 'This pull request has not seen any recent activity.' close-issue-message: 'This issue was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.' close-pr-message: 'This pull request was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.' days-before-issue-stale: 300 days-before-issue-close: 0 days-before-pr-stale: 7 days-before-pr-close: 14 stale-issue-label: 'stale' stale-pr-label: 'stale' exempt-issue-labels: 'no-stale' exempt-pr-labels: 'no-stale' operations-per-run: 1000 optuna-3.5.0/.github/workflows/tests-integration.yml000066400000000000000000000051721453453102400226360ustar00rootroot00000000000000name: Tests (Integration) on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-integration: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.8', '3.9', '3.10', '3.11'] test-trigger-type: - ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'Scheduled' || '' }} exclude: - test-trigger-type: "" python-version: "3.9" - test-trigger-type: "" python-version: "3.10" steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test-integration with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev libopenblas-dev - name: Install run: | python -m pip install --upgrade pip # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu pip install --progress-bar off .[integration] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest tests -m "integration" - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest tests -m "integration and not slow" optuna-3.5.0/.github/workflows/tests-mpi.yml000066400000000000000000000051301453453102400210720ustar00rootroot00000000000000name: Tests (MPI) on: workflow_dispatch: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-mpi: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.8', '3.9', '3.10', '3.11'] test-trigger-type: - ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'Scheduled' || '' }} exclude: - test-trigger-type: "" python-version: "3.9" - test-trigger-type: "" python-version: "3.10" steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test-mpi with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev - name: Install run: | python -m pip install --upgrade pip # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu # TODO(not522): Remove this line when torchmetrics can be installed with extra-index-url pip install --progress-bar off torchmetrics pip install --progress-bar off .[integration] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests run: | export OMPI_MCA_rmaps_base_oversubscribe=yes mpirun -n 2 -- pytest tests/integration_tests/test_pytorch_distributed.py env: OMP_NUM_THREADS: 1 optuna-3.5.0/.github/workflows/tests-storage.yml000066400000000000000000000101711453453102400217520ustar00rootroot00000000000000name: Tests (Storage with server) on: workflow_dispatch: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: # TODO(masap): Modify job name to "tests-storage-with-server" because this test is not only for # RDB. Since current name "tests-rdbstorage" is required in the Branch protection rules, you # need to modify the Branch protection rules as well. tests-rdbstorage: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.7', '3.8', '3.9', '3.10', '3.11'] test-trigger-type: - ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'Scheduled' || '' }} exclude: - test-trigger-type: "" python-version: "3.8" - test-trigger-type: "" python-version: "3.9" - test-trigger-type: "" python-version: "3.10" services: mysql: image: mysql:5.7 ports: - 3306:3306 env: MYSQL_ROOT_PASSWORD: mandatory_arguments MYSQL_DATABASE: optunatest MYSQL_USER: user MYSQL_PASSWORD: test options: >- --health-cmd "mysqladmin ping -h localhost" --health-interval 10s --health-timeout 5s --health-retries 5 postgres: image: postgres:10.1-alpine ports: - 5432:5432 env: POSTGRES_DB: optunatest POSTGRES_USER: user POSTGRES_PASSWORD: test options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 redis: image: redis ports: - 6379:6379 steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test-storage-with-server with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup environment run: | sudo apt-get update sudo apt-get -y install openmpi-bin libopenmpi-dev - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Install DB bindings run: | pip install --progress-bar off PyMySQL cryptography psycopg2-binary redis - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests MySQL run: | pytest tests/storages_tests/test_with_server.py env: SQLALCHEMY_WARN_20: 1 OMP_NUM_THREADS: 1 TEST_DB_URL: mysql+pymysql://user:test@127.0.0.1/optunatest - name: Tests PostgreSQL run: | pytest tests/storages_tests/test_with_server.py env: OMP_NUM_THREADS: 1 TEST_DB_URL: postgresql+psycopg2://user:test@127.0.0.1/optunatest - name: Tests Journal Redis run: | pytest tests/storages_tests/test_with_server.py env: OMP_NUM_THREADS: 1 TEST_DB_URL: redis://localhost:6379 TEST_DB_MODE: journal-redis optuna-3.5.0/.github/workflows/tests-with-minimum-versions.yml000066400000000000000000000054341453453102400246060ustar00rootroot00000000000000name: Tests with minimum versions on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-with-minimum-versions: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.7', '3.8', '3.9'] services: redis: image: redis options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test-with-minimum-versions with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/setup.py') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/setup.py') }} - name: Setup pip run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools - name: Install run: | # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Install dependencies with minimum versions run: | # Install dependencies with minimum versions. pip uninstall -y alembic cmaes packaging sqlalchemy plotly scikit-learn pip install alembic==1.5.0 cmaes==0.10.0 packaging==20.0 sqlalchemy==1.3.0 numpy==1.20.3 tqdm==4.27.0 colorlog==0.3 PyYAML==5.1 pip install plotly==5.0.0 scikit-learn==0.24.2 # optional extras - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest tests -m "not integration" - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest tests -m "not integration and not slow" optuna-3.5.0/.github/workflows/tests.yml000066400000000000000000000061351453453102400203150ustar00rootroot00000000000000name: Tests on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests: if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') runs-on: ubuntu-latest strategy: matrix: python-version: ['3.7', '3.8', '3.9', '3.10', '3.11', '3.12'] test-trigger-type: - ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && 'Scheduled' || '' }} exclude: - test-trigger-type: "" python-version: "3.8" - test-trigger-type: "" python-version: "3.9" - test-trigger-type: "" python-version: "3.10" - test-trigger-type: "" python-version: "3.11" services: redis: image: redis options: >- --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5 steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Setup cache uses: actions/cache@v3 env: cache-name: test with: path: ~/.cache/pip key: ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-${{ matrix.python-version }}-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup pip run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools - name: Install run: | # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] --extra-index-url https://download.pytorch.org/whl/cpu - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests run: | if [ "${{ github.event_name }}" = "schedule" ] || \ [ "${{ github.event_name }}" = "workflow_dispatch" ] ; then target="not integration" else target="not integration and not slow" fi if [ "${{ matrix.python-version }}" = "3.12" ] ; then # TODO(not522): Remove ignores when BoTorch supports Python 3.12 ignore_option="--ignore tests/terminator_tests/ \ --ignore tests/visualization_tests/test_terminator_improvement.py" else ignore_option="" fi pytest tests -m "$target" $ignore_option env: SQLALCHEMY_WARN_20: 1 optuna-3.5.0/.github/workflows/windows-tests.yml000066400000000000000000000103551453453102400220040ustar00rootroot00000000000000# Run tests and integration tests on Windows, which are triggered by each master push. # Currently, Python3.9 is only used as an environment. # This is mainly for the sake of speed. name: windows-tests on: push: branches: - master pull_request: {} schedule: - cron: '0 23 * * SUN-THU' workflow_dispatch: concurrency: group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }} cancel-in-progress: true jobs: tests-windows: runs-on: windows-latest # Not intended for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.9 uses: actions/setup-python@v4 with: python-version: "3.9" - name: Setup cache uses: actions/cache@v3 env: cache-name: windows-test with: path: ~\\AppData\\Local\\pip\\Cache key: ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] pip install PyQt6 # Install PyQT for using QtAgg as matplotlib backend. pip install "kaleido<=0.1.0post1" # TODO(HideakiImamura): Remove this after fixing https://github.com/plotly/Kaleido/issues/110 - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Scheduled tests if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} run: | pytest -m "not integration" env: SQLALCHEMY_WARN_20: 1 MPLBACKEND: "QtAgg" # Use QtAgg as matplotlib backend. - name: Tests if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} run: | pytest -m "not integration and not slow" env: MPLBACKEND: "QtAgg" # Use QtAgg as matplotlib backend. tests-integration-windows: runs-on: windows-latest # Not intended for forks. if: (github.event_name == 'schedule' && github.repository == 'optuna/optuna') || (github.event_name != 'schedule') steps: - name: Checkout uses: actions/checkout@v3 - name: Setup Python3.9 uses: actions/setup-python@v4 with: python-version: "3.9" - name: Setup cache uses: actions/cache@v3 env: cache-name: windows-test-integration with: path: ~\\AppData\\Local\\pip\\Cache key: ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }}-v1 restore-keys: | ${{ runner.os }}-3.9-${{ env.cache-name }}-${{ hashFiles('**/pyproject.toml') }} - name: Setup MPI uses: mpi4py/setup-mpi@v1 with: mpi: "msmpi" - name: Install run: | python -m pip install --upgrade pip pip install --progress-bar off -U setuptools # Install minimal dependencies and confirm that `import optuna` is successful. pip install --progress-bar off . python -c 'import optuna' optuna --version pip install --progress-bar off .[test] pip install --progress-bar off .[optional] pip install --progress-bar off .[integration] pip install "distributed<2023.3.2" - name: Output installed packages run: | pip freeze --all - name: Output dependency tree run: | pip install pipdeptree pipdeptree - name: Tests # Skip allennlp tests since it's not supported on Windows. run: | pytest tests -m "integration" ` --ignore tests/integration_tests/allennlp_tests/test_allennlp.py ` env: OMP_NUM_THREADS: 1 optuna-3.5.0/.gitignore000066400000000000000000000032121453453102400150140ustar00rootroot00000000000000# macOS metadata .DS_Store # Ignore files that examples create t10k-images-idx3-ubyte* t10k-labels-idx1-ubyte* train-images-idx3-ubyte* train-labels-idx1-ubyte* training.pt test.pt catboost_info/ # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python env/ build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ *.egg-info/ .installed.cfg *.egg # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *,cover .hypothesis/ .pytest_cache/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # PyBuilder target/ # IPython Notebook .ipynb_checkpoints # pyenv .python-version # celery beat schedule file celerybeat-schedule # dotenv .env # virtualenv .venv/ venv/ ENV/ # Spyder project settings .spyderproject # Rope project settings .ropeproject # PyCharm .idea # VSCode .vscode .devcontainer # MyPy .mypy_cache # Sphinx tutorial/**/example.db tutorial/**/example-study.db docs/_build/ docs/source/reference/generated/ docs/source/reference/multi_objective/generated/ docs/source/reference/visualization/generated/ docs/source/reference/samplers/generated docs/source/tutorial/** !docs/source/tutorial/index.rst # asv .asv # Dask dask-worker-space/ # PyTorch Lightning lightning_logs/ optuna-3.5.0/.readthedocs.yml000066400000000000000000000014141453453102400161140ustar00rootroot00000000000000# Read the Docs configuration file # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the version of Python and other tools you might need build: os: ubuntu-22.04 tools: python: "3.11" # Build documentation in the docs/ directory with Sphinx sphinx: configuration: docs/source/conf.py # Optionally build your docs in additional formats such as PDF and ePub formats: all # Optionally set the version of Python and requirements required to build your docs python: # `sphinx` requires either Python >= 3.8 or `typed-ast` to reflect type comments # in the documentation. See: https://github.com/sphinx-doc/sphinx/pull/6984 install: - method: pip path: . extra_requirements: - document optuna-3.5.0/CODE_OF_CONDUCT.md000066400000000000000000000004651453453102400156320ustar00rootroot00000000000000# Optuna Code of Conduct Optuna follows the [NumFOCUS Code of Conduct][homepage] available at https://numfocus.org/code-of-conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at optuna@preferred.jp. [homepage]: https://numfocus.org/ optuna-3.5.0/CONTRIBUTING.md000066400000000000000000000264421453453102400152670ustar00rootroot00000000000000# Contribution Guidelines It’s an honor to have you on board! We are proud of this project and have been working to make it great since day one. We believe you will love it, and we know there’s room for improvement. We want to - implement features that make what you want to do possible and/or easy. - write more tutorials and [examples](https://github.com/optuna/optuna-examples) that help you get familiar with Optuna. - make issues and pull requests on GitHub fruitful. - have more conversations and discussions on [GitHub Discussions](https://github.com/optuna/optuna/discussions). We need your help and everything about Optuna you have in your mind pushes this project forward. Join Us! If you feel like giving a hand, here are some ways: - Implement a feature - If you have some cool idea, please open an issue first to discuss design to make your idea in a better shape. - Send a patch - Dirty your hands by tackling [issues with `contribution-welcome` label](https://github.com/optuna/optuna/issues?q=is%3Aissue+is%3Aopen+label%3Acontribution-welcome) - Report a bug - If you find a bug, please report it! Your reports are important. - Fix/Improve documentation - Documentation gets outdated easily and can always be better, so feel free to fix and improve - Let us and the Optuna community know your ideas and thoughts. - __Contribution to Optuna includes not only sending pull requests, but also writing down your comments on issues and pull requests by others, and joining conversations/discussions on [GitHub Discussions](https://github.com/optuna/optuna/discussions).__ - Also, sharing how you enjoy Optuna is a huge contribution! If you write a blog, let us know about it! ## Pull Request Guidelines If you make a pull request, please follow the guidelines below: - [Setup Optuna](#setup-optuna) - [Checking the Format, Coding Style, and Type Hints](#checking-the-format-coding-style-and-type-hints) - [Documentation](#documentation) - [Unit Tests](#unit-tests) - [Continuous Integration and Local Verification](#continuous-integration-and-local-verification) - [Creating a Pull Request](#creating-a-pull-request) Detailed conventions and policies to write, test, and maintain Optuna code are described in the [Optuna Wiki](https://github.com/optuna/optuna/wiki). - [Coding Style Conventions](https://github.com/optuna/optuna/wiki/Coding-Style-Conventions) - [Deprecation Policy](https://github.com/optuna/optuna/wiki/Deprecation-policy) - [Test Policy](https://github.com/optuna/optuna/wiki/Test-Policy) ### Setup Optuna First of all, fork Optuna on GitHub. You can learn about fork in the official [documentation](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo). After forking, download and install Optuna on your computer. ```bash git clone git@github.com:YOUR_NAME/optuna.git cd optuna pip install -e . ``` ### Checking the Format, Coding Style, and Type Hints Code is formatted with [black](https://github.com/psf/black), and docstrings are formatted with [blackdoc](https://github.com/keewis/blackdoc). Coding style is checked with [flake8](http://flake8.pycqa.org) and [isort](https://pycqa.github.io/isort/), and additional conventions are described in the [Wiki](https://github.com/optuna/optuna/wiki/Coding-Style-Conventions). Type hints, [PEP484](https://www.python.org/dev/peps/pep-0484/), are checked with [mypy](http://mypy-lang.org/). You can check the format, coding style, and type hints at the same time just by executing a script `formats.sh`. If your environment is missing some dependencies such as black, blackdoc, flake8, isort or mypy, you will be asked to install them. The following commands automatically fix format errors by auto-formatters. ```bash # Install auto-formatters. $ pip install ".[checking]" $ ./formats.sh ``` ### Documentation When adding a new feature to the framework, you also need to document it in the reference. The documentation source is stored under the [docs](./docs) directory and written in [reStructuredText format](http://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html). To build the documentation, you need to run: ```bash pip install -e ".[document]" ``` Note that the above command might try to install PyTorch without CUDA to your environment even if your environment has CUDA version already. Then you can build the documentation in HTML format locally: ```bash cd docs make html ``` HTML files are generated under `build/html` directory. Open `index.html` with the browser and see if it is rendered as expected. Optuna's tutorial is built with [Sphinx-Gallery](https://sphinx-gallery.github.io/stable/index.html) and some other requirements like [LightGBM](https://github.com/microsoft/LightGBM) and [PyTorch](https://pytorch.org) meaning that all .py files in `tutorial` directory are run during the documentation build if there's no build cache. Whether you edit any tutorial or not doesn't matter. To avoid having to run the tutorials, you may download executed tutorial artifacts named "tutorial" from our CI (see the capture below) and put them in `docs/build` before extracting the files in the zip to `docs/source/tutorial` directory. Note that the CI runs with Python 3.8 and the generated artifacts contain pickle files. The pickle files are serialized with [the protocol version 5](https://docs.python.org/3/library/pickle.html#data-stream-format) so you will see the error with Python 3.7 or older. Please use Python 3.8 or later if you build the documentation with artifacts. ![image](https://user-images.githubusercontent.com/16191443/107472296-0b211400-6bb2-11eb-9203-e2c42ce499ad.png) **Writing a Tutorial** Tutorials are part of Optuna’s documentation. Optuna depends on Sphinx to build the documentation HTML files from the corresponding reStructuredText (`.rst`) files in the docs/source directory, but as you may notice, [Tutorial directory](https://github.com/optuna/optuna/tree/master/tutorial) does not have any `.rst` files. Instead, it has a bunch of Python (`.py`) files. We have [Sphinx Gallery](https://sphinx-gallery.github.io/stable/index.html) that executes those `.py` files and generates `.rst` files with standard outputs from them and corresponding Jupyter Notebook (`.ipynb`) files. These generated `.rst` and `.ipynb` files are written to the docs/source/tutorial directory. The output directory (docs/source/tutorial) and source (tutorial) directory are configured in [`sphinx_gallery_conf` of docs/source/conf.py](https://github.com/optuna/optuna/blob/2e14273cab87f13edeb9d804a43bd63c44703cb5/docs/source/conf.py#L189-L199). These generated `.rst` files are handled by Sphinx like the other `.rst` files. The generated `.ipynb` files are hosted on Optuna’s documentation page and downloadable (check [Optuna tutorial](https://optuna.readthedocs.io/en/stable/tutorial/index.html)). The order of contents on [tutorial top page](https://optuna.readthedocs.io/en/stable/tutorial/index.html) is determined by two keys: one is the subdirectory name of tutorial and the other is the filename (note that there are some alternatives as documented in [Sphinx Gallery - sorting](https://sphinx-gallery.github.io/stable/gen_modules/sphinx_gallery.sorting.html?highlight=filenamesortkey), but we chose this key in https://github.com/optuna/optuna/blob/2e14273cab87f13edeb9d804a43bd63c44703cb5/docs/source/conf.py#L196). Optuna’s tutorial directory has two directories: (1) [10_key_features](https://github.com/optuna/optuna/tree/master/tutorial/10_key_features), which is meant to be aligned with and explain the key features listed on [README.md](https://github.com/optuna/optuna#key-features) and (2) [20_recipes](https://github.com/optuna/optuna/tree/master/tutorial/20_recipes), whose contents showcase how to use Optuna features conveniently. When adding new content to the Optuna tutorials, place it in `20_recipes` and its file name should conform to the other names, for example, `777_cool_feature.py`. In general, please number the prefix for your file consecutively with the last number. However, this is not mandatory and if you think your content deserves the smaller number (the order of recipes does not have a specific meaning, but in general, order could convey the priority order to readers), feel free to propose the renumbering in your PR. You may want to refer to the Sphinx Gallery for the syntax of `.py` files processed by Sphinx Gallery. Two specific conventions and limitations for Optuna tutorials: 1. 99 #s for block separation as in https://github.com/optuna/optuna/blob/2e14273cab87f13edeb9d804a43bd63c44703cb5/tutorial/10_key_features/001_first.py#L19 2. Execution time of the new content needs to be less than three minutes. This limitation derives from Read The Docs. If your content runs some hyperparameter optimization, set the `timeout` to 180 or less. You can check this limitation on [Read the Docs - Build Process](https://docs.readthedocs.io/en/stable/builds.html). ### Unit Tests When adding a new feature or fixing a bug, you also need to write sufficient test code. We use [pytest](https://pytest.org/) as the testing framework and unit tests are stored under the [tests directory](./tests). Please install some required packages at first. ```bash # Install required packages to test all modules without integration modules. pip install ".[test,optional]" # Install required packages to test all modules including integration modules. pip install ".[integration]" -f https://download.pytorch.org/whl/torch_stable.html ``` You can run your tests as follows: ```bash # Run all the unit tests. pytest # Run all the unit tests without integrations. pytest -m "not integration" # Run all the unit tests defined in the specified test file. pytest tests/${TARGET_TEST_FILE_NAME} # Run the unit test function with the specified name defined in the specified test file. pytest tests/${TARGET_TEST_FILE_NAME} -k ${TARGET_TEST_FUNCTION_NAME} ``` See also the [Optuna Test Policy](https://github.com/optuna/optuna/wiki/Test-Policy), which describes the principles to write and maintain Optuna tests to meet certain quality requirements. ### Continuous Integration and Local Verification Optuna repository uses GitHub Actions. ### Creating a Pull Request When you are ready to create a pull request, please try to keep the following in mind. First, the **title** of your pull request should: - briefly describe and reflect the changes - wrap any code with backticks - not end with a period *The title will be directly visible in the release notes.* For example: - Introduces Tree-structured Parzen Estimator to `optuna.samplers` Second, the **description** of your pull request should: - describe the motivation - describe the changes - if still work-in-progress, describe remaining tasks ## Learning Optuna's Implementation With Optuna actively being developed and the amount of code growing, it has become difficult to get a hold of the overall flow from reading the code. So we created a tiny program called [Minituna](https://github.com/CyberAgentAILab/minituna). Once you get a good understanding of how Minituna is designed, it will not be too difficult to read the Optuna code. We encourage you to practice reading the Minituna code with the following article. [An Introduction to the Implementation of Optuna, a Hyperparameter Optimization Framework](https://medium.com/optuna/an-introduction-to-the-implementation-of-optuna-a-hyperparameter-optimization-framework-33995d9ec354) optuna-3.5.0/Dockerfile000066400000000000000000000014531453453102400150230ustar00rootroot00000000000000ARG PYTHON_VERSION=3.8 FROM python:${PYTHON_VERSION} ENV PIP_OPTIONS "--no-cache-dir --progress-bar off" RUN apt-get update \ && apt-get -y install openmpi-bin libopenmpi-dev libopenblas-dev \ && rm -rf /var/lib/apt/lists/* \ && pip install --no-cache-dir -U pip \ && pip install ${PIP_OPTIONS} -U setuptools WORKDIR /workspaces COPY . . ARG BUILD_TYPE='dev' RUN if [ "${BUILD_TYPE}" = "dev" ]; then \ pip install ${PIP_OPTIONS} -e '.[benchmark, checking, document, integration, optional, test]' --extra-index-url https://download.pytorch.org/whl/cpu; \ else \ pip install ${PIP_OPTIONS} -e .; \ fi \ && pip install ${PIP_OPTIONS} jupyter notebook # Install RDB bindings. RUN pip install ${PIP_OPTIONS} PyMySQL cryptography psycopg2-binary ENV PIP_OPTIONS "" optuna-3.5.0/LICENSE000066400000000000000000000057101453453102400140360ustar00rootroot00000000000000MIT License Copyright (c) 2018 Preferred Networks, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. == Optuna contains code that is licensed by third-party developers. == SciPy The Optuna contains the codes from SciPy project. Copyright (c) 2001-2002 Enthought, Inc. 2003-2022, SciPy Developers. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. == fdlibm Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. Developed at SunPro, a Sun Microsystems, Inc. business. Permission to use, copy, modify, and distribute this software is freely granted, provided that this notice is preserved. optuna-3.5.0/MANIFEST.in000066400000000000000000000000201453453102400145540ustar00rootroot00000000000000include LICENSE optuna-3.5.0/README.md000066400000000000000000000207161453453102400143130ustar00rootroot00000000000000
# Optuna: A hyperparameter optimization framework [![Python](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org) [![pypi](https://img.shields.io/pypi/v/optuna.svg)](https://pypi.python.org/pypi/optuna) [![conda](https://img.shields.io/conda/vn/conda-forge/optuna.svg)](https://anaconda.org/conda-forge/optuna) [![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/optuna/optuna) [![Read the Docs](https://readthedocs.org/projects/optuna/badge/?version=stable)](https://optuna.readthedocs.io/en/stable/) [![Codecov](https://codecov.io/gh/optuna/optuna/branch/master/graph/badge.svg)](https://codecov.io/gh/optuna/optuna) [**Website**](https://optuna.org/) | [**Docs**](https://optuna.readthedocs.io/en/stable/) | [**Install Guide**](https://optuna.readthedocs.io/en/stable/installation.html) | [**Tutorial**](https://optuna.readthedocs.io/en/stable/tutorial/index.html) | [**Examples**](https://github.com/optuna/optuna-examples) *Optuna* is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, *define-by-run* style user API. Thanks to our *define-by-run* API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters. ## Key Features Optuna has modern functionalities as follows: - [Lightweight, versatile, and platform agnostic architecture](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/001_first.html) - Handle a wide variety of tasks with a simple installation that has few requirements. - [Pythonic search spaces](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html) - Define search spaces using familiar Python syntax including conditionals and loops. - [Efficient optimization algorithms](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html) - Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials. - [Easy parallelization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html) - Scale studies to tens or hundreds or workers with little or no changes to the code. - [Quick visualization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/005_visualization.html) - Inspect optimization histories from a variety of plotting functions. ## Basic Concepts We use the terms *study* and *trial* as follows: - Study: optimization based on an objective function - Trial: a single execution of the objective function Please refer to sample code below. The goal of a *study* is to find out the optimal set of hyperparameter values (e.g., `regressor` and `svr_c`) through multiple *trials* (e.g., `n_trials=100`). Optuna is a framework designed for the automation and the acceleration of the optimization *studies*. [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/optuna/optuna-examples/blob/main/quickstart.ipynb) ```python import ... # Define an objective function to be minimized. def objective(trial): # Invoke suggest methods of a Trial object to generate hyperparameters. regressor_name = trial.suggest_categorical('regressor', ['SVR', 'RandomForest']) if regressor_name == 'SVR': svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True) regressor_obj = sklearn.svm.SVR(C=svr_c) else: rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32) regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth) X, y = sklearn.datasets.fetch_california_housing(return_X_y=True) X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0) regressor_obj.fit(X_train, y_train) y_pred = regressor_obj.predict(X_val) error = sklearn.metrics.mean_squared_error(y_val, y_pred) return error # An objective value linked with the Trial object. study = optuna.create_study() # Create a new study. study.optimize(objective, n_trials=100) # Invoke optimization of the objective function. ``` ## Installation Optuna is available at [the Python Package Index](https://pypi.org/project/optuna/) and on [Anaconda Cloud](https://anaconda.org/conda-forge/optuna). ```bash # PyPI $ pip install optuna ``` ```bash # Anaconda Cloud $ conda install -c conda-forge optuna ``` Optuna supports Python 3.7 or newer. Also, we also provide Optuna docker images on [DockerHub](https://hub.docker.com/r/optuna/optuna). ## Examples Examples can be found in [optuna/optuna-examples](https://github.com/optuna/optuna-examples). ## Integrations Optuna has integration features with various third-party libraries. Integrations can be found in [optuna/optuna-integration](https://github.com/optuna/optuna-integration) and the document is available [here](https://optuna-integration.readthedocs.io/en/stable/index.html). Integrations support libraries such as the following: * [Catalyst](https://github.com/optuna/optuna-examples/tree/main/pytorch/catalyst_simple.py) * [Catboost](https://github.com/optuna/optuna-examples/tree/main/catboost/catboost_pruning.py) * [Dask](https://github.com/optuna/optuna-examples/tree/main/dask/dask_simple.py) * [fastai (v2)](https://github.com/optuna/optuna-examples/tree/main/fastai/fastaiv2_simple.py) * [Keras](https://github.com/optuna/optuna-examples/tree/main/keras/keras_integration.py) * [LightGBM](https://github.com/optuna/optuna-examples/tree/main/lightgbm/lightgbm_integration.py) * [MLflow](https://github.com/optuna/optuna-examples/tree/main/mlflow/keras_mlflow.py) * [MXNet](https://github.com/optuna/optuna-examples/tree/main/mxnet/mxnet_integration.py) * [PyTorch](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_simple.py) * [PyTorch Ignite](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_ignite_simple.py) * [PyTorch Lightning](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_lightning_simple.py) * [TensorBoard](https://github.com/optuna/optuna-examples/tree/main/tensorboard/tensorboard_simple.py) * [TensorFlow](https://github.com/optuna/optuna-examples/tree/main/tensorflow/tensorflow_estimator_integration.py) * [tf.keras](https://github.com/optuna/optuna-examples/tree/main/tfkeras/tfkeras_integration.py) * [Weights & Biases](https://github.com/optuna/optuna-examples/tree/main/wandb/wandb_integration.py) * [XGBoost](https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py) ## Web Dashboard [Optuna Dashboard](https://github.com/optuna/optuna-dashboard) is a real-time web dashboard for Optuna. You can check the optimization history, hyperparameter importances, etc. in graphs and tables. You don't need to create a Python script to call [Optuna's visualization](https://optuna.readthedocs.io/en/stable/reference/visualization/index.html) functions. Feature requests and bug reports welcome! ![optuna-dashboard](https://user-images.githubusercontent.com/5564044/204975098-95c2cb8c-0fb5-4388-abc4-da32f56cb4e5.gif) Install `optuna-dashboard` via pip: ``` $ pip install optuna-dashboard $ optuna-dashboard sqlite:///db.sqlite3 ... Listening on http://localhost:8080/ Hit Ctrl-C to quit. ``` ## Communication - [GitHub Discussions] for questions. - [GitHub Issues] for bug reports and feature requests. [GitHub Discussions]: https://github.com/optuna/optuna/discussions [GitHub issues]: https://github.com/optuna/optuna/issues ## Contribution Any contributions to Optuna are more than welcome! If you are new to Optuna, please check the [good first issues](https://github.com/optuna/optuna/labels/good%20first%20issue). They are relatively simple, well-defined and are often good starting points for you to get familiar with the contribution workflow and other developers. If you already have contributed to Optuna, we recommend the other [contribution-welcome issues](https://github.com/optuna/optuna/labels/contribution-welcome). For general guidelines how to contribute to the project, take a look at [CONTRIBUTING.md](./CONTRIBUTING.md). ## Reference Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD ([arXiv](https://arxiv.org/abs/1907.10902)). optuna-3.5.0/asv.conf.json000066400000000000000000000153101453453102400154360ustar00rootroot00000000000000{ // The version of the config file format. Do not change, unless // you know what you are doing. "version": 1, // The name of the project being benchmarked "project": "Optuna", // The project's homepage "project_url": "https://optuna.org/", // The URL or local path of the source code repository for the // project being benchmarked "repo": ".", // The Python project's subdirectory in your repo. If missing or // the empty string, the project is assumed to be located at the root // of the repository. // "repo_subdir": "", // Customizable commands for building, installing, and // uninstalling the project. See asv.conf.json documentation. // // "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}"], // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"], // "build_command": [ // "python setup.py build", // "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}" // ], "build_command": [ "python -m pip install build wheel", "python -m build --wheel -o {build_cache_dir} {build_dir}", "python -m pip install .[optional,test]" ], // List of branches to benchmark. If not provided, defaults to "master" // (for git) or "default" (for mercurial). // "branches": ["master"], // for git // "branches": ["default"], // for mercurial // The DVCS being used. If not set, it will be automatically // determined from "repo" by looking at the protocol in the URL // (if remote), or by looking for special directories, such as // ".git" (if local). // "dvcs": "git", // The tool to use to create environments. May be "conda", // "virtualenv" or other value depending on the plugins in use. // If missing or the empty string, the tool will be automatically // determined by looking for tools on the PATH environment // variable. "environment_type": "virtualenv", // timeout in seconds for installing any dependencies in environment // defaults to 10 min //"install_timeout": 600, // the base URL to show a commit for the project. "show_commit_url": "https://github.com/optuna/optuna/commit/", // The Pythons you'd like to test against. If not provided, defaults // to the current version of Python used to run `asv`. // "pythons": ["2.7", "3.6"], // The list of conda channel names to be searched for benchmark // dependency packages in the specified order // "conda_channels": ["conda-forge", "defaults"], // The matrix of dependencies to test. Each key is the name of a // package (in PyPI) and the values are version numbers. An empty // list or empty string indicates to just test against the default // (latest) version. null indicates that the package is to not be // installed. If the package to be tested is only available from // PyPi, and the 'environment_type' is conda, then you can preface // the package name by 'pip+', and the package will be installed via // pip (with all the conda available packages installed first, // followed by the pip installed packages). // // "matrix": { // }, // Combinations of libraries/python versions can be excluded/included // from the set to test. Each entry is a dictionary containing additional // key-value pairs to include/exclude. // // An exclude entry excludes entries where all values match. The // values are regexps that should match the whole string. // // An include entry adds an environment. Only the packages listed // are installed. The 'python' key is required. The exclude rules // do not apply to includes. // // In addition to package names, the following keys are available: // // - python // Python version, as in the *pythons* variable above. // - environment_type // Environment type, as above. // - sys_platform // Platform, as in sys.platform. Possible values for the common // cases: 'linux2', 'win32', 'cygwin', 'darwin'. // // "exclude": [ // {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows // {"environment_type": "conda", "six": null}, // don't run without six on conda // ], // // "include": [ // // additional env for python2.7 // {"python": "2.7", "numpy": "1.8"}, // // additional env if run on windows+conda // {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""}, // ], // The directory (relative to the current directory) that benchmarks are // stored in. If not provided, defaults to "benchmarks" "benchmark_dir": "benchmarks/asv", // The directory (relative to the current directory) to cache the Python // environments in. If not provided, defaults to "env" "env_dir": ".asv/env", // The directory (relative to the current directory) that raw benchmark // results are stored in. If not provided, defaults to "results". "results_dir": ".asv/results", // The directory (relative to the current directory) that the html tree // should be written to. If not provided, defaults to "html". "html_dir": ".asv/html", // The number of characters to retain in the commit hashes. // "hash_length": 8, // `asv` will cache results of the recent builds in each // environment, making them faster to install next time. This is // the number of builds to keep, per environment. // "build_cache_size": 2, // The commits after which the regression search in `asv publish` // should start looking for regressions. Dictionary whose keys are // regexps matching to benchmark names, and values corresponding to // the commit (exclusive) after which to start looking for // regressions. The default is to start from the first commit // with results. If the commit is `null`, regression detection is // skipped for the matching benchmark. // // "regressions_first_commits": { // "some_benchmark": "352cdf", // Consider regressions only after this commit // "another_benchmark": null, // Skip regression detection altogether // }, // The thresholds for relative change in results, after which `asv // publish` starts reporting regressions. Dictionary of the same // form as in ``regressions_first_commits``, with values // indicating the thresholds. If multiple entries match, the // maximum is taken. If no entry matches, the default is 5%. // // "regressions_thresholds": { // "some_benchmark": 0.01, // Threshold of 1% // "another_benchmark": 0.5, // Threshold of 50% // }, } optuna-3.5.0/benchmarks/000077500000000000000000000000001453453102400151435ustar00rootroot00000000000000optuna-3.5.0/benchmarks/README.md000066400000000000000000000265721453453102400164360ustar00rootroot00000000000000# Benchmarks for Optuna Interested in measuring Optuna's performance? You are very perceptive. Under this directory, you will find scripts that we have prepared to measure Optuna's performance. In this document, we explain how we measure the performance of Optuna using the scripts in this directory. The contents of this document are organized as follows. - [Performance Benchmarks with `kurobako`](#performance-benchmarks-with-kurobako) - [Performance Benchmarks with `bayesmark`](#performance-benchmarks-with-bayesmark) ## Performance Benchmarks with `kurobako` We measure the performance of black-box optimization algorithms in Optuna with [`kurobako`](https://github.com/optuna/kurobako) using `benchmarks/run_kurobako.py`. You can manually run this script on the GitHub Actions if you have a write access on the repository. Or, you can locally execute the `benchmarks/run_kurobako.py`. We explain both of method here. ### How to Run on the GitHub Actions You need a write access on the repository. Please run the following steps in your own forks. Note that you should pull the latest master branch of [Optuna](https://github.com/optuna/optuna) since the workflow YAML file must be placed in the default branch of the repository. 1. Open the GitHub page of your forked Optuna repository. 2. Click the `Actions` below the repository name. ![image](https://user-images.githubusercontent.com/38826298/145764682-0c4a31aa-f865-4293-a3c7-2ca6be5baa03.png) 3. In the left sidebar, click the `Performance Benchmarks with kurobako`. 4. Above the list of workflow runs, select `Run workflow`. ![image](https://user-images.githubusercontent.com/38826298/145764692-a30a74c0-5ebe-4010-a7cd-4ebcdbb24679.png) 5. Use the `Branch` dropdown to select the workflow's branch. The default is `master`. And, type the input parameters: `Sampler List`, `Sampler Arguments List`, `Pruner List`, and `Pruner Arguments List`. 6. Click `Run workflow`. ![image](https://user-images.githubusercontent.com/38826298/145764702-771d9a6f-8c7d-40d5-a912-1485a1d7dcfa.png) 7. After finishing the workflow, you can download the report and plot from `Artifacts`. ![image](https://user-images.githubusercontent.com/38826298/145802414-e29ca0ba-80fd-488a-af02-c33e9b4d5e3b.png) The report looks like as follows. It includes the version information of environments, the solvers (pairs of the sampler and the pruner in Optuna) and problems, the best objective value, AUC, elapsed time, and so on. ![image](https://user-images.githubusercontent.com/38826298/146860092-74da99c6-15b6-4da4-baef-0457af1d7171.png) The plot looks like as follows. It represents the optimization history plot of the optimization. The title is the name of the problem. The legends represents the specified pair of the sampler and the pruner. The history is averaged over the specified `n_runs` studies with the errorbar. The horizontal axis represents the budget (`#budgets * #epochs = \sum_{for each trial) (#consumed epochs in the trial)`). The vertical axis represents the objective value. ![image](https://user-images.githubusercontent.com/38826298/146860370-853174c7-afc5-4f67-8143-61f22d2c8f6c.png) Note that the default run time of a GitHub Actions workflow job is limited to 6 hours. Depending on the sampler and number of studies you specify, it may exceed the 6-hour limit and fail. See the [official document](https://docs.github.com/ja/actions/learn-github-actions/usage-limits-billing-and-administration) for more details. ### How to Run Locally You can run the script of `benchmarks/run_kurobako.py` directly. This section explains how to locally run it. First, you need to install `kurobako` and its Python helper. To install `kurobako`, see https://github.com/optuna/kurobako#installation for more details. In addition, please run `pip install kurobako` to install the Python helper. You need to install `gnuplot` for visualization with `kurobako`. You can install `gnuplot` by package managers such as `apt` (for Ubuntu) or `brew` (for macOS). Second, you need to download the dataset for `kurobako`. Run the followings in the dataset directory. ```bash # Download hyperparameter optimization (HPO) dataset % wget http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz % tar xf fcnet_tabular_benchmarks.tar.gz # Download neural architecture search (NAS) dataset # The `kurobako` command should be available. % curl -L $(kurobako dataset nasbench url) -o nasbench_full.tfrecord % kurobako dataset nasbench convert nasbench_full.tfrecord nasbench_full.bin ``` Finally, you can run the script of `benchmarks/run_kurobako.py`. ```bash % python benchmarks/run_kurobako.py \ --path-to-kurobako "" \ # If the `kurobako` command is available. --name "performance-benchmarks" \ --n-runs 10 \ --n-jobs 10 \ --sampler-list "RandomSampler TPESampler" \ --sampler-kwargs-list "{} {}" \ --pruner-list "NopPruner" \ --pruner-kwargs-list "{}" \ --seed 0 \ --data-dir "." \ --out-dir "out" ``` Please see `benchmarks/run_kurobako.py` to check the arguments and those default values. ### Multi-objective support We also have benchmarks for multi-objective optimization in `kurobako`. Note that we do not have pruner support for multi-objective optimization yet. Multi-objective benchmarks can also be run from GitHub Actions or locally. To run it from GitHub Actions, please click `Performance Benchmarks with mo-kurobako` in Step 3. To run it locally, please run `benchmarks/run_mo_kurobako.py`. ```bash % python benchmarks/run_mo_kurobako.py \ --path-to-kurobako "" \ # If the `kurobako` command is available. --name "performance-benchmarks" \ --n-runs 10 \ --n-jobs 10 \ --sampler-list "RandomSampler TPESampler NSGAIISampler" \ --sampler-kwargs-list "{} {\"multivariate\":true,\"constant_liar\":true} {\"population_size\":20}" \ --seed 0 \ --data-dir "." \ --out-dir "out" ``` ## Performance benchmarks with `bayesmark` This workflow allows to benchmark optimization algorithms available in Optuna with [`bayesmark`](https://github.com/uber/bayesmark). This is done by repeatedly performing hyperparameter search on set of `scikit-learn` models fitted to a list of [toy datasets](https://scikit-learn.org/stable/datasets/toy_dataset.html) and aggregating the results. Those are then compared to baseline provided by random sampler. This benchmark can be run with GitHub Actions or locally. ### How to run on the GitHub Actions 1. Follow points 1 and 2 from [Performance Benchmarks with `kurobako`](#performance-benchmarks-with-kurobako) 2. In the left sidebar, click the `Performance benchmarks with bayesmark` 3. Above the list of workflow runs, select `Run workflow`. ![image](https://user-images.githubusercontent.com/37713008/156530602-480921f5-f55e-4c14-85d7-1f285ebd17ef.png) 4. Here you can select branch to run benchmark from, as well as parameters. Click `Run workflow` to start the benchmark run. ![image](https://user-images.githubusercontent.com/37713008/156531474-a92f48f6-ec02-4173-acb7-3b3a33a06ad3.png) 5. After finishing the workflow, you can download the report and plots from `Artifacts`. ![image](https://user-images.githubusercontent.com/37713008/156532316-0cd02246-3803-44af-bcd7-383e79e2fec8.png) `benchmark-report` contains markdown file with solver leaderboards for each problem. Basic information on benchmark setup is also available. ![image](https://user-images.githubusercontent.com/37713008/156562609-6fcc72fe-541a-4053-8db0-370c5f2a12d8.png) `benchmark-plots` is a set of optimization history plots for each solved problem. Similarly to `kurobako`, each plot shows objective value as a function of finished trials. For each problem, average and median taken over `n_runs` is shown. If `Include warm-up steps in plots` checkbox was not selected in workflow config, first 10 trials will be excluded from visualizations. ![image](https://user-images.githubusercontent.com/37713008/156562987-dbaba38c-755c-448a-bc45-7aefb3fd8efd.png) See [this doc](https://bayesmark.readthedocs.io/en/stable/scoring.html) for more information on `bayesmark` scoring. ### How to run locally CI runs benchmarks on all model/dataset combination in parallel, hovever running benchmark on single problem locally is possoble. To do this, first install required Python packages. ```bash pip install bayesmark matplotlib numpy scipy pandas Jinja2 ``` Benchmark run can be started with ```bash % python benchmarks/run_bayesmark.py \ --dataset iris \ --model kNN \ --budget 80 \ --repeat 10 \ --sampler-list "TPESampler CmaEsSampler" \ --sampler-kwargs-list "{\"multivariate\":true,\"constant_liar\":true} {}" \ --pruner-list "NopPruner" \ --pruner-kwargs-list "{}" ``` Allowed models are `[kNN, SVM, DT, RF, MLP-sgd, ada, linear]` and allowed datasets are `[breast, digits, iris, wine, diabetes]`. For more details on default parameters please refer to `benchmarks/run_bayesmark.py`. Markdown report can be generated after benchmark has been completed by running ```bash % python benchmarks/bayesmark/report_bayesmark.py ``` You'll find benchmark artifacts in `plots` and `report` directories. ## Performance Benchmarks with `NASLib` This workflow allows to benchmark optimization algorithms available in Optuna with [`NASLib`](https://github.com/automl/NASLib). NASLib has an abstraction over a number of NAS benchmarks. Currently only NAS-Bench-201 is supported. This benchmark can be run on GitHub Actions or locally. ### How to run on the GitHub Actions Please follow the same steps as in [Performance Benchmarks with `kurobako`](#performance-benchmarks-with-kurobako), except that you need to select `Performance benchmarks with NASLib` in step 3. ### How to Run Locally In order to run NASLib benchmarks, you need the following dependencies: * [`NASLib`](https://github.com/automl/NASLib) and necessary data files (Currently, [`nb201_cifar10_full_training.pickle`](https://drive.google.com/file/d/1sh8pEhdrgZ97-VFBVL94rI36gedExVgJ/view?usp=sharing), [`nb201_cifar100_full_training.pickle`](https://drive.google.com/file/d/1hV6-mCUKInIK1iqZ0jfBkcKaFmftlBtp/view?usp=sharing) and [`nb201_ImageNet16_full_training.pickle`](https://drive.google.com/file/d/1FVCn54aQwD6X6NazaIZ_yjhj47mOGdIH/view?usp=sharing) are needed.) * [`kurobako`](https://github.com/optuna/kurobako) * [`kurobako-py`](https://github.com/optuna/kurobako-py) * `gnuplot` Please see each page for the detailed instructions. In short, `NASLib` can be installed by cloning the [NASLib](https://github.com/automl/NASLib), downloading all the data files under `NASLib/naslib/data/` repository from GitHub, and running ``` $ pip3 install -e . ``` You also need to set up `kurobako` command in the same way as we have described. After this, `kurobako-py` can be installed with ``` $ pip3 install kurobako ``` Finally, you can run the script of `benchmarks/run_naslib.py`. ```bash $ python3 benchmarks/run_naslib.py \ --path-to-kurobako "" \ --name "performance-benchmarks" \ --n-runs 10 \ --n-jobs 10 \ --sampler-list "RandomSampler TPESampler" \ --sampler-kwargs-list "{} {}" \ --pruner-list "NopPruner" \ --pruner-kwargs-list "{}" \ --seed 0 \ --out-dir "out" ``` Please see `benchmarks/run_naslib.py` to check the arguments and those default values. optuna-3.5.0/benchmarks/__init__.py000066400000000000000000000000001453453102400172420ustar00rootroot00000000000000optuna-3.5.0/benchmarks/asv/000077500000000000000000000000001453453102400157345ustar00rootroot00000000000000optuna-3.5.0/benchmarks/asv/__init__.py000066400000000000000000000000001453453102400200330ustar00rootroot00000000000000optuna-3.5.0/benchmarks/asv/optimize.py000066400000000000000000000064421453453102400201540ustar00rootroot00000000000000from __future__ import annotations from typing import cast import optuna from optuna.samplers import BaseSampler from optuna.samplers import CmaEsSampler from optuna.samplers import NSGAIISampler from optuna.samplers import RandomSampler from optuna.samplers import TPESampler from optuna.testing.storages import StorageSupplier def parse_args(args: str) -> list[int | str]: ret: list[int | str] = [] for arg in map(lambda s: s.strip(), args.split(",")): try: ret.append(int(arg)) except ValueError: ret.append(arg) return ret SAMPLER_MODES = [ "random", "tpe", "cmaes", ] def create_sampler(sampler_mode: str) -> BaseSampler: if sampler_mode == "random": return RandomSampler() elif sampler_mode == "tpe": return TPESampler() elif sampler_mode == "cmaes": return CmaEsSampler() elif sampler_mode == "nsgaii": return NSGAIISampler() else: assert False class OptimizeSuite: def objective(self, trial: optuna.Trial) -> float: x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2 def multi_objective(self, trial: optuna.Trial) -> tuple[float, float]: x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return (x**2 + y**2, (x - 2) ** 2 + (y - 2) ** 2) def optimize( self, storage_mode: str, sampler_mode: str, n_trials: int, objective_type: str ) -> None: with StorageSupplier(storage_mode) as storage: sampler = create_sampler(sampler_mode) if objective_type == "single": directions = ["minimize"] elif objective_type == "multi": directions = ["minimize", "minimize"] else: assert False study = optuna.create_study(storage=storage, sampler=sampler, directions=directions) if objective_type == "single": study.optimize(self.objective, n_trials=n_trials) elif objective_type == "multi": study.optimize(self.multi_objective, n_trials=n_trials) else: assert False def time_optimize(self, args: str) -> None: storage_mode, sampler_mode, n_trials, objective_type = parse_args(args) storage_mode = cast(str, storage_mode) sampler_mode = cast(str, sampler_mode) n_trials = cast(int, n_trials) objective_type = cast(str, objective_type) self.optimize(storage_mode, sampler_mode, n_trials, objective_type) params = ( "inmemory, random, 1000, single", "inmemory, random, 10000, single", "inmemory, tpe, 1000, single", "inmemory, cmaes, 1000, single", "sqlite, random, 1000, single", "sqlite, tpe, 1000, single", "sqlite, cmaes, 1000, single", "journal, random, 1000, single", "journal, tpe, 1000, single", "journal, cmaes, 1000, single", "inmemory, tpe, 1000, multi", "inmemory, nsgaii, 1000, multi", "sqlite, tpe, 1000, multi", "sqlite, nsgaii, 1000, multi", "journal, tpe, 1000, multi", "journal, nsgaii, 1000, multi", ) param_names = ["storage, sampler, n_trials, objective_type"] timeout = 600 optuna-3.5.0/benchmarks/bayesmark/000077500000000000000000000000001453453102400171215ustar00rootroot00000000000000optuna-3.5.0/benchmarks/bayesmark/optuna_optimizer.py000066400000000000000000000075551453453102400231170ustar00rootroot00000000000000from __future__ import annotations from typing import Any from typing import Dict from typing import Union import numpy as np from bayesmark.abstract_optimizer import AbstractOptimizer from bayesmark.experiment import experiment_main import optuna from optuna import pruners from optuna import samplers from optuna.integration.botorch import BoTorchSampler from optuna.integration.cma import PyCmaSampler _SAMPLERS = { "GridSampler": samplers.GridSampler, "RandomSampler": samplers.RandomSampler, "TPESampler": samplers.TPESampler, "CmaEsSampler": samplers.CmaEsSampler, "NSGAIISampler": samplers.NSGAIISampler, "QMCSampler": samplers.QMCSampler, "BoTorchSampler": BoTorchSampler, "PyCmaSampler": PyCmaSampler, } _PRUNERS = { "NopPruner": pruners.NopPruner, "MedianPruner": pruners.MedianPruner, "PatientPruner": pruners.PatientPruner, "PercentilePruner": pruners.PercentilePruner, "SuccessiveHalvingPruner": pruners.SuccessiveHalvingPruner, "HyperbandPruner": pruners.HyperbandPruner, "ThresholdPruner": pruners.ThresholdPruner, } Suggestion = Dict[str, Union[int, float]] ApiConfig = Dict[str, Dict[str, str]] class OptunaOptimizer(AbstractOptimizer): primary_import = "optuna" def __init__(self, api_config: ApiConfig, **kwargs: Any) -> None: super().__init__(api_config, **kwargs) try: sampler = _SAMPLERS[kwargs["sampler"]] sampler_kwargs: dict[str, Any] = kwargs["sampler_kwargs"] except KeyError: raise ValueError("Unknown sampler passed to Optuna optimizer.") try: pruner = _PRUNERS[kwargs["pruner"]] pruner_kwargs: dict[str, Any] = kwargs["pruner_kwargs"] except KeyError: raise ValueError("Unknown pruner passed to Optuna optimizer.") # We are using negative log-likelihood for classification # and MSE for regression. Both are minimized. self.study = optuna.create_study( direction="minimize", sampler=sampler(**sampler_kwargs), pruner=pruner(**pruner_kwargs), ) self.current_trials: dict[int, int] = dict() def _suggest(self, trial: optuna.trial.Trial) -> Suggestion: suggestions: Suggestion = dict() for name, config in self.api_config.items(): low, high = config["range"] log = config["space"] == "log" if config["space"] == "logit": assert 0 < low <= high < 1 low = np.log(low / (1 - low)) high = np.log(high / (1 - high)) if config["type"] == "real": param = trial.suggest_float(name, low, high, log=log) elif config["type"] == "int": param = trial.suggest_int(name, low, high, log=log) else: # TODO(xadrianzetx) Support `suggest_categorical` if benchmark is extended. raise RuntimeError("CategoricalDistribution is not supported in bayesmark.") suggestions[name] = param if config["space"] != "logit" else 1 / (1 + np.exp(-param)) return suggestions def suggest(self, n_suggestions: int) -> list[Suggestion]: suggestions: list[Suggestion] = list() for _ in range(n_suggestions): trial = self.study.ask() params = self._suggest(trial) sid = hash(frozenset(params.items())) self.current_trials[sid] = trial.number suggestions.append(params) return suggestions def observe(self, X: list[Suggestion], y: list[float]) -> None: for params, objective_value in zip(X, y): sid = hash(frozenset(params.items())) trial = self.current_trials.pop(sid) self.study.tell(trial, objective_value) if __name__ == "__main__": optuna.logging.disable_default_handler() experiment_main(OptunaOptimizer) optuna-3.5.0/benchmarks/bayesmark/report_bayesmark.py000066400000000000000000000217241453453102400230520ustar00rootroot00000000000000from __future__ import annotations import abc from collections import defaultdict from dataclasses import dataclass import itertools import os from typing import Dict from typing import Generator from typing import List from typing import Tuple from jinja2 import Environment from jinja2 import FileSystemLoader import numpy as np import pandas as pd from scipy.special import binom from scipy.stats import mannwhitneyu Moments = Tuple[float, float] Samples = Dict[str, List[float]] class BaseMetric(object, metaclass=abc.ABCMeta): @property @abc.abstractmethod def name(self) -> str: """Metric name displayed in final report.""" raise NotImplementedError @property @abc.abstractmethod def precision(self) -> int: """Number of digits following decimal point displayed in final report.""" raise NotImplementedError @abc.abstractmethod def calculate(self, data: pd.DataFrame) -> list[float]: """Calculates metric for each study in data frame.""" raise NotImplementedError class BestValueMetric(BaseMetric): name = "Best value" precision = 6 def calculate(self, data: pd.DataFrame) -> list[float]: return data.groupby("uuid").generalization.min().values class AUCMetric(BaseMetric): name = "AUC" precision = 3 def calculate(self, data: pd.DataFrame) -> list[float]: aucs: list[float] = list() for _, grp in data.groupby("uuid"): auc = np.sum(grp.generalization.cummin()) aucs.append(auc / grp.shape[0]) return aucs class ElapsedMetric(BaseMetric): name = "Elapsed" precision = 3 def calculate(self, data: pd.DataFrame) -> list[float]: # Total time does not include evaluation of bayesmark # objective function (no Optuna APIs are called there). time_cols = ["suggest", "observe"] return data.groupby("uuid")[time_cols].sum().sum(axis=1).values class PartialReport: def __init__(self, data: pd.DataFrame) -> None: self._data = data @property def optimizers(self) -> list[str]: return list(self._data.opt.unique()) @classmethod def from_json(cls, path: str) -> "PartialReport": data = pd.read_json(path) return cls(data) def summarize_solver(self, solver: str, metric: BaseMetric) -> Moments: solver_data = self._data[self._data.opt == solver] if solver_data.shape[0] == 0: raise ValueError(f"{solver} not found in report.") run_metrics = metric.calculate(solver_data) return np.mean(run_metrics).item(), np.var(run_metrics).item() def sample_performance(self, metric: BaseMetric) -> Samples: performance: dict[str, list[float]] = {} for solver, data in self._data.groupby("opt"): run_metrics = metric.calculate(data) performance[solver] = run_metrics return performance class DewanckerRanker: def __init__(self, metrics: list[BaseMetric]) -> None: self._metrics = metrics self._ranking: list[str] | None = None self._borda: np.ndarray | None = None def __iter__(self) -> Generator[tuple[str, int], None, None]: yield from zip(self.solvers, self.borda) @property def solvers(self) -> list[str]: if self._ranking is None: raise ValueError("Call rank first.") return self._ranking @property def borda(self) -> np.ndarray: if self._borda is None: raise ValueError("Call rank first.") return self._borda @staticmethod def pick_alpha(report: PartialReport) -> float: # https://github.com/optuna/kurobako/blob/788dd4cf618965a4a5158aa4e13607a5803dea9d/src/report.rs#L412-L424 # noqa E501 num_optimizers = len(report.optimizers) candidates = [0.075, 0.05, 0.025, 0.01] * 4 / np.repeat([1, 10, 100, 1000], 4) for cand in candidates: if 1 - np.power((1 - cand), binom(num_optimizers, 2)) < 0.05: return cand return candidates[-1] def _set_ranking(self, wins: dict[str, int]) -> None: sorted_wins = [k for k, _ in sorted(wins.items(), key=lambda x: x[1])] self._ranking = sorted_wins[::-1] def _set_borda(self, wins: dict[str, int]) -> None: sorted_wins = np.array(sorted(wins.values())) num_wins, num_ties = np.unique(sorted_wins, return_counts=True) points = np.searchsorted(sorted_wins, num_wins) self._borda = np.repeat(points, num_ties)[::-1] def rank(self, report: PartialReport) -> None: # Implements Section 2.1.1 # https://proceedings.mlr.press/v64/dewancker_strategy_2016.pdf wins: dict[str, int] = defaultdict(int) alpha = DewanckerRanker.pick_alpha(report) for metric in self._metrics: samples = report.sample_performance(metric) for optim_a, optim_b in itertools.permutations(samples, 2): _, p_val = mannwhitneyu(samples[optim_a], samples[optim_b], alternative="less") if p_val < alpha: wins[optim_a] += 1 all_wins = [wins[optimizer] for optimizer in report.optimizers] no_ties = len(all_wins) == len(np.unique(all_wins)) if no_ties: break wins = {optimzier: wins[optimzier] for optimzier in report.optimizers} self._set_ranking(wins) self._set_borda(wins) @dataclass class Solver: rank: int name: str results: list[str] @dataclass class Problem: number: int name: str metrics: list[BaseMetric] solvers: list[Solver] class BayesmarkReportBuilder: def __init__(self) -> None: self.solvers: set[str] = set() self.datasets: set[str] = set() self.models: set[str] = set() self.firsts: dict[str, int] = defaultdict(int) self.borda: dict[str, int] = defaultdict(int) self.metric_precedence = "" self.problems: list[Problem] = [] def set_precedence(self, metrics: list[BaseMetric]) -> None: self.metric_precedence = " -> ".join([m.name for m in metrics]) def add_problem( self, name: str, report: PartialReport, ranking: DewanckerRanker, metrics: list[BaseMetric], ) -> "BayesmarkReportBuilder": solvers: list[Solver] = list() positions = np.abs(ranking.borda - (max(ranking.borda) + 1)) for pos, solver in zip(positions, ranking.solvers): self.solvers.add(solver) results: list[str] = list() for metric in metrics: mean, variance = report.summarize_solver(solver, metric) precision = metric.precision results.append(f"{mean:.{precision}f} +- {np.sqrt(variance):.{precision}f}") solvers.append(Solver(pos, solver, results)) problem_number = len(self.problems) + 1 self.problems.append(Problem(problem_number, name, metrics, solvers)) return self def update_leaderboard(self, ranking: DewanckerRanker) -> "BayesmarkReportBuilder": for solver, borda in ranking: if borda == max(ranking.borda): self.firsts[solver] += 1 self.borda[solver] += borda return self def add_dataset(self, dataset: str) -> "BayesmarkReportBuilder": self.datasets.update(dataset) return self def add_model(self, model: str) -> "BayesmarkReportBuilder": self.models.update(model) return self def assemble_report(self) -> str: loader = FileSystemLoader(os.path.join("benchmarks", "bayesmark")) env = Environment(loader=loader) report_template = env.get_template("report_template.md") return report_template.render(report=self) def build_report() -> None: # Order of this list sets metric precedence. # https://proceedings.mlr.press/v64/dewancker_strategy_2016.pdf metrics = [BestValueMetric(), AUCMetric()] report_builder = BayesmarkReportBuilder() report_builder.set_precedence(metrics) for partial_name in os.listdir("partial"): dataset, model, *_ = partial_name.split("-") problem_name = f"{dataset.capitalize()}-{model}" path = os.path.join("partial", partial_name) partial = PartialReport.from_json(path) ranking = DewanckerRanker(metrics) ranking.rank(partial) # Elapsed time is not used as a voting metric, but shown in report # so it gets added to metric pool *after* ranking was calculated. elapsed = ElapsedMetric() all_metrics = [*metrics, elapsed] report_builder = ( report_builder.add_problem(problem_name, partial, ranking, all_metrics) .add_dataset(dataset) .add_model(model) .update_leaderboard(ranking) ) report = report_builder.assemble_report() with open(os.path.join("report", "benchmark-report.md"), "w") as file: file.write(report) if __name__ == "__main__": os.makedirs("report", exist_ok=True) build_report() optuna-3.5.0/benchmarks/bayesmark/report_template.md000066400000000000000000000052271453453102400226570ustar00rootroot00000000000000# Benchmark Result Report * Number of Solvers: {{ report.solvers|length }} * Number of Models: {{ report.models|length }} * Number of Datasets: {{ report.datasets|length }} * Number of Problems: {{ report.problems|length }} * Metrics Precedence: {{ report.metric_precedence }} Please refer to ["A Strategy for Ranking Optimizers using Multiple Criteria"][Dewancker, Ian, et al., 2016] for the ranking strategy used in this report. [Dewancker, Ian, et al., 2016]: http://proceedings.mlr.press/v64/dewancker_strategy_2016.pdf ## Table of Contents 1. [Overall Results](#overall-results) 2. [Individual Results](#individual-results) 3. [Datasets](#datasets) 4. [Models](#models) ## Overall Results |Solver|Borda|Firsts| |:---|---:|---:| {% for solver in report.solvers -%} |{{ solver }}|{{ report.borda[solver] }}|{{ report.firsts[solver] }}| {% endfor %} ## Individual Results {% for problem in report.problems %} ### ({{ problem.number }}) Problem: {{ problem.name }} |Ranking|Solver|{%- for metric in problem.metrics -%}{{ metric.name }} (avg +- std)|{% endfor %} |:---|---:|{%- for _ in range(problem.metrics|length) -%}---:|{% endfor %} {% for solver in problem.solvers -%} |{{ solver.rank }}|{{ solver.name }}|{{ solver.results|join('|') }}| {% endfor -%} {% endfor %} ## Datasets * [Breast Cancer Wisconsin](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer) * [Diabetes Data Set](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html#sklearn.datasets.load_diabetes) * [Digits](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits) * [Iris](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html#sklearn.datasets.load_iris) * [Wine](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_wine.html#sklearn.datasets.load_wine) ## Models * [AdaBoost](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) * [Decistion Tree](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) * [kNN](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) * [Linear Model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) * [Multi-layer Perceptron](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html) * [Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) * [SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)optuna-3.5.0/benchmarks/kurobako/000077500000000000000000000000001453453102400167605ustar00rootroot00000000000000optuna-3.5.0/benchmarks/kurobako/__init__.py000066400000000000000000000000001453453102400210570ustar00rootroot00000000000000optuna-3.5.0/benchmarks/kurobako/mo_create_study.py000066400000000000000000000017101453453102400225170ustar00rootroot00000000000000import json import sys from kurobako import solver from kurobako.solver.optuna import OptunaSolverFactory import optuna optuna.logging.disable_default_handler() def create_study(seed: int) -> optuna.Study: # Avoid the fail by `flake8`. seed n_objectives = 2 directions = ["minimize"] * n_objectives sampler_name = sys.argv[1] # Sampler. sampler_cls = getattr( optuna.samplers, sampler_name, getattr(optuna.integration, sampler_name, None), ) if sampler_cls is None: raise ValueError("Unknown sampler: {}.".format(sampler_name)) sampler_kwargs = json.loads(sys.argv[2]) sampler = sampler_cls(**sampler_kwargs) return optuna.create_study( directions=directions, sampler=sampler, pruner=optuna.pruners.NopPruner(), ) if __name__ == "__main__": factory = OptunaSolverFactory(create_study) runner = solver.SolverRunner(factory) runner.run() optuna-3.5.0/benchmarks/kurobako/problems/000077500000000000000000000000001453453102400206035ustar00rootroot00000000000000optuna-3.5.0/benchmarks/kurobako/problems/wfg/000077500000000000000000000000001453453102400213665ustar00rootroot00000000000000optuna-3.5.0/benchmarks/kurobako/problems/wfg/problem.py000066400000000000000000000642151453453102400234100ustar00rootroot00000000000000from __future__ import annotations import math import sys import numpy as np import shape_functions import transformation_functions from kurobako import problem class BaseWFG: def __init__( self, S: np.ndarray, A: np.ndarray, upper_bounds: np.ndarray, shapes: list[shape_functions.BaseShapeFunction], transformations: list[list[transformation_functions.BaseTransformations]], ) -> None: assert all(S > 0) assert all((A == 0) + (A == 1)) assert all(upper_bounds > 0) self._S = S self._A = A self._upper_bounds = upper_bounds self._shapes = shapes self._transformations = transformations def __call__(self, z: np.ndarray) -> np.ndarray: S = self._S A = self._A unit_z = z / self._upper_bounds shapes = self._shapes transformations = self._transformations y = unit_z for t_p in transformations: _y = np.empty((len(t_p),)) for i in range(len(t_p)): if isinstance(t_p[i], transformation_functions.BaseReductionTransformation): _y[i] = t_p[i](y) else: _y[i] = t_p[i](y[i]) y = _y x = np.empty(y.shape) x[:-1] = np.maximum(y[-1], A) * (y[:-1] - 0.5) + 0.5 x[-1] = y[-1] f = x[-1] + S * np.asarray([h(m + 1, x[:-1]) for m, h in enumerate(shapes)]) return f class WFG1: """WFG1 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConvexShapeFunction(M) for _ in range(M - 1)] shapes.append(shape_functions.MixedConvexOrConcaveShapeFunction(M, 1, 5)) transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(4)] transformations[0] = [transformation_functions.IdenticalTransformation() for _ in range(k)] # transformations[0] = [lambda y: y for _ in range(k)] for _ in range(n - k): transformations[0].append(transformation_functions.LinearShiftTransformation(0.35)) # transformations[1] = [lambda y: y for _ in range(k)] transformations[1] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for _ in range(n - k): transformations[1].append( transformation_functions.FlatRegionBiasTransformation(0.8, 0.75, 0.85) ) transformations[2] = [ transformation_functions.PolynomialBiasTransformation(0.02) for _ in range(n) ] def _input_converter(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[3] = [ transformation_functions.WeightedSumReductionTransformation( 2 * np.arange(i * k // (M - 1) + 1, (i + 1) * k // (M - 1) + 1), lambda y: _input_converter(i, y), ) for i in range(M - 1) ] transformations[3].append( transformation_functions.WeightedSumReductionTransformation( 2 * np.arange(k, n) + 1, lambda y: y[k:n], ) ) self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG2: """WFG2 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments // 2 assert (n_arguments - k) % 2 == 0 self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConvexShapeFunction(M) for _ in range(M - 1)] shapes.append(shape_functions.DisconnectedShapeFunction(M, 1, 1, 5)) transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(3)] transformations[0] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for _ in range(n - k): transformations[0].append(transformation_functions.LinearShiftTransformation(0.35)) def _input_converter0(i: int, y: np.ndarray) -> np.ndarray: indices = [k + 2 * (i + 1 - k) - 2, k + 2 * (i - k + 1) - 1] return y[indices] transformations[1] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for i in range(k, n // 2): transformations[1].append( transformation_functions.NonSeparableReductionTransformation( 2, lambda y: _input_converter0(i, y) ) ) def _input_converter1(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[2] = [ transformation_functions.WeightedSumReductionTransformation( np.ones(k // (M - 1)), lambda y: _input_converter1(i, y), ) for i in range(M - 1) ] transformations[2].append( transformation_functions.WeightedSumReductionTransformation( np.ones(n // 2 - k), lambda y: y[k : n // 2], ) ) # transformations = [transformations[0], transformations[1], transformations[2]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG3: """WFG3 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments // 2 assert (n_arguments - k) % 2 == 0 self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.zeros(M - 1) A[0] = 1 upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.LinearShapeFunction(M) for _ in range(M)] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(3)] transformations[0] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for _ in range(n - k): transformations[0].append(transformation_functions.LinearShiftTransformation(0.35)) def _input_converter0(i: int, y: np.ndarray) -> np.ndarray: indices = [k + 2 * (i + 1 - k) - 2, k + 2 * (i - k + 1) - 1] return y[indices] transformations[1] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for i in range(k, n // 2): transformations[1].append( transformation_functions.NonSeparableReductionTransformation( 2, lambda y: _input_converter0(i, y) ) ) def _input_converter1(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[2] = [ transformation_functions.WeightedSumReductionTransformation( np.ones(k // (M - 1)), lambda y: _input_converter1(i, y), ) for i in range(M - 1) ] transformations[2].append( transformation_functions.WeightedSumReductionTransformation( np.ones(n // 2 - k), lambda y: y[k : n // 2], ) ) # transformations = [transformations[0], transformations[1], transformations[2]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG4: """WFG4 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConcaveShapeFunction(M) for _ in range(M)] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(2)] transformations[0] = [ transformation_functions.MultiModalShiftTransformation(30, 10, 0.35) for _ in range(n) ] def _input_converter(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] # transformations[1] = [] for i in range(M - 1): transformations[1].append( transformation_functions.WeightedSumReductionTransformation( np.ones(k // (M - 1)), lambda y: _input_converter(i, y) ) ) transformations[1].append( transformation_functions.WeightedSumReductionTransformation( np.ones(n - k), lambda y: y[k:n], ) ) # transformations = [transformations[0], transformations[1]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG5: """WFG5 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConcaveShapeFunction(M) for _ in range(M)] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(2)] transformations[0] = [ transformation_functions.DeceptiveShiftTransformation(0.35, 0.001, 0.05) for _ in range(n) ] def _input_converter(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[1] = [] for i in range(M - 1): transformations[1].append( transformation_functions.WeightedSumReductionTransformation( np.ones(k // (M - 1)), lambda y: _input_converter(i, y) ) ) transformations[1].append( transformation_functions.WeightedSumReductionTransformation( np.ones(n - k), lambda y: y[k:n], ) ) # transformations = [transformations[0], transformations[1]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG6: """WFG6 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConcaveShapeFunction(M) for _ in range(M)] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(2)] transformations[0] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for _ in range(n - k): transformations[0].append(transformation_functions.LinearShiftTransformation(0.35)) def _input_converter(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] # transformations[1] = [] for i in range(M - 1): transformations[1].append( transformation_functions.NonSeparableReductionTransformation( k // (M - 1), lambda y: _input_converter(i, y) ) ) transformations[1].append( transformation_functions.NonSeparableReductionTransformation( n - k, lambda y: y[k:n], ) ) # transformations = [transformations[0], transformations[1]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG7: """WFG7 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConcaveShapeFunction(M) for _ in range(M)] def _input_converter0(i: int, y: np.ndarray) -> np.ndarray: return y[i:n] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(3)] transformations[0] = [ transformation_functions.ParameterDependentBiasTransformation( np.ones(n - i), lambda y: _input_converter0(i, y), 0.98 / 49.98, 0.02, 50, i, ) for i in range(k) ] for _ in range(n - k): transformations[0].append(transformation_functions.IdenticalTransformation()) transformations[1] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for _ in range(n - k): transformations[1].append(transformation_functions.LinearShiftTransformation(0.35)) def _input_converter1(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[2] = [] for i in range(M - 1): transformations[2].append( transformation_functions.WeightedSumReductionTransformation( np.ones(k // (M - 1)), lambda y: _input_converter1(i, y) ) ) transformations[2].append( transformation_functions.WeightedSumReductionTransformation( np.ones(n - k), lambda y: y[k:n], ) ) # transformations = [transformations[0], transformations[1], transformations[2]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG8: """WFG8 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConcaveShapeFunction(M) for _ in range(M)] def _input_converter0(i: int, y: np.ndarray) -> np.ndarray: return y[: i - 1] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(3)] transformations[0] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for i in range(k, n): transformations[0].append( transformation_functions.ParameterDependentBiasTransformation( np.ones(i - 1), lambda y: _input_converter0(i, y), 0.98 / 49.98, 0.02, 50, i, ) ) transformations[1] = [transformation_functions.IdenticalTransformation() for _ in range(k)] for _ in range(n - k): transformations[1].append(transformation_functions.LinearShiftTransformation(0.35)) def _input_converter(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[2] = [] for i in range(M - 1): transformations[2].append( transformation_functions.WeightedSumReductionTransformation( np.ones(k // (M - 1)), lambda y: _input_converter(i, y) ) ) transformations[2].append( transformation_functions.WeightedSumReductionTransformation( np.ones(n - k), lambda y: y[k:n], ) ) # transformations = [transformations[0], transformations[1], transformations[2]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFG9: """WFG9 Args: n_arguments: The number of arguments. n_objectives: The number of objectives. k: The degree of the Pareto front. """ def __init__(self, n_arguments: int, n_objectives: int, k: int): assert k % (n_objectives - 1) == 0 assert k + 1 <= n_arguments self._n_arguments = n_arguments self._n_objectives = n_objectives self._k = k n = self._n_arguments M = self._n_objectives S = 2 * (np.arange(M) + 1) A = np.ones(M - 1) upper_bounds = 2 * (np.arange(n) + 1) self.domain = np.zeros((n, 2)) self.domain[:, 1] = upper_bounds shapes: list[shape_functions.BaseShapeFunction] shapes = [shape_functions.ConcaveShapeFunction(M) for _ in range(M)] def _input_converter0(i: int, y: np.ndarray) -> np.ndarray: return y[i:n] transformations: list[list[transformation_functions.BaseTransformations]] transformations = [[] for _ in range(3)] transformations[0] = [ transformation_functions.ParameterDependentBiasTransformation( np.ones(n - i), lambda y: _input_converter0(i, y), 0.98 / 49.98, 0.02, 50, i, ) for i in range(n - 1) ] transformations[0].append(transformation_functions.IdenticalTransformation()) transformations[1] = [ transformation_functions.DeceptiveShiftTransformation(0.35, 0.001, 0.05) for _ in range(k) ] for _ in range(n - k): transformations[1].append( transformation_functions.MultiModalShiftTransformation(30, 95, 0.35) ) def _input_converter(i: int, y: np.ndarray) -> np.ndarray: indices = np.arange(i * k // (M - 1), (i + 1) * k // (M - 1)) return y[indices] transformations[2] = [] for i in range(M - 1): transformations[2].append( transformation_functions.NonSeparableReductionTransformation( k // (M - 1), lambda y: _input_converter(i, y) ) ) transformations[2].append( transformation_functions.NonSeparableReductionTransformation( n - k, lambda y: y[k:n], ) ) # transformations = [transformations[0], transformations[1], transformations[2]] self.wfg = BaseWFG(S, A, upper_bounds, shapes, transformations) def __call__(self, z: np.ndarray) -> np.ndarray: return self.wfg.__call__(z) class WFGProblemFactory(problem.ProblemFactory): def specification(self) -> problem.ProblemSpec: self._n_wfg = int(sys.argv[1]) self._n_dim = int(sys.argv[2]) self._low = 0 self._high = 2 params = [ problem.Var(f"x{i}", problem.ContinuousRange(0, self._high * i)) for i in range(self._n_dim) ] return problem.ProblemSpec( name=f"WFG{self._n_wfg}", params=params, values=[problem.Var("f1"), problem.Var("f2")], ) def create_problem(self, seed: int) -> problem.Problem: return WFGProblem() class WFGProblem(problem.Problem): def __init__(self) -> None: super().__init__() def create_evaluator(self, params: list[problem.Var]) -> problem.Evaluator: return WFGEvaluator(params) class WFGEvaluator(problem.Evaluator): def __init__(self, params: list[problem.Var]) -> None: self._n_wfg = int(sys.argv[1]) self._n_dim = int(sys.argv[2]) self._n_obj = int(sys.argv[3]) self._k = int(sys.argv[4]) self._x = np.array(params) self._current_step = 0 self.wfg: WFG1 | WFG2 | WFG3 | WFG4 | WFG5 | WFG6 | WFG7 | WFG8 | WFG9 if self._n_wfg == 1: self.wfg = WFG1(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 2: self.wfg = WFG2(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 3: self.wfg = WFG3(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 4: self.wfg = WFG4(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 5: self.wfg = WFG5(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 6: self.wfg = WFG6(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 7: self.wfg = WFG7(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 8: self.wfg = WFG8(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) elif self._n_wfg == 9: self.wfg = WFG9(n_arguments=self._n_dim, n_objectives=self._n_obj, k=self._k) else: raise AssertionError("Invalid specification for WFG number.") def current_step(self) -> int: return self._current_step def evaluate(self, next_step: int) -> list[float]: self._current_step = 1 v = self.wfg(self._x) v = v.tolist() if math.isnan(v[0]) or math.isinf(v[0]): raise ValueError if math.isnan(v[1]) or math.isinf(v[1]): raise ValueError return [v[0], v[1]] if __name__ == "__main__": runner = problem.ProblemRunner(WFGProblemFactory()) runner.run() optuna-3.5.0/benchmarks/kurobako/problems/wfg/shape_functions.py000066400000000000000000000057421453453102400251400ustar00rootroot00000000000000import abc import numpy as np class BaseShapeFunction(object, metaclass=abc.ABCMeta): def __init__(self, n_objectives: int) -> None: self._n_objectives = n_objectives def __call__(self, m: int, x: np.ndarray) -> float: assert 1 <= m <= self.n_objectives assert x.shape == (self.n_objectives - 1,) return self._call(m, x) @abc.abstractmethod def _call(self, m: int, x: np.ndarray) -> float: raise NotImplementedError @property def n_objectives(self) -> int: return self._n_objectives class LinearShapeFunction(BaseShapeFunction): def _call(self, m: int, x: np.ndarray) -> float: if m == 1: return x[:-1].prod() if m == self.n_objectives: return 1 - x[0] return x[: self.n_objectives - m].prod() * (1.0 - x[self.n_objectives - m]) class ConvexShapeFunction(BaseShapeFunction): def _call(self, m: int, x: np.ndarray) -> float: if m == 1: return ( 1 - np.cos( x * np.pi / 2, ) )[:-1].prod() if m == self.n_objectives: return 1 - np.sin(x[0] * np.pi / 2.0) return (1.0 - np.cos(x * np.pi / 2.0))[: self.n_objectives - m].prod() * ( 1.0 - np.sin(x[self.n_objectives - m] * np.pi / 2.0) ) class ConcaveShapeFunction(BaseShapeFunction): def _call(self, m: int, x: np.ndarray) -> float: if m == 1: return np.sin(x * np.pi / 2.0)[:-1].prod() if m == self.n_objectives: return np.cos(x[0] * np.pi / 2.0) return np.sin(x * np.pi / 2.0)[: self.n_objectives - m].prod() * np.cos( x[self.n_objectives - m] * np.pi / 2.0 ) class MixedConvexOrConcaveShapeFunction(BaseShapeFunction): def __init__(self, n_objectives: int, alpha: float, n_segments: int) -> None: super().__init__(n_objectives) self._alpha = alpha self._n_segments = n_segments def _call(self, m: int, x: np.ndarray) -> float: if m == self.n_objectives: two_A_pi = 2 * self._n_segments * np.pi return np.power( 1 - x[0] - np.cos(two_A_pi * x[0] + np.pi / 2.0) / two_A_pi, self._alpha ) raise ValueError("m should be the number of objectives") class DisconnectedShapeFunction(BaseShapeFunction): def __init__( self, n_objectives: int, alpha: float, beta: float, n_disconnected_regions: int ) -> None: super().__init__(n_objectives) self._alpha = alpha self._beta = beta self._n_disconnected_regions = n_disconnected_regions def _call(self, m: int, x: np.ndarray) -> float: if m == self.n_objectives: return ( 1 - np.power(x[0], self._alpha) * np.cos(self._n_disconnected_regions * np.power(x[0], self._beta) * np.pi) ** 2 ) raise ValueError("m should be the number of objectives") optuna-3.5.0/benchmarks/kurobako/problems/wfg/transformation_functions.py000066400000000000000000000123231453453102400270770ustar00rootroot00000000000000from __future__ import annotations import abc from typing import Callable from typing import Union import numpy as np class BaseIdenticalTransformation(metaclass=abc.ABCMeta): @abc.abstractmethod def __call__(self, y: float) -> float: raise NotImplementedError class BaseBiasTransformation(metaclass=abc.ABCMeta): @abc.abstractmethod def __call__(self, y: float) -> float: raise NotImplementedError class BaseShiftTransformation(metaclass=abc.ABCMeta): @abc.abstractmethod def __call__(self, y: float) -> float: raise NotImplementedError class BaseReductionTransformation(metaclass=abc.ABCMeta): @abc.abstractmethod def __call__(self, y: np.ndarray) -> float: raise NotImplementedError BaseTransformations = Union[ BaseIdenticalTransformation, BaseBiasTransformation, BaseShiftTransformation, BaseReductionTransformation, ] class IdenticalTransformation(BaseIdenticalTransformation): def __init__(self) -> None: pass def __call__(self, y: float) -> float: return y class PolynomialBiasTransformation(BaseBiasTransformation): def __init__(self, alpha: float) -> None: assert alpha > 0 and alpha != 1.0 self._alpha = alpha def __call__(self, y: float) -> float: return np.power(y, self._alpha) class FlatRegionBiasTransformation(BaseBiasTransformation): def __init__(self, a: float, b: float, c: float) -> None: assert 0 <= a <= 1 assert 0 <= b <= 1 assert 0 <= c <= 1 assert b < c assert not (b == 0) or (a == 0 and c != 1) assert not (c == 1) or (a == 1 and b != 0) self._a = a self._b = b self._c = c def __call__(self, y: float) -> float: a = self._a b = self._b c = self._c return ( a + min(0, np.floor(y - b)) * a * (b - y) / b - min(0, np.floor(c - y)) * (1.0 - a) * (y - c) / (1.0 - c) ) class ParameterDependentBiasTransformation(BaseReductionTransformation): def __init__( self, w: np.ndarray, input_converter: Callable[[np.ndarray], np.ndarray], a: float, b: float, c: float, i: int, ) -> None: assert 0 < a < 1 assert 0 < b < c self._w = w self._input_converter = input_converter self._a = a self._b = b self._c = c self._i = i def __call__(self, y: np.ndarray) -> float: w = self._w a = self._a b = self._b c = self._c i = self._i u = (self._input_converter(y) * w).sum() / w.sum() v = a - (1.0 - 2 * u) * np.fabs(np.floor(0.5 - u) + a) return np.power(y[i], b + (c - b) * v) class LinearShiftTransformation(BaseShiftTransformation): def __init__(self, a: float) -> None: assert 0 < a < 1 self._a = a def __call__(self, y: float) -> float: return np.fabs(y - self._a) / np.fabs(np.floor(self._a - y) + self._a) class DeceptiveShiftTransformation(BaseShiftTransformation): def __init__(self, a: float, b: float, c: float) -> None: assert 0 < a < 1 assert 0 < b < 1 assert 0 < c < 1 assert a - b > 0 assert a + b < 1 self._a = a self._b = b self._c = c def __call__(self, y: float) -> float: a = self._a b = self._b c = self._c q1 = np.floor(y - a + b) * (1.0 - c + (a - b) / b) q2 = np.floor(a + b - y) * (1.0 - c + (1.0 - a - b) / b) return 1.0 + (np.fabs(y - a) - b) * (q1 / (a - b) + q2 / (1.0 - a - b) + 1.0 / b) class MultiModalShiftTransformation(BaseShiftTransformation): def __init__(self, a: int, b: float, c: float) -> None: assert a > 0 assert b >= 0 assert (4 * a + 2) * np.pi >= 4 * b assert 0 < c < 1 self._a = a self._b = b self._c = c def __call__(self, y: float) -> float: a = self._a b = self._b c = self._c q1 = np.fabs(y - c) / (2 * (np.floor(c - y) + c)) q2 = (4 * a + 2) * np.pi * (0.5 - q1) return (1.0 + np.cos(q2) + 4 * b * (q1**2)) / (b + 2) class WeightedSumReductionTransformation(BaseReductionTransformation): def __init__(self, w: np.ndarray, input_converter: Callable[[np.ndarray], np.ndarray]) -> None: assert all(w > 0) self._w = w self._input_converter = input_converter def __call__(self, y: np.ndarray) -> float: y = self._input_converter(y) return (y * self._w).sum() / self._w.sum() class NonSeparableReductionTransformation(BaseReductionTransformation): def __init__(self, a: int, input_converter: Callable[[np.ndarray], np.ndarray]) -> None: assert a > 0 self._a = a self._input_converter = input_converter def __call__(self, y: np.ndarray) -> float: a = float(self._a) y = self._input_converter(y) n = y.shape[0] indices = [(j + k + 1) % n for k in np.arange(n) for j in np.arange(n)] q = y.sum() + np.fabs(y[indices].reshape((n, n)) - y)[:, 0 : int(a) - 1].sum() r = n * np.ceil(a / 2) * (1.0 + 2 * a - 2 * np.ceil(a / 2)) / a return q / r optuna-3.5.0/benchmarks/naslib/000077500000000000000000000000001453453102400164135ustar00rootroot00000000000000optuna-3.5.0/benchmarks/naslib/__init__.py000066400000000000000000000000001453453102400205120ustar00rootroot00000000000000optuna-3.5.0/benchmarks/naslib/problem.py000066400000000000000000000053201453453102400204250ustar00rootroot00000000000000from __future__ import annotations import sys from typing import Any from kurobako import problem from naslib.utils import get_dataset_api op_names = [ "skip_connect", "none", "nor_conv_3x3", "nor_conv_1x1", "avg_pool_3x3", ] edge_num = 4 * 3 // 2 max_epoch = 199 prune_start_epoch = 10 prune_epoch_step = 10 class NASLibProblemFactory(problem.ProblemFactory): def __init__(self, dataset: str) -> None: """Creates ProblemFactory for NASBench201. Args: dataset: Accepts one of "cifar10", "cifar100" or "ImageNet16-120". """ self._dataset = dataset if dataset == "cifar10": # Set name used in dataset API. self._dataset = "cifar10-valid" self._dataset_api = get_dataset_api("nasbench201", dataset) def specification(self) -> problem.ProblemSpec: params = [ problem.Var(f"x{i}", problem.CategoricalRange(op_names)) for i in range(edge_num) ] return problem.ProblemSpec( name=f"NASBench201-{self._dataset}", params=params, values=[problem.Var("value")], steps=list(range(prune_start_epoch, max_epoch, prune_epoch_step)) + [max_epoch], ) def create_problem(self, seed: int) -> problem.Problem: return NASLibProblem(self._dataset, self._dataset_api) class NASLibProblem(problem.Problem): def __init__(self, dataset: str, dataset_api: Any) -> None: super().__init__() self._dataset = dataset self._dataset_api = dataset_api def create_evaluator(self, params: list[float]) -> problem.Evaluator: ops = [op_names[int(x)] for x in params] arch_str = "|{}~0|+|{}~0|{}~1|+|{}~0|{}~1|{}~2|".format(*ops) return NASLibEvaluator( self._dataset_api["nb201_data"][arch_str][self._dataset]["eval_acc1es"] ) class NASLibEvaluator(problem.Evaluator): def __init__(self, learning_curve: list[float]) -> None: self._current_step = 0 self._lc = learning_curve def current_step(self) -> int: return self._current_step def evaluate(self, next_step: int) -> list[float]: self._current_step = next_step return [-self._lc[next_step]] if __name__ == "__main__": if len(sys.argv) < 1 + 2: print("Usage: python3 nas_bench_suite/problems.py ") print("Example: python3 nas_bench_suite/problems.py nasbench201 cifar10") exit(1) search_space_name = sys.argv[1] # We currently do not support other benchmarks. assert search_space_name == "nasbench201" dataset = sys.argv[2] runner = problem.ProblemRunner(NASLibProblemFactory(dataset)) runner.run() optuna-3.5.0/benchmarks/run_bayesmark.py000066400000000000000000000157571453453102400203760ustar00rootroot00000000000000from __future__ import annotations import argparse import json import os import subprocess from matplotlib import cm from matplotlib import colors from matplotlib.axes import Axes import matplotlib.pyplot as plt import numpy as np import pandas as pd from xarray import Dataset _DB = "bo_optuna_run" def run_benchmark(args: argparse.Namespace) -> None: sampler_list = args.sampler_list.split() sampler_kwargs_list = args.sampler_kwargs_list.split() pruner_list = args.pruner_list.split() pruner_kwargs_list = args.pruner_kwargs_list.split() if len(sampler_list) != len(sampler_kwargs_list): raise ValueError( "The number of samplers does not match the given keyword arguments. \n" f"sampler_list: {sampler_list}, sampler_kwargs_list: {sampler_kwargs_list}." ) if len(pruner_list) != len(pruner_kwargs_list): raise ValueError( "The number of pruners does not match the given keyword arguments. \n" f"pruner_list: {pruner_list}, pruner_keyword_arguments: {pruner_kwargs_list}." ) config = dict() for i, (sampler, sampler_kwargs) in enumerate(zip(sampler_list, sampler_kwargs_list)): sampler_name = sampler if sampler_list.count(sampler) > 1: sampler_name += f"_{sampler_list[:i].count(sampler)}" for j, (pruner, pruner_kwargs) in enumerate(zip(pruner_list, pruner_kwargs_list)): pruner_name = pruner if pruner_list.count(pruner) > 1: pruner_name += f"_{pruner_list[:j].count(pruner)}" optimizer_name = f"{args.name_prefix}_{sampler_name}_{pruner_name}" optimizer_kwargs = { "sampler": sampler, "sampler_kwargs": json.loads(sampler_kwargs), "pruner": pruner, "pruner_kwargs": json.loads(pruner_kwargs), } # We need to dynamically generate config.json sice sampler pruner combos (solvers) # are parametrized by user. Following sample config schema. # https://github.com/uber/bayesmark/blob/master/example_opt_root/config.json config[optimizer_name] = ["optuna_optimizer.py", optimizer_kwargs] with open(os.path.join("benchmarks", "bayesmark", "config.json"), "w") as file: json.dump(config, file, indent=4) samplers = " ".join(config.keys()) metric = "nll" if args.dataset in ["breast", "iris", "wine", "digits"] else "mse" cmd = ( f"bayesmark-launch -n {args.budget} -r {args.n_runs} " f"-dir runs -b {_DB} " f"-o {samplers} " f"-c {args.model} -d {args.dataset} " f"-m {metric} --opt-root benchmarks/bayesmark" ) subprocess.run(cmd, shell=True) def make_plots(args: argparse.Namespace) -> None: filename = f"{args.dataset}-{args.model}-partial-report.json" df = pd.read_json(os.path.join("partial", filename)) df["best_value"] = df.groupby(["opt", "uuid"]).generalization.cummin() summaries = ( df.groupby(["opt", "iter"]) .best_value.agg(["mean", "std"]) .rename(columns={"mean": "best_mean", "std": "best_std"}) .reset_index() ) fig, ax = plt.subplots() fig.set_size_inches(12, 8) warmup = json.loads(args.plot_warmup) metric = df.metric[0] color_lookup = build_color_dict(sorted(df["opt"].unique())) for optimizer, summary in summaries.groupby("opt"): color = color_lookup[optimizer] make_plot(summary, ax, optimizer, metric, warmup, color) handles, labels = ax.get_legend_handles_labels() fig.legend(handles, labels) fig.suptitle(f"Bayesmark-{args.dataset.capitalize()}-{args.model}") fig.savefig(os.path.join("plots", f"optuna-{args.dataset}-{args.model}-sumamry.png")) def make_plot( summary: pd.DataFrame, ax: Axes, optimizer: str, metric: str, plot_warmup: bool, color: np.ndarray, ) -> None: start = 0 if plot_warmup else 10 if len(summary.best_mean) <= start: return ax.fill_between( np.arange(len(summary.best_mean))[start:], (summary.best_mean - summary.best_std)[start:], (summary.best_mean + summary.best_std)[start:], color=color, alpha=0.25, step="mid", ) ax.plot( np.arange(len(summary.best_mean))[start:], summary.best_mean[start:], color=color, label=optimizer, drawstyle="steps-mid", ) ax.set_xlabel("Budget", fontsize=10) ax.set_ylabel(f"Validation {metric.upper()}", fontsize=10) ax.grid(alpha=0.2) def build_color_dict(names: list[str]) -> dict[str, np.ndarray]: norm = colors.Normalize(vmin=0, vmax=1) m = cm.ScalarMappable(norm, cm.tab20) color_dict = m.to_rgba(np.linspace(0, 1, len(names))) color_dict = dict(zip(names, color_dict)) return color_dict def partial_report(args: argparse.Namespace) -> None: eval_path = os.path.join("runs", _DB, "eval") time_path = os.path.join("runs", _DB, "time") studies = os.listdir(eval_path) summaries: list[pd.DataFrame] = [] for study in studies: table_buffer: list[pd.DataFrame] = [] column_buffer: list[str] = [] for path in [eval_path, time_path]: with open(os.path.join(path, study), "r") as file: data = json.load(file) df = Dataset.from_dict(data["data"]).to_dataframe() df = df.droplevel("suggestion") for argument, meatadata in data["meta"]["args"].items(): colname = argument[2:] if argument.startswith("--") else argument if colname not in column_buffer: df[colname] = meatadata column_buffer.append(colname) table_buffer.append(df) summary = pd.merge(*table_buffer, left_index=True, right_index=True) summaries.append(summary.reset_index()) filename = f"{args.dataset}-{args.model}-partial-report.json" pd.concat(summaries).reset_index(drop=True).to_json(os.path.join("partial", filename)) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--dataset", type=str, default="iris") parser.add_argument("--model", type=str, default="kNN") parser.add_argument("--name-prefix", type=str, default="") parser.add_argument("--budget", type=int, default=80) parser.add_argument("--n-runs", type=int, default=10) parser.add_argument("--sampler-list", type=str, default="TPESampler CmaEsSampler") parser.add_argument( "--sampler-kwargs-list", type=str, default='{"multivariate":true,"constant_liar":true} {}', ) parser.add_argument("--pruner-list", type=str, default="NopPruner") parser.add_argument("--pruner-kwargs-list", type=str, default="{}") parser.add_argument("--plot-warmup", type=str, default="true") args = parser.parse_args() os.makedirs("runs", exist_ok=True) os.makedirs("plots", exist_ok=True) os.makedirs("partial", exist_ok=True) run_benchmark(args) partial_report(args) make_plots(args) optuna-3.5.0/benchmarks/run_kurobako.py000066400000000000000000000120321453453102400202140ustar00rootroot00000000000000import argparse import os import subprocess def run(args: argparse.Namespace) -> None: kurobako_cmd = os.path.join(args.path_to_kurobako, "kurobako") subprocess.run(f"{kurobako_cmd} --version", shell=True) if not (os.path.exists(args.data_dir) and os.path.isdir(args.data_dir)): raise ValueError(f"Data directory {args.data_dir} cannot be found.") os.makedirs(args.out_dir, exist_ok=True) study_json_fn = os.path.join(args.out_dir, "studies.json") solvers_filename = os.path.join(args.out_dir, "solvers.json") problems_filename = os.path.join(args.out_dir, "problems.json") # Ensure all files are empty. for filename in [study_json_fn, solvers_filename, problems_filename]: with open(filename, "w"): pass # Create HPO bench problem. datasets = [ "fcnet_tabular_benchmarks/fcnet_naval_propulsion_data.hdf5", "fcnet_tabular_benchmarks/fcnet_parkinsons_telemonitoring_data.hdf5", "fcnet_tabular_benchmarks/fcnet_protein_structure_data.hdf5", "fcnet_tabular_benchmarks/fcnet_slice_localization_data.hdf5", ] for dataset in datasets: dataset = os.path.join(args.data_dir, dataset) cmd = f'{kurobako_cmd} problem hpobench "{dataset}" | tee -a {problems_filename}' subprocess.run(cmd, shell=True) # Create NAS bench problem. dataset = os.path.join(args.data_dir, "nasbench_full.bin") cmd = f'{kurobako_cmd} problem nasbench "{dataset}" | tee -a {problems_filename}' subprocess.run(cmd, shell=True) # Create solvers. sampler_list = args.sampler_list.split() sampler_kwargs_list = args.sampler_kwargs_list.split() pruner_list = args.pruner_list.split() pruner_kwargs_list = args.pruner_kwargs_list.split() if len(sampler_list) != len(sampler_kwargs_list): raise ValueError( "The number of samplers does not match the given keyword arguments. \n" f"sampler_list: {sampler_list}, sampler_kwargs_list: {sampler_kwargs_list}." ) if len(pruner_list) != len(pruner_kwargs_list): raise ValueError( "The number of pruners does not match the given keyword arguments. \n" f"pruner_list: {pruner_list}, pruner_keyword_arguments: {pruner_kwargs_list}." ) for i, (sampler, sampler_kwargs) in enumerate(zip(sampler_list, sampler_kwargs_list)): sampler_name = sampler if sampler_list.count(sampler) > 1: sampler_name += f"_{sampler_list[:i].count(sampler)}" for j, (pruner, pruner_kwargs) in enumerate(zip(pruner_list, pruner_kwargs_list)): pruner_name = pruner if pruner_list.count(pruner) > 1: pruner_name += f"_{pruner_list[:j].count(pruner)}" name = f"{args.name_prefix}_{sampler_name}_{pruner_name}" cmd = ( f"{kurobako_cmd} solver --name {name} optuna --loglevel debug " f"--sampler {sampler} --sampler-kwargs {sampler_kwargs} " f"--pruner {pruner} --pruner-kwargs {pruner_kwargs} " f"| tee -a {solvers_filename}" ) subprocess.run(cmd, shell=True) # Create study. cmd = ( f"{kurobako_cmd} studies --budget {args.budget} " f"--solvers $(cat {solvers_filename}) --problems $(cat {problems_filename}) " f"--repeats {args.n_runs} --seed {args.seed} --concurrency {args.n_concurrency} " f"> {study_json_fn}" ) subprocess.run(cmd, shell=True) result_filename = os.path.join(args.out_dir, "results.json") cmd = ( f"cat {study_json_fn} | {kurobako_cmd} run --parallelism {args.n_jobs} " f"> {result_filename}" ) subprocess.run(cmd, shell=True) report_filename = os.path.join(args.out_dir, "report.md") cmd = f"cat {result_filename} | {kurobako_cmd} report > {report_filename}" subprocess.run(cmd, shell=True) cmd = ( f"cat {result_filename} | {kurobako_cmd} plot curve --errorbar -o {args.out_dir} --xmin 10" ) subprocess.run(cmd, shell=True) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--path-to-kurobako", type=str, default="") parser.add_argument("--name-prefix", type=str, default="") parser.add_argument("--budget", type=int, default=80) parser.add_argument("--n-runs", type=int, default=100) parser.add_argument("--n-jobs", type=int, default=10) parser.add_argument("--n-concurrency", type=int, default=1) parser.add_argument("--sampler-list", type=str, default="RandomSampler TPESampler") parser.add_argument( "--sampler-kwargs-list", type=str, default=r"{} {\"multivariate\":true\,\"constant_liar\":true}", ) parser.add_argument("--pruner-list", type=str, default="NopPruner") parser.add_argument("--pruner-kwargs-list", type=str, default="{}") parser.add_argument("--seed", type=int, default=0) parser.add_argument("--data-dir", type=str, default="data") parser.add_argument("--out-dir", type=str, default="out") args = parser.parse_args() run(args) optuna-3.5.0/benchmarks/run_mo_kurobako.py000066400000000000000000000142171453453102400207160ustar00rootroot00000000000000from __future__ import annotations import argparse import os import subprocess def run(args: argparse.Namespace) -> None: kurobako_cmd = os.path.join(args.path_to_kurobako, "kurobako") subprocess.run(f"{kurobako_cmd} --version", shell=True) if not (os.path.exists(args.data_dir) and os.path.isdir(args.data_dir)): raise ValueError(f"Data directory {args.data_dir} cannot be found.") os.makedirs(args.out_dir, exist_ok=True) study_json_filename = os.path.join(args.out_dir, "studies.json") solvers_filename = os.path.join(args.out_dir, "solvers.json") problems_filename = os.path.join(args.out_dir, "problems.json") # Ensure all files are empty. for filename in [study_json_filename, solvers_filename, problems_filename]: with open(filename, "w"): pass # Create ZDT problems cmd = f"{kurobako_cmd} problem-suite zdt | tee -a {problems_filename}" subprocess.run(cmd, shell=True) # Create WFG 1~9 problem for n_wfg in range(1, 10): if n_wfg == 8: n_dim = 3 k = 2 elif n_wfg in (7, 9): n_dim = 2 k = 1 else: n_dim = 10 k = 2 n_objective = 2 python_command = f"benchmarks/kurobako/problems/wfg/problem.py \ {n_wfg} {n_dim} {n_objective} {k}" cmd = ( f"{kurobako_cmd} problem command python {python_command}" f"| tee -a {problems_filename}" ) subprocess.run(cmd, shell=True) # Create NAS bench problem(A) (for Multi-Objective Settings). dataset = os.path.join(args.data_dir, "nasbench_full.bin") cmd = ( f'{kurobako_cmd} problem nasbench "{dataset}" ' f"--metrics params accuracy | tee -a {problems_filename}" ) subprocess.run(cmd, shell=True) # Create solvers. sampler_list = args.sampler_list.split() sampler_kwargs_list = args.sampler_kwargs_list.split() if len(sampler_list) != len(sampler_kwargs_list): raise ValueError( "The number of samplers does not match the given keyword arguments. \n" f"sampler_list: {sampler_list}, sampler_kwargs_list: {sampler_kwargs_list}." ) for i, (sampler, sampler_kwargs) in enumerate(zip(sampler_list, sampler_kwargs_list)): sampler_name = sampler if sampler_list.count(sampler) > 1: sampler_name += f"_{sampler_list[:i].count(sampler)}" name = f"{args.name_prefix}_{sampler_name}" python_command = f"{args.path_to_create_study} {sampler} {sampler_kwargs}" cmd = ( f"{kurobako_cmd} solver --name {name} command python3 {python_command}" f"| tee -a {solvers_filename}" ) subprocess.run(cmd, shell=True) # Create study. cmd = ( f"{kurobako_cmd} studies --budget {args.budget} " f"--solvers $(cat {solvers_filename}) --problems $(cat {problems_filename}) " f"--repeats {args.n_runs} --seed {args.seed} --concurrency {args.n_concurrency} " f"> {study_json_filename}" ) subprocess.run(cmd, shell=True) result_filename = os.path.join(args.out_dir, "results.json") cmd = ( f"cat {study_json_filename} | {kurobako_cmd} run --parallelism {args.n_jobs} -q " f"> {result_filename}" ) subprocess.run(cmd, shell=True) # Report. report_filename = os.path.join(args.out_dir, "report.md") cmd = f"cat {result_filename} | {kurobako_cmd} report > {report_filename}" subprocess.run(cmd, shell=True) # Plot pareto-front. plot_args: dict[str, dict[str, int | float]] plot_args = { "NASBench": {"xmin": 0, "xmax": 25000000, "ymin": 0, "ymax": 0.2}, "ZDT1": {"xmin": 0, "xmax": 1, "ymin": 1, "ymax": 7}, "ZDT2": {"xmin": 0, "xmax": 1, "ymin": 2, "ymax": 7}, "ZDT3": {"xmin": 0, "xmax": 1, "ymin": 0, "ymax": 7}, "ZDT4": {"xmin": 0, "xmax": 1, "ymin": 20, "ymax": 250}, "ZDT5": {"xmin": 8, "xmax": 24, "ymin": 1, "ymax": 6}, "ZDT6": {"xmin": 0.2, "xmax": 1, "ymin": 5, "ymax": 10}, "WFG1": {"xmin": 2.7, "xmax": 3.05, "ymin": 4.7, "ymax": 5.05}, "WFG2": {"xmin": 2.0, "xmax": 2.8, "ymin": 3.0, "ymax": 4.8}, "WFG3": {"xmin": 2.0, "xmax": 2.8, "ymin": 3.0, "ymax": 4.8}, "WFG4": {"xmin": 2.0, "xmax": 3.0, "ymin": 0.0, "ymax": 3.6}, "WFG5": {"xmin": 2.0, "xmax": 3.0, "ymin": 2.5, "ymax": 5.0}, "WFG6": {"xmin": 2.0, "xmax": 3.0, "ymin": 3.4, "ymax": 5.0}, "WFG7": {"xmin": 2.0, "xmax": 3.0, "ymin": 4.0, "ymax": 5.0}, "WFG8": {"xmin": 2.0, "xmax": 3.0, "ymin": 3.4, "ymax": 5.0}, "WFG9": {"xmin": 2.0, "xmax": 3.0, "ymin": 0.0, "ymax": 5.0}, } for problem_name, plot_arg in plot_args.items(): xmin, xmax = plot_arg["xmin"], plot_arg["xmax"] ymin, ymax = plot_arg["ymin"], plot_arg["ymax"] cmd = ( f"cat {result_filename} | grep {problem_name} | " f"{kurobako_cmd} plot pareto-front -o {args.out_dir} " f"--xmin {xmin} --xmax {xmax} --ymin {ymin} --ymax {ymax}" ) subprocess.run(cmd, shell=True) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--path-to-kurobako", type=str, default="") parser.add_argument( "--path-to-create-study", type=str, default="benchmarks/kurobako/mo_create_study.py" ) parser.add_argument("--name-prefix", type=str, default="") parser.add_argument("--budget", type=int, default=120) parser.add_argument("--n-runs", type=int, default=100) parser.add_argument("--n-jobs", type=int, default=10) parser.add_argument("--n-concurrency", type=int, default=1) parser.add_argument( "--sampler-list", type=str, default="RandomSampler TPESampler NSGAIISampler", ) parser.add_argument( "--sampler-kwargs-list", type=str, default=r"{} {\"multivariate\":true\,\"constant_liar\":true} {\"population_size\":20}", ) parser.add_argument("--seed", type=int, default=0) parser.add_argument("--data-dir", type=str, default="data") parser.add_argument("--out-dir", type=str, default="out") args = parser.parse_args() run(args) optuna-3.5.0/benchmarks/run_naslib.py000066400000000000000000000110351453453102400176510ustar00rootroot00000000000000import argparse import os import subprocess def run(args: argparse.Namespace) -> None: kurobako_cmd = os.path.join(args.path_to_kurobako, "kurobako") subprocess.run(f"{kurobako_cmd} --version", shell=True) os.makedirs(args.out_dir, exist_ok=True) study_json_filename = os.path.join(args.out_dir, "studies.json") solvers_filename = os.path.join(args.out_dir, "solvers.json") problems_filename = os.path.join(args.out_dir, "problems.json") # Ensure all files are empty. for filename in [study_json_filename, solvers_filename, problems_filename]: with open(filename, "w"): pass searchspace_datasets = [ "nasbench201 cifar10", "nasbench201 cifar100", "nasbench201 ImageNet16-120", ] for searchspace_dataset in searchspace_datasets: python_command = f"benchmarks/naslib/problem.py {searchspace_dataset}" cmd = ( f"{kurobako_cmd} problem command python3 {python_command}" f"| tee -a {problems_filename}" ) subprocess.run(cmd, shell=True) # Create solvers. sampler_list = args.sampler_list.split() sampler_kwargs_list = args.sampler_kwargs_list.split() pruner_list = args.pruner_list.split() pruner_kwargs_list = args.pruner_kwargs_list.split() if len(sampler_list) != len(sampler_kwargs_list): raise ValueError( "The number of samplers does not match the given keyword arguments. \n" f"sampler_list: {sampler_list}, sampler_kwargs_list: {sampler_kwargs_list}." ) if len(pruner_list) != len(pruner_kwargs_list): raise ValueError( "The number of pruners does not match the given keyword arguments. \n" f"pruner_list: {pruner_list}, pruner_kwargs_list: {pruner_kwargs_list}." ) for i, (sampler, sampler_kwargs) in enumerate(zip(sampler_list, sampler_kwargs_list)): sampler_name = sampler if sampler_list.count(sampler) > 1: sampler_name += f"_{sampler_list[:i].count(sampler)}" for j, (pruner, pruner_kwargs) in enumerate(zip(pruner_list, pruner_kwargs_list)): pruner_name = pruner if pruner_list.count(pruner) > 1: pruner_name += f"_{pruner_list[:j].count(pruner)}" name = f"{args.name_prefix}_{sampler_name}_{pruner_name}" cmd = ( f"{kurobako_cmd} solver --name {name} optuna --loglevel debug " f"--sampler {sampler} --sampler-kwargs {sampler_kwargs} " f"--pruner {pruner} --pruner-kwargs {pruner_kwargs} " f"| tee -a {solvers_filename}" ) subprocess.run(cmd, shell=True) # Create study. cmd = ( f"{kurobako_cmd} studies --budget {args.budget} " f"--solvers $(cat {solvers_filename}) --problems $(cat {problems_filename}) " f"--repeats {args.n_runs} --seed {args.seed} --concurrency {args.n_concurrency} " f"> {study_json_filename}" ) subprocess.run(cmd, shell=True) result_filename = os.path.join(args.out_dir, "results.json") cmd = ( f"cat {study_json_filename} | {kurobako_cmd} run --parallelism {args.n_jobs} -q " f"> {result_filename}" ) subprocess.run(cmd, shell=True) # Report. report_filename = os.path.join(args.out_dir, "report.md") cmd = f"cat {result_filename} | {kurobako_cmd} report > {report_filename}" subprocess.run(cmd, shell=True) cmd = ( f"cat {result_filename} | {kurobako_cmd} plot curve --errorbar -o {args.out_dir} --xmin 10" ) subprocess.run(cmd, shell=True) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--path-to-kurobako", type=str, default="") parser.add_argument("--name-prefix", type=str, default="") parser.add_argument("--budget", type=int, default=100) parser.add_argument("--n-runs", type=int, default=10) parser.add_argument("--n-jobs", type=int, default=10) parser.add_argument("--n-concurrency", type=int, default=1) parser.add_argument("--sampler-list", type=str, default="RandomSampler TPESampler") parser.add_argument( "--sampler-kwargs-list", type=str, default=r"{} {\"multivariate\":true\,\"constant_liar\":true}", ) parser.add_argument("--pruner-list", type=str, default="NopPruner") parser.add_argument("--pruner-kwargs-list", type=str, default="{}") parser.add_argument("--seed", type=int, default=0) parser.add_argument("--out-dir", type=str, default="out") args = parser.parse_args() run(args) optuna-3.5.0/docs/000077500000000000000000000000001453453102400137565ustar00rootroot00000000000000optuna-3.5.0/docs/Makefile000066400000000000000000000022271453453102400154210ustar00rootroot00000000000000# Minimal makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = -W --keep-going SPHINXBUILD = sphinx-build SPHINXPROJ = Optuna SOURCEDIR = source BUILDDIR = build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) doctest: sphinx-autogen source/**/*.rst @$(SPHINXBUILD) -M doctest "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) # Copy from https://sphinx-gallery.github.io/stable/advanced.html#cleaning-the-gallery-files clean: rm -rf $(BUILDDIR)/* rm -rf source/reference/generated/ rm -rf source/reference/visualization/generated/ rm -rf source/tutorial/10_key_features rm -rf source/tutorial/20_recipes rm -f ../tutorial/**/*.db rm -rf ../tutorial/20_recipes/artifacts rm -rf ../tutorial/20_recipes/tmp rm -rf ../tutorial/20_recipes/best_atoms.png optuna-3.5.0/docs/image/000077500000000000000000000000001453453102400150405ustar00rootroot00000000000000optuna-3.5.0/docs/image/favicon.ico000066400000000000000000000420761453453102400171720ustar00rootroot00000000000000@@ (D(@€   ÿ€€™f3™3U™f¿€@ÿÿÿ€€€™f3Uÿÿ̸Žq¹•{µ”r&®€W&º˜|%º“v»ªˆÿÿÿ™3€€@ÿÿÿªUUªUÿÿÿ°ˆl-¦zY\ŸmGˆe:«™a0Ƙ]'Ø—^+å™`-í˜[!í™`1ì˜^*â™_,Õšb2Àh<¢ mF|¦wSM³‘o€@€€€€@@ÿÿÿ©€_>žlE‰›c7É—\&ò›[ÿžZÿžYÿœWÿšVÿ˜Tÿ—Sÿ—Tÿ—Sÿ™UÿšWÿXÿžZÿž[ÿ˜Zÿ˜]'çœf;·¢uQq·–|'€@€€€ªUU€ʪ•£uSo›d8É–Y!þœXÿ›Vÿ—Tÿ”Sû”Sø•Tø•Tû–Uÿ˜Wÿ˜Wÿ™Xÿ™Wÿ˜Wÿ—Uþ•Uû•Tø•Tø•Tý™VÿXÿ[ÿ˜^+ñŸnGªªƒcH™f3™f3¿ŸŠ¤yY™b7äœZÿšTÿ•Sÿ”Tù•U÷–TþœYÿ¢^ÿ¢aÿ bÿ›`ýœ`ôŸg3ñe/ëŸg0ñœaõ›_ÿ aÿ£aÿ¢_ÿZÿ–Tþ–V÷–Uù—TÿžXÿš\"ÿžjA»ª€`H€€€€€€€Uÿÿÿ¤vTj™a3àžZÿ—Rÿ“Rû–V÷•TüœYÿ£a ÿb$ÿ i4Û£m=«¥rC}©}V\¯€\@®‚W/¹œ…,º˜ƒ%¹—z,­…W2ª~UEªzP`¥sG…¢m:¯žg/Ýžc$ÿ¤a ÿžZÿ–Vü–Vö•SþWÿš]$ÿ nH©·–|'€€@ªªU·•}5œiD›Yÿ˜Rÿ“Rú–V÷–TþžZÿ¢cÿ i5Ù¤rC‰²‡aB̦Œ¸™z°…aG¥rEˆ¡i5Ó¡c ÿ¡]ÿ—Vÿ•Vö”RÿŸYÿ™_,í¦{Yj™f3¿€@©}an˜^1÷›Tÿ“Pþ•U ø•UüœYÿ aÿŸi4Å­Z]ϯŸϯ¯‡eS¤qD´b&ÿ¡]ÿ–Uþ–Uõ˜Tÿž]ÿ¡pK¦Ûȶ€@™f3ÿÿÿ¡pM˜œ[ÿ”Oÿ”Sú–Vù–Tÿ¢` ÿ i4Û®\aÿÿÿÿÕÕ¹›l!±‰ZA®Q_ªzFu­}G…¦r †ª{B‡«|Jy­Kc±‡\H³Œf(Õ¿ª ¶pB£qC· c"ÿž[ÿ–Vö•SüŸYÿi@θœ€$™3™f3èèÑ ¡oMµWÿ’Nþ”Tû•UúšWÿŸb$ÿ§xO—ɼ¡f̳™³ŒcP¬|G¦t6Ƥo'ë¤m ÿ©pÿ«pÿ¬p ÿ¬pÿ¬p ÿ¬pÿªpÿ¥nÿ¤n'î¦t7˨yB“±‰`R̳™¯‡aYŸh2Ú¢_ÿ•Uý•U÷Vÿ›e9ãµp0™3¿€@èèÑ žlH»Wÿ‘Oû•U ü”TûžZÿžf1î°‡gW’$º˜v4¬~L¦s5Ú¥o$ÿ«o ÿ¨lÿ¥jÿ¢hÿ¢hÿ£jÿ¤kÿ¢iÿ¢hþ¡iü¡iû hý£iÿ¨lÿ¬o ÿ¦n#ÿ¦s7Ø­O…è†&̳Œ§wMŸ£e$ÿ˜Vÿ•VôœVÿ˜^*篊p0f3ªªUŸnK°X ÿ‘Oú”U ý”SûŸ[ÿ k9Õ¹—z,f½™u2«{G¦¤o,÷ªnÿ¥jÿ£hÿ¦lÿªqÿ¬tÿ©tÿ§t$ú¨t'ò¦pê¨v+ô§s üªtÿ®vÿ­sÿ¨nÿ¢iþ¡hø¡gü§kÿ«nÿ¥q3î­€P†Î¶ž­„cjd-üœYÿ•Uô›Uÿ›e;弡†&™f3€€€£sSŽœX ÿ‘Nü”U ý”Sû¡^ÿ¤sJÌȱ›€Õ¸œ¬L¤o*ù©mÿ fÿ¤jÿ«rÿ¦rÿ©w/Ø­~=¢°…Jq³ŽYM¼•`5¹™l(³ˆM¾ w+¸’^6´UN¯…Is¬=£ª{4Ö§s þ¯u ÿ¨oÿ¡iø hø¤hÿ«n ÿ¦t:ܳŽeQ¶’qFžf0íYÿ•UôVÿœg>ÏÒÃ¥€€@€¨~b[š[#ÿ‘Mÿ”T ü”TûžZÿ¤rGÌçεŽgH§t:Û¬oÿ fÿ¡hü­rÿ¨t(ò®ƒK¤·]JØĉ êÕª ·“`@¯…Jªz4à¯wÿ©nÿ¢i÷¡hù¦hÿ¦n#ÿ¬€Q‹ÿÿÿ·•x5 k;êœXÿ”TôŸYÿ¡qO«€€€€€@¹›ƒ!™`4ê•Oÿ”Tú”SûXÿ j:ÙÌ­¬‚S~¤m%ÿ§iÿ¡hö§mÿ©t#ÿ¯ƒJ£¼–i.€ª€+¶’I¶’Iª€+™f™fª€Ÿ€ Ÿ€€@ʵ•µŽ\uªz4à¯uÿ£jÿ¢jö eÿ¬oÿª{J´Øı ·•x5že/í›Xÿ”Røž\ÿ¤vTp€ÿÿÿžkG¥›Tÿ’Qú”Tü™Uÿe0ò»—x1ÿÿÿ«}N£«oÿ eÿ¡hö­qÿ©x3ᵑ^OŸ€@ª€Uÿ¿€€UÑ®‹°…O’©t$ÿ¨mÿ¡jø fû¬mÿ¥o+½Äv ´sGœc,ý•Sÿ•Rÿ–\'ô±‹o.™f3™f3¬eG–X!ÿ‘Nÿ”Tû”Rÿž`ÿ®ƒaaÿÿÿ¥q2ª¬n ÿŸeü¡gù¯uÿ®€E¿Ð³Žª€Uª€Uÿß¿¿Ÿp0º•\Y¶GwµŽJŠ»—b•´‹B„¶Im½š[IÁ§rŸ€ ™f¸”dO©x4ë«pÿ¢iù fúªjÿ§u>¼ÿÿã ­ƒ`m¡c"ÿ”RúœWÿžkC¸ÿ€€€€€žkH¾™Rÿ’Qû”Tú \ÿ¤rH£©zI§«kÿ eû¡hú­rÿ­C«ÿÿÿ™fÌ™fÄ¢q4¶‘R‰³‰>β†/÷·ˆ'ÿºŠ!ÿ»Šÿ»ŠÿºŠÿ¹‰ ÿ³†'ÿ°†.é´Š>°»˜_^Ûȶ¿€`Ũ€,¨w.Úªoÿ¡iú fú¬nÿª{J¢¥uO¥Ÿ]ÿ’Oûœ]$ÿ©}^\€™f¬…hG–Y$ÿ’Oÿ“Tù™Uÿf5èÅ¥Œ¬€SŠ¬nÿždý¡hø­rÿ®‚J³Ì™f¿€€Â¥v6·M©±…0ù¼‹ÿ¸ˆÿ±ƒÿ°„ü¯„"ú°„#ù°„#ù°…#ú°„û¯‚ÿ´…ÿº‰ÿ¸‰)ÿ³ŠBÔ¼˜a\ª€UªU+Å u#ªy7ã©nÿ¡hùždýªmÿ±†^rѹ—žg6á—Uÿ˜Sÿše8ÐÕÕ¿ ªUUžjE£›Uÿ’Rú”Rü _ÿª}Wr±‡cU¦m'ÿŸbÿ¡g÷ªoÿ¬|=ÐÝÌ»ª€U€€ÿÿÿ¸’R…±…2ù»‰ÿ²ƒÿ°„ø±…&÷²†'û³ˆ&ÿ¶‹'ÿ¸*ÿ¸Œ)ÿµ‰&ÿ±†%ÿ±†'û±…'ù°„#ù¯ý·†ÿ·ˆ'ÿ¶O¨Õƪ¿€@ª€+¾žy?¨u.ú¤iÿ¡h ù¡dÿ£m.÷º˜{4­]]Ÿa!ÿ“Oü]!ÿ§yZcªµŒk–\*ç–Qÿ“SöœXÿžf3ÔÃ¥‡¿¤€£m1à¤eÿ g ÷¤iÿ§u,÷¿Ÿv8ª€ªªÓ¹•´ŒD¿º‰ÿ²ƒ ÿ¯ƒ ù±†&ø³‡#ÿ¼*ÿ¾“6ÿµŽ:ý¶?ëµ<á·‘?åµ?ö»’:ÿ¾“2ÿ¸Œ'ÿ±†$ý²†)û°„#ú®€ÿ»‰ÿ´ŒEÒÖºŸ%™™3U´‹[{®uÿ¡gü fú¨iÿ¨wC»¡m@¼›Wÿ˜Sÿšc7Èÿÿÿ€€€¤sN_™Wÿ‘Pý”Rü ^ÿ¨zUu©zM’ªkÿŸdù gý­sÿ°‡TˆU™™fе&³‹DÛº‡ ÿ®ü°…&ú±†#ü»Ž(ÿ¸>ÿ¹•N½œ_t¿¢j<Å­{ªmÌ­z¦t.Á¢lZ»›Zž¸’Fè¿”7ÿµ‰#ÿ²†'û°…%ý¯ùº‡ ÿ´‹GÚѶ’¶’mÿÿÛ¬}?Ê«oÿ¡iøŸcÿ¦k!ÿ±‰e[©~VG›_ ÿ“Oÿ—Z!ÿ«bI€@@œf> œVÿ’QöšVÿ›b*æ´–q"¹–x3¢k-ö bÿ fù¦jÿ©y8âÓ¼¦¿€`Ó¼›³‰Bع†ÿ®ù°…%ü±…þ¾’2ÿº•OÌħvNν”¿ždŒ·‘BúºŽ(ÿ±†%ú±…%þ®ù¹†ÿ´‹DÂÿÿÿ¶’I·‘^Qªu#ÿ¡gÿŸgø§hÿ¦tBÆÿÿÛÿÿÿ k>¾›Wÿ™UÿjC¡ı ˜^/Ó—Qÿ“R÷Ÿ[ÿ£oE£¨yI“¨iÿždû fü­r ÿ°†R…ªUµJ³º†ÿ®€ù°„#ý°„ÿ½“:ÿ¾œaÿÿÿÌ™3¶’IªªUªªUªªUÕªU¶’I€ǯ€@¹•Oß¼-ÿ±…%û°…#þ®€û»‰ÿ·O‡¿€ÿÿÿ«|=À¬pÿ høŸcÿ£j"ÿ°Š`J§yPc]ÿ–Qÿ—\)๗€°‚f-–Z(õ’Oÿ“Rýœ]ÿ©€[\Á§„¢m3â¢cÿŸfú£hÿ¦r&ö¸˜m/¶m$¹”Zi¹‡"ÿ­~ü¯ƒ!ý°„ý¼‘3ÿ¾žd–ÕªU¿¿€€€¿€@Õª€UÖÁ¥%º—SÚº$ÿ±…%û°„!ü°€ÿ°ƒ,ø¡q6ªq ²‰Uc«tÿ¡gÿŸeøªjÿ§t@œµ”s›b.ã™Sÿ•Xÿ¥{ZU¥wTO˜Xÿ‘Oü—Tÿš`(鱎j$¯†`]£hÿžbÿ fùªnÿªz9ÄÿÿÿÏ¿°‚+Û³ÿ¯û°„ ü»Ž'ÿ»–S¾ªªUÿÿÿææÌ È±z.Ë´€DÊ°€:нŽ¿¿€ª€ȱ….·’Gî·Š$ÿ°„#ü¯ùº‡ÿ´ŒH¤Uħ€§u,ã§lÿ g÷¤eÿ¢n2ßȦ¡lAªXÿ™Uÿœg> kCk›Wÿ‘PùœWÿŸh9Áÿÿÿ§uD™§hÿŸcüŸfú®rÿ¯‚IªU¹”Wiµ„ÿ­} þ¯ƒü³†ÿµ9úǧt7ÕªUÿÿÿƬrkÀ¢]ѾŸSö¾ŸTÿ¾ŸTú¾ VæÁ¥d­Ë³ƒJ¿¿€¿@Á¢nk»‘8ÿ°„ÿ°„#ú±€ÿ°„/òÁ¨v)€¬|=¨¬qÿ gúžbÿ£j%ÿ³ŠfF¥vOn›Zÿ™Sÿ˜`1ÁÿÿÿžkFƒœVÿ“Q÷žZÿ n@—ÿÌÌ£m3Ħfÿžeû fü­sÿ¯…Nf³‡=µ¶ƒÿ®€û°„ ú¼Ž&ÿº–T¶ªªUĨlœÌªVÿÄ¢Iÿ¼›Gÿ½œCÿ EÿÀžGÿÇ¥MÿäWÿÄ©k”ÿÿÿÕª€ÿÿÿ·’IÇ»'ÿ°ƒ!û­€ü¸‡ÿ¸’SlŸ`¯‚Gl«rÿ¡fþždú©jÿ«{Pvª€^<˜\!öšSÿ—[(㵌kžlB“œVÿ’Pøž\ÿ¤uLv¹ž{¡k.á£dÿŸeú fÿ¨rÿ±ˆRKƪ„¯‚-ã³ÿ¯û°ƒü¾‘1ÿ¼`rªªªªUÇ®y_ʨUÿ»šDþ»›J÷¿žJÿ¾ŸUø¾ YéÀŸNÿºšGüºšEÿÌ©Qÿ§gŸ¶’I¿žal»4ÿ°ƒÿ®ù¹†ÿ³ˆ>¥™f¶Ž]?¥pþ¢hÿŸeø¨iÿ¦sA¿›€›a.á™Rÿ”X"ù§|`@˜[”›Vÿ’Qúœ[ÿ¥sI[¬}M+Ÿfî¢dÿ eù¢hÿ¤n ÿ«}*=¾™a7®€#ø°ÿ¯‚û°ƒÿ¹7ÿÀŸcM¿€@À£]¯Æ¤Lÿ¼›KöÁ IÿÀ¢\Þεˆ>ÕƜƩn}¾ŸRÿ»›Iý»™EûÊ©Wÿɯyc¶’IȪy*µ:ñ´‡ÿ¯‚ø¶ƒÿ²†5Êÿÿ̼žq"¦s'ì¥jÿŸfø§gÿ£n2·ßß¿f6ÊšSÿ–Vÿ¡rN\šc2•œVÿ‘Oû›^ÿ«~WO²r8 h+õ¡bÿŸfú¢hÿ¥oÿ¯€@@¶ŽEF­~ÿ¯ÿ°‚û°„ÿ¸9ÿãjH¿€@½žNºÅ£Mÿ»šHøȦOÿÁ¦hx¦d£Æ¤Mÿ»šJöÆ£Jÿ¿¡\Äÿÿÿèѹ ¶CÕ¸Š!ÿ°ƒø´ƒ ÿ¯€ ÙȤmÃ¥‡§s'ܧkÿŸfø¥fÿ¤p9ÉÿÿÕi<·šTÿ™Vÿ nFtžkEŒœW ÿ‘Mûžc0ÿ·—~Qµo7Ÿh*õ¡bÿ fû¡fþ«sÿ²ŠVS¹”QE­~ÿ¯ÿ°‚û°„þ»5ÿ¿œ`X¿€@Á¥c¥Ç¥Mÿ»šIöȦNÿħf…ª• ƪocÆ¥Qÿ»šHû¿žHÿ½žWèÑ¿›ÿÿÿ·’Kĺ‹!ÿ¯‚øµ„ÿ¬{Ù¶€ѹt §u#Ô¨lÿŸfø¤eÿ¤r>ÓææÌ œb+§šUÿ™Uÿœf@„ŸjDx›Xÿ‘Nüœ_$ÿ®„dO°€U*Ÿf í¢dÿ fû gü®tÿ®‚Fn¿l4®&ö°ÿ¯‚ü°„!û¿’0ÿ¼™V‰ªU¿€@ʱ€RƦVÿºšEü½œIÿ¿¡YìϺŠ;çj®Å£Lÿ»œJù¿žHÿ¾ŸVæÖ™ÿÿÿ·‘Hɹ‹ ÿ¯‚ø´‚ÿ°‚,Øر‰ ѹ‹ ¨t%Õ¨lÿŸfø¥eÿ£o7ËÿÿÕ›b!¤šUÿšVÿžlE“¤sQ_šXÿ‘Oý›[ÿ£qDVÄ€¢k/Ý£dÿ eû gú®sÿ¬C̳€°‚*Û´ÿ¯û°„"ü¹Œ%ÿ¸“IÔææÌ ¿¿€Ĩl¥ÍªQÿº™Eû½Iÿ¾ŸVìÁ£`œÀ£_º¿ŸQÿ»›Hþº›IõʧLÿÀ¤b³ÿζ’µŽ>ᶈÿ¯‚ø¶ƒÿ³‡>Âÿÿÿª†¦s+à¦jÿŸeø§gÿ¢m2»›b-©™UÿšUÿ›d4”¥wV>”Wÿ‘Nÿ\ÿ¥tJnÿÿ¿£o7¿¦fÿždû¡hû©nÿ©x3Òæ̳ ²ˆ>©·„ ÿ¯€û°„!ý±†"þ¼’:ÿÁ ifª€ÿÿ€ææÌ Ä¨k¬É¨Uÿ IÿÀŸFÿƤKÿ HÿºšEý»›GÿɦKÿ¾¡ZóÏ·‡5ªŽ9 ¢lG´Š5ÿ±„ÿ®ø¹† ÿ´‹F•™f»•j)¦q%ñ¤hÿŸeø¨hÿ¥q:¢h;»˜SÿšU ÿ mI“°r–\*ç”PÿZÿ£pE§wFŽ§h ÿŸcü gý¢iÿ§r!þ¸–d=»—_V²‚ÿ­~ ÿ¯ƒý°…%ü¸Œ&ÿ¸“IæÑÁ›!¶’IªªUƬtl¾ ZÕ½žRÿÄ£PÿƤPÿÆ¥Qÿ¿ŸRÿÀ¢^É̳„<ªªU»—W¥½Ž)ÿ¯‚û­~ ý´„ÿ»“YSŸ€ ¶Ž`M§qÿ fÿŸcù¨i ÿ¦t@{ãƪ ›b/ΗRÿ™U ÿžjD„ÿÿÿ™a2¾™Sÿ™UÿŸj=¼ÿÿÿ±†bN¡gÿžbÿ fü¡hü®tÿ¯‚I“ÿÿÿ²†9ɶƒÿ®‚ú°„"ÿ±†'û½’1ÿ»šVÄÿÿæ Ì™3Ì™fëØÄ Ê®xHħjqÄ«q…ĨmuƬwGÿÿÿ¿€@¶mĨwIµŒ:þ²…ÿ°‚÷µ‚ÿ²‡8ÚÒÖ’m¯K‚¬q ÿ füžaþ¤iÿ®ƒ[L·p ™_)ç•Pÿ˜Vÿ mGsžkE†›W ÿ”Pÿ›c0轜„ÒÃ¥£p:Ù£dÿŸeü¡hü§mÿ©v)濤vÁŸnJ³„)ÿ®~ÿ°„!ü±…%ÿ²‡(û¾“3ÿ¼˜SÁØĪªUŸ€@¿Ÿ@€É®†&¸‘FḊ"ÿ¯„ ú­~ ü¹‡ÿ¹•[x¿€ÿÿÿ«{=êmÿ f÷£dÿ¡l/åÄ€«€[F—Z þ“Nÿ–Vÿ¢tNX©eG–Y%ÿ“Nÿš]ÿ¬€\V«|P¨i ÿžcü¡gþ¢iü®u ÿ²†L‡@¸’VŒ»‰ÿ­€ú°„#ý±†'ÿ³ˆ*û¾”3ÿ¹–KߧpT€¶mª€™3ª€ª€Æ«zC·‘G๋!ÿ°„!ü¯ù·„ÿ²ˆ9Íèѹ ªq9 ¹•j:¥q&ú£hÿŸeö©hÿ¥r?¥£rK}›Zÿ’Mÿ•X ÷¨{W8Û¶¤™b7Ò—QÿœXÿ¢oF¥È¤‰¢l1ç¢dÿ¡gû¡iü§mÿ©w-ó¼›t.¶N±½‹ÿ®ú±…$ü±†(ÿ²ˆ+ûº-ÿ»•Aÿ¼™T¶Â¦rS×¼”É®}/¼š[•µŽ=ý¸‹$ÿ°„"ü¯‚ù²ÿ°„1ùÀ¡p9ª€U±†Sœ­qÿ eùžbÿ¤j!ÿ¯†\Pÿÿÿži>¾™Uÿ”Oÿ–[*ൕu¡oM…šXÿ•Qÿœc1켡†&­„[v«mÿcû¡hÿ¢jú®sÿ®‚B¹ÿèè ¸‘S³¼Šÿ®€ÿ°…$ú²‡(þ³ˆ,û³‰)ÿ¿•2ÿ½–Bÿ¸“IÚ¼™V§¼œ]~¿žcgĦsd¼œYj¼›YŠº•O¼¶Aö¿“0ÿ³‡ ÿ°…$ú¯‚ù´‚ÿ²…/ÿ½›jYªUÌ™f½žu2§v2÷¤hÿ fö§gÿ¥s?ÊÿÿÛ±m1™_,õ’Nÿ˜Sÿœe=¹™f3´‘t,—],ñ”Oÿž]ÿ¤sIÿÿߨvCǨhÿ fù¡iÿ¢jü°xÿ³‹X€ªUÿÿÿ¹”V‘¶‰-ÿ¶†ÿ°ƒü°…%ü±ˆ+ü±‰,ú²‰(ÿ¼‘.ÿÁ–5ÿÀ–9ÿ½•<ÿ¾•?ÿ½”8ÿÀ•5ÿ¾’-ÿµ‰#ÿ±…$ø°„#ù®ý¹†ÿ°„1ó¿žjT¿€€«}@³«oÿŸgøžbÿ¥jÿ°ˆcZ€¢rI„ž[ÿNùšWÿ mI~›e;œ™Uÿ–Rÿœc.罜„¹–s3£n/ô¢eÿ¡hú¢ký¤kÿªw%ÿ¹•h[’I¼›gT³Š=غŠ$ÿ·‡ÿ±„ÿ±…#ü±†)ù²ˆ,ö³ˆ)÷²ˆ)û±‡&ý±‡%ý±‡'ü±†%ø±…$õ±…"ø°ƒÿ·†ÿ¸ˆ"ÿ´‹CÂãw/ªªUUµŽ^u­tÿ fýŸeù¨hÿ¦vDµ¹—€ša-á–RÿNÿ•Wú¨€[8¿€@°ˆl-—],ö’OÿŸ\ ÿ¤tM˜@°ˆ_^§m#ÿ dÿ¢jû¢jü§nÿ©x,ù·VGªUUÝ̪º•[v³‰=Ö´‡+ÿ»‹ ÿ¹Šÿ¶ˆÿ³†ÿ±„ ÿ°„$ÿ°„!ÿ±„ÿµ†ÿ¹‰ÿ»Šÿ³†'ÿ³Š@Ó¼˜cjÿÿÿ€€ªªª€·“iN¦s'ý£hÿ g ÷£dÿ£n3î»›ƒ)’I¥tPž[ ÿ‘Oø—Qÿša6Çÿÿÿ¢qOŽœXÿ’Pýœa&ÿ¯ˆeI€±†]|ªo ÿŸcÿ¢jû£lû¨nÿ¨w&÷»—kVªŽ9 Øر »˜`RµF˜´‰:˱‡0ë±…,ýµˆ)ÿ´†ÿµ‡'ÿ±…+þ±†/<зH›»˜]RææÌ ªªU¿€€º‘_C¦r#ò§kÿ g ÷Ÿaÿ¥l$ÿ´‹eXŽU ¹™€(›b1ð•QÿOù›Xÿ¦yYr€@ªUUÉ®”›c7Ý—Rÿ˜Tÿžh8àʪ•@¯ƒZ‚«q&ÿ¡eÿ¢jú£lú¨oÿªw)ÿµŽ[xª€¶mßß¿ÁŸn%¾ša?º”UQ´‰,Rº”OQ½—]B¿Ÿl(Õ¿ª ¿€¿€€¿€¹’e[¨u-ø¦kÿ¡höŸbÿ¬p(ÿ°†^x€@¢qI³œWÿ’R÷”Nÿ™`3ä­ªUU€@¬„lS›^'ÿ’Oüž[ÿ£nB¹ÿÿÿ±ˆ`r¥n'ÿ¥hÿ¡iú£lù¥lÿ¯xÿ¯ƒD¯Å¨ƒ#™f™€ ªUÕªUÕªUªUÿãã ±‡TŒ©t"ÿ¥jÿ hõ¡dÿ¨m"ÿ±Šeƒ€+©~^wž]ÿ’OþNùXÿ£tT}€ÿÿÿ¤uV’Ÿ[ÿ’Oø¡`ÿ¥vO¢€µŽkO¥r6è«mÿ¡gÿ¢k÷£ký¯uÿªz2ìµ[yÙ̦ªU™€ ªŽ9 Ÿ€ ¶’I¶’mŸ€@ªŽ9 ªŽ9 ª€+»–fK«|?Ï®tÿ¡gþ f÷§hÿ£l+þ±‡al’I®†eL™^&ý•Qÿ’R÷–Oÿšc9Û̳™ªUUªªUÿÿÿ nGÁWÿ”RøŸ\ÿ£qC™€@Á¢ƒ!ª{G°©p ÿ©lÿ¢iû£lö§nÿ±yÿ¬~;ß±‡N}¿œt,ÒÃ¥·`R­~@´ªt$ÿ©nÿ¡gö eÿ«mÿ¥q6Ù´mD’$¿Ÿ‹@œd4ò—Qÿ’R ùLÿ›[$ÿ«b[€™f3ƪ—œf:Ú›Uÿ”Rø£cÿ¥tM­èèÑ Ž9 ²‰`]§u:Ô«oÿ©lÿ¡iü¢jø¨nÿ¯wÿªx.ò­€A»°†K~µTL¸šk+Å¢tÝ̪ïßÏϯȤ€º“]4±ˆN\¯G–ª{:Öªt#ÿ«pÿ füŸeÿ«lÿ£m*û«~M’Õ¸œ€U€¸™AŸkBò™Sÿ’RùNûœUÿ oM£€€€™f3¾ ˆ+œg>çšTÿ”Sö¡^ÿ¡n@̹—z,ı ¬~Ki§u9Ñ©pÿ«oÿ¤jÿ¢iü¥kÿ¬qÿ­uÿ§rþ¨u)í©w&Û¬}<Ô®‚LÖ©x1Õ¨v+á§r$ó¨qÿ¬rÿ¨mÿ¢gÿ£hÿªn ÿ£n*ú§u<¥·j5™f °ŠlWœc0ô˜Rÿ’Rú‘Pù™Qÿ›f>ÓÏ¿¯¿€@f3ºš…0œf=åœUÿ”Söž[ÿžf/ñ­‚^j@ããÆ ³‹^Q¬|G¨¦s1ç¦nÿªpÿ¨mÿ§lÿ¨mÿªnÿ¬pÿ«oÿ«nÿ¬pÿ«nÿªmÿ©lÿ¨kÿ§l ÿ£k þ¦r4Ù«|IŒ¹‘n3™™ÿÿÿ©}Z…œa(ÿ—Rÿ’Rû‘Qù—Oÿ™a5ê¹—€,™f3€¼š€&g<ÓŸZÿ•S÷˜Wÿ¢cÿ¤qE¹³p9ħ€±†WR¬~KŽ¨w;»§s0Ù¤o&ì¤n"÷¢jü£kþ¢jþ£kû¤n"ö¤o&é¦s3Õ§u:³©zB€°‡ZDØı €@€µŽq4¢mA¿¡_ÿ”Qÿ“S û‘Pù—Pÿ–Z#b9€™3Ì¿¦ oI³ ]ÿ”Rÿ•Tø¡]ÿc)ü£o<œ·Œo5Õ¸Ž¹“f(´Š`=¨u/Fª€RN¬~OMªz;E¶c;º‘g%ϯŸ’$½ ƒ#¤tK›b*øžYÿ“Qü”T ú‘Oú™Qÿšc9è­‰i8€™f3¥xV˜])ÿ›Vÿ•T÷–Uÿ£`ÿd)ø£pC¯®„`UÒÃ¥@²Œj5¢rF’e0ê¡^ ÿ—Sÿ“Sù“SúMÿœUÿ›e=Ì·–ƒ'€@¿€@­ˆg>œh=ÌŸ]ÿ˜Uÿ–T÷—Vÿ¢_ÿŸbÿ i6ܦvKš«~WU¹›t!ÿÿÿÕ¿ª ²Œj5¨zQu l<¿›a%ø ]ÿšVÿ”Sú”T÷‘Oþ—Pÿ™\(ÿ¢sR™èѹ €ªUUÿÿÿ¦yWx›b3èŸ\ÿ˜Uÿ”Tù–VûœZÿ¢_ÿ bÿže*î¡k6Æ¢m:š¦tDu¨}TX®ƒ]B¨€W2¨v8)²‚S+¤s7*¯„^6¬~VG©yQ_¦uI¡l;ªŸh4×›a$û¡_ÿžZÿ˜Uÿ”Sû“Sö’Pü—Pÿ›Xÿšd;Ô¬†iP¿€@ÿÿ€€@½¥Œ¥wV‹š`/æž\ÿœWÿ•Rþ–V÷–VûšXÿ \ÿ£`ÿ¤bÿ¡cÿžb ÿœ`ù›_óbô›^ô›a!ûžaÿ¢bÿ£a ÿ¡]ÿœXÿ—Uÿ•Tû”Tø”S÷’Oÿ™Rÿ›Yÿ™a9á¤wWxêÕ¿ ªªUÿ€€€¹•{¢qNsœg<Ì™]$ÿŸZÿ›Wÿ–Tÿ•Uú–V÷–Vø•Uû–Uþ˜Wÿ™Xÿ˜Wÿ™Xÿ—Vÿ–Tþ–Uú•Tø•U÷•Tø“Rû“Qÿ˜SÿWÿ—Y!ÿše<Í£tUr­™™fÿÿÿ¿€@ããÆ ©~^A¡nF‹œf9Η]&ø[ÿžZÿXÿšWÿ˜Vÿ–Uÿ•Tÿ–Vÿ–Uÿ–Uÿ–Tÿ—Tÿ™TÿšUÿœWÿ›Yÿ•Z#ý˜_0ÕŸmG”©~]GããÆ @ªªU¿€@€ÿÛÛ¯Šp0£tLažkA‘œe7¹™^)Ô˜\%è—["õ•Wû“Sú“Rú”T û–Z"ø—[&í˜_.Ýšc8ÃiAž¢tQn«„e:¿ª• U¿€@ªUUªUѹ‹ ¹r®‚g/¦tL9’R8’M8g-9­‚e5±Žj$ƪŽªUªUUÿ€€€ªUU€@€€@€@@€€€ÿÿoptuna-3.5.0/docs/image/optuna-logo.png000066400000000000000000000372231453453102400200210ustar00rootroot00000000000000‰PNG  IHDRèÒît€gAMA± üasRGB®ÎéPLTE®®®ª¬¬ª««ªªªªªªª««ª««ªªªª««©ªª©ªª©ªªDyª¨¬¬ªªª§§§ªªªªªªb™_šª««©««l $o¡ªªªY–¥¥¥ª««©««ª««P’NNKNª««Q’ª««Y—ª««H‹ª««ª««¨­­ªªªªªªY—KŽJMŽH‹Y—KŽKŽªªª©©©\™H‹ s§Y—W•ªªªªªªH‹m£S“HŒª««ª««fŸH‹\™fŸp¤H‹Q’ª««R’]šH‹Y—p¤G‹ª««`›i¡HŒªªª[˜V•aœª««IŒT“Q’j¡x©yª@”·0Š³fŸ2Š²yªªªª<‘¶H‹©¬¬ª««B–¹h \™i i i¡i¡S“x©G‹ª««Hš»_›S“j¡H‹yªN6´yªyªHš»ªªªª««_š\™]™j¢T”m£1‰²y©3Š²i¡Y˜ª««ªªª<‘¶Q’2‰²©©©HšºP‘5Œ³x©3Š³x©H›ºi¡Q’3Š³x©4‹³1‰²ª««Gšºh HšºS“i i y«Oi h Fšº1‰²z«=’·Z˜1‰²Hš» v¨k¢Œ¥°$‚®ª««KŽI›»G‹JMaœP‘Nddžg LOR’i¡bœežIŒ\™HŒ#®fžQ‘c_›'ƒ¯H‹Z˜h gŸ^š`›}¬T“_š%‚¯­-†±]š-‡±)„°|¬*…°1‰³ ­­{«/ˆ±S“!€®~¬xªj¡zª$‚®R“{«gž,…°{«T”k¢ w©4‹³K2Š³Y—6´w¨G™ºO;x¹tRNS +ÀWi’Š!ŸòéöAQ6¦ Ëær–<-T]=&à­ßmÙ8y%i¨g3èC}J³ÇòERtѺÌ*¶ˆûí•Õž`"ºÂ3Êç“Pù…K1s]´ŠYÄB” ¥¶€R:¦ã$ÂKC:‰Ò°ò¿ô\œUÃôëj±kíD™Å¬d½¨ÜÙÞv•ÕäÐð.).ÓÓ×Øké‚¥˜çåvŽÆ÷H†íâüÈà6ùü÷w·õ]ùïòõõNâ«Aró IDATxÚíxUúÿ/‹T‰¨ $„’F…‚@"¡…z—"H¤*MAQŠ (¬]Ä‚Ý]w÷y¢‚‰RŒùÊê‚C,ˆüå?§Î9gfnËLrcÞï³+;÷ÎÍùÜ·ž÷¸\e¢ð¶)mÚ L\2=½g|RG¤¤¤øžé}væ@XaQÑíLï¸ÝDûöiÿÛwàÀCIé‰ÝbÚ†ÂÇU8ÅÆ œž„‘þÈ ô3Î;¡ýPÇôm¢‚àƒ* ämG$&™nB<ÇÁ~(iɈ(øü@ ÀWD›Ä¤ä{®^ݳgÏŽ;öè"ÿ¸ºG¦9òȲggÇ/ˆ `µm—Î 9ÚÐ]%¬$Øõìì!}ۄÇ ¤¢ºÅsÄÌ»¬¥ã.vÆzöôHÏ@ç±è)CNhÞK•£‹ýHÇ}FûU;ÉÍi¨IŒ„Ï E÷ÝN1×!WW¤À¾ƒvbÖ)êÙñ# \CáÝ’[ÎבÎ7Jâ]·ìëõ! báÊ]m—覜B."žk%wJ;g£ŽYïÛ>e¨\•2y•aÎ)×/.ÎÍ=wý7ÿ¿øÜ9í?ÅwsÔ?­zötÖA rļ'·æ˜r9!CU$‹ü°¸˜ñNi7šu«gO«•#æWUÊ©øþåò_,üø\jÛÖÍQ_Ÿ8T曎1'œS—=ŸøêÅò_¸æmZ½!³_W¬~™V/š§ãN`'v]BTÛ(éÙ¡°•­bûb§]´æ”r 9å{ëê~ã%¤FÆÆË/™š0ú±~‹0ëÄ‹WXWCõŽmàƒÊN¡ tÌ©ÏžOvÝoê::&Òs'kPDJÚøÌyvÁ…Ǩ3«¾÷À#¥C­ *+µé(Xsž~+.æÏë7zL¬OKNC#Ó7‘ܼˆúŽ« uJzv7øøA ²PÎÁ‰˜cN)ßš9(Ù¿`:(*­ëVnÖ‘/øïܨÇCþr^ݶë^;¶½Üe×(_4>¦tËÎBSÇ/B¨ç˜ Î’r`ÔA ‡Õ6IÆ<_0æ‹Æ%Û1þ-,eü<êÀÓPøïB¤*Csά¹–O¶oTpL¿bŽ:ÔuÒ³£áR€@N)6žGç{YlN0ÏŒ ¶û\ãæ ¡:uß@NrZm¨9§%5Ä91æã)z·YMP—Œ: Ô§ÃìWØÄ9/³àü—Õ Î5¬¥nÐIߣfß¡%rÀmß.Fç¹ÔœoHuö´)ÏrÔ÷(úH¸* ½ŠÙ¾ºí»¯=3Ùù3GoÊÜw)P‡”d«ºéá9áyí¼Ç<($"jrò°Ô1cRS‡%OŽŠñ!EŸö¶vN\hS“ïÐûÙ¨%"çÔœoòÊižš–1ªÃÊ…;e-\ÙavFZêp¯Rj!‰‚û.5ÄŽ€kÙ¤žÛñºsâ¶ç`η¦y|Zø°û›¥þ™¦/?ãúý]¾C¯„Éž³y‘›”@HìUTK·3·½¨(ÑÃpÖˆÞ³>#ˆùå'F}‰q§¼[øè dO¦}4 ÔÁ¦ƒ@(r᜸íùÈœ¯Žt÷„°äŒY; â„ñO i'¬[8*Á}ckÛMœtɦCœ•^)(Ý~U¨ª+ä.&óØÂÜŽ#¨?'ú‚Šþ“ñ.²^Ð!Í-ëã‘Q7zï{J+½¬Æ8Öz»Ó aá˜Qþé'r î¤B"ôW<µívÂzAÁ£cܬ‹‰~“¾C©²…ËÙÇ9J·¶<6væL91å”q ÷nE„xB»n؉U/X˜aÝPñ,#ý#¡ÊÖvZJÍùG:çÅ‹,£óa4P±-Ç”SÆÖGLÄxg°cÔ™U?þ¨u>‘yï¢M‡©‘ P)9Gñù^ÊyW ¢‚0̑î9ëÔŒkLç™J§[öOÖê³,‹ô#pJN!½/\+È_EïSìy;óãB-ĘÊ?G¶œrJõEEíÌ°KfýXÁñã+‡Y¼«TݦÃ$9¨Ô»èŒó­)æÖœbþ%5æŒrñ_5ýgÕÜ ÷­YÜ[Óâ533fo{󈻀:‹Õ5Ùü}µgB:¤ÞA ¿Ôvâ\·ç«MsdAi2æ%…œrùª ÷ nl° ‹›¼8cÛÌ:uá©UgþûÑQæ+Q#6Iï‹VA ?‘Äî„ó~¦­k©ŽÑL;¼„sF9‚üYÔ9ƒ×²íãƒ(à’@>+¸§ÀyNnq¢ÙA]:ãÁùç8GŒ9ñÖ»¸Wç 3ûÝ<‚úêÇ`š€ÝÀÖ²í;pÂtÈ_õå q¨¿½xœ™×žQ€œlìµ£ØYó<Žù¶Å¾Œj í=—£Ž2ðܨ=Ù+ÈŠt%L„«ù¦û„=7×̃N ™sæ´cÌ·Ììâó)ûdÌÑQÿä””ä`–” yÖ@zO¸l Oj³OL¸›•Õ‚zPΙ9×}öU½ýï¼æeŠ:sßqöýä~³ï™MjÒ¨û NȉÎ;lÖy¯$ÖùŠî&ö|ÐqÄ9vÛI®cþD§Ò;hͻȨK¤?º€ Ñ$!' r ÷ZÂt”p7®b íPÀÃóÏ™9ǘ¿<¸ôgÀ"u-Pg6}ÿþ1Æ#£õ0šô¸x —jw@ÐósÇÝö…,<×£s ówÖØó†¯:BÜw¨5ÔÛ±y‘|u ìêy ï:eºO8ZpìK՜Ϸí=ÜGÝwF:òÞ÷÷27^uÞÛÁõ¼QXO–qGœgŠØ£¿»íóÿ ¶ó] S Ô©÷>Êøf7ÐÌ;k›éû¬‚@Þh 5踲ö ¡ƒ|ÔQî·KœÏ¶{I¸©–è¤hλ éQ´ÆÆMú@¸‚ g¥@ý*uÜÕþÓàÇ ç‚=ÇY¸Þö¿“1GhEÝ­Mo³‹™tBúXÜyTh€z†_é&Ïh¦ÿ”37`Óú=,?õFMÜßK°A¿Š=wyækì?‘AçŽ;ÐýuŽY.NÆœBþ-Òÿ°ð_¿e°K¨ÿ¼Ö*ƒßgOÈQ“®vÈ×LºžŽsxŽwxºÃo¿ÁÓÞàbÝ«V³Á–4/%èµý}7©¯äÉ@ß\GyBu[>òZ†_©³ ×tä+¿¥§û¯®[ƒ®GèR&.lÖ~š‰“t³‘¿×¼vä´ÿ€=v9âû'Q˜wÊ:òàµ`ý< Õ¾´ÌbØܰݸoF7é‡ÌMúçÞÿz»7÷x³ºå :Róþºí {z'ezÝf^|ô-ïµzz_â¹ã½›’ˆ2qœs“>™ÉS¿GæüëÓ§ ˜cC®¡ý Ò‘ðß¾A°ÃŽQ?}'à5£>u²Uê›t¼Žíð“qF“®×ÒóÝ«Ôóö.¿cZ9ƒ®½…õºÝ ßQΠ?ÒÙÛïyóIä!Þåž3O‚(f?ÍÄ镵‹¦}¯Ë.·]Àü˜qøY]”÷Ÿ¨]×QG¤_šhþûÍÅÎûg–™÷H tÇ|÷¦=|¸ÏG–7èYYµèö‚îáª: ú=>|úMÍSîºç.•Ö"2ƒ.:î /Ðe*â\‹Î9æ˜r‚ø)Y„vb× êéËM[X»¼£9Î{¤|@W:ÖѼ{5ßnôêåzV›t{AoYž ·òéó¯f|¨ìÜs— ú(dÐ tƒŽ9ÂPö^ú3 ϱ9ǘÿaNàþI`¡ŽŒ:ÎÉ}÷FúTÓn»Å…B1}¿Fzùñɤ;³¹ê-¾ÞéõÊô¬¬Æº­ gµ(?ÐkøxZ^a à¹KÛ/%6ô‹¿¾c؇eÙÏ—‰Û~ðàÁsN¹F÷o¢(ìu̓GF»ï˜ôï¾ti²•óNJlÄw?¬Ì…MÂ:Ò3ÓÊ÷[½^ùƒ^Š{@7-_U)/Ї–ú[>$I÷Üß[ÉãžÄ•Ð©ã¾XyzðrsbΑ5g”#¶ÿŸ(vÌ:Õ©ûNlú³áð“Y>Ž™ôYòã#D“îÄöL#ý¹×k”?èþßôº—v²l@¯æÇ%PÚd ]q»ví•ÆMô"}§dÐÕý®µ:çÜœcÌå ï×6¾?å…W^yî¹W^x|Ê«ë4Öé?aÔ%ÒÍÖ¾fð|5ér‰-âmÑw²óõþÝì#Ët·w&€î3èµ)ÐÿêWÏRké5¦ ©81É•|x¿Ð+C úËJçkŸç¦e5Ê96çÈ`sS¾nÊs3äÕçAwOz|#E%à›þ½…M›#tÍ Ð“ž¸C˜?a­Ÿ7{Íò=ëÝFг†– è·ûu ªŠ/Ñ6›·¿îÍè DèØqlàüòwß öœšsnÍ_}åi‹·~×s¯bÔµH]³é8%÷5&ýÒ¥›tÉ݇¢t’x'&]^™#úî¶é­ü½Û›ènÐm½{ërý?/‚˜z˜}ˆŠMaŒ`ÐÙbÕq'œeäœ`¾î…»Ý¾ý»íu߉÷ŽI×LúóÆ YÈË%b”~øð£òà +[ìÒ§Õ)ksj'èYõtAwÓIïè­»ûynÔ×='‘ôy_A+ë©8¡Ï‘[ÒB¦^ºü=æüÄ9sÛ)æ¯>äù7}ºïŒôó_á0}¹¹IÿTèxWj鉤áƒ>Äæ‘5ü¿Ý«èYMtAÏê_æ ×÷û*ÜÂ_#:›„èÈ 'J¥5æ¹ã:æü¢Ò·s~žrþ­dÎ_áÝï:…‘~âàÁ´0ý+¦o¶ŠÒq-ƒÞËà»ó[Š­œßliÐ{4h^«IõúÍï¸Ñò“~=@oïÏËmõZÀƒÞ¼¬AŸfaÐ{´ºâž{›Þ[wEÎ÷¨^LÔ»erÄN“YÜ Ó]}›|þ/] wó±ß®s¾î!ï‡Ö!ÒQ•Žw“Ô{FáR:îI)/÷ö=HQúûóÓ §Vtv4J7€^cd Io©_õFŸmµè\ÏLµÔ“Ô©WËì¸æõ«üYA·ôœÝülÿºØâ\åþª¦G­`ÞwRö!â¹ïØõ`ˆbÐÐq„.Ã7ñsÜdñ9åüŸ~°)gIAýM½#“þ¼a°Åp½ãݬÂÖWÒmÝ4ãÙÞ°ØxšùQÓÐo59(¨ÿ-ÍoÇíìK ÉŸt«åª-Ýlajã8Ž¦ÍÜÄŽÑ8çNBôñRÊ]ðÜi„>[zÙN.á :IÄ}û-ã\óÚïòõ7y‘þ?¦ãÌû…e†£æòt]Þ쵤§ÛyãÝjæ—š&Ùú7òµiÒ^Ðñf+o5´ïãP Ç•t«xÌ!Ðoòú«Û,¯B Ò!Df4%V=÷¼¼‹Ry;äyÍ £Œ;OÄ!Α9Áße¶éˆôÓÌy¿b¨¦÷.$K[Žá¦™Ã‡K+ÏÙ6Û×µ˜8N,.aÃ6Èþƒî jî\¦@w»\Õ!ÐkxßsYËÊÐÅÏýêŽ]»ÞrÕ–Šèx9‹lÐ_Ô :ÉÄ‘ûíùõË6»Õs¤9Î7Ðo«íT+.€îÆOsôú¾Í9¨j–ŸIÉfUtiA‹æ¹Ë!ºæ¹ß'eâ®ÐºdÐ×¹+«Ý=é½—~à~é½IÇ=ô_’ûÁ2ñ~í™aýîÒ ™ÐuЧÛwßÕóÑ¿ßðå_ö ßô ½t w÷"Çz»# ßáÛâçVfß í²qÐÇ)žû~)D¿ØG1è?Ë9îO[ž|ÆKÏ|,ê÷LYåãPâ™ô•&» Òû¡Éqôt“î·zȨ·]¿•=è-ITfÐ;7ó\árô–•qo?¾Î8Dדîúʵ칟<~ü˜¾]ª­-»¢§â˜Aÿý·Xœ9ô½¿lÐõ—LP¿ëµŸH×Ì™ó$÷o¥Ž lŸè _KM$#"èña¶Ýw |]¬¡^îî5Ëô¦¾äot/@o¬ZÊ:Ó<Þ*¶€n¨úxð I¢Û]®°x!é©gµ®Ñ\ž-ó9Ñ{ /÷oâ¹³úO¸„þ¸ùyÃÞøØ\×ß3üO¼k¾û%“tÒõ93×ÒÄ…Þ8;fÔ/ÔžžÐ܉D˜ ·Ðm½FPIVG@7xˆU}¼a›¡ñ¯B_œÞf6ã}äýGgÍ”s×kk8B·HÄMú×ÇVºþ@¨‰Iÿöÿþúî—M|w²°EÏÆõ2I»°u»–†Í|ít«nÇ’’R‚þHmÝ^Л³²ëËtÃwv-/PÔèžÍ·hyV?2“‚.„è’ç¾VÑOÕ©§~3mˆ {øc7ºþwƒû>™tî»_¸ò¼ò]°˜‚¾“¶ÌHÙ¸¡ Ö¶Ö¸†|üœÓ@n-{Ðoë Û zUc)£ž§´™- ?åë %u}qmÜ.ÃA×ûâP[ÜaÚ.Ãú_Åѯ“ÿ !:*¢ãT2èSL3íÿØ-èןQmú$’Žû­aSóîÉ…rþ¤¸áÙƒdãÚ:ºÇ²¸!íùW½âƒÞÀ伩‡ÆÐÿâmŸ»UÅ¥6Z‹Îrq;„¥kÉ×èt¸ Îʼn³Ü'"Ð/+©8³ŒûŒ=èúõ‡•§¯#é8¼† ®lÒÔí–Î[f¾&§~[_©éèï¢ÆúŸô–.W•F¢¸Îeº§KYßzO¡Œ®oÅ”v÷ű¤ûâ -¿rå’ ú©SûÃ9"ý9sß.V½påN%ƒ¿¥„¶Ìu-RÚ=T˜7¶ü,:€þgµÂµðІ¨ ‡&e³ÁbumœØ‡W´äeˆFôK´[Ò£Rý÷~ÆHºRd{K Ò/]¸pEÙ‹m®’v—êkPÚÄè)•tÈºÛ :n0_á~¹jó}l¶°¤E7«…ê*£k ‹sÖ7ÐÙJt¢O2žî%é‡qÿô*è(ßþÄ+é¨eæ%77Ášt»gbÐ÷E7vMè¥P«6Cµt› Ðc²…9R@·ô[M{i¹jý} :o%K¾&÷Ëî>2Ax™ð#ÐfÃâP»Ì)c—û?d»=É2oÒYÚý }™9è;)è£,@o[¹Ao¢Þ@/%è7áÞãîƒm  'Š óqîVA÷mþÇ|-ú'H£»1ç.‡â[?výúò3§íîèß™4ÁÞ·›n”N;fºª Óf÷¨Ê zKX¦j7è÷˜gÖÅåªÕô¾b«;=FŽ—‡ËtúCït§ ¯3´¹ß-ôÖi:KÐY·û&  ­qZ€[©A¯›ƒ'ì lêfîf=‚~Uým¾j%Áº˜ùFÕµ t*]ºö¾á\ÏIœ?#?ø†[×]·è¾ƒžÉAïW©A7Îí —ô‘©õ::ÎCôžèºEO¸¦O†¤ ‹ÕµùtÉ¢sq/¹ñÜÕlÜ$5F× é?AÇ1ºè(臲“B+3è­š9]©AgùõõÖ3[(èñ"èÖ ¿ÓÇ=è/xÑßp÷àõ§Õ¬»[Ð3ÐÅd\Ø&º}'W@Ðï7N8kàÐK zc+²þbuåô$ôXkÐç„XN\wãNkrqí%é1¹5öºâ×»6*®»’Œ› ¸î"è¡rÐÓ+1è+LFVÐK :ÆehFâËU[(èEÐÛZÇè[‚ 1ºdÑÕ…)® yyênÜúëò·€+øC%·L~|¶ºØ0ñ6Ñ—TZÐo3Ý×ü½´ µL®ók±¢L@oRå/nô”áÐAÇY÷±fYwR^{9Xʺë嵃 +ýc±ü¦¬i»®l£þôY:Y¾¦4ÌlSÊkÂHKWÛ]ô•ôþõM·Ò®}€^ZÐõeà7׶X®:²L@÷UèÒ¢–a††™—ÅòÙ[°åè§è¯xˆÑÅ =T™E¡´º»þq–-_;oÖö¦Ò03Hx0eo˜i÷§ýþ*­%=ukÝƵšY\ëæ.½´  ×]Í®gÝkž HÐù2Uµ3®D¶èzg$uöqY÷?æ­s†™3ŠAw=þ_ÚË@—÷dW]Üg1F½ÍŸtŸT@/5èÂ1à º\õž@-jổ{Ý·ˆµª çÿ¸B·cb½îÆòÚ?Ôå¨oà}^‚ßSþ’úÌ t¶¨E^¦:|·²¨EÜ”©›úX]ø  —ôzî.1ɼ  '‰»Òê5¼Lõ]iv2Y½Æ†=£Õk¯Þ\ˆaöë¿~㇠?5ÌŒ{íž,|V¦CŽÑ@ÿôSa™ªØÓž¨Çèº#ý¯•t1üih>¼i€‚/‚®7¯VOJVu™4xm˜üZ¨GßÝBªãîzá,ݬ… ž¸¤–Ñ×ì–×£?)Vþ2w±õèöõËü@oàÐKzgñ±¦ËUû.èú„}ë5>a†’;ãp!]%eõ|·WœçUld{'Ó 3W6«ÕµBi”çË%+`IDATÔJ1~p¶èÙ¹õÚŸô{t@—ÇF&P™0( §‹ ?Èmã8ÃpH)!Ö嶕*_§jls½áçÆÕ­¥û,žfý2r.Îõ„0ö¤Ò/EBtúÝ‘f™Ê úÒƒ†îCÔOÓ:@AŸž-LÍá ;…)°tÜsoéÌkùòµÓt ì«&ïï?ì¹k ]œ;ñoy‹‡>…»å)°£IwzÝâÐý] €ÔÍnü›q÷¤@}I6ÝÀaÏŽ»r¢õúš¾©¯Í”Î̇Æñ ý¬ÉTwãpHeT¤Ésž>ECôXÒ}¹|Àà#t‡ct®û9éÎ6pHÐù”Gt[@WJ¯›,WU÷r Ðd ;µäð“X\_A—6dÂ+ÒÉfªdwt-H7ñÝ]¡nmúÃf[µi]Þ’éÂù€™,Ƕd“îýv±-™lÜb±¢ƒ~cM€n è-•‡Õ u¦êëz7 ôDþÒ³XÇÌ1RH/”öo@Sã.(;8¬ 6{‹Ö©÷gf˜?ãÔY^E§!zùˆm¨ŒNö^ÃIw1òà.¶ÉbO€NïΛ]ô{ M€ž0­\@ojÌ…5 LÐÛ Ýy³»¾Ëb/¥^øŽ<Æa3‰}wóvwL®¹ûþÌ$ó_é}bÐOèí2Šç÷în´m2ª®‘m“ÅÑc÷rД+è­ôæA®Šº ‘žÐ:«,@—Æ@"©(4®[&è)tšã UéþèÇó´{²têᬒ~†çÝ7Z¼ËI;gfXûÜ©³RqM]™õÜû[~¬à¨º}C; ô«¤_¦ «ýã¾ï¦Ú¿œ@ï>Òåú3€ÞÂÃn-Щ¿g㊠zÛlÒ1CAÖƒt}E:YÖrŸ|î;¥}“ñf-¯X½Ï»ß{øïú^-o<luà]žÅ©8açµçCäC&°X¶vM ÑûîÚ@G¹¸¶vÞw--–*YÊ°Šq}ù€Þ¤Š«B‚nÈÓö¡¯»ãfèÆßÓ0¢u½À=¤#+¤_E ë+>;–ºÝKvÏ•ÏÝé kwg=3ÿ}ín7ïõ®Mš4iÆÝaî~¡÷eƒŽ<÷ÍJzo‹Þ‡A—ÚeÞf¹¸Cñ¶Þw·ûÚg¦®î^³@ïQãf—«b‚ÞÚ×µwÕ|ÍàèÆi\5 _­µÊôFU¸Ó=  £¡q¤ Õ×önà¯=HØfñä»o 11éßëé8Ó ‘¾éÄ9*¢d;,^ºÐE-®‘ï¼&.FÞËAOtôÆ®óˆß ¶Ìkó ôëÕmèrUTÐoîîãšœ&¾úú~^ÛxÝÕ©¯µ{愽®é“…–2df˜‚λ“ý×IGû2½Rª_gÒ©S¼¶ÆRqÊ)×=D'¹8ñ]ßË×®µ±õ¾3Lliêþxä¡f®²½G³ªMZÜärPŽƒnXê ˜¾¦éý½ŽÑE2le £¤h}5Áæè€Ì’ÇÆ}±;Ãe4é—©I§[ªšt¼{¯¿cΉAÿš¤âÔÚZøR\ã!ú!Ý´‹'Ýcm½ïªûøAtdM‰ô­›ªúëúÖs9.ÇA7‚»Â7WßÃæ~‚n ­  ÇÈiwÝå$éx]˛ʋ ç¥tÒ4ƒHw¦»×ÝžBMqB„~ùÒ…eÊA‹(!º8/.e¯:i€M·÷¶«æ#¸ –# ßê*'9ºátŸÿlå `Þl™ht>t´A:évÿhÇ®]9úº–.BN lÊîÅ®ÍB”ŽM:ÚTõ.9_Çw¡ŸÇýõÕ¶),‘BtSÏýÀ¡nöÞv†œnÖëî*Ë‘ÔPeÝh'Ý( ÛOµt9zk“cW ÐC“èx¬^`s’ƒôƒï¶–TØh-½T¤?ýá)æ¸ ]©¡»:aƒ.,]“Ûâôv™±öÞv†KeÁ¢‡Ù¦Ý*è7ø4óÎ8ݺ½3 ?erLPË :Ú}M˜&•3žœ*®kAÍqsBTî¸I?}ðG꼟]ç÷þÐo˜óŸ¾¥ËYp›û¥åêa³yΆèâ\Șž‹‹·;u{–é”sÝãÐå®L ƒn7éµGzd™­uô¿x÷5 ·Ë—¤ ¾{ÐJ¤Ó¼ûbõõ&b“®;ïšM?{ê5ß3rÏ!ÎI€NKk8åþ–rX—#» 圻˜sëšÃר´û¾«n¼‚–M­5{84ߥ2nòÕZ§µÕ±Wç&g@_ïå{ DÐÅ&Ø»rrbÄtí‚%=3ÊÂœy§íqd©±=åc•-lÊïˆs ÿÈ÷KkÔ3Ž°qqÄ ïûÜ£A§¹¸»o;ãÖ¤Yu,Jl5M’35t¾Zo´ ½½ñÐF Ý8.üI¡9§ãƨ/2••Øé8L?{êÕ_Üö çØž#Çeâ4ƒ~§z`Ÿwð™ÄsûïGçð=>ÌîÛîo7š\ÃƦ4šížÐ@/}þ3+«»Ùf35Íìi}WY‚n˜@ ã ](°é=3®ÑĤðRú\Ã+¾…L: Ó‘Iÿé õS¯=çíï:EsÛµøœpŽ[eˆãþ|¸©A'›1Ͻƒø:›Pq¬hhÿ­]ßì"ÞnX%ÝÔô’Û4j¹RdºÅjÀtsuS¼îqô[½ÿV <ÐÛI-3{õéȤó…-¸”~d°á%ÓÌ;ÓqBeäNÚèU¤öÂkÔmÿ†Øs _¾ta²Á ¿£t>syÑ&G/®EÚkßk~U¿¿5õÖ¼§†Eþµ€î»,QZ}}Mµþ­‹ÎæÙ6ľ–—‰™ÎôH’ãé¸Õar”ÎK韗ì6Fé®ùšó.’ŽâtͦŸ:õê Öü•£œÿ$rŽJèK GOÈ#©8â¹+ݵA=݉{»uWyƒª;Wm`Ý7QçfÝwÕìn=«AçÎïhYÇò€e zÓŠ:ù¬wÁ ¥ôЈHTaSfDb-%5¶ó4õŽsïبŸÚø`7oæéÇ_C˜#·ýê·#ÎI€>Ñpøä ß\;ðAÇó¤Ät\¾àþ¦ê‰w´Zµd÷‘N¦¯=‘’Ž:A[uuûY <ý‹ö£o¾ÁN;òÚ)æ´®6ßeé¸ó…è'÷¯óùi„sÜ-ïàh•þ¾[˜‘.½jìó^ÕUN 'ÿ èíôRú5JwõÚ/DéÈyÓ|Câ·¦"Ò¿:õ)êÿû ÁŽhG¼ã?п‰1GÖüÄAÌ9î{½üóÔ.f/=á)­ñ=T÷§ †?ˆkkÄsáä þ×îåÈyeÝgÒ«•èæ( ôØlqrœfÒs…e!Á+1è¼ÄV¨lϤ{ÐËðR6ͨŸFF£ŽY×`G¸#âÑšþG9÷Ú¿þ sþ¢i—úƒã.–qËÏa³eöÅ;z‡·näË]×ýu€^Jô‰óz6ŸÝÐÍ¿” t<"RLÇåd ÏšÌM:ÎÇiÎûL‹×k-6êÈ}'¨SÖ1íº¾%cÊ5ÌItŽÒpóM_µSžî¸“.÷⢙Èܽ+n„÷xÃö>”y¦¹ôÒgä|ønmì*OÐM'PèÑÙòF9ùâÕœxœ÷#‹­Î°ôdÔ êȧ¬k´cÞ¿EÈO¨˜ÿ|§ùš·>sŽìÆMîziMH›™›e0èI!Žßä#ë”Cö·ƒî jâíëS®rÝ8Z0ð@w¥‹m°¨Â¶(DÓ©I§m3G†Y"tó÷—Q¤NPÿá ;üàØiG^»ÆùÔN/¸Šô¾ ͯbÆÝÕ.?‡¯Dß7¢ îò*^õf6ŸæÐmÒSÞt%ÞèÄæS¾nVø4ÐÛdËóÝsòÅ\˜IG ÓQ˜þÎdË“„/ÓüwlÕ¿Fy9Ä:¢]ÐAFùdÍ1æ,µú>ß&9î('õ¾FaÇÖ֎Й©á¡ÐÖ½~kN[iAw¹ú7÷€yKGö˜óôô°x¾­ê¥çF OŒ°ÿ¨¦^¸{÷»Ã­O2ñŒúù¯±]G°Ü©~øyìgP#Áüù¥–ÉÒ¹ˆóιæ¸?!>Þ/_ˆÐÛ”ÕÞ°nsËÁýuÚ¯pæ¾é` Xé@¯}›ã§¼¹±u¿R³êý:k=»"ŒïÑ–pÂç­èêY‚.VبIóq®áÿÔHç™w”›ÓÅ͉Âæ/ç¨cÃ~óŽ ?M ÿšóËkç[bÄ9g!OÊz;nÐ5ÐÓËòfoxÓ-Í[*å¶îÍ:½Ç1ëÖºq‹j‚Z´x¤¼@¯ÛJ|#ÕlXg}ä†êUÕÌ\íÛkµXïÜ)ï•~Ñ=Åc‘?˜j-ªÙr7ÜÖBºòÕZyúb»I¹@Âö!I<G—¶äŠ#V]Éj˜îÖ¦£Úĵ˜õ¯Î3Ú©¾¦k”#c¾ÙÍF0òÛy€Ž8?º_Ü&ÙYœ/t¿F—ùÿÈ­u«µªQ½I“ê5Zµ¸¿é4ÈYռ醷hŸxõC[­¸g}øH|Õˆl)§™ô\iëá1j˜~$¯“‡—ì2qù͈ö¯ÎkÀS¡Ê×nîä®Á!åáv—|!:îR=øYRZ#} \DÈ“B“8èWi”¾A¢°7vÞé(õž×Û㫆tšxçÔï íßaá?¾ÿþƒåËæwqÿÜÉs œŸ|T:b|n¾Þ·/ ."äQmÔö¸ü\)óîJ aúÎ/)é»óîóê•ûtZ:ñÅ;—¯];uêÔµk—¿¸lâü·B=>«wáüS󣳤´zBn>3èçÝà‚@^¨'Òi{\~n´J: Ó©M?2Û§K“×OÀœJö|€4<&2WÈÄm·Æ3ô§TŠlÒéó¤R–kÌI=!G½÷U}y/]Våq¿uÄí,(8zT.¬ÅmÊ{e¢á‚@^IèxgÎ{¦|Ä0JúNFºWºïZs1OŠÏ5Î}RŽé3sÅLܸ| wŠÍ6:ïãåCÚ.T¼wôÙv÷£uy"O±çh䓱ÒAãç¼´–—òRíÄ%=M>$b%·é¤sF Ô· ¶óM„es^Xò9Ï5{>@æ|t®ä¸ÇÀżVºnÒi˜ž›«ìXô(#ôÈ¡”\ÞlûVõž“GÓp,ߎýö•òç„sR&wÈEf+ýqšIߪ֧GSÒQ=&ßóÞYc϶ ™sšn§œ£„{¹“+gÜCáÒ@>¨ý5é”ôMꎩ')é$%‡Ý÷¼7mð߇=q‘šsžs¿}”|\ʹb Ç},\8¨4Î{rÞ7¨3b-ÅRr%8'—·ªt¨õ^õ+ã›sóÑò‘ÉçrÅŒûövpÙ@ ßÔVÏÇ¡Ì;Ó‹ûú\Fs›®‘®¹ïØÏ{³·ß³ùúÌÜrQ4çzºýèÑ1*çÅd- uÜûÂU|î„=@÷mÁλfÓ»K ¡Žº†ú–™þ4ЄŽ™û+Ã\4ç˜ó•±öœ:îPYüP¢!óžoFzX¯tÝÏËÛ¶Ø7òBO˜C1gE5ºüsÞKqR‹ŠQa ­eÙqµÊDÂ%|WXO‘t’+.6’u”}×ýwÄúŹ‹½µë]Ïþ¡œb.™ó‚ãÇ·ÝS”Ë:eˆãÞ®䢆H r¤šnbÓ]®A’Q×üwêÀk¬¯ÊèÝÅ}À:yñ„7(ǘãd;7çªÎAšfÏõ5kçáz@þ)ƒŽW·\ÝÁ½÷~&ËÃâFaÒ¹ÿŽJmÔ¬ç]¼øòÜ™‹;õ1Ô¸Ãâ†^3aÛœ_å*æÌœ£.™…ª9wæœÓD ›üV;ݤc›Žr¹ÏšÅÞ‘³¨Qç¨sÃ~ñâ–UsgOȘyŸ¦™33&Ì}âåw~%ŒcÈ5Ê‘Ï^øÇœ¹í½ _]UÎ{ÂÚTÈ-ÃtJú"Ó.cQÿ£Ží:¥ãŽÁÆÑÇcÊ 5ÊEÌ5·}–a$]øóÚwÐö¤¸T P)Ô×CÞûÖÓƒ{ þ;F™u݉·vرˎ9Α9j8Kò<‘s”pã@ R)(ÝHºfÓÏ¥™Þ{¥ŒºÆú…ö#2ðô'rä²Á(ÿŒcž`hÖNûgv†»Ž8aœ@Ž(?Þ!Í¢V¦™sž y8à²MÁé¼EŽ÷Èá”\‘û%àq½°s'ÃýKbÞu‘±4Ì<ÖÛ*òOÞDÜv¶Žòí ýZ‚r²MÇîûO5ì. ½fq–±uçÚ)0~låc Ã-›âãºþR$¥áö Î;¶… ÙªR7,ÔsÏó-%ÖmŠHN×uÑ/Ì”—=7_Âœ˜ónAð±ƒ@e­ˆ%Ȩ“¤F]7ë”u vL{Ñ¢ÌQãÒR#ÛÆFÄ…kŠ‹ˆŠLI—˜¹i+cœC.PÎœv´W*,aÊEQÓ©ÿ®gå¨U§¬ãÜa½è¬­óæÍ[4oÞVò¯_Š˜»®SÎ"sÙš÷Œ„*/E¦ëþ»nÕ©YÏ'†»ñœwUø!qÑ–Êkƒd@ òE]²êuÂ:3ìIJk°çžÃæ]ú È%vœC.PNRpÛ㡤•»Ú.ÑI×P§fûðíVÊgŒ3ʹώœv°æ P@(¢[GâÀë<ƒ]¤=ß<þYŽrJ9JÁÁ25(`“Nͺ`× ìˆvwsí¥ŒcÈuÊ;Œ… (E LP—YÇ°ïµBœBÎ(¿J1Ÿ Ÿ*p JI²O`ÃÎpçÈ‹xsÄäÄ–£\;0æ P *4:±#eÀ~ÑÎp—‘ß%üô3åÛ{vƒ9Î P`+lìÀžû$î™ö«ˆv xÊ7fü*3äåC¦€ÝÎA  ¡ˆ˜ñÌ°sÚ©3/J|š>0:>=¨"ÁÝ­o‡]æý#•ðíÛ“ú¶K5¨ PETpTt»Ó“†0ÞÏÿ5$~ú‚ѱ°2 ªØ ŠˆŒÑmÁ’¾é=ã“:b%Å÷L [›è¶€8TFúÿHäY”¶/–DIEND®B`‚optuna-3.5.0/docs/image/sampling-sequence.png000066400000000000000000003063271453453102400212010ustar00rootroot00000000000000‰PNG  IHDRvfü9¯zTXtmxGraphModelMVÇΫL}šOšYüWä° &™`2›9çÌÓO3ß]ŒdᦫútÕ¡ÒÊug^µÙuCZåU–þ ¯A ˜þ"ÿi‚PFÁÇþ 1Ð/‡eýÕ=ŽãO:GÇŸjøEEÖÿ•}†»jÛèð?ýË«út8°Ôlð€!°²`ö»< ìßàÇ6ó²X©Ö篡›zƒZà9®5œ©BNɇ .œõÉWH*LÙAÕã|r*ÝÞÍxbµÔ+ ¢õ´å›; Yx_ ›ãTë=àÓ™wiÂP88È Óýë …t`My¹Ä²AUh1)‡)¶wajÀ[ ‘•Mê»ZÄSN7ªbšÎÁùuòI¥O1ª:pí{ŠœËFu'áH QBP(h ®f‡6œj¹È¿\©P™7¯Ó"ÐÕ5|€Kºlš6Ħ‡­ª6»£Žf„Û›Ožø«õdm†*«ÙΧ>ÏÑäCÚék—è†?G7‰Ý‚zk& †„ÏQC[÷f½(LÜ1‡P¬¼á©}EÔÆkv,bò#J9‚•×·ô9^ ¤R ÕâÅ‘GÃû‰QˆåůóŽ #Õô0õý&¿"ª¬)«ŠNI/íq"–4„ƒë äûn6ô†ÐÞy“ÛŒ¢Ýd,\øÅûóUùÄàïÆæS9§¹á „Œ“;Ëz½Çb¤-ÌÎpÍöùrE|éä¼j‹.ÍûÅ)_Ýov^~ÍFNóléÛo9ž£M"6^nìõÃý[~ ÃzOL§{QM‚ñü¨”„M¦çÄ(B8Vݸæ'bÍéÝh‚McÇðòå ¾Û¶^Ð÷i ápHGÅqu ª «ã¸€‡ìP³ îp°½¥ŽBJR>½Aq`¿VYÌ[tGFúŽ=4)‰û2Í5­Fw#Nþ´ÞÄ_†&ðùØþv¡Ç‘[l|ÃÛ®Ó6ª¶wuî£8hœlíS«“²6$ÛfC óE>fe|Ó€ð‰Ÿ¢7ÄäßøknA5Å’£šÊîQиö|è`íBˆÙ^B ¯¨±2PÚžœO]pý‡¸V|³‰_“k!XÍò~¾ÃVY)Á¼H \¡CQy‹àÈ—Ð=ÚÙ¶syã’å÷S2ô{íyBpõWë¸)`@ßøŠXêçÞ+^ý»:2Î+ûݞȕöH§>ñÚô$g¹£_²†PÅÐÃ…Õ«8UÎé\ê{ÝѹJY5õ($ŠôA†Ú^ï10l}©JEj%öR˜Ms)yÕÂ}Á>ž‹€¢Ò[¨LÍâ;aÕ2–‘·ÙŒÐx†ˆÌÍç§l¦½vMøA¯øa­N1 sNµË^hal‘‘L€èôv^TJ“•(tX‡«9 Z›éÃ0p_Nýãé…€6ºÐ}1“*'M¯‹Àq kv£Ï‘þÖùáçÆ3.¼«|·{'‰¬{p=û¸¼÷}«(U›Àû^‘H(X®FëŒb¯‡ÚÕ†ìÏ‚0¯ Fu«#Ë^äXãyal PZ(èÛ,‹|løœfho#Ó~Ñßô¶ ÂÝf+û^ÅúþÀ¹¤¨rY œ^7joÏ£Ý]r¿ô%À1Ö=¾¦p Ü|R¿Fˆ>í§ƒÅ¹p¦¥Ã›Z(€QCHjuïQKI½´ÐTˆÏôœÎUåŽä³;„<>,?ÙÄæ* ö4A wšªÞËÝÑ(’?®üäôµ-)%Nys\¹ ¿õ_évŽ·:¨¥Ünt«G¬9îB²¾`ªc!FƒÅ7¬k„еÀpB{§ˆøo…= ¼²ýìf.Ò`TI¾²vUæ¥x¤~&Ñð4×öyÅ|3BoÛð“Øz(%Ó{‘oœdôú&VÂiÂm%_c/¿§ xª?©fo~ h´V”¯…û€v´q;7Hé¦Ý2ЖÙÏ´.–H-ùý¿oÄVF¿‘OÑjè'zŸÙr—¾ÝCùÁ?CÊ=dÿÿ7‚÷¿ó6ÊÿµòL IDATx^ì´Õõ‡·Š"ØP0Ø¢X"h4XЀ–¨0…Ä¢QÁ^¨b•¢ XPQÁŒ¢Ö€Š+ bCű þ×=7óæͽwî½SÏ|³+ñ½™söþö™ýf~sÎ>«ˆÈÂ@€ @€ )=ô¬¢ÂN‡¤cÇŽ™2c!@€ @€@^ :T ÂÎàÁƒeÈ!ye߀ @€ @ SVYe„LE c!@€ @€ÀÏv € @€ @ £v28̆ @€ @;Œ@€ @€ Q; fC€ @€ „Æ @€ @È(„Œ³!@€ @€Âc€ @€ d”ÂNF‡Ù€ @€ @a‡1@€ @€2Ja'£Ãl@€ @€ €°Ã€ @€ @%€°“ÑÀa6 @€ @@Øa @€ @€ Œ@ØÉhà0€ @€  ì0 @€ @€@F ìd4p˜ @€ @€v€ @€ @ £v28̆ @€ @;Œ@€ @€ Q; fC€ @€ „Æ @€ @È(„Œ³!@€ @€Âc€ @€ d”ÂNF‡Ù€ @€ @a‡1@€ @€2Ja'£Ãl@€ @€ JaçÛï¿‘×>xN–._$ï-_,K?["ß}ÿ Ñ‚@âš®Ý\6Zwsٸɲe³V²Ñº¿LÜ& H–ù*Yþô^œùŠÑá%@¾bL¤•ù*­‘IÎ.òUrìé¹4´æ«Ô ;¯}ø¼ÜóÜdùâëå²vÃ&²Îšúo}YmµÕcH–À?ÊWß~)_~ý©|þõ§ŽØ¸óæíeŸí‘† %k½'B€|•v: B€|„R®Î!_å*ÜÙr–|•­xÅ`-ù*ÈtQç«Ô;߬\!w/˜$/¿7_Ö_«¹´Ýf?i¼Æ:Õç*Ä@൞‘ç–Ì‘µÖXGh}¤lÛü71ôJi @¾JC°¡ä«JhÙu.ùÊ®xæÁòU¢ìï#ù*¿±ÏªçiÊW©vn~jœ¼öás²Í/v’Ö[ü!«±Åîœøä¿ïË“oÜ'_}ó…×qˆ4[{ãœȧ»ä«|Æ=ë^“¯²Áêì'_UÇ«’%@¾J–R½“¯’"O¿µHK¾J…°óâÒy2ýékd›æ­¥õæjáʵˆÀWß~.ÿ~~ŠèzËcÿ00öþé0^ä«xyÓ[¸ÈWáòL{kä«´GûJ _åk|¯òoÛ¼MC¾J\Øùò›ÏdÜCƒdÕÖ”N;.«­ÒÀ¶8ãO,úèE™¿èAÙãWûKÇ–æÀã|ºH¾ÊgÜmóš|e[Dýý!_å#ζ{I¾²=Â?ùG¾ÊGœm÷2é|•¸°óø÷Ë/O“½ýiÒø¶Çÿ,&0ç•;D§âµßå{™o×ÈWùŽ¿MÞ“¯lŠ¦¿/ä+ûcœÉWöGš|eŒóâa’ù*qaç®g'Ê;ËIÇíþœ—x㧥Þ[þ–<öÚÝÒi£#äwmÚ[êe¾Ý"_å;þ6yO¾²)šþ¾¯ìq^<$_Ùiò•ý1΋‡Iæ«Ä…ëæ^(«®º†üvë.y‰7~ZJàóË¿_˜";4ê(»ló{Ùb‹-,õ4¿n‘¯ò{Û<'_ÙÑúþ¯ìq^<$_Ùiò•ý1΋‡Iæ«Ä… î=Þ)š¼Ãfíòoü´”À÷?¬”;æ—­×ÜE6[c;Ùu×]e­µÖ²ÔÛ|ºE¾ÊgÜmôš|ecTëúD¾²?Æyñ|e¤ÉWöÇ8/&™¯vÎûWÙ~Ó¶²Ý&»ç%Þøi1iÿ+[¬±£lÑpGiÞ¼¹´jÕÊboóçù*1·Ùcò•ÍÑ!_Ùß¼yG¾²;âä+»ã›7ï’ÊW;yiø)s#o¾Æ²æškJÛ¶m#íÆã%ÀƒG¼¼é-Zä«hù&Ý:ù*éИÈWaÒL_[ä«ôÅ‹ª'T¾BØ©>f\ zÜ7²þ²]»v²úê«CÊýûÈG÷,ÆA” çU[#ªxÈŸ /SQöçg£±aµUW“]·«¯¿jµtèÔ¡k¾& Œ*q‚|U ­ìF¾JÂëÙÌ–×^y½Ðõ민îºUØÑY/COêSg–ür‹_Ö1A¿8©ú»bÅ×rÑ:"OÔ‚Mæ¡(L®Aû­ä¼¤näJläÜê „ñàqï÷ÉEç^$'}’ü©ÛufûéÏ;o¹K.t±œ6ä49¤G7gæNÜJ)QÅ-ìTO²º+p²ón;'6+§”å~ÂNužÆsù*ÎIõF¾JÊvw¿Q ^ÿ‚;ßÿ½\Õ rÓÄ›äÜ Ï‘=öÚ£ Þh{úÌôqÿ‰W]/£®åÌÄ4Ï0ÕÌ" 3;aÒ¤­0 $•¯>úà#9ëïgK‹­¶ƒÔY±ðþÒ÷eØéÃdý¦È9ž]çwaú^m[q ;<·V¡¤ž¯vÄJ§?óŸg|§æ~øþ‡rF¿3¤cç=¥WŸ#Å;…X—üõ¸Þ¾3nüfâ¨rüÔ£O9_ô U§ýö–~§ö“›'N-,ÅR¡iÊuSêM½ûï—ÿuD™-¶j!}N>¶ÎR㦩F]s™ì°Ó¾Þ¿ðì Òÿ˜rä±GJÏ£{È7_ãدB–.ÁЖW^xEÚïÙ^ŽþûQÎÔcHÅ wß~×ñG§ÿê±Wç½äÈ>GȶÛm[§ÿW_~Un˜0IfÍœUï “¯™ì\û›]~#w?H:ô :Ë*õþ½ó–;厩wÖ;ï½wßs¾ò.|q¡c¦Ú¯K>ýdya‰ØF7—Óû.þù@9ôÈCëpÖ—¥)×Ý(#¯!›m¾™µÉ¬J„b<ÞŸ›ÿxѹ2çÁ9άEÍ¥ûì¿ôíßG6Þtサ22Ëèôkÿ]·Üíð.“JìªvP“¯ª%—ëjÍWiñ²”°c„˜?÷ü³Œ¿d¼|÷ÝwrÎgËÇ}\o)–>ch4˦t©Eç;Ë‘Ç!ëý<(ˆ°£³qôéÐ#už‘ÜËf ³Ï–æÌôÖcðÈAβ|ãÇ©ƒN‘{o¿Oîÿ×ýÎ žÇô”}öëTø{àç¯æŒiS¦ËÔë§:KwõCâ_zÿEöìÜQ4hPÕgŸ~&·N¾ÕÉãÞóž{úy'_›£Ýží-9KË8 _¥%ÑØ‘T¾2bçßúýÕwi§Î°ÖÊæyL½×g¢I×L.,¥ßä—›H»Žíœw/óÌàþ®Kîõ¹MËx軕~´{åÅWäŠã ïzÇ 8NZíÐÊëwm±g>ïR¬ ÏW¦ý~§'7ýófY²h‰ó±¬ëa]ë׶çÖJ–æÖ2Ò“ÊW;¢öï{CFø~™ñ^þÆ«oÊŒ»fÈÜYsåØ“•Í[l.›þr“ÀÂά³äÌÎr];®Ó‘Ÿšû¤|õÕ Ù½ÝîÎW, -|i¡£:½dˆì¸óŽõ<3ÂþB§áj‚ñ;Œ@³üÓå2ìÒ¡²úê«;ö?x~Óõõ¦Í7i.wN½Kžöy¹xÜÅΔ^?aG“Ç ƒñç€nû;ÝýkÚ=òì¼gëÌ€ÒºEgŸtŽ³þû9çÝvã4ùâ³/œ™C+W~_ëÖÛn zñž’Ô¯—ùí­ÖÇf?&Çt?V®™zµü¾ÃO_qý½oúÞ×yP1Õ66ßP~ßáw…Ü0ùÚÉÒqŸŽrÊÀŽHãvôžsá™3k®üùˆ?;÷áËÏ¿,·NºUÚ¶ÿmá딾4\6ì2¹÷Îûœ%`­Û´–óȤk'ËÑ'-‡ôìæü÷øK¯”ö{µ—ßíÑÖw¾ôjAØi½ëoäò‹¯Þ{¿N~1yAípNѯÝAlòã•°3à˜ÎìD½ôEIó®2Òeæ+]Œ4Ošvf?øˆô<º§SOIëœ)ëýÚ·e»j¹ÉWµÐKÿµµæ«´xXNØvÆyγC¯¾½dµUWuêê2÷ì}>:µïiÒê×­œg}®ybΓróÄ›Üh„ö ÂŽ¾èý4ç2i¹}Ë¢˜´­K†]*ãn¸Â©g£ÿ­¶6^«±cƒþ ˜ÿäÓ2éêIÒ·_éÕ÷HYmµÕê-ÙÒ™Ö:3üíÅïÈa=L6Ú¤y!güíø¿Iï¾½qÇ” ÐgDõI—¥>4óag‰ï™ÃÏtòö£?êäq}¹ÜnÇV²ÝÛù~`HKìÝv¯Ò•ðlJ*_©«ï)M6XßÉ4Û ¤S¼÷óž¶ú«;ï+ë6YWôƒ¸ 7;î´cAÈõþ ×RÏÌ{Ö¹ß÷û¿ýdÑ‹¤{¯CE—–ßôÏ›dµ äÂË/}Î3×6ûE3Ùv»_•|æs ;AŸùÌ3¦Ön=òØ#Ÿ·úÕVÎ?ïaÛsk±wßðFòO-%•¯vDÒ½–Z:í×IÚ´ÝÕQV›nØ´Þ×ïƒA±9ÞŸ›ä²á/6¬3P –žqü™Î×yvT½ô¼KåÛo¿«SÌX6ôö°Ë†ùmÖ‡ƒÓû!»¶Ýµì2!óÐ̇䒫F:3…TØ™÷Ä<ç¿ÍƒÌ—_|骿óÑO?(™këý—:ïƒæ™Ïýœàv‚>_™ö^4ЗüfŸm{në®Kêù a'`„uÒ¯¾!3îœ!³xÄùê­‡IôÝ] Ï] &¨°£/[ýàÌ rÍ7‰`ó-·(2úrpåeWt×®A§ –vm-Gö9Ò÷&5/XºÄ Ü2&÷ƒ…N+TÁf5Vw Ü;€é ¯ËÂÜv–¾³TN9öT9àêÍ Ò:Þ÷ c¿~¡Ò/iªÛáBÛ,÷À0”‘ž–Ô©S4š°£‚©.I,·S‹wfŠùzsùıuî%#Öþ¶ýîu Ý=kçÝvr^@ž~òig™èšüÐœrõè«4\3ö™÷Ä|¹èò Yy¥D•RÂŽ™²ûÇö),ÇÒûV—è—(ýúÔ¦R6Û˽#V%K±Tعþö‰uŠ#zE²0í¸ÓrÙù£dù'ŸÖ›5i„ý-·ÙJþ~Æ ²`þsÎl¨rvÕr{’¯j¡—þk“|Q “N9aǯnMкx謺›¯ŸZVØYòÖGìÜgÿNõþx»×{—9=ÿì …¥QÛï¸s?jŸÅý¤;ú½plÙ‚ ;>é×ç £¯–Åo.r„ mÕ} Ò—ÍeAlÒÂÑ ×lXÏìr»b¹wĪTØñn%ï~ðùõo¶ôÒTŽ‘‰‰ÖÛPñFù˜Ã»”ÕäÏRv™š=ÕÞ²ä«jÉeãºZóUZ¼ KØÑüôùgŸË’·Þ–%o-–§ŸzÚy‘[gÝuêÌ–)µIƒÎòÑ%§/=ÿRYaGgõè³ÍÐK‡¿¶MÞÐ*»ý]ƒõ™’ÅÄlÑÉgŸäé:{sÇ]~Sòùa'-£;¼Ò”¯t¦›®xàÞ÷<}îÒZYÍ6lV0û«ÿ~%ú¬¦ïcºTý¡ûvJe˜¿Ù~âI±Z¤Þû²Øó‹ßs‚vÌGørÏ|ú|õâs/žË=GØöÜ×—ÔóÂNÖ›ìÒó.“w—¼ã,c0òª™±SLØ1ú¢cfÚ±GU_EsÓÄ›º5ZGRüŽZkìvŒB½Å–[ø ;Þ¯mA•c„+—×L ÖZÖ*{ ä©3æ¼ Ì,½çŒˆ çéÒH­qãwÝõ®œhatYÃyg— Æžï*88·¿3½9h_ÅUSJ@ ʲ#„šoA¨€@­ùª‚®"=5 agñ›‹eÄàòð¿g;_áuI¹.yøæë¯åÉGŸ ,쨣µÔØ)%ìtÚwoG ñvÊí47"ìD:Ti¼iÍWºyŽ®¦Ð¥ëº¤RgJ:”L¿év§V©–åضկ¤é/šÉ#®ˆEØÑZbF ñ ;åžùô÷Aß»ô\Ûž[k¢]Š°³Éî‹ëä%‹Þ–3?ÓÙEƻӋ±áŸãÿéì²Pl¶÷«ºY¯©SïÝ_X*YŠen6ýr¤»¹\sùµNa¼b6[u èå_îÌÄió»6¾M‘A- ¦ËºÌ®X¦ø©{½©.źuÒ-2büO_åÝ3v¿µØI†Zد”]fv‚w)–ö«¢™. ÑÚZíÎãùôãG Ö³Té—-~YTl15¨´°§ÙEÊ=3Î=#Ï,ÅÒÚú¥×ýÇÚ3^0ïÙÂ2ƒbQ-¶ÌH¿8z‰3#Qk•«±£í›%EèÔA6\ÃÙeÊ,ñ2"u›ül úòb\´ø°)\jÚ»ïÎûäš±×^äŠ=à]ŠU £ K±¶Ý¾¥3»é™ÿ<ëûE­’²rwqRåìâ÷á¨5_…cEí­Ô*ìè΂Z—ð­7ÉÙÃÏ’-¶Ú¢°\Ýû5:È$³+–.KÕâÅî™wÆ[SƒP?¨™º‡Ú¶kÖç%ÝEÇfƉgþÝ)–ïöW?¨é‹›¥6½Ðç¥bK±4çi­•¦³ ¨±Sû˜¤…ð $•¯®÷™ûàœ¢ÏIîûwÝ_n˜pƒÜyë]2èâNS†Ã aÍØñ{~1K±´\‡yæ3ÂÎæ[nîÌ& ò|UÉs„mÏ­á\ÿ“z¾bÆN™kád-ì«xõâ;ýºNý]g=ôôa²åÖ- õg¼fvÍÒ·ß­SØØ[Y†œ:DÖZgmGnܸ±cÙ1J »k㘛mÇwpÎÑ‚|Z@¯Ô¡/‚CO*+¾Zá¬ïôV@_úöRç!bÅŠ¯Ý¨t§#L9/0¿:? IDAT®b_f†M6ÛØñ]ëùOþôãO¿µ¾†úâzíå×ÉsóÈà‘ƒeõÖñ-ž¬;ŒvÜib¶ òÀ× [¬Ÿ¤nä¤ýÎKÿañlGpÕº ~…Í}ª_«T˜Ñkƒ;:®Ÿpƒ¼ðÌóΰØa§êÔýÒ¾‚Øä7¦*vtyêñGž ç\pŽ³¤MÍçZLZg™:GA…°•+ž|ÞèaÒ¡S‡¢_Ô*y +w_’¯ÊÊöïÃÈWi P«°c–3é MMÔeYž{¡³»Ÿ»pq¹H*P_Õ rÓÄ›äÔ§È>ìSx¹S^úÌôqÿøyç¬Q…‚ô¦x²>[™|¬yVwØÒ]GïWóéÒ0Ý¥ÙýÌç-žäùªÒ盞[ãºó’z¾BØ a} ÒåFÏ=ýœ³œ@gºè¬•7¾îRÖ%HÃG/T7×ݤTìÑíÎÛî·ÎÏÍ6æ:äÙ{ß½ítç<8GVk°šìÔf§‚`cDý™n‡ùÁÒ䆫'9ë6Ý…AÕl³£ƒþÓµÙf[Þr.é”dÝ•å…/:EÁvûýnÎ%óŸç|]W1gÈÈÁ‘È-ìhŸƒ»,7*lGn|÷[faüÙ éÒû¸^Î6ŸZ“è{tFL]"sžnxP÷¯J¯¼ÞÙÖsðˆÁÎl ?®å|û÷IÝÈqû™×þÂxðÐyýJ<ú‚1²Ón;É]÷—6ÝXÞ÷=gVÚŸ'}ú÷q¶7¢™±c¶äÖm3UœÐ-²ÝÛÝzÿX›­µõEåÐ#ÿ,í÷l/‹ÞX,·ß|»S(Ô¬÷Û‚{îCså®[îv¶ÉÝïà}Åh×Øÿ¥WwG°qowî^§mD=×[ð9¨Mµ ;:shèiÃdÑ›‹äˆcz:9ä_Óî‘ï¾ýV–ú™3 I×ÍvÂbTÉvçå–ˆ•[_î>%_•#”í߇‘¯Ò@ VaGg쌹h¬“÷ôCÑovm-úkúMÓeه˜eª•;F$¾fÌ5rõ˜k¤óŸ:Ë~ÿ·¯¬Ûd=yóÕ7åÞ;îÕsÎùg˾ï[gc µCŸ‡:îÓÁãõRó¾ûyÈë¯ùˆ6ÿÉùŽýš{õù펛ï¶í[Ø5Çœ§5çéß mKÿ¦\<îb§«ú}z¿Óe»·wlÛ¡õ¯Ùî< ƒ$©|eÄÕQÃG93éþ°÷²ñf›ÈW_þWä çݯÿ¹ýÕ:;oòµSœ•æ™êóåŸË·Üél°óÉÇŸê…ÖZcGÿþëR¯ƒ=HöìÜQ^zî%Ñ¢êÞg>·°ôùªRaǦçV¶;8Ù$u#Wê–Ît¹ëÖ»dά¹Î •N¯ÝnÇí¤Ë­¸ÝÕÒõ\ýã}ß÷9[ë—wpîÿ׿eÒÕ“œ›__°tëb-¶¥‡™‰£Š¬î¸¥3ZôÅJ Òógg›Í¶Ø¬^Q<£4Ÿ1ì çË{ÐCoþ‡f<ä,mzæ©gœËvÞ}gÇÞ=»ìYG±.ìêÕ|CÙvûmÛÔ‡Nûí-}NîãLkÖ£XýŒW_~Un˜0IfÍœ%+¿[éô£ Ò½“˜^ï>Oo¼?u;@zõé嬇×ë_Õ  ¢8¥(¨¦§Í°ò•Þço¾ö–L»qš<òÀ#Žp«"ƒîRÕíðn²Õ¯¶¬33ÐÒÓ…{n¿GfÜ5SšoÔ\Ž8¶§Ðõ€:W ÐûR:tg*}@Ñ~:ØYzüíð:yË{žŠØš£vo·»c‹Ú¬õ#Ɖ¾8háóï¿ÿÁw¹©çõõ×ßøÖý j“7ò•ÌØÑkõëôU£&8;‘éËÔ_zw—6mÛÈeÃ/“³†ŸU‘°crœ›eµŒ4ÿªh¦miLt:w÷^Ý¿'¦RPÁ©–»ƒ|U ½ô_V¾JÚÓZ…ÍyºÃÍõ®w>^éÿ×çÃÿv˜³)ÅéýÎó.æ|¸«df°ÎRÖûwúÓåø*šèsa§ý:ÉA>°ÎÖäÊд­³†ô™HŸñ´>Ç_ûýUT¬wï¬êÝéKm¾uò­N׿Ú¾ðuëѵÎóšû<Ý0ÃûuªóÌç­¿äùªRaG£jËs«ÎÜŒãH*_1c'ŽèF؇ޜº¦NÑuï–a—E›®µ0j6‡ÝgR7rØ~О?$<‚Ĥš?ÖAÚå; ¯ìŒ«ñ*íùÊnúÕ{t«öê{Èæ•ä«lÆ-¨Õä«ÿ‘ÊÓ³œ­¾&•¯v‚fœžgŠ¬:Åðž’øtZ­Û£"Ó/6n^g{ ÑEfRR7rdÑpiðÐå”çŸ}\zõ%õêgJx ¯ìiÏWvÓ¯Þ;­³v÷mwˈñ×™UY}‹v\I¾²#ŽÅ¼ _åSرõ¹5©|…°“Á<©BŽþÑ×õ–=ò¸\8ö‚¢;\ÅåÞ ßçŸ}AÆ]2Núöï[Ѳ°¸lŒ£Ÿ¤nä8|£Il x9öº´çù§Ÿw¦ú¿÷îûráåȆÍ7,w¿Ï9ò•Ý€¥lÅW—›ësÝÿ¸QÖoºAຉÙò²zkÉWճ˕ä«| ;¶?·&•¯v²í<6êîYZüxÞóåøSwŠ,7hÐ 1O¾ûî;7r¼ó0¢ÅøN:óÄÜ~eJêFN,ø9ë8­ _Z(ý 6”N?^öê²W=9 î$@¾ *£§¥5_egäfÏ5W{ŠSwç”$®Z‘;Rä«@¦´òU¾„ÛŸ[“ÊW;)Mp˜•MIÝÈÙ¤•=«yðÈ^Ì°¸8ò•Ý£ƒ|ew|óæùÊ¯ìŽoÞ¼K*_!ìäm¤áo¤’º‘#uŠÆ xð`0ØD€|eS4ëûB¾²;¾yóŽ|ewÄÉWvÇ7oÞ%•¯vò6Òð7RIÝÈ‘:Eã;Œ+ ¯¬ +ùÊî°æÖ;ò•Ý¡Gر;¾yó.©|…°“·‘†¿‘HêFŽÔ)çE‰1`%ò••a%_ÙÖÜzG¾²;ô;vÇ7oÞ%•¯vò6Òð7RIÝÈ‘:Eã¼(1¬$@¾²2¬ä+»Ãš[ïÈWv‡aÇîøæÍ»¤òÂNÞFþFJ ©9R§hœ%Æ€•ÈWV†•|ewXsëùÊîÐ#ìØß¼y—T¾BØÉÛHÃßH $u#Gêó¢Ä°’ùÊÊ°’¯ìkn½#_Ùz„»ã›7ï’ÊW;yiø)¤näH¢q^”V _YVò•ÝaÍ­wä+»C°cw|óæ]Rù a'o# #%Ô©S4΋cÀJä++ÃJ¾²;¬¹õŽ|ewèvìŽoÞ¼K*_¥BØÙnãÝdûÍ~—·˜ã¯…’º‘-D™J—ôÁƒ|•ÊÐ`TÈWU@ËÐ%ä«  SË _•E”éÈW™Æ{$•¯v®xè\Y§á²ÛÖÈ4/¿^.3Ÿ¿AZ­ù{Ù°ÁŽ/-Z´pþqØA€|eGñB„|eÿ( _Ùã¼xH¾²?Òä+ûcœ“ÌW‰ ;·Í¿ZÞ]¾HöÙ¡G^⟖xçÓ×åÉ×ï•]ï/W]aÇÂ8“¯, jN]"_Ùxò•ý1΋‡ä+û#M¾²?Æyñ0É|•¸°3÷õûä¡Wîn»˜—x㧥^xçQyí½g¥ÝÚ‡N¾ÊGœm÷’|e{„ò|•8ÛîeÒù*qaGüÈk÷Èì…wÉ^¿>LÖo¼¡í1Ç? <òÊtùôËd÷Æ’ÕVY½àá6Ûl#›m¶™…ç×%òU~co‹çä+["YÞòUyFœ‘nä«tÇ'LëÈWaÒ¤­$$¯R!ì(ø±ž%ß}¿Röhy°¬½f“$bAŸ¨Š€N¹[øÞ|ÙvÍßJó[Õic§v’&MÏUMñEä«ÓJ _åo€¯òs[<&_ÙÉà~¯‚³âÌtHC¾J°óÆG/ÊOŽu"´ã/Û˶í’®ha <¾øúSùÏ3åÓ¯>”õVÛP~Ó¨S=F{챇¬¶Új°³Œùʲ€æÀòU‚\ÄEòU~cŸUÏÉWY\ív“¯jgH ñHS¾J°£!xþͧeæ+7ÉŠ>—u×\_6^+i¶ö&ÒtMdõÕÆ%zƒ€ϾZ&˾\*}þŽ¼ûéëο\}{iÑ°u½³×\sMiÛ¶--%@¾²4°¹E¾²(˜5ºB¾ª —GN€|9âÌt@¾ÊL¨rkhZóUª„ï¿ÿ^žxâ YøåSòîw¯äv°àx6¬¹ÊÚ²]£v²öªøL}lıZ+ÉWÕ’ãº$¯’ žž>ÉWÑÄâ½7—9 o¼U³h:Èi«ä«œþg·ÉWùŽÖ¼OS¾J•°£üä“Oä¹çž“ä{ùêûÏäë¿”?|éü7’$°Š¬"k¬ÚH­²¶¬¹ê:Òp•Æ¾æèMµöÚkË®»þow¬$í¦ï询cK˵ _ÕÆÏÆ«ÉWáGõ¢Ó.w=säßÃo @€@ìvbGžL‡;Ép§W@ 2;•ñâl@ 9ºÌ]Ñ£G'g=C€~®G¬3ÞW‘u™Ž.×á°Sïì‹)AÀF¼(ÙU|‚€¢íŒ+^AÈ"fìd1jØ @€ („DñÓ9 ¸ ì0 @€ P!„ q: (Vp$Š?òÎv"GL€ @¶`é¨mÅØM€š«öÇ—;vÇï @€ @ Çvì>3vìŽoÁ;¦Þå$и Œ0`€|ÿý÷2f̘Œ{‚ù€ @ =vÒ‹(,A؉‚j ÛäFNaP0 ¨G`ï½÷–•+WÊìÙ³¡@€ >ô‡2¥Í ì¤40a›…°6QÚƒ¢ €°UÚ„ @°™ÂŽÍÑuù†°““@ã&2Na'ãÄ|äˆÅ“sl\… r;)PXæ1õ.,’´DI€;QÒ¥m@ Llw&MÚ‚ Z ìÔBk!@€rIa'—aÇi@©$€°“Ê°` @€@š ì¤9:Øx °‚Ãî1°cw|ñ€ @ Ô؉*MB‘ æjdhSÑ0ÂN*€€ @€ h ìDÃ5-­"ì¤%ÛÁÔ»ˆÓ<  SN9EV®\)cÆŒ ¥= @Aر{ ìØß‚wÜÈ9 4nB ã:uê$ß~û­<òÈ#÷ó!@€@zð¡?=±ˆÂ„(¨¦°M„“ zv€ @¨ŒÂNe¼2{6ÂNfC‡áÈ„\…g!iOÎtø0€€Uv¬ gqg˜z—“@ã&2N€; æC GØîòõ×_Ëœ9s @€ >ô‡2¥Í ì¤40a›…°6QÚƒ¢ €°UÚ„ @°™ÂŽÍÑuù†°““@ã&2Na'ãÄ|äˆÅ“sl\… r;)PXæ1õ.,’´DI€;QÒ¥m@ Llw&MÚ‚ Z ìÔBk!@€rIa'—aÇi@©$€°“Ê°` @€@š ì¤9:Øx °‚Ãî1°cw|ñ€ @ Ô؉*MB‘ æjdhSÑ0ÂN*€€ @€ h ìDÃ5-­"ì¤%ÛÁÔ»ˆÓ< @€ ”@ØIi`B2 a'$io†9íÂ>¤ƒÀÂ… åºë®“¡C‡J£Fj2ê¼óΓ½öÚKÚµkW´G}TŽ?þx™:uª´lÙ2PS¦L‘I“&‰þoÓ¦M]ã=IýìÞ½»Œ7®¨}ü±ôèÑCŽ8âçÃ:Âd–M´@€€ÝøÐow|vìŽoÁ;„œ7!P#cÞ}÷]5jTMÂNá¤ZSãvªµ¯Üua1.׿‡ @È„|ÄYvrhÜ„@ÂvŠ",Æ5†šË! P<¹F€\@¡@Ø eºbê]ºãƒupÐåIíÛ·/ühØ°a2pà@ç¿u¶Êœ9sdÝuו‘#GJëÖ­ ˘ôw={öô½ÎüÐ{Ž¹~óÍ7—þýûË„  ×Ï;·°L)HÛæB¯ý}úô‘ .¸@Î>ûlg¹ÕÌ™3úóC=T Pg)–…,XP°eòäÉ…åPAfì¬X±¢¤?¦~ýúÉøñãÅôåf]l)VÞsÔþ®]»–´‰»È¶;ÏV¼°€€ÍvlŽ.¾A™#`#ªqAëÔ¨¸c~ï ÔI2mÚ´‚@â½ÎˆBÞÚ4z 1¦^ßl’ m{A{g츅·`ä­±ã7ÓÇEæºrÂŽñ½E‹…%e¦ #¹Å£b¬ý„ ,¼ç˜¾N;í4GœbÆNænK †€/„ ¤…ÂNZ"€@î ›!¢¢„Š*h̘1Ù©ã.6ì H· ÒªU+ßBÀ^!Å+:iÛ¯8r1aGms×ïñ ;~¢‡…6ÝtÓ‚¸Uªxr1áÇ-b-[¶Ì)žlÄ73S̹Y³fu˜a¡×øe63­Ô÷K.¹$”:F¹¿a„ ì$º‡*"À ŽŠpeîd„Ì… ƒ![ ©Kã'Zèϼb2*µ«“Šƒ * 4³V¼ÂJ5mk£Å„#Îø )Þ]±Š-I+7c§ØŒ·ˆ¤ýû 0nf]ºt©#ìa¡³„ÊíòÅŒ[ï`üÊjìä-âø l æj¶ãWÎz„r„ø= ˜x—ùu[LØq×Öñ^g–¹ë¾˜Ú:^ÃOØ Ò¶·Ïj…o}wmš 3vÌìµÇ»³—WØ9ñÄeìرu¶Y/'ì”c¡ýú‰ln>;1ÝPt@€@Ž݃aÇîø¼cê]N›™&P‹°Sji’B)¶Œ¨ÜR¬r³cŠ¯FØ1œ½¢L¥K±¢œ±SŽs±Y=;™¾51€ y;™aIvìŽ/ mNâ‹›v(¶tÊ-’,Z´H¼âB±%\îŸ+!ÝiË]¸Xæ-Ö\¬Æθqã ;d¹…"ïÏM$ªvLMS(ÚÛV·nÝ×Øñ›5VR,ŠÕØqϺå–[¨±cÇ-‹€ ÌàCfBU•¡;UaËÞE(´Ù‹ç“€Whñ.-š>}z=aGIywbò^·dɧ¦ŒGÜâŒnõ]jÇ©rm7jÔ¨^°¼"•wÖ¹À-x˜;*^™]ºÜ»i™ÀÊÍ"ªdW,µÃ¢6b”adW,¿¥_åvÎ*g>G>^C€ TKa§Zr»a'cÃÜ\(V4X¡”Ü5tô\ï–èÞú5;wvjÌh­3KÆ}Ž©Ícúu×—ñ¶í0S Yû¹öÚkeøðáR®x²[È1mªè4k֬¶ìº3X¹%QÞvLM!S ÙøÙ¯_??~¼¨¸¥‡Ûçb3¨Êqöãån·ã\zœ‡@ P<9ƒAÃd@–@ر4°^·˜z—“@ã& R»Š…Ö A±=z´|ðÁ¢;ØÕz°Ýy­¹€Â"€°IÚ «Ù~Þ*‡q9  ºfÏž-½{÷–Áƒ×$ð ìä`Àà" Œ@ØÉH 0€â#`–‘õéÓ§Þ¶éñYAO€@ØŒ°Ó AY¹reMÂNØÑ¡=@ J¬àˆ’nòm#ì$,€ @ˆ€vLWµ<Ô؉!`t„F€š«¡¡LeC;© FA€ „MÀ+ì„!ð„m#íAˆ‚ÂNTÓÓ&ÂNzb©%L½‹/C!8ýôÓåË/¿tvªâ€ 6bÂOؤiH„´E$\{vÂå™ÚÖ4Ѐ @Á tíØAŽm¶~ð |Îì|ëí5]ÏÅ€ À‡þ0(¦· „ôÆ&TË4Ðz3wèÐ!Ôvi €@˜¦L™"_ýµuÔQa6K[€'N”Å‹—¤aêîôêÕKz·ÙU¾™=K¶h¿GÅ?{{‰,_¼Xv*FÇ€ P!„ eõt¦Þe5rØ |Øwß}eùòåòøãçËq¼…b!Pj)–[Ð2dˆ³úë·Ü,oÜ:U:ž}nÅö-šóˆè?„ŠÑq THa§B`Y=©wYvC _¨±“¯xã-â&à'ìø :Æ.„¸#D€ P „j¨q  @€@清R‚N˜ÂNË‘£ œt @ l;a¥=@€ Tp ;ZCÇ,¹*fl3vR¡yퟀ@XÁ‘õøúD؉5=A€ $H@_ltÖL9Aǘˆ°“`°è•5WCÅ™ºÆvR ‚ @ˆ‚€goÒ¤Ià¦v£âD@ åvR ÍCØ©`V.gê]V"…È73Ïûì3¹òÊ+ó ï!T@ØIE0ÂNSÜÂNŠƒ¦iÜÈaÒ¤-@ *ûï¿¿,[¶Lž|òɨº ]@ ìFʼn€@Ê ð¡?åªÑ<„får„¬D ;!o;ùŽ?ÞC mvÒì ?;9;9 4nB ãv2@̇€ev, (î@°”ÂŽ¥õºÅÔ»œ7!qÔØÉx1–@ر, ¸i—  IDAT@ÀR;–· @€j#€°S?®† x ìÄÙ^ @€2Fa'cÃ\@ (VpØ=8vìŽ/ÞA€ TIa§Jp\¤Ž5WS’P BØ 'A€ ØBaÇ–Hâ €°c÷@ر;¾ï˜z—“@ã&2NଳΒåË—Ë•W^™qO0°ÂŽ QÄ@@ ìØ=vìŽoÁ;näœ7!qp€|øá‡òÔSOeÜ̇l €°cCñP|è·{ ìØ_„œÄ7!` „["‰°ƒÂŽqÄ @¶@ر=Â?ûÇŒœ7!q; æCÀ2;–w XJaÇÒÀzÝbê]N›È8jìd<€˜Ë ìXPÜ `)„K‹[€ @µ@Ø©WC€@<vâáL/€ @#€°“±€a. P”+8ì;vÇï @€ª$€°S%8.ƒRG€š«© I¨!ì„Š“Æ @€l!€°cK$ñ@ر{ ìØß‚wL½ËI q'pÎ9çȲeËd„ ÷ó! ìØE|€”ÂŽÝãaÇîø¼ãFÎI q'pàÊÒ¥KeÞ¼y÷ó! ìØE|€”úí;vÇa''ñÅMØBaÇ–Hâì €°cGñ€€ívlðÏþ1c''ÆMdœÂNƈù°ŒÂŽeÅ@–@ر4°^·˜z—“@ã&2N€; æCÀ2;–w XJaÇÒÀâ @€@mvjãÇÕ€ „x8Ó  @€@Æ ìd,`˜ %À »ÂŽÝñÅ;@€ * ìT ŽË Ô æjêBªA;¡â¤1@€ [ ìØIü€vì;vÇ·àSïrhÜ„@Æ hñä>úH®¾úêŒ{‚ù€€ vlˆ">@JaÇîq€°cw| Þq#ç$и Œ8è ƒäwÞ‘ùóçgÜ̇l €°cCñP|è·{ ìØ_„œÄ7!` „["‰°ƒÂŽqÄ @¶@ر=Â?ûÇŒœ7!q; æCÀ2;–w XJaÇÒÀzÝbê]N›È8jìd<€˜Ë ìXPÜ `)„K‹[€ @µ@Ø©WC€@<vâáL/€ @#€°“±€a. P”+8ì;vÇï @€ª$UagÊ”)Ò³gOéÓ§Œ5J5jTs½¹hòäÉÒ£GŠÚˆóä?þرoæÌ™2wî\i×®]EÝ«¿o¾ù¦ 8йî¼óΓAƒÉ°aà ?«¨Á\ëX³ •ŽiÓ¦ÉÔ©S¥Y³fθlÑ¢EU÷aØl¨¹6Ñtµ‡°“®x`  @€@JäQØY¸p¡tïÞ],XPˆB5bIœ!¬EØñqv‚Gaç¬Ì½Ó¶mÛ‚cÆRÄQ„àã:‹g"ìd1jUØÌÔ»* q  ;sÏ=W>üðC¹úê«cï›!x dUØ©%’æåTÛÐY-[¶¬¥¹X® [؉ÅhK:AØù_ ýDœG}TÚ·o_õì¹0‡ ÂN˜4Ó×ÂNúb‰EÜÈ‘`¥Q@ d|°,Y²Dž~úé[¦9@•Ȫ°ã}Ùþꫯê,Uš5k–³ÔHÖ­[ï,ý½{9W©%ZnAhèС2xð`gÖOÐåLn›wÙe§_=Ìl¡R}vÜ?7Ñ7ö¬X±Bú÷ï/&L¨30´?ÃGÏ=õÔS çyg]øÍì1/òÞþ*}~±ðÎœòúàŽ¥»?c§ùYçÎEÛoÚ´©¸ÝqÇrå•W:KÚLÜÝcÇ\ïæàŽ›~œ9úè£ëõ{·­~3ÆÜí¸í½ÿþûeO&†~K½ýùo¬Ë±6m¼ÿþûuÑZDÇJÇL¹óùÐ_ŽP¶°“íø¶a'0*N„$€°“ |º†ê°QØñ ³yÑŸ1c†S›Ç}˜—åK.¹¤ ¹o^ºý^ÈÝÂL¹áå'd»Æ_²o¿—g¿—u·Øâlܶy…­»ã7+Åof“ŸÚv¥µŽŠµã%Jù瀼¢ŽWÜÑÿ6õ‰ÜTØhÓ¦M½ey^qÇØúÇ?þQ>øàƒ:KøôÜJ– CîvJù­ç¹E« kíÓ°6‚ž»µÇ-U¢ÜýÂï!à%€°““1°““@ã&2Na'ãÄ|XFÀFaÇýâéž]b^Pý ¿—Vï  ½©ÍS©¡×º_ÂÝ/ÀAú6EjÝÅ“M{îÙFäpÛ¤ÆN)Ç´¥³MÕ=Ì6÷K}%uŠ¼6¹Û1þ”š-äµi£6*ÌÐ1Ü)\ªn‹[lòÖ‚ñvÜBI¥3v¼3s‚Œ™rl¼¢ŠWŒ)UÌz¯½örv£ÒÃœç]ŠUéŒR¬K-· ƒE9VA~Ï Ž ”²{ÂNvc‡å€ @@ØùIØ R‹GÃà­1SIhŠ½üéÛ[cÇ=üŒ»E‚jfì¨/Þ n!Ä-øõtyš_=íÛ[ÇÌñ«ã-xíço9aÇí¯_-sý¼yó|kËT#fø-ërÇÍ;[©Tµ_ +¦\ªÆN5¬³Pc‡š«•d¤ì‹°“½˜a1 @€@ v~vJÕña1Ë“* Q1! Hß~ˆJ-Ãñ:Œ­*¼ù曾õRÜâ_ý—b}VRgÆ-âøñ+· ™^ãã×Na§Ôr6cG-3­¼v•ZöäW<ÙÏ/oáè øCo±p¿óªaæ]±v*ÉHÙ;a'{1«Êb¦ÞU…‹ ˜ 4H–.]êL3ç€ 4„Ÿ„sx… wݘ¨–béÛOØñ Bj롇ZoV‘WL(%ì›M㧥 5W2žýDožbEݵbÜöø-ãò«Oä¶ÓOXs/I SØÑ~ýús‹%ÞXÏš5Ëáôð hîÙ^¥já”óQÛ.ÇÚ»Kœ¹oŠý¼’±Ö¹;a‘Lg;;éŒKèVq#‡Ž”!èN#ú¥ôÙgŸ uš„ P¬ ;•yÉÙÈ E“ðȽsšnµ®‡èÚ¶m+£F’F6môèÑrðÁK‹-_SêD>ô‡‚1µ ì¤64ᆰ.OZƒ¢!€° WZ…ª#€°S7sU¹­¢Íy~Ëšjë9}Wç™E¹Í4Z~[´ûE1ÍÂŽŸˆcf4•*¼\l´ª3{ölg¶Áƒ‡&ð¤ïîÀ¢0 ì„A1m ìd H˜ÂƒH„Ú¢‘g1ÃK.Ï,ò"ìhÌÕ×iÓ¦9u¦LQoý¹ {{‰,_¼Xv*F—š vRŠh aê]´|i‡€O~÷Ýwåºë® §AZ P°…}〠„@©¥XnAgÈ!ÎVèaä+„ ‘Iç9;éŒKèV±ÝyèHiˆ€@·nÝäõ×_— DÐ:MB¨Œ@/J ?©Ð)ÂNeü9y&à'ìø :†Qù a'»#a'»±«Èr„Špq2 „„ÀÓ- àK Œ%„ P ·°SJÐ SØi9rTÁTÄ‘;Ù‰UM–"ìÔ„‹!˜ ìÄšn @vaâ$@ naGkè˜%Wź"_E„ 5‰°“¡`Õb*5vj¡Çµ€@\¨±iú‚àE)%΢  ïo:k¦œ cú&_E…ì´‰°“Xa) @€@ŒxQŠ6]Au,_¾\š4i˜ ù*0*+ODر2¬8@€ P+^”j%Èõ€@\ÈWq‘Ng?;éŒ VA€ $L€¥„@÷€@`ä«À¨¬<aÇÊ°â @€@­xQª• ×Cq _ÅE:ý ì¤3.¡[EñäÐ‘Ò  ÁƒË;ï¼#×]w]­Ó$ Êð¢T/Ά’#@¾JŽ}zFØICb°íÎc€L€@Í9äyõÕWå¹çž«¹-€ P+^”j%Èõ€@\ÈWq‘Ng?;éŒKèV!ì„Ž”! ìD•&!ª ð¢T5:.„b&@¾ŠxʺCØIY@¢2a'*²´ „Ia'Lš´ÔJ€¥Z r= òU\¤ÓÙÂN:ãºUÔØ ) B ÆNPi¨š/JU£ãB@ f䫘§¬;„”s @€ÒA€¥tÄ+ òÈWåÙ|ÂŽÍÑÅ7@€ ª ð¢T5:.„b&@¾ŠxʺCØIY@0€ @ xQJG°(O€|Už‘Íg ìØ]|ƒ @¨š/JU£ãB@ f䫘§¬;„”$*s(žYÚ…Â$0tèPY´h‘üóŸÿ ³YÚ‚ P^”ªÂÆE€@ÈW @OQ—;) F”¦°Ýy”ti‹À¡‡*/½ô’¼ð a5I;€ª&À‹RÕ踈™ù*fà)ëa'e‰Ê„¨ÈÒ. &„0iÒ P+^”j%Èõ€@\ÈWq‘Ng?;éŒKèV!ì„Ž”! ìD•&!ª ð¢T5:.„b&@¾ŠxʺCØIY@¢2‡;Q‘¥]@ LÔØ “&mAµàE©V‚\ÄE€|étöƒ°“θ` @€@ÂxQJ8t&@¾ ŒÊÊv¬ +NA€ ÔJ€¥Z r= òU\¤ÓÙÂN:ã‚U€ @ àE)áÐ= ˜ù*0*+ODر2¬8@€ P+^”j%Èõ€@\ÈWq‘Ng?;éŒKèVQ<9t¤4D@@‹'¿õÖ[2qâÄZ§I@•àE©2^œ $G€|•û4ôŒ°“†(Ä`ÛÇ™. š tïÞ]^xáyñÅkn‹ ÔJ€¥Z r= òU\¤ÓÙÂN:ãºU;¡#¥A@ ;@¥I@ j¼(UŽ !˜ ¯bž²îvR¨ÌA؉Š,íBa@Ø “&mAµàE©V‚\ÄE€|étöƒ°“θ„n5vBGJƒ€@¨±Tš„ª&À‹RÕ踈™ù*fà)ëa'eÁ@€ tàE)qÀ @ <òUyF6Ÿ°cstñ € @ j¼(UŽ !˜ ¯bž²îvRÌ @H^”Ò¬€Ê _•gdó;6Gß @€ª&À‹RÕ踈™ù*fà)ëa'e‰ÊŠ'GE–v!0 6L^ýu¹á†Âl–¶ TE€¥ª°q òUÐSÔ%ÂNŠ‚¥)lw%]Ú†Â"pØa‡É³Ï>+/¿ürXMÒ ª ð¢TÝ£>*³fÍ’VÍ8© §L™"“&MýߦM›2cÅŠ2xð`9ꨣ¤eË–òñÇK=äˆ#Žpþ7¯ÒyòU:â”;I‘¹_„˜Ó P„ª°q ^”ŠƒU‘£ÿþ²é¦›æFØQ!ëøã—©S§:ÂÇOvÒ1ÈWéˆCRV ì$E>æ~vbNw€@UvªÂÆE€@DxQBØq@Øñ;%  ›%_U̲Óv, h1w¨±““@ã&2N€; æCÀ2¶¿(™Y7:ûdæ̙ο>}úȨQ£œHꌜ &¢:wî\i×®]a††ž¯GëÖ­Y,óæÍ«·¼É¼ôëuºdKÅ‘óÎ;O:wî, p®ðÁå–[nqfÿlµÕVÒ³gÏ:í›cÚîÚµ«Œ?^,X úwDû1¿óÚhÚò[Š¥?3}»ýÒk¼¿Ó~úõëWXŠÕ¦MéÞ½»œvÚiu–eyý×výÚ*·œM™ 4¨ §F~¤ÝR>V2LÌÝKÒ4O?ýtaÌhŒË-u+å×Â… ¦C‡•+¯¼ÒŸz˜»ÓM)¿ÌyÞ1áµÏøï7æ³ÚlÏWYˆA’6"ì$IŸ¾!@€RKÀö%÷‹¬m4æ¸E‹ŽÈ£â 2íÛ·sžßR,?±ÄOØÑvÜ„ێɓ';ˆùÙ¢E‹ŠŠ¦í÷ß¿Îò(#tëÖ­°LLm9rdá<¯­~¶«è ~qÂ;cÇ=SEE Âô0Ìôÿ{¯Ñ6§M›V°ÃOøñÞ^;üØi·œ7.ˆyåƃ›g³fÍœ˜©ðâ^nßÊùe⨂WT4B¡ʼõ’¼mû ÷9ÚŽúPj̧6Qýl˜íù*íü“¶a'éÐ? @€@* Øþ¢dïË·W1Áq¿À]c§aÇ-³£ÜÒ§b¢ˆ÷¥^í÷ö1}úôÂì"óRï-‚lÄ€qãÆ93•J ;* øÙ«¶¼ûسdÉßY=^ÑÌ}3©edìôÎr·ÛªU+ßBÏnwÙe_qÊ/®Æ®=öØCºtéâ´í[Š .Æ·Zý2µŽŒ¨T.vî8˜YNfüèl)ÝŸߘZd;©df{¾JŠkVúEØÉJ¤°€ @ V¶¿(ù½\Y¼[¸Ø|óÍëO*ìx {É*ì¸_êKÍ€qÛ7cÆŒ¢»by—ªœ°ã­5ãýïb‚Y¹5f™‘{ù“ûF¨¦]?°ã-ˆí'Š¸û/f1»Ìµåüò kæºbB–þ¾¿¼"“WàôŽù´̶=_Åšü3ØÂNƒ†É€ @Ñ°ýE©”°ã®3â&mÄ…¸„÷ò/oÄýoï5¦®ŠWØq×h1>êµZã%èŒ#,˜:ZoÆ-byëÀxm3ËÐüF¶{Y’þÞ-òm·œ~ÂN)¡ÏØY­°£×—òKwâ‰'ÊرcëìBæ'˜yë2¹cgf+ygy…rca'úœKÕ@Ø©ž]¦®¤xr¦Â…±È-áÇ˫¯¾*7ÜpCnà8 yvÊmc^KJfìT+ìx—æxG•{öβeË|—HUºKûpÏêÑ‚Ðz˜ÂÈ~3šªíÞÚB~E«½í›åâ·Ëû¨…·^¿¼Âš9×íK±¢Õn¿‚ ;åÆ|5±ŠóÛóUœ,³ØÂN£V…Ílw^4.b'pøá‡;;j¼òÊ+±÷M‡€¼lQ*¶ª˜áþyÐ;Þ¢µ~Ë«J-ŪTØÑ[:äþ¹»ÆŽþÍñëÇÌp ºKûöî,¥¢ŽÎÑ£ÜÒ"33(È讟£ufÜ3‹¼ˆ¶«G9‹-Å*6 ÏsÏ=WŽ>úhñŠiå–bùùéç—_í #ª(WÎ/åï7&ÜãNwcó`VûÂã‚Ä´ÖslÏWµò±ýz„Û#ü³;9 4nB ãv2@̇€elQ*&¨øíŠå7ãÃû²ì=ǽە٢:aÇo$ob¿;î]´üvdò 3¥– éÒ ¿í¾½»W•›Sì÷ÞÑåÚ5…›KùXLØ)·KZ±ÂÌ¥„ ~™™TšV¦Nê,ÇòÆÖ/Ö¥bWl§4S„Ù½+V©Z>iLu¶ç«42O“M;iŠF„¶ ìD—¦!Ð 섆’† $õ¢týõ×K‡œ­—£ø@Z´hQ˜±£ÿߺ½¹žã>z÷îíœÏ9âlÿŸÿÆ÷…Žï}ñÉ‹/H›ÿ~!ÝÏ¿ âdˆ°S1²Ô]€°“ºDcÂN4\i—ÂN¸cÆ ¹ùæ›Eëìè—ðÁƒ³¥ymááj@ÀE Ì|¥Í2c'[à a'[ñªÚZjìTŽ ! Pc'FØt”%Å‹’Î<Ñ;ÂNöcÈŠ'ÂÄI€@ÂÎ?ÿ|y饗dÊ”) [B÷€D’và@ V;µÌÞõ;Ù‹YU³ÝyUظˆ™ÀG!O<ñ„¼öÚk1÷Lw€ê@ØaT@¶àC¿-‘ô÷aÇîø¼CØÉI q'€°“ñb>,#€°cY@q€€¥v, ¬×-„œ7!q; æCÀ2;–w XJaÇÒÀzÝbê]N›È8jìd<€˜Ë ìXPÜ `)„K‹[€ @µ@Ø©WC€@<vâáL/€ @#€°“±€a. P”+8ì;vÇï @€ª$€°S%8.ƒRG€š«© I¨!ì„Š“Æ @€l!€°cK$ñ@ر{ ìØß‚wL½ËI q'pá…Ê /¼ S¦Lɸ'˜Ø@adž(â  vì;vÇ·à7rN›È8#ôÛ=vìŽ/ÂNNâ‹›°…ÂŽ-‘ÄØAaÇŽ8â Û ìØáŸýcÆNN›È8„Œó!`„ËŠ;€,%€°ci`½n1õ.'ÆMdœ5v2@̇€ev, (î@°”ÂŽ¥Å-@€ Ú ìÔÆ«!@ ;ñp¦@€ Œ@ØÉXÀ0(J€v„»ã‹w€ @U@Ø©—A©#@ÍÕÔ…$TƒvBÅIc€ @¶@ر%’ø ìØ=vìŽoÁ;¦Þå$и ŒÐâÉÏ?ÿ¼Üxã÷ó! ìØE|€”ÂŽÝãaÇîø¼ãFÎI q'ЫW/™;w®¼ñÆ÷ó! ìØE|€”úí;vÇa''ñÅMØBaÇ–Hâì €°cGñ€€ívlðÏþ1c''ÆMdœÂNƈù°ŒÂŽeÅ@–@ر4°^·˜z—“@ã&2N€; æCÀ2;–w XJaÇÒÀâ @€@mvjãÇÕ€ „x8Ó  @€@Æ ìd,`˜ %À »ÂŽÝñÅ;@€ * ìT ŽË Ô æjêBªA;¡â¤1@€ [ ìØIü€vì;vÇ·àSïrhÜ„@Æ \tÑEòì³ÏÊÍ7ßœqO0°ÂŽ QÄ@@ ìØ=vìŽoÁ;näœ7!qýë_åᇖ·Þz+ãž`> `„¢ˆ€€àC¿ÝãaÇîø"ìä$¾¸ [ ìØIü€€vìˆ#^@°ÂŽíþÙ?fìä$и Œ@ØÉx1–@ر, ¸@ÀR;–ÖëSïrhÜ„@Æ Pc'ãÄ|XFaDz€â K ìXXÜ‚ @¨ÂNmü¸€â!€°gz @È„Œ s!¢XÁa÷à@ر;¾x@€ P%„*Áq :Ô\M]HB5a'Tœ4@€ ` „["‰€ÂŽÝcaÇîø¼cê]N›È8#FÈÓO?-7ß|sÆ=Á|@À;6D %€°c÷8@ر;¾︑shÜ„@Æ üío“Y³fÉ¢E‹2î æC6@ر!Šáû0eÊéÙ³§ôéÓGF%5 ¿“*[<ï¼ódРA2lØ08p`E­<úè£2iÒ¤‚OQû¹bÅ éß¿¿ó7_ûZ¶l™tïÞ]ºuëV±í9šÓ“ùÐowàvìŽ/ÂNNâ‹›°…ÂŽ-‘ÄØAaÇŽ8†íEÔ‚G-öV+ìøùµŸ*$µoß¾ B¡g„ 2wî\i×®]-(¸¹"€°““p3c''ÆMdœÂNƈù°ŒÂŽe ɨZÌ SØ©ÅŽr×q Ûjf•ë“ßCÀf;6G×åSïrhÜ„@Æ Pc'ãÄ|XFa'Þ€š—zw¯~37ŒxaÎëܹ³³”§iÓ¦²páBg9ãÆ=wæÌ™ÎOž=Lÿf–‹iÏ+€vJ1óþΰÒÿu/9»ä’K|—yùñ(ÇÂÄi£6*ÄNûsÇoêÔ©Ò²eË è8¹&€°“ëðã< @€@1;ñ ?QG{oݺµ¸_ðýD=ψ(¦NË‚ ê¯m5oÞ\î¿ÿþ:¿3‚O1ÌÉÞóÜ¢$¼ýzí/GÔÏãÛŒ3¡Å{¸íðvÊ1?~|Aèrûêvts]:å¬ÜŽŸ‚°"Ž±«Ühá÷ø„F @€| ìÄ7,¼‚„{©Ž™•â7ËÃÌ`1ŠZ¬3vT`1×¹Å#J4nÜØ)Ü«õ\Œ02}úô‚pbD·^Å\§}š¶Ìuú³j–¹…w[~3Yü–3y9a¦³b‚ÔØùꫯ 3’¼3ˆü¸–bQlf‘Û'÷õñD{{b‡½±UÏvìŽ/ÞA€ TIa§JpU\æ©R®ÆŠwVˆŸ°cÄ?‘È-¼x…ïnW^QeÞ¼yu–(¹?×+Y’e8x¯ñ.Áòöcx•ª±SŒYPaGwÿòŠUÞþ¼K°¼vª_×^{­ >ÜÕüÄ›jëU1ìru 5Wí7ÂŽÝñÅ;@€ * ìT ®ÊËŠ-ò›=ãíÂ+ìèïÍ®b³@¼³TÌŒJ…%K–f E%ì”[&VLØqû^ŒY%ÂŽ{ÐرcåÄOtj •[†eúFØ©òæá2„ ¦¸ „'LÓ˜z&MÚ‚¢" Å“çÏŸï<Œs@HšÂNrpÏPñ«1ã] ¦°ã-SÉŒZë«=cx”›ýãíâ„Š1«DØñÎ~Ò"Ôn!̯æŽw•[nÅŒhî;„h¸¦¥U„´D"b;¸‘#Ló€@(Ž:ê(yàdñâÅ¡´G#€j!€°S ½à×[*å­ŸsË-·8E~ýŠ‡)ì¨åÕÖØ)Wȸ•bÂŽ{•±ÍODñŠ"æ¿K1«DØQû½³‡Ü˩ܱ,Å‚;åFBø¿çCøLÓÔ"ÂNš¢¡-;Â¥i@ 4;¡¡¤!@ ;!@ ØD©2f™Q©sÂvüÌ63^üÄ—b¶U»+–w9˜ÚSl©Z)%3·°cüVæ[mµUZBZcG· äç_ìŠðÆà4$€°TÖOCØÉz±ù €°“8ã%²Ba'ÞHùm“í-¢ìž-âW¯¥M›6N½=j©±Ó·o_éÝ»·³»–î%VAfÕè5•Š:zM±¶M$¼³e¼ßL˜rÌzôè!ÞZ<¥„·Èä'@éïKjvÿ~£6r|nÚ´©ã¢ß.^ñŽBzƒ@6 ìd3n[ÍÔ»Š‘q jì$.!¢vò58ʉ*ù¢­·~[µkf¶O¹]Ñ¢µ.­=Z>ø`iÑ¢E: ŠT@ØIux0€ @ );I‘O¦ß¨„R;S¹=-W9*Ñõê'â˜Gµ¡ŽÎêøZÖó³gÏvfŽ <'>ô™ì a'“aÃh@€ ¨ ìDM8]í#ìÄ#x-Z´ÈYŽµlÙ2g]Û¶meÔ¨QbjúÄkUzz3ÂNƒ dåÊ•5 <¬àHOl£°a' ª´ @€ y;™!@ ³Œ°c¨Uà¡æjf‡B Ãvaâ$@€ ¼@ØÉ[Äñé!àvjxvÒÛ(,A؉‚j Ûdê] ƒ‚I€@=#GŽ”§žzJn½õVè@HœÂNâ!Àä–@1a§ZaÇî¡„°cw| Þq#ç$и Œ8æ˜cdÆŒòöÛogÜ̇l †°Óå¶;l@€@Ê tíØAŽm¶~MVv¾õöš®çâä ì$Ç>ÖžvbÅMg€@•vªÇe€@$ÂvnZk½‚ml[I˜hV˜8q¢,^¼¸¤o¦îN¯^½¤w›]å›Ù³d‹ö{TÌã³·—ÈòÅ‹a§bt©¹a'5¡ˆÖ„hùÒ: „p8Ò  0„†ÇŸT0F—Vp@B ÔR,· 3dÈg+ô0òÂNȤó„tÆ%t«¨±:R„" @ Ò$ P50^”vªÆÏ…È5?aÇOÐ1ÂÈW;Ùr;Ù–C€ DH Œ%„DÓ°˜€[Ø)%è„)ì´9ª@”¥£Ù\;ÙŠÖB€ ÄDa'&ÐtÔ#àv´†ŽYrU ù*߃a'ßñÇ{@€ "xQbh@IPaGgÍ”tŒ}䫤"•Ž~vÒ¬€ @H^”RÌ@Ž,_¾\š4iØcòU`TVžˆ°ceXë;Eñäœ7!q—^z©<ñÄrë­·fÜ̇l À‹’ QÄäƒù*q.æ%ÂNNâÏvç9 4nB ãŽ=öX¹÷Þ{åwÞɸ'˜Ø@€%¢ˆÈòU>⌰“ï8 ÂNÎîC #v2(Ì„@N𢔓@ã&, @¾² ˆ5¸ÀŒàeéR„,E [!_;ù=žC xQJcT° ð#@¾Ê÷¸@ØÉIü©±““@ã&2N€; æCÀ2¼(YPÜ€ÅÈW7€k; q  @€@þ𢔿˜ã1²J€|•ÕÈ…c7ÂN8i€ @À2¼(YPÜ€ÅÈW7€k; q  @€@þ𢔿˜ã1²J€|•ÕÈ…c7ÂN8i€ @À2¼(YPÜ€ÅÈW7€k; Ùp Å“mˆ">@À~—]v™<öØcrÛm·Ùï,B©'À‹RêC„€ÀÏÈWù ;9‰?Ûç$и ŒèÛ·¯Üu×]²téÒŒ{‚ù€€ xQ²!Šø| _å#ÎżDØÉIüvrhÜ„@Æ ìd<€˜Ëð¢dY@q _YÜ®!ì€dÃ);6D `?„ûcŒ‡È^”²-l…@¾ ¯ò„œÄŸ;9 4nB 㨱“ñb>,#À‹’eÅXL€|eqp¸†°§@€ ä/Jù‹9C «ÈWY\8v#ì„ÑV @€,#À‹’eÅXL€|eqp¸†°§@€ ä/Jù‹9C «ÈWY\8v#ì„ÑV @€,#À‹’eÅXL€|eqp¸†°’ §P<Ù†(âì' Å“çÎ+Ó§O·ßY<„RO€¥Ô‡!Ÿ ¯ò=vr¶;ÏI q'pÜqÇÉwÜ!ï½÷^Æ=Á|@À¼(ÙE|€@>¯òçb^"ìä$þ;9 4nB ãv2@̇€exQ², ¸‹ ¯,n×v@²á„¢ˆ°ŸÂŽý1ÆCd‰/JYŠ¶BàÿÙ»p­Ê:ßÿ·hÀhz tÂ`çdx¦éà0'ðrìtÂs~¿3Ó‘‚®$þG …bÆLØFSx4AI,tÀ’²‰ âOÂ53æøJKr¦ÙPt¬@±ø3ù§}®{Ñý¸öÚëyžµžõç{ÿyïëò*a­ûÏë{?O­Ï^ë^a ð}vý v©?{ìRh¦‰€ãì±ãx>ž p¡äYA™ ð}åqq3L`'‡ € €„'À…Rx5gƸ*À÷•«•+gÜ;å8Ò  € €€g\(yVP¦ƒ€Ç|_y\Ü S#ØÉ€Ä! € €á p¡^Í™1® ð}åjåÊ7ÁN9Ž´‚ € à™Jž”é à±ßW7ÃÔv2 ùp›'ûPE怀ÿË—/WßúÖ·Ô¦M›üŸ,3Dë¸P²¾D ~#À÷UØK`'úóºó@ Í4p\àúë¯Bçž{Îñ™0|ðA€ %ªÈC€ï«0êÜl–;ÔŸ`'B3M Øq¼€ ϸPò¬ Lø¾ò¸¸¦F°“ɇCv|¨"s@À‚ÿkÌ pI€ %—ªÅX[€ï«°ëO°HýÙc'B3M`Ç ÈððL€ %÷ zâÄ ÕÝÝ­®¹æ5jÔ(÷&àÀˆ1¶³H|_ÙY—ºFE°S—4ý € €8%À…’S劻gÏ5kÖ,µqãF‚ŠÊ‡qE°›åûª  ã§ì8^@† € PJÕ¸VÙ*¡C•º'ÛƸzãNzàûª5Î!Øñ§–Ì@@ D.”JÄÌÑ”/^¬&Nœ¨æι{÷n5~üø(T˜0aB£µ™3gªeË–©Aƒ©|PM›6­ñw‹-RzﶩS§ªéÓ§Gÿi~ô±K—.îì1b„š3gNt‡ÏÖ­[£t»“'OŽú_±bE4ýçúG·;þü¦3ÒdžÆÖ "¯±±Hó7§ßJ3Öµ»ãŽ;Ô‚ RkšcÙ(ßWa/‚°ëÏì@@šp¡$³4L°môHLp`‚è¿3áNòn’ÇgvV­ZÕt›f:`Ò}:´ñgñ&©¤CT$C'$éÉ„::4ÒU¼¿dˆ•æ°nݺƘô¹ºO=^3N3B+=ÇVãêÄØôÿÈ#4Â+ã®ç¦Ãxÿñ )9nsÜðáÃ[†g2+Óî^ù¾²»>UŽ`§jaKÚgódK Á0@ ¥ÀwÞ©vîÜ©¾üå/#…ˆ p¡$S¨Ä€dP`Ff’yóæEN‘`'¥-úϲÉ°"íž p¡$Sд`§Ù7É?ÏìèÅlökÞŠUf°ßÈX+ƃ ó¦®f{ÙLš4)ºÛ'Ï#iÉM;¹cÇ챓ܿ§qÚFÐzÎñ?7{ìdÙ¹Ùcw2«Ñ^ù¾r§VUŒ”`§ UÚD@p^€ %™¶{©Õ[•ÒB†äÞ6&80üTìè·bÅ_‹ž$õôô4ÞbkT«=vÌüLøPôkÓ‹ì±£ƒ¦d@¤ÛOú53Ž¿+ùÆ2ÝNr´·š¥õ'³ Ýë•ï+÷jVæˆ vÊÔ¤-@@o¤.”Ö®]«.¿ürÕÕÕåež‰´Ú4Ø„2¦½ä«ÀÓ^ëmBýªoý£ÏÑÁˆ~“ÖÆUÁŽ~ êÌ3ÏTúçú'þÊtýïñqš¹èPfûöí×–?óÌ3*íѨäž4ºí»îºKÍž=»ñF«NîØI¾v=ñ Aƒ¡9/l5{+Í!i•gí„|¬Ô÷UÈæ6Í`Ǧj0@@kê¾PÒÎÂ… •¾“£»»;úïü¸'Ðì­XîÍ„»$P÷÷•K6!Œ•`'„*3G@@Üu](Å}§Žtôþˆü¸)@°ãfÝ\u]ßW®;ù:~‚_+›˜›'Rh¦‰€ãzód}ú£>êøL>ø Põ…Ž«¤ÿvü¬«í³ªúûÊöù‡>>‚@V¯;¤ÐLÇ>ò‘¨ 6¨ŸÿüçŽÏ„á#€€U](èø°:˜v Tõ}e×,M3‚@ÖÁN …fš8.@°ãx>ž ”}¡´ÿþÆ:wîÜ<æg×®]ý6Kæ|ôú`môÝDœÏEëÏÅg.Ÿ f/[Þö»-y@ÏcßRúŸÓgÝØø+‚ÜŒ¢'ìˆòÓ9 € €€­e;;~wTt·ÎOúSÕÕÕÕ¸cGÿwó£_o®‰ÿ̘1#:žcTôúw|^[¬ >z5$?Ï?ý”{ì—jÊ'—äþJ%ØÉMfÝ ;Ö•„!€ €Ø PF°c~¾eË–hsx½ÏŽþMxww7¯4·¡ÈŒOÊü¾Ò$ܱãÖ Øq«^Œ@@ &*.”ô'zžšŠH7"`¾¯ŠL—G±ŠèÉžK°#ëOï € €– T옩ðXZt†…€£úQ,ýþ9räHôOügíڵꢋ.R£Gn:×\Úø;îØqk!ì¸U¯ŽGËæÉÓq"Ô(ðÙÏ~V}ãßP>úh½Ò .Pe°CÀêCªôþ;úŸøÏW\¡ôþLW_}u¦n v21YsÁŽ5¥¨v ¼î¼Z_ZGrfÏž­zè!uèСr¤@ €@ÁOq*¤ ¤;7Ýt“ºòÊ+£²üìdQ²ç‚{jQéHv*å¥q(I€`§$HšARê v’Þ‡Gÿv@ ¯@Z°“· ‚¼b²ÇìÈú×Ö;ÁNmÔt„v àq*”. ì”> Dàv‚+¹"Ø ¤æì±H¡™&Ž °ÇŽãdøx&@°ãYA™ à©ÁŽ§…eZ € €ÅvŠùq6 €@=;õ8Ó  € €€c;ŽŒá"€@Sžàð{qìø]_f‡ € СÁN‡pœ†Ö °çªu%)u@;¥rÒ € €€/;¾T’y €ÁŽßk€`Çïú6fÇ­wši"à¸ÀÝwß­¶mÛ¦¾úÕ¯:>†>ìøPEæ€Z€`Çïu@°ãw}³ãƒH¡™&Ž ÜtÓMêPÏ?ÿ¼ã3aø àƒÁŽUd  øE¿ßë€`ÇïúìR_¦‰€/;¾T’y à‡ÁŽud €€ï;¾W©e IDATø7óãŽ@ Í4p\€`Çñ2|< Øñ¬ LðT€`ÇÓÂ&§Å­wši"à¸{ì8^@†€g;ž”é €ž ìxZX¦… € PL€`§˜g#€Ô#@°S3½ € €8&@°ãXÁ.4à ¿ÁŽßõev € € ìtÇi `{®ZW’RD°S*'!€ €ø"@°ãK%™ìø½vü®ocvÜzH¡™&Ž èÍ“·lÙ¢¾öµ¯9>†>ìøPEæ€Z€`Çïu@°ãw}³ãƒH¡™&Ž Ì™3G­Y³F½ð ŽÏ„á#€€;>T‘9 €€àý~¯‚¿ëK°H}™&¾ìøRIæ€;~Ô‘Y €¾ ìø^áßÌ;v)4ÓDÀq‚Ç ÈððL€`dz‚2@ÀS‚O ›œ·ÞRh¦‰€ãì±ãx>ž ìxVP¦ƒx*@°ãia™ € €@1‚b~œ PÁN=Îô‚ € à˜ÁŽcc¸ ÐT€'8ü^;~×—Ù!€ €t(@°Ó!§!€€uì¹j]IJÁN©œ4† € à‹ÁŽ/•d @°ã÷ Øñ»¾Ùqë] …fš8.°råJµyóæè~@i‚é Ð?”%@°S–¤íìØY—ÒGŹtRD >úѪիW«_|±‚ÖiÈ'@°“Ï‹£@À^~ÑoomÊÁNŠ´A°ã@‘"(‚Ø$@°cS5  €@3‚@ÖÁN …fš8.@°ãx>ž ìxVP¦ƒx*@°ãia“ÓâÖ»@ Í4p\€=v/ ÃGÀ3‚Ï Êt@Ov<-,ÓB@(&@°S̳@ê Ø©Ç™^@@ Øq¬` š ð‡ß‹ƒ`Çïú2;@@v:„ã4°N€=W­+I©"Ø)•“Æ@@| Øñ¥’Ìvü^;~×·1;n½ ¤ÐLÇî¹çõµ¯}MmÞ¼Ùñ™0|ðA€`LJ*2Ð;~¯‚¿ëÛ˜ä@ Í4@@ 4‚Ò(i„øE¿p*îž`§b`[š'ر¥Œ@\ Øq¥RŒ[€`'úìRh¦‰ €¥ ì”FIC € ìTˆkSÓÜzgS5 ÕìÙ³Gmß¾]ÍŸ?¿š2´še ‡VS§NUÓ§Oþ3ËnwÖ¬YjãÆjÔ¨QYNéwLÖ~/^¬<¨–-[¦ ÔQ_É“Nœ8¡º»»Õ5×\ÓñøK €@.‚\\Œ $@°#O· €@™:8˜3gŽ>|¸X°Såê vʬ‹i«ŒñW1.ÚDÖ;¬@v\¨cDÚTªdůr e#YïØÉ:ß<Ç•1þ<ýq,”#@°SŽ#­ €€¼OpÈ× ÊìT©KÛ €@ŒèG¶nÝý3sæÌè‘ ý£ïÈYµjU£ÅÝ»w«ñãÇ+XèãõÏèÑ££G–žxâ µnÝ:õàƒª¡C‡FgŽÕçéG¶tà =š8q¢š;wntÌ7¿ùMõðÃGwÿœþùjÚ´i}ÚM{ªÕ{ì1uæ™gª¥K—Fc»÷Þ{ÕÂ… û<Š•<_w¸hÑ¢ÆÝGYƒ=W3Þd¦=Wã«1Æ汫´G±tÿ&LhØ'ÏÑ‘<ÆŒ¿Õ˜r,E‚tºDJØsµVk%ر¦ B0ÁŽoLhcºººû¾˜Á—v·Œ²;:°ˆñq¬_¿>ÚÇüYOOOŸ (^³fcÐAK<¤IÞ9“v';}ûÔ”)SÔ¼yó¢þ³;:yä‘Gûð˜vÛñãÇ£¶t¨“œ—ž‡ÙS'ì˜`&i?'yLZ€Vt Ð?Ì ‚ uúD*vªPµ§M‚{jQéH¸õ®R^G ŒÄýßuh ïvIn¬xè¿Ùród­X±"º‹ÉüÄ° /¼0 vÌÝJæ˜ä¹ñ~MÔì<]yå•©›A›»¡ôüžyæ™Â›?—²Èhr ìäââ`°X€`Çââ”04‚]h‚² UbŒ¡ ¤#ÍBmZFŒÑq°“¼“¤Ù^9;ÍîJ{+– YöîÝ-s'Q»`§ÙØâ!R³&9ßd ¤ïhŠ_z\ñs&OžÝ]” •âë¹]èkŸù#`«ÁŽ­•a\ W€_ôçsëx‚·êÕñh v:¦ãDjhìÄ÷Ö‰Èì§SW°“r˜ñä}Ì;ÉýuÌc[yîØivWSZ°£÷ŠßÕ“%Øi¶ôXßýîwGûï$ÂÚ>:t„@e;•ÑÒ0 €@‰;%bÚÜÁŽÍÕalœhì´{yÞP%¾yrž;vªvâ”™MžµGž`§Ê;vÚí“Üï(m=sÇŸrÜ Øq³nŒM€`'Šsë] …fšN 4{ªÙ4ñ?ϺǎyÔiÒ¤I·bI;W]uUô™þ1›ëÿžÜø¸Ý£XÉÍ–ÍB(käccYñŠïÝ£ÇÓ. rzñ2x< Øñ´°L ðL€`dz‚2pW Y°“ 9ôk¹Ó‚ŒäÛœ’ÇÄßvewJ»“¤Õ;í9J{£T»=v’o³ÒÔ¶`Á‚Ì{ì˜s²¾+ùªøøÇÚ½Ëôe6®Öwµ{sÖÚîÃãîÊeäø+@°ãom™ à“ÁŽOÕ¬`.úÕÆ;wîT3f̨ ušD¸@³@EeÌ9É=]ÌÝ)úïÍßÅÿLÿù¶mÛ¢W‚›G»Êv’cÐß!í‚xcæ¦_G®Ìµ:”鎰ÄÛÑoÂÒ?& ›8qbôÊsýþ‰¿ŠÝŒåàÁƒ}î J:Æ_oúJo7^¿´sù$ €€;vÖ…Q!€@~žàÈoæÒ;.U«Æ±ê‹±Ûn»M­Y³F½þõ¯WúßÏ:ë¬G@W €€Œ@òŽ™QÐ+Ø @°cC”!Àž«e(ÚÛÁŽ½µY2й馛”þ‡PG¤tŠ5 ´ºkªæ¡ÐX @°cA”"@°S £µìX[šrÖîÖ;r½i Ü0R™WÈ5ʽI0b(U€`§TNCA‚Aüº&ةن.š} tl¨c@@vl¬ cBNÚý¢¿“69Ç‚{jQéH’ÁN¥Ü4Ž |ãßP?þ¸ºå–[:lÓ@òvʳ¤%@êvª³µªeì\}õÕ}6Ef«ÊÄ`^à/þâ/ÔÝwß­Ž;¼ /@°#_F€ Ð^€`§½‘Gè[ï^}õÕèÈúç·û·ÕÑ£G•þó;vô™ãW\½âÜüp >f-°6ø\èµPåw‚Þ×æ?øß?;óYæ³\õgÙ—5vߟÍTÿúÅ…þ¿àé³nìóV¨1NF@ E€`' e¡¿Z¸p¡Z»v­4hºä’KÔäÉ“Õõ×_ßGA¿â\k~ºººÔŒ38æ7ø°6ôRàsÁw‚ùRä;ï¾üýÿ 4n¬zþ駢û‘#G¢ÌÏÖ­[£ÿ:qâÄ–ÿorÀ%—ìôÿ·™* !@°#¡.Üg<àѯ1×cÝxã¼Ò\¸.t €ö èÿÿÿÅל9s¢Á.[¶,ó õü €lž,¡^_Ÿ;õY[×u%a@ € `©@2ØÑûéŸn¸!óˆ v2Sq ”,ÀëÎKµ¬9‚Ë "1 uúD@—’ÁN'c'ØéDs@  ‚2ímƒ`ÇÞÚ”:²,·Þð”JNc ÐÀç>÷9µiÓ&µeË–Îæ@ :‚êliª Ø©ÞX²‚IýûÎóA&ੱ0t…}n¾ùfu×]w©ãÇ#ƒ € P’@–_ô—ÔÍì Kt™'Ø1ã3~õy|³@‰ñÓ'„!@°F™% € €@y;åYZÝR'ÁŽÕbp à¥ÁŽ—eeRx) ß*ª–/_îåü˜ €€;;îÔªÐH¹õ®'#€@Mì±S4Ý €@a+®¸"jcÇŽ…Û¢@ŠìÑã\@@ v‚,;“F¬ ر², @@Àf‚›«ÃØ@ )À~¯ ‚¿ëËì@@*` Pi*`ÏÕÊh­h˜`ÇŠ20@@@ª Ø©ÆÕ–V vl©DÅãàÖ»ŠiJЛ'éK_RÛ¶m+¥=A@PŠ`ÇïU@°ãw}³ãƒH¡™&Ž |üã^|âÄ ÇgÂð@@{øE¿=µ¨b$;U¨ZØ&ÁŽ…EaH ÐO€`‡E € €@>‚|^ÎM°ãlé8A ìUn&‹€Ólžìtù< à•ÁŽWål>n½ ¤ÐLÇØcÇñ2|àu盩"€– ìX^ †‡ € `ŸÁŽ}5aD €@¨;¡Vžy#€ €t,@°Ó1'"€€€Op ×Ø%ÁNØt… € à‡{ìøQGf@(ì¹êw¥ vü®/³C@@\€`Çï@°ãw}³ãÖ»@ Í4p\`õêÕjãÆêë_ÿºã3aø € €€=;öÔ¢Š‘ìT¡ja›|-, CB~ŸøÄ'Ô’%KToo/: € €% ð‹þ’ -m†`ÇÒ”=,‚²Eiª Ø©B•6@@| Øñ¹º±¹ìRh¦‰€ã;Žá#›'Tl¦ŠX.@°cyÊ·Þ•%I; P¥{ìT©KÛ P¦¯;/S“¶@ŠìÑã\@@ v‚,;“F¬ ر², @@Àf‚›«ÃØ@ )À~¯ ‚¿ëËì@@*` Pi*`ÏÕÊh­h˜`ÇŠ20@@@ª Ø©ÆÕ–V vl©DÅãàÖ»ŠiJ¸ÿþûÕ† Ô׿þõRÚ£@@¥vü^;~×·1;>Èši"à¸À­·Þª>ùÉOªÞÞ^ÇgÂð@@{øE¿=µ¨b$;U¨ZØ&ÁŽ…EaH ÐO€`‡E € €@>‚|^ÎM°ãlé8A ìUn&‹€Ólžìtù< à•ÁŽWål>n½ ¤ÐLÇØcÇñ2|àu盩"€– ìX^ †‡ € `ŸÁŽ}5aD €@¨;¡Vžy#€ €t,@°Ó1'"€€€Op ×Ø%ÁNØt… € à‡{ìøQGf@(ì¹êw¥ vü®/³C@@\€`Çï@°ãw}³ãÖ»@ Í4p\à _ø‚Z¿~½úæ7¿éøL> € `ÁŽ=µ¨b$;U¨ZØ&d ‹Â@ ŸÀ‚ ÔâÅ‹Uoo/: € €% ð‹þ’ -m†`ÇÒ”=,‚²Eiª Ø©B•6@@| Øñ¹º±¹ìRh¦‰€ã;Žá#›'Tl¦ŠX.@°cyÊ·Þ•%I; P¥{ìT©KÛ P¦¯;/S“¶@ŠìÑã\@@ v‚,;“F¬ ر², @@Àf‚›«ÃØ@ )À~¯ ‚¿ëËì@@*` Pi*`ÏÕÊh­h˜`ÇŠ20@@@ª Ø©ÆÕ–V vl©DÅãàÖ»ŠiJЛ'¯[·Nmß¾½”öh@@@)‚¿WÁŽßõmÌŽr …fš8.ÐÝÝ­-Z¤z{{Ÿ ÃG@ìàýöÔ¢Š‘ìT¡ja›;…!!€@?‚ € €ùvòy9{4ÁŽ³¥cà%@°T¹™,N °y²Óåcð €€W;^•³ùd¸õ.B3M`Ç ÈðH€×Tl¦ŠX.@°cy € €€};öÕ„!€¡ ì„Zyæ € бÁNÇtœˆ<Á!€^c—;5bÓ € €€ì±ãG™¡°çªß•&Øñ»¾Ì@@@ p‚¿ÁŽßõmÌŽ[ï)4ÓDÀq5kÖ¨x@mß¾Ýñ™0|@@À‚{jQÅHvªPµ°M>È…!!€@?… ªÛn»Mõöö¢ƒ € P’¿è/ ÒÒfv,-LÙÃ"Ø)[”ö@  ‚*Ti@@Àg‚Ÿ«›ÁN …fš8.@°ãx> °yr@Åfª €€å;–¨¬áqë]Y’´ƒU °ÇN•º´e ðºó25i @ ˆÁN=ÎE@R€`'Ȳ3i@ÀJ‚+Ë @@l ر¹:Œ ’<Áá÷š Øñ»¾Ì@@ öØ©•&@ 2ö\­ŒÖŠ† v¬(ƒ@@@@ ‚j\mi•`Ç–JT<n½«˜æ@ x@}þóŸW;wî,¥=A@PŠ`ÇïU@°ãw}³ãƒH¡™&Ž ,Z´Huww«ÞÞ^ÇgÂð@@{øE¿=µ¨b$;U¨ZØ&ÁŽ…EaH ÐO€`‡E € €@>‚|^ÎM°ãlé8A ìUn&‹€Ólžìtù< à•ÁŽWål>n½ ¤ÐLÇØcÇñ2|àu盩"€– ìX^ †‡ € `ŸÁŽ}5aD €@¨;¡Vžy#€ €t,@°Ó1'"€€€Op ×Ø%ÁNØt… € à‡{ìøQGf@(ì¹êw¥ vü®/³C@@\€`Çï@°ãw}³ãÖ»@ Í4p\€Í“/ ÃG@+v¬,Kiƒ"Ø)Òî†ø Û]F‡'/^¬,X z{{!A@(I€_ô—ii3;–¦ìaì”-J{ P…ÁNª´‰ € à³ÁŽÏÕÍ`'B3M Øq¼€ €Ø<9 b3U@Àr‚Ë TÖð¸õ®,IÚA*Øc§J]ÚF2xÝy™š´… PD€`§ˆç"€ €)@°dÙ™4 `¥ÁŽ•eaP € €6 ìØ\ƆIžàð{Mìø]_f‡ € P{ìT€J“ P™{®VFkEÃ;V”A € € € PÁN5®¶´J°cK%*·ÞU Ló PŠÀºuëÔý÷߯vîÜYJ{4‚ €  ÁŽß«€`Çïú6fÇ9B3M¸ýöÛÕüùóUoo¯ã3aø € €€=ü¢ßžZT1‚*T-l“`Ç¢0$è'@°Ã¢@@@ ŸÁN>/g&Øq¶t  v‚*7“EÀi6Ovº| ðJ€`Ç«r6Ÿ ·ÞRh¦‰€ãì±ãx> ðºó€ŠÍT@Ëv,/ÃC@°O€`Ǿš0"@ T‚P+ϼ@@: Ø阎@@@€'8Ðkì’`§FlºB@ðC€=vü¨#³@ ö\õ»Ò;~×—Ù!€ € €.@°ã÷ Øñ»¾Ùqë] …fš8.ðàƒªûî»OíܹÓñ™0|@@À‚{jQÅHvªPµ°M>È…!!€@?%K–¨O|⪷·@@’øEI–6C°ciaÊÁNÙ¢´‡UìT¡J› € €> ìø\ÝØÜv)4ÓDÀq‚Ç ÈðH€Í“*6SE, ر¼@e [ïÊ’¤¨R€=vªÔ¥m(S€×—©I[ €EvŠèq. € €@;A–I#€V ìXY… € `³ÁŽÍÕal à ¿×ÁŽßõev5ìÙ³GÍš5KmܸQ1BÍ™3G >\ÍŸ?¿ãÞõã(ëÖ­Sú?‡šÚξ}ûÔ”)SÔŠ+Ôøñã;î+y¢žÏöíÛãÏ2–2:?qâDd·jÕª¨¹Ý»w—:¯VcÔ}www«k®¹F5J>|XM:UMŸ>=úO~@H °Çk\`ÏU—ª•¬;ùÍ8>RÁNe0áJÑ`ª“±Åu¸RçdßuΓ¾@@ Øñ»î;~×·1;n½«®Ð;åØÖugPÚh vÊ©a­°yrŠ´ €ô Øñ{Eìø]ßÆìø 7/´¾¨Ÿ0aBã€E‹õyŒÊ<ò´wïÞÆ1ëׯo<¢ÓI°cõÙºukÔæèÑ££G¹Ì*&ä˜8q¢š;wn£ßøãIibµkW7”|äÉô=lØ°hNÉ1=ñÄÇÂV®\©ô|“ˆé_ºticYÆa&•þó™3gª[o½U]{íµý‡Š÷eƬ™úá¨,X5«Ý’clVg}Ü´iÓúÔÿúë¯ï÷(VÚ8ãõˆ?¾Õn,|ít4ÍO}êSê–[nQ½½½ÏI € €ýøE¿ß«‚`Çïúì´©¯¹¨7èæâ\ïY£÷ÈI OL@`ÎÉì˜6'MšÔg.˜6͸âE²ßäØšµ›¸tuu©eË–©AƒEˆ9&m ø4Ï<óL‚Åäã[YÆ‘V–ä;Íö¹I vt•¬arŽqßd“wì$û6ÿoÓÔÄ|ñ0«ÕXùÊéxš;Óq" € €@ ;ž;vúºYp /Ø/^ú•ƒ6BÝJ2ÈÈì趛µ©Û×˦M›úÜcF?÷À}6OÖ—¼›ÆŒ5ÞnrSfsÌe—]¦®ºêª~›?Ç—Áƒ÷ûûdÀ”e:PJþ vLgÚŒ·¥ÿ,m#äxu`e6ÀNÛ<¹Ùcbñ¹š~Z¥ÙFØ| eš&ÁN&&B Ø<Ù‚"0@H€`'…À­wý ÷­RÍåÉì´Úœ8lÙ²%õ­Xñ¾ôŒÌ[±.¼ðÂ(¼H† ú˜x»iAU\&m|ÉP£Õ¿7 7’ãH 8Š;É·WÅÛ:tèPÛ·‡µ»c'-ŒÓsŠŸ,,þ&-ɽƒ\üzc«Æ˜S€×‡YwfØ(@°ccUS-ÉG›Ò:M»IÞÕ’'ØI>ï3ì<öØc}îJ iÁŽÙ'9ýH×êÕ«Õí·ßý•y +y\–`'ˆ]|ñÅ}îàIî­“6Žf¯p/;Ø1—é`'ùøXr\­‚Sï4·¬ÁNüîàÝ IDATq¸Z7 €T.@°S91 €d ØÉÅaþ ´ v’1™Ç‡Š<ŠUõ;É;WâUk6Ÿ´câ¯;O.ñ9Lž °çªÏÕe¿«ËìÚ $ߊ¿«Å„===WgÇ_ym^‹ž7ØÉóV,ýÚïøÛ«âovÊòV¬ä]I­ÞîdÞä” `Òöˆ‰?rõ»O›_»»£ôyiý$C0ÓNòíí”Vu־ɨ‹¼«ÝXÚ.J@@@ d‚’A-kŽ;v,+HUÃáÖ»æ²Í6EÖgăÓ‚@¶oßÞxUüJi¯ Oë9¹MüµæñCÿùܹs£&L˜¡yŠ(+V¬ˆî*Ò?Év“ç¤Í)yLÜCÏU[Íîò1wΘ1™¹fGÒ%-ØIúë K¿&~Þ¼y™ïØ1cËZgÝÇ­·Þª®½öZi’cIºey={Ò©ªÏ»Ëí>ôÐCêsŸûœÚ¹s§ËÓ`ì € €€U;V•£ôÁì”Njgƒ|í¬K‘Qå}«W‘¾8VË—/Wï{ßûTWWWa¨OúÓêæ›oV½½½…Û¢@@“ü¢ßï•@°ãw}³#Øñ¯ÐÉ79ù7Cf䊀þ? »víR3fÌPÝÝÝ…‚WªÎ8@@l ر¥ƒ`§bà›OÛç§Æîé ~&Ø0`€zå•W <;,0pE€Í“]©ãDü Øñ¿ÆÑ ¹õ.B3ML°cº.ð°ÇŽ@é:àuç±q €@; Ò$ ’@2Ø)#à ɹ"€€›;nÖQ#€> ìøXUæ„Ô(Ð,Ø!ੱt…µ ìÔNN‡ P@€'8 à9p*ÁŽE*cˆúÕÁæÿ€”Ñm € \õ‡—«?vv§§GçMüâ— ÏÉ €@QöØ)*Èù P§{®Ö©]_;õ›‹ôØÓÓ£Ö¬Y#Ò7"€€ßú»eÿþý-'iöݹúê«ÕŒ±cÔ¯vmW#'\–æÅPGöï'ØÉ-Ç  € ²ÁŽßÕ'Øñ»¾Ùqë] …fš´z+è,\¸0zú¿<¼Aýë7ª?¼åÖÜ£íyì[JÿÃ;¹é8@ Øñ»ø;~×·1;>Èši" ì¤:fh;E¢K@Z€_ôû]~‚¿ëK°H}™&’ñ`§U Sf°3jé²Æ”õ]@ü € € ªÁN •çŽ@ Í4ˆ;zóÈU³¡”qÇÎé³nl4¯ûç¨[€Í“ë§?@f;¬ n½ ¤ÐLýý¢ïši蘡ì‰.@ t^w^:) "€t(@°Ó!§!€œ8räˆ:묳2sìd¦â@°X€`Çââ04@ 0‚À Ît@i‚é Ð?”!@°S†"m €@]<ÁQ—´L?;2îôŠ+@°lé™8^ °ÇŽWåd2x/Àž«~—˜`Çïú2;@À:‚ëJ€@@Às‚¿ L°ãw}³ãÖ»@ Í4p@€`Ç"1D@ðJ€`Ç«rö› ÁŽßõmÌŽr …fš8 @°ã@‘" €x%À/ú½*'ÁŽßål>;‚P+ϼ°O€`Ǿš0"@@w¸cÇÝÚå9ÁN..F v*Ä¥i¨M€Í“k£¦#@6;,n½ ¤ÐLv(CD¶¼î¼- €Ô$@°S4Ý €œ Øa% €€;>T‘9 €~ìøQGf8#@°ãL©(´ Øay €€K<ÁáRµò•`'¿g € Ø)€Ç© `{ìXS ‚Øs5’Çì8\<†Ž¸(@°ãbÕ3 €¸,@°ãrõÚ`§½‘Gpëedx!@°ãE™ €8$@°ãP±:*ÁNh.žÂÙŪ1fü Øñ³®Ì @ìàýöÖ¦Œ‘씡è@;‰!"ˆÁN …fš € €µìÔÂ,ß ÁŽ| œ Øa% €€lžìC™ à‡ÁŽul; n½kKÄ P“ÁNMÐtƒ• ðºóJyi@ ‡ÁN,E(.@°SÜ@@^€`G¾Œ@à¤Á+@ V‚Z¹é * Ø©–f@ žà¨„ÕšF v¬)AÂ Ø £ÎÌßØcÇ÷ 3?ü`ÏU¿ê™œ ÁŽßõŸÝÏŸý¢øBÀ¯_9®^}ù¨8è!L—9:.ðüÓO©ç¿÷´êšpYî™9°_9p@ ÷Îƹ]]]¹Ûá@@ $Ûn»Mé»v.¿üò¦-2×!CO þo¯µo‚Z¹å:“ºõîû7EnÒõ|Ê)Jõö4a¦Š € €X(0ì­ïWo¸àµŽŒ`§Vn¹Î¤n½ÓÁÎFNPÃÞü.¹ÉÐó÷w&úòÐ_"ü `»bÙ^!Ƈ €¾ Hý¢ß7Çv󉮮ËvÚUÆ“¿'Øñ¤M¦A°ãw}}›ÁŽoe> € €Z€`‡uP©ÁN¥¼â숗€ä ØÉÅ¡ `­›'[[†ˆ ìˆÑ‡Ñ±Ô­w<ŠUÏúÒÁαãÔÑcÕ±cÇÔÑ£GÕ¹çž[Oçô‚@Nƒ;¶«ŸìÜ®.šö¡œg*õÜwöFÿ üŸ¯=vxÑEån‡@¢;E9ðOàÇWªã§]]—éý’‘#GV>QŪœ8ìvê©<Øy饗Ô!CÔyçWOçô‚@NžÍ£lþué¬ä§þù»?PW¾çÔÍsf¨ÿ5W 2(ªÉ‘#¿T³ÿ|©ú­ßú-u÷®^zéåLÇzê)ÑæÌ[¾ñ÷}B¡_üâ¨úȼ¥êÔSOÚ3ýTµöwƒ:þâõüë&«—OJ°S4í–&@°S% !€€ ÁŽ >]#€– ìXZ†ULÀ¦`çŸþÿï«ÿ1eŽZ{ïm꿾û}&¶é«ÛÕÍÝw«/­û´zùåW2wÁï¾9 vôŠ z]£Í7þZ¶ò!µáóKÔ[÷ÍÅ[œýâÏžR?ùÁßFwêè;vÌwìTFNÃ%씀H .@°#^€9¤žàÈ1D/%Øñ¢ŒL")`S°³çþY}øÏEáÍúý ú UÿÝ„÷^«vo[ýy–ã.}aÓשÇÛ«êq¬W_>¦ž}â>õrïêðé“û̇`‡Ï¢Í;6W‡±!€@VöØÉ*Åq `ƒ€Ôž«6̽Î1ìÔ©M_µ „ìü·I7ªm_¹[]:î•xxú‹êØ ÿ¦ŸþAõÊ©gìT¢L£UìT¡J› € €@s‚zVÁN=ÎÁö"uëMÁN»G±ô[­^û—êر-Å2Ç|ó¹Ñ;ƒ½Nýåm‰6P6?úQ¬•«¿¤ºÿv5rÄî^{ëutÀÉ Ÿã?o{ÛÛÔ›Þô¦Òû¥AÊ Ø)C‘6@@ì;Ù­ŠI°SDsÛ H}m vÚmžüÒ˯¨ÏÝy‹zõÕ_·Ü<Ù7pà€(ØÑoÅúëÏRýÇQo‰êðüó/ª™7-Qú5èŸY<[~úimë“ç€V`é:2D3¦¶W­ç;Ç" vX € €@½R¿è¯w–ò½ìÈ×Àëì\Õ7Ëk̳g^w¾êó›Ô¸1oW³þ¿¨³Ï:CÝ·æ+êç‡^P_¸§»ö”¹¸Z=‚¥ë "@°Ã²¨T@êÖ;ÉG±*UJ™`G÷“|Ýy}óVª´)!@°#¡NŸ P¶¯;/[”ö@÷vܯ!3h–XŽœ †½ù]ÞùÔìð–wK(Ø ì[z&Ž€W;^•“É €¥ì”ÂH#¶ pÇN9i<‚5ðu,å-X<‚UŽ3­Ô#@°S3½ €@µ;ÕúÒ:”+ õG¹³°¿5‚ûkÄ; Øé-q `7¤»v쪣AÎØc§37ÎB©=Wef+×+ÁŽœ==W(às°S![Ÿ¦y«.iú©K€`§.iúA@N ìÔ³vêq¶©[ïvŠ-9Á*æÇÙv ìØYF… €þ ìÔS[‚zœƒíEêƒL°Óù’ËòÖرý_yÞyœ‰@=;õ8Ó  € `¤~ÑZvB«xÍó%Ø©¼„îx«Dš°R€`Çʲ0(@@‚;9½µÁŽ[+äµG°Þ©Ž ¸¸ßày –[õd´}vX àƒ›'ûPE怔+@°S®'­%¤n½ãQ¬üK‘G°ò›q†[;nÕ‹Ñ"€@º¯;ge €$vX^ ìä/+`å7ã ·vܪ£E‚Ö €@6‚lNå˜ÁN¾‚µ{ëüóÏW#FŒÈ×(G#`™ÁŽea8 БwìtÄÆI $ õ‡Ðtź%Ø£§ã*v²ëòVv+Žt[€`Çíú1z8)À;¬pI@jÏU—ŒÊ+ÁNŠ´a€^؃_ž|&w™´+α#=êÄ/¢ŸþAõÊ©g÷9\7N <¸]3ü=Ö ìX_"ˆ €ž ìÔSP‚zœƒíEêÖ»ïÿÝduŠ:Eõ+ŸoâG¦¿‹G°ò9r´Ý;vׇÑ!€ €€;õÔT;ÃÞú~õ† > víÚ¥ºººÔÈ‘#+ï\×wÇŽê¥Toww·Z¸paåÒAýRäxb¹sçÎú'îxºnC† QcÇŽu|& ×vX  € €@½R¿è¯w–ò½qÇŽ| ¼ÁŽ›å=õÔS£P‡G°Ü¬£N Øae € €ø(@°ãcU-šÁŽEÅÈ1ÁÊÅ¡Îì8S*Š-Ø<™å  ØaMT* uë]|aoÞ¼Y 8PvÚi•ÎÕ—ÆÏ8ã 5fÌ_¦Ã<hì°@À^wîC™ P®ÁN¹ž´f‰@|aoܸQwÞyjøðá–ŒÎîaœsÎ9jРAv’Ñ!ÐÁNhœ‚Ö ìXW„ˆ 숗€T! µ°«˜ m"€@9;å8Ò È ìÈúÓ;äz‚#ß(Ý?Zêú—·b¹¿v¬žÔ¶…Á!¸ÁNà €é#à‰{ìxRH¦@ R{®ÂÛ˜¦Ôõ/ÁNh+­æùJ-욧Iw C€`'‡"€ €%씀˜¡ ©ë_‚ Åñá©[虜5cø*@°ãke™ €Ø*@°SOe¤® vꩯx/Rd©…-Î@ ©Á‹@¨W@êýõÎR¾7©ë_‚ùÚ×2‚ЂŸ?ûÅZ\CïäׯW¯¾|T ôÆÐ)˜çŸ~J=ÿ½§UׄËrœuòÐ#ö«#¨ãÞÙ8·««+w;œ€ € à·À¡¿§ÿ‡·×:I‚Z¹Ãë,´`G ø©^à”S”êí­¾z@@@ò {ëûÕ.ø@žS K°S˜Z HÝz'µ°£~GNPÃÞü.F…ßßý™èËRiòƒ@VÅÊ*Åq `³¯;·¹:Œ Bˆ_‡Öi!uýË£XuV9À¾¤6ÁN=‹`§gßz!Øñ­¢Ì0v¬;³F7v.t£RŒÒ ‚'ÊÔñ v:¦ úD‚ ËÏäðF€`Ç›R2‚z‚C —`‡`GjíyÙ/ÁŽ—emLJ;ÇŽSGŒUÇŽSGUçž{®ß“fv…îØ®~²s»ºhÚ‡r·õÜwö*ýÏÀÿùÚã]tQîv8(*p÷ÝwGMÜpà E›â|@ rì|øÃVW_}uå}ÙÐÁ¯TÇO»$ºNÑ?úe#GŽ¬|hR׿<ŠUyiÃî@jaó(V=ë.ì¼ôÒKjÈ!ê¼óΫ§szqV góߨ›ÿF]:ë#¹çðãÇW?þö?ªÓ>ô's/½ôÒÜíp € ’€6æÌ™£n¼ñÆ ¦}ü{®N{ã{Õ€aïQ===;AT=€IJÝzG°ã÷âŠ;& çÕÓ~×¼ŒÙñ(VŠ´ €dzKrö–{¤èuèo^.³k×®Ú%îØ)wýXÛšÔYôÅ[±*_;•{ÙÁŽ—eeR € `±€Ô/ú¥HD¯C v¤Êî¿;þ×Xb†;êî÷I°ã~ ™ € `³Á›'Û¼>;ÁNÇtœØB€`‡åщÁN'jœƒ¶ ÜtÓMÑ–/_nÛÐ ¼ÁÁŽ—©[ïD?P<ŠUùZ&Ø©œØËv¼,+“B 8^w\É™08$ zÊ£X­†šI@ôE°“©FE"Ø)¢î¹;áÖž™#à“ÁŽOÕd. à›€èu(ÁŽoˉùˆ~ v*_€;•{ÙÁŽ—eeR'@°\É™0N H=Á!…&zJ°#Uvú­J@ôE°SUYíìTNìe;^–•I!œ{ìWr&Œ€ÓR{®J¡‰^‡ìH•~«ý@ìTUV‚Êeýî€`Çïú2;@°O€`§K9²òÂH]ÿêúîرC¢”êíîîV Ù<¹òbKt uëÔÂŽú%Ø©|©qÇNåÄ^v@°ãeY™ €X,@°C°cñòdhY¤>È;Y+äæq;nÖMzÔ;Ò(¿ÿ'N¨9sæ¨áÇ«ùóç«={ö¨Y³f©7ªQ£F•ß¡¥-îÛ·OÝÿýê¶ÛnSƒ j;Jí4aÂ5zôèT+ó÷ÍÚ½{·?~|ô׺ï)S¦¨yóæ©©S§¶í›@ÂúE¿”²èu(bI•Ýÿ~ vü¯±Ä v$ÔÝï“`Çý&g@°sRdñâÅêàÁƒjÙ²e™‚}¼g~úÓŸªI“&E¡XüÇ;ñÇüýƒ>¨¦M›¦Ö¯_9;þ}®˜ €@ç;<ŠÕùê±øL‚‹‹Ódh{þáŸÕ¬~Fm\³Dº Kýú׿VÛw}[?ñïêþûåVLˆ`ÇŠ287‚çJÖvÀ;ùƒÃ‡GÌôéÓÕøÃè.'Ö :´áÝ*Ø1æú`$8p€;vÚ®Ôr`óär=i (S€`‡`§ÌõdM[R·Þ‰~ ßc'ì~þˆšzí|5}ÊWS§ü7+ÖÁŽepn;ö•Ì„«V­Š—öh¾»dÁ‚ÁOœ8±Dtì$3Š÷™lÏtšt˜;Uöîݦçðä“O6 ÓfB”­[·FÇ,Z´(úÏøÝ5Iý÷É;ešyĈÑ£hÆ/íÜdÕuˆ³téÒè,ý££Z±bEãÑ*ýg­‚ý÷º&&:tèÁNÍ-^w^38Ý!€9D¯Cy+G¥8Ô ÑÁNåk„`§rb/; ر«¬&ôèêêjÎ^ôeq°ó£?§–¯ükµyënµïÙýjäˆßQÿ˨[çý‰zóyçFK!~ÇNôÛÜ·¨½ßýAôw£ßñ¶Æ#Z’ë†`GRßݾ vìª]ã*Yîعøâ‹£ð'†$ƒœø]-f~É0kÓ¦M»gâ›=ÇÏݲeK#xŠ?.wÖ`ÇŒ1~‡N2L‹þwà7›+·ÚcÇü{ìÔÿ¹"Ø©ßœ@ s©'8:q±3E¯C vŠ³íü@ {ó»¢WžÛösð'?SÓÿt:í´êÚ½O}ÖêÛO~OÝûùGÔ%cÞ®V-¿E}ö™}‚7ûµûïÿYÝö—÷©+ÿë»Ô{þð5ú÷/Pgœ1Dlz¯¼tT=ûøJutÀXulà¸hú·Ýú~h%@°c×úÈFèQ7{d+o°|ÊÜm“7عð {ÔÄßeBý¶¨}ìc©wþÄï˜Y²d‰ºåÄ • IDAT–[úD&X1á•þwý¸”~ä+mÌúï³Z¦…Miwµz+Vòq9‚ú?Wì±S¿9="€@çR{®v>âbgJ^‡¾`§Xñ8Û>Ñ”¥wìè»tî¸kºwùÇ£M‘ÍÏý<ª>»jcãNÛ÷ØÙÿÝ êø‹Ô󯛬^>åäfŸ;ö}mÁŽ=UI>ÔìÝñýuÌÞ:Ï<óLã•æyƒ´HÿYò‘*s‹ßÁ2lØ°Ô=eâwû\ýõQø£Cžä§Lc‚øþ8ñ ¥íý?6òd v’ûý$WC|ï¢v{ìÄÏ%رçsÅH@vºÔÈ‘#+/Ôõ¯®ïŽ;Ô)J©ÞîînµÍ“+/¶DR·ÞI-ì¨_Kƒfõ×A·ÿl‘úÒºO«ÿôûô¹cG@6mžüâÏžR?ùÁßFwêè;vÌÁŽÄ§Û½> vì©Y–`§Y¸ ÿ¼ÓG±ÒLx¤ƒ’«®ºªí;eß±“ ‘²T)>f e vÒ¹J ®tE°“¥ ƒ E€`‡`'Ë:áˤ>È;ÍÆÑ£ÇÕ¿üðGêßöÿDýýãßU_ýÛ]Ñ~;»·­Vã/½ÈÚ`çÕ—©gŸ¸O½Ü{†:|úä>$رü‹À’áìXRˆß £Ù~7&¤xç;ß©î¼óÎèíMÉýgyä‘èÏ;¹c'©å*=ÖiÓ¦Eo«*{³qs|ÿœv{ÅǬïjì´ Ò’{ÿè·{M˜0¡ßÛ¹ÒVwìØõ¹b4 €€mR¿è—r½åQ,©²ûß/ÁŽ=5þù¡ÔÍÝŸUŸ_÷Õhœ‹Þñ6õŽ·¿UûÆ¡jÁ'ïµ>Ø9ðôÕ±þM>ýƒê•SÏ&رgi93‚»J•öV¬ø"z´:\ˆ?rdþÞ<¦”7ØɲQpò˜ø¾þWêÕW­îüôÇÔàÁ¯‹ÖÛ¯~õRôÆ«Oýï/Xì´zK‡C† QcÆŒQú¿óƒ@+‚ÖG•&ÄÑÞ§&í'þ:÷øµªmÛ!°|ùrõ¾÷½¯”78òºs;jÊ(@4ÑëPÅbQú& ú²0ØÑwìÜuï5ÿö{ÕÌ_¥®|ϨŽüR­}èkê{û~¨~~èˆú»GîLÝcçŪk?²8Z"vÍûÕ¸‹¯Ö×·zK‡9ãÆSƒöm 3Ÿ v*@ °ÉfûÕ$ïŠI»K¦ÕÆR5eý‹®]»vEw›é—†è½á:ý!ØéTŽó@êD¯C vª/0=Ô+ ú²0ØÑúú®ûÖ~E­¼ï‹ÑfÉ—O£><í¨±ÿù?ª?½DýÉô?R×|èûmž¬C¡G7ïŠöáé9ðÔ¶¯Ü­.÷ŽZ jÁ::`œ:6ðµ·`™Îßò–·ÔòúÀZ&K'• ìTNLi¯׉-[¶L™W·§=b¥x0XL40Á΀Ô+¯¼R(à!ØaQ!€€KROpH‰^‡ìH•~«ý@YìTe]U»<‚U•l¸íì„[{fŽ€´€ vÌ8Š<ì±#]MúG<R{®æc™ÇŠ^‡ì”YJÚ²A@ôE°SÊà¬Ri$&@°Ãr@)d°SFÀ#5úEòìtÕò„Ôõ¯®ïŽ;”Þí´W?k¼Í“ó|>œ9VêÖ;©…õK°Sx}6Áx‰:–ò,Á*Ld;A–I#`…@³`‡€ÇŠò0¨P€`‡`§ÂåEÓu H} vêªpùýðVù¦´xR h°3ùö%ê;??' P¹ÀUx¹úÓagêgâ¿\è|NFÊúEcï¤ ÑëPÅê¤dœ“E€`'‹ÇÄx‹õP•@Ñ`ç¾uëÕ¡·¿¶qx‘·ÚT5GÚE;Ö¬Y£öïßßrpfß«¯¾ZÍ;Fýj×v5rÂe¹'ôâ¨#û÷+‚Ütœ€ ØáQ¬Â‹ÈÆvl¬Š½câ,{kãÃÈŠ;=}K>ëÆ…þ ? €@VbŽ5Ëø¾"ØÉRŽAÊ Ø!Ø)wEYÒšÔ­w¢(öØéhõeykìØþ¯<ï¨3N R Œ %‚ —“F °@Z°“蘎Êø¾"Ø)\6@r ˆ^‡ò(Vîzq‚å¢(‚ŽV`uÄÆI9ʸP"ØÉΡ Ðˆ;­2ƒQK—5úçÑQ# P€èu(ÁN=E¦—úD?P;¹ ýÚ#XïTÇ\Üï|Þ‚•›”RvX %vô:æ‘«fãáûJªRô‹e H=ÁQö<²¶'zJ°“µL犀èŠ`'×2á¬\\\@€ ¥xœŠ…ô…¾k¦] c:áûª7'#€€ER{®Jˆ^‡ìH•~«ý@ìä*+`åââà\(ÀãT($päÈuÖYgenƒï«ÌTˆ– ìt©‘#GV^%©ë_]ß;v¨S”R½ÝÝÝÑo0øñO@êÖ;©…õK°“y!·{ëüóÏW#FŒÈÜ"ÐJ€ %Ö¸"À÷•+•bœ ÐN€`‡`§Ýáïú ìØ¿8xËþù6B.”|«(óAÀ_¾¯ü­-3C 4©_ôK9‹^‡ò(–TÙýï7Ä`gðëÏSƒÏä.“v«ûØ‘uâ—?Q‡Oÿ zåÔ³û®×͸qãÔàÁƒÛ5Ãß#Y€ ¥ÌTˆÂ|_ €î@vx«Ã¥c÷iá;“Õ)êÕkwY¬ÝÑéoÁâ,kJäÕ@¸PòªœL¯ø¾òº¼L<ÐÁΰ·¾_½á‚¨]»vEç³ÇŽÇejR·ÞÙ”îܹ3”2—6O2D;¶´6i#À…k\àûÊ•J1N@ ¯€ סuJlžÌ' R>P;ùK|ꩧF¡`å·ãŒö\(µ7â°C€ï+;êÀ(@¼6\‡ìä­Ç[+`Ê`'ÿòà¬üfœ‘]€ ¥ìV‰²|_ÉúÓ;”' õGy3È×’ ס;ùjÆÑ ØðÚ¼y³8p :í´Ó,–²ghgœq†3fŒ=b$Þ p¡ä]I™Þ ð}åmi™Á Hí¹*mÃu(ÁŽTõé·t>P7nTçwž>|xéóó±ÁsÎ9G 4ÈÇ©1'K¸P²¤ Ú ð}Õ–ˆ@À‚6Ovd©2ÌVR·ÞÙìÔ™”² @ ½Jí8ìàûÊŽ:0 (.@°C°S|Ñ‚¸€Ô™`G¼ô ë¸P²®$ šð}ÅÒ@_¤~Ñ/åÚu(oÅ’Zi5÷K°SOB[sYé'¸Pr²l  ø¾ ²ìL< ØY¸Ðƒ22…¤ÁÁŸ làBÉ–J0h'À÷U;!þ°S€`‡`ÇΕYpTR·Þ…ö*X&NG .”‚(3“DÀ ¾¯¼(#“@B»åQ¬ySíU§-}!àªJ®VŽq#žßWáÕœ#€€¡]‡ìø±n­Eh(k ÁÀ°H€ %‹ŠÁP@ ¥ßW,ðE@ê )¿Ð®C v¤VZ ý†ö ¤¬LB\(âãd¨Q€ï«±é *ÚsµÒIµh<´ëP‚©•H¿¡} )+ÓD J…ø8jàûªFlºBJvêy™ŽÔõ/ÁN¥{—ºõNjaKõkOÅ ö p¡domôàûŠ¾ììø²–ƒž‡ÔY*`‘ê7èEÆäÈ(À…RF(Cq¾¯ÄKÀ@ $©_ô—4üÜÍH]JõË;¹—ˆ›'ìԓк¹:5õ p¡T¯7½!€@ç|_unÇ™ €€¤€TÀ"Õ/ÁŽäj«±o‚‚—]!ÐR€ %¸"À÷•+•bœ €@_©€Eª_‚@>R·ÞI-l©~YNLB\(âãd¨Q€ï«±é (Q@êzPª_‚MõZØRý²@ ½Jí8ìàûÊŽ:0 @ ¯€Ôõ T¿;yWÇçZØRýæÂá`àB)ÐÂ3màûÊÁ¢1dHz‚CªR׃RýìH­´@ú•ZØRýRV¦‰@!.” ñq2Ô(À÷UØt…• Hí¹Zé¤Z4.u=(Õ/ÁŽÔJ ¤_©…-Õo eešàB©'#€@|_ÕˆMW P©ÁN=/Ó‘º%Ø©ôãcOãR·ÞI-l©~í©8#AÀ^.”ì­ #C¾|_±"@À‚‚_ÖrÐóú K,Rý½È˜<¸PÊÅa .À÷•x ”$ õ‹þ’†Ÿ»©ëA©~¹c'÷qó‚zZ7W£F ^.”êõ¦7è\€ï«Îí8 X¤ú%Ø‘\m5öM°C°Sãr£+Z p¡ÄAWø¾r¥RŒè+ °HõK°È'@êÖ;©…-Õo ˉi"PH€ ¥B|œŒ5 ð}U#6]!€% H]JõK°Sââ¡©þR [ª_Ö´àB©½G €€|_ÙQF五ê—`'ï áø\R [ªß\8Œ@ \(Zx¦€ƒ|_9X4†Œ©ROpH•CêzPª_‚©•H¿R [ªß@ÊÊ4($À…R!>NFø¾ª›®@ R©=W+T‹Æ¥®¥ú%Ø‘Ziô+µ°¥ú ¤¬LB\(âãd¨Q€ï«±é * Ø©çe:Rס;•~|ìi\êÖ;©…-Õ¯=g$Ø+À…’½µad ÐW€ï+Vø"@°C°ãËZzRd©€Eªß “G £J¡8 Äø¾/@’¤~Ñ_Òðs7#u=(Õ/wìä^"nž@°SOBëæê`ÔÔ+À…R½Þô† ð}Õ¹g"€’R‹T¿;’«­Æ¾ vvj\nt…@K.”X  àŠßW®TŠq"€}¤©~ vùHÝz'µ°¥ú d91M p¡Tˆ“@ F¾¯jĦ+@ D©ëA©~ vJ\<4Õ_@jaKõË@ö\(µ7â°C€ï+;êÀ(@¼R׃Rýìä]!ŸK@jaKõ› ‡ƒT€ ¥@ Ï´pP€ï+‹Æ@ U@ê ©rH]JõK°#µÒéWjaKõHY™&…¸P*ÄÇÉ P£ßW5bÓT* µçj¥“jѸÔõ T¿;R+-~¥¶T¿”•i"PH€ ¥B|œŒ5 ð}U#6]!€@¥;õ¼LGê:”`§Ò=KÝz'µ°¥úµ§âŒ{¸P²·6Œ ú ð}ÅŠ@_vv|YËAÏCêƒ,°Hõô"còdàB)#‡!€€¸ßWâ%` P’€Ô/úK~îf¤®¥úåŽÜKÄÍvêIhÝ\Œz¸Pª×›Þ@ s¾¯:·ãL@@R@*`‘ê—`GrµÕØ7ÁÁNË®h)À… \àûÊ•J1N@ ¯€TÀ"Õ/ÁN Ÿ©[虜T¿,'¦‰@!.” ñq2Ô(À÷UØt…”( u=(Õ/ÁN‰‹‡¦ú H-l©~Y Ð^€ ¥öFvð}eG W@êzPª_‚¼+„ãs H-l©~sáp0 p¡há™6 ð}å`Ñ2¤ H=Á!U©ëA©~ v¤VZ ýJ-l©~)+ÓD J…ø8jàûªFlºBJ¤ö\­tR-—º”ê—`Gj¥Ò¯Ô–ê7²2M p¡Tˆ“@ F¾¯jĦ+¨T€`§ž—éH]‡ìTúñ±§q©[虜T¿öTœ‘ `¯JöÖ†‘!€@_¾¯X à‹ÁÁŽ/k9èyH}¥©~ƒ^dLŒ\(e„â0àûJ¼ JúEIÃÏÝŒÔõ T¿Ü±“{‰¸yÁN= ­›«ƒQ#P¯JõzÓt.À÷Uçvœ‰H H,RýìH®¶û&Ø!Ø©q¹Ñ-¸Pb €€+|_¹R)Ɖô X¤ú%Ø ä uëÔ–ê7åÄ4($À…R!>NFø¾ª›®@¤®¥ú%Ø)qñÐT©…-Õ/kÚ p¡ÔÞˆ#@À¾¯ì¨£@ò H]JõK°“w…p|.©…-Õo.F P.”-<ÓFÀA¾¯,CFT©'8¤Ê!u=(Õ/ÁŽÔJ ¤_©…-Õo eešàB©'#€@|_ÕˆMW P©€Ôž«•NªEãR׃RýìH­´@ú•ZØRýRV¦‰@!.” ñq2Ô(À÷UØt…• ìÔó2©ëP‚J?>ö4.uëÔ–êמŠ3ìàBÉÞÚ02è+À÷+| Ø!Øñe-=©²TÀ"ÕoЋŒÉ#Q€ ¥ŒP†â|_‰—€ €@IR¿è/iø¹›‘º”ê—;vr/7O Ø©'¡usu0jêàB©^ozCÎø¾êÜŽ3@I©€Eª_‚ÉÕVcß;;5.7ºB ¥J,pE€ï+W*Å8@¾R‹T¿;|¤n½“ZØRý²œ˜&…¸P*ÄÇÉ P£ßW5bÓ P¢€Ôõ T¿;%.šê/ µ°¥úe €@{.”ÚqØ!À÷•u` €@^©ëA©~ vò®ŽÏ% µ°¥úÍ…ÃÁ*À…R …gÚ8(À÷•ƒEcÈ * õ‡T9¤®¥ú%Ø‘Ziô+µ°¥ú ¤¬LB\(õçÛ³gÚ¾}»š?~![Nn.²ñáÇÕÔ©SÕôéÓ£ÿ,òóàƒª¥K—ª7ªQ£FõkJÿýºuë”þÏ¡C‡éÊŠsù¾²¢ JÚsµ„¡wÔ„Ôõ T¿;-NÊ* µ°¥úÍêÂq„,À…RßêŸ8qBÍ™3G >œ`§¢FèÆ;/,¾¯:·ãL°K€`§ž—éH]‡ìØõy«l4R·ÞI-l©~++ #à‘J;u/g‚îØétÍñ}Õ©ç!€€m;;¶­IÆÓ€ÔY*`‘ê·ƒÒp Á øx¡d‚ýhÊÖ­[£fÎœ©–-[ÕWß‘³jÕªF­wïޭƯÌúxý3zôèè—'žx¢ßã,æX}ž~dK?Z´xñb5qâD5wîÜèüo~ó›êᇎîþ9ÿüóÕ´iÓú´›öèŒ>`ß¾}jÊ”)ê¶ÛnS÷ÜsO4~ý³hÑ¢~wé>,XИ‹îß%iÁŽn' ÄDZ~ýú(H1ÖÓÓÓtßìÝ»·å¸¾È?xð`c.¦m½“!VšC|Óç¼yó¢qš×m™ý[ŒU«` ScÓߤI“áUrÿÓZH“ ?ô±É€+þx];»xðeìâuIÖÓ˜8p åZÕ.šä1‹¯ãaÈf{ìè@1>%×¼¶ˆ{é¹èŸ5kVÓÀHêËÎÇï+)KúEê X¤ú%Ø©su öE°C°#¸üè>>^(%s'I³f㶃î·ÇNž`' 4G»‹ædÈb ?Oÿ™V¬XÝm”v̈#¢¹˜Ð!n<ì0… ztÉ0Ëô?oÓ¦M©›ù¶3N ’~úΙx—öQÎò˜—ñme7lØ°(àŠßÉÒj]ÏC‡Eu‰»%ƒÓ¶ Ö²¬Ëgžy&uîY6ONÛ\9^³4×dðeËצßW¶Ø2@ J©€Eª_‚*W“EmKÝz'µ°¥úµ¨ä k|¼PJ»¸ovQž¼è6aHüÁNò‡f!CÖ`'<4 |’ñ˜;8Òæ’\ˆñ»Pôß™»aL°C»`gÉ’%ê–[né$µ3nÖ®>/n¯Ã,w‘èsô]*­îd1Íì’á‹>>O°ÓªvW^yeã­XúQ¯´ð-i–ö8`ò˜.eæ™ÖŸ1Ôkâ±Çk<®gþ<.v µÿv)ÿ¿¯ÊW¢E@À>©ëA©~ vì[ƒ^HjaKõëUñ˜  øx¡Ô*Øi÷ó"{ìä¹c§]°“ö˜UüŽ½OPZ­îÄ0K(í±'ýwe;y³Þ!ÒîN§fÓþsÏ=í£—jgW$ØiW»´`§YZ¸¨çÛÎ$ëyñÇÓ¶oßÞg力¾z:jÖÇï«Ž 8 œz‚C NêzPª_‚©•H¿R [ªß@ÊÊ4($àã…R³G š]äÆÿ<ë;ÉÍ~Ó.°[=Š•%ØiµO‹~4§ÙÞ)<òH`¤Ý±Óìq¢ä¾<Š¥7lÖ{ì¤íßÓθٞ.ɽ`²<Š•öˆo¬7 ngW4ØÉ»ÇN;³*÷ØÑ{/Å×…öÑoz‹ïÝTèK¦Ä“}ü¾*‘‡¦@À!©=W¥ˆ¤®¥ú%Ø‘Ziô+µ°¥ú ¤¬LB>^(5 TÒÞ>”¶oM2dHß—Å<ÆTE°£ kÞH•$épȼÉܽ¡ÿ¬Ý;z~&ü1û²˜G“Šì±£ƒó6©ø›ÇòÇߊ•|{S»»S´A³à*~—’ IZÙ vZÕ.Ë[±š™ÅÇJ{sYòËÀì7“˜ù³ä«âwü Z…¾\*8ÙÇï« ˜h Ø©çe:Rס;|Ë¢Ô­wR [ªß2jEø.àã…R«7#%7ËÕõmvÿ»ø«¾õŸoÛ¶- GÌ#4U;×_½Z¹reôêlý"âAŽY£úâýºë®S3f̈ÞÊd6æM{Ì'¾ÇŒi[ÿ§¹“Eÿ÷ä#Eí6O6¯ïÔ8¹×Nr_œ,ÁN<܉ï[“ +’õLÚÅ—ÒoÇjÅïF2oÅjU»´Gϲ˜é1Äë¦}L?&L vôÝ@zþsçÎþºÙ¦Ò­6±¶å{ÐÇï+[lÔ+@°C°S·J¤>ÈR‹T¿•FðL€ %û ÚìuÜö”%\® cçûŠÏø" õ‹~)?©ëA©~¹cGj¥ÕÜ/ÁN= mÍe¥;œàBɾ²¹pmŸš#r¹vÍÞ e‡ìÉQð}eS5  ]@*`‘ê—`'ûÚpúH‚‚§0ƒ÷J€ %ûÊér8`Ÿf½#r±vfÏ›÷Ö1UäûªÞõLo €@YR‹T¿;e­ËÛ‘ºõNjaKõkù2`xX!À…’e` A€ï« H‚X( u=(Õ/ÁŽ…‹Ð§!I-l©~}ªsA *.”ª’¥]([€ï«²Ei¨G@êzPª_‚zÖU°½H-l©~ƒ-4œLóò IDATG ‡J9°8Dø¾å§s(Q@ê Ž§«)©ëA©~ vr-Î+ µ°¥úÍëÃñ„(À…RˆUgθ)À÷•›ucÔ Ð_@jÏU©ZH]JõK°#µÒéWjaKõHY™&…¸P*ÄÇÉ P£ßW5bÓT*@°SÏËt¤®C v*ýøØÓ¸Ô­wR [ª_{*ÎH°W€ %{k#=2󶤙3gªeË–©AƒI©ÑÿâÅ‹Õ‚ Ô¢E‹ÔüùóskÏž=jݺu9U=Ï'N¨9s樞ž¥û:tèš2eŠš4iRî±çš¨‡ó}åaQ™ ììºôýš¶ÔY*`‘êׯUÃl¨F€ ¥j\}hµêÀ£ˆQ§ÁNÚœªž§’&L˜Ð¡LгjÕ*µ{÷n5~üø"AË÷UPåf²x- õ‹~)T©ëA©~¹cGj¥ÕÜ/ÁN= mÍe¥;œàBÉɲÕ2èª"“(3Ø)2Žvç6 qŒm'wµëÓç¿çûÊçê27ðY@*`‘ê—`ÇçÕ›ÁÁN Ki: À…RýE2õñžÓîÜ0á…9nâĉѣŽfÁÎáÇ£öMñqeÕŒ·}ñÅGãÓ?¦s—‹i/€4 vZ™%ÿÎXéÿœ6mZÃèŽ;îH}Ì+Í£…©Ó¹çžÛ¨î/^¿7ªQ£Fe¥ ú8¾¯‚.?“G‡¤©~ v^¬y†.uëÔ–ê7OM8P¸Pª·òi¡ŽÁèÑ£Uü?-„ÐÇ™ÅìÓ²wïÞ~ÐmsÎ9jÛ¶m}þÎ>ÍÆ`N…L ‘ì79þvªic0sÛ²eK´$âãH vÚ™­\¹²tÅçš vž|òÉèÑ©x`pLø”Å"K8ÆãXíVËkÏ÷Uv+ŽDlº”ê—`ǦÕçáX¤¶T¿–)!PºJ¥“¶l0HÄÕ1w¥¤Ýåaî`1ŠîDß±£s^<|0¡ÄàÁƒ£{õ~.&Ù´iS#81!N|ɀŜ§û4m™óôŸuòXQ<؉·•v'KÚãLIÇ,fú®˜,{ì?~¼qGRò¢4×VÍî,ŠÏ)~~½«Ñ½Þø¾r¯fŒÐR׃Rýì°î+ZØRýVŠIãx"À…R½…LÞ©Òn•ä]!iÁŽ ÒB¢xð’ v’o»J†*O<ñDŸG”âGšZžG²ŒCòœä#XÉ~ŒW«=vš™e vôÛ¿’aU²¿ä#XÉqêy­^½ZÝ~ûíQ¨–ÞtºOP½+Ö®Þø¾²«Œ:z‚£ó;SêzPª_‚bë…³ÛH-l©~Y Ð^€ ¥öFeÑì‘¡´»g’}'ƒý÷æ®fw$ïR1wìä v8иK¨ª`§ÝcbÍ‚øÜ›™å vâwÝu×]jöìÙÑžBíÃ2}ì”ý©9ÙßWÕ¸Ò*Ô/ µçjý3=Ù£Ôõ T¿;R+-~¥¶T¿”•i"PH€ ¥B|…OŽß¡’¶ÇLòQ 2ƒäÝ2yîØ)º/L³½gŒG»»’w»Ä¡ffy‚äÝOzêx–¶çNr1´{ÜŠ;vò|ø¾ÊoÆ `§ÁN=/Ó‘º%رósWú¨¤n½“ZØRý–^8DÀC© ¥µkתË/¿\uuuy¨š>¥fJ%÷ÏyøᇣM~Ó6 .3ØÑ£ìtv·+j³`'þ•[Zˆ’ EÌ¿·2Ëìèñ'ïŠ?N¯e+ öØi·òý½Ô÷U¾Qr4 Ð^€`‡`§ý*áë¤>ÈR‹T¿Ö/ˆ€u_(é@gáÂ…ª§§GuwwGÿ=¤ŸV{ȘnjZSv°“foîxI _š­Ó·b%Óãiö¨Z«%‹Y<Ø1óÖæçŸ~Ÿ½„ô;ú'(¥Í/‹oÅ*÷Ó]÷÷U¹£§5@à5©_ôKÕ@êzPª_îØ‘Zi5÷K°SOB[sYé'êºPŠ:úNèèÿSâOÚk²“›(ÇïIÛ¯eìرÑ~7ú§È;×]wš1cFôv-ýÄ*Ë]5úœ¼¡Ž>§YÛf=$ï–I@iw´3›:uªJîÅÓ*؉‡Li”þûV5ÇÿþÜsÏæ’m°F¡M¶@p Œ9QBà ˜HLA cŽWà#D@b¤)»;b¥Ö0ÂÂNØŠÃê „˜(!ðP€ÀBŒWCøI€ °œ€”À"ea§7@jéTaKÙ-¤œ½„œ(!ðôJC xf¼ê‚[±úÐãY@³šJÙEØ™­NxÊ“€TaKÙõÄB3M ¤°c@GŸÃ£oâ@À‡€¾KÿÑŸK—.Mþ˜Ï‰'&ÿºyóæÆ®®ú… Ó¿çðdê´ П€Ô|PÊ.ÂNÿš¡‡R…-e—b€Ú H;í^Ñ€@3sçÎ)ýÇ|öìÙ3ù׃z£CØñFEC@``R;8û;©ù ”]„ïÒ á,¤ [Êî,Œx¥@Ø)-ãÄ <T…/~ñ‹“À>þñ{ˆ°ãŠ†€ÀÀ¤Î\8 ïî¤æƒRvv¼Kƒ†³*l)»³0â”Fa§´Œ/ò Pvf‰ agj< Aa'Ìe:RóP„!Þ’úZz'UØRv(\„€8„ñà 0„ ñ  „„hŠGf' õ"K ,RvgÏOB ;åäšH!@ˆƒ€Ô/ú¥¢—šJÙeÅŽT¥¶‹°F¡ œVÌA I;I¦ §!@€@2¤)»;É”f?GvvúUOC`8;ñ¤'@@ŽÀwÞ91þ¹Ï}NÎ ,C€€“€”À"ea§AjéTaKÙ-¤œ½ ìôÂÇÀ@$æææ&žœ:u*p€ ©ù ”]„jTR…-ewT˜tL ìd’H€@áv /‡¢& 5”²‹°u9¦ïœTaKÙM?cDñ ìŒÏ €ÀøvÆgŒ@`8R;8†‹ [ORóA)»;ÝêƒÖ H¶”ÝŽxh" ì™v‚†@v8c'»”²& uæªT©ù ”]„©J+Ä®TaKÙ-$­„ ^vzáãa@€ ЙÂN˜Ët¤æ¡;_‰4Zz'UØRvÓ¬¼†@X;ayc €  ì ìðd@@êE–X¤ìfP*„Ñ ìŒŽ€ @`©_ôK¥Aj>(e—;R•Ø.ÂN…6pZ1$ ì$™6œ† @ÉX¤ì"ì$SšýEØAØéWA< á ì Ç’ž 9ž,ÇË€ÚH ,RvvÚ*"“¿—Zz'UØRv3)€À¨vFÅKç€@ \w4f Ì@@j>(eag†"áR…-eן -!P.„rsOäȉÂNNÙ$@ 7RóA)»;¹UpdñH¶”ÝÈðã¢$€°eZp èHa§#0šC¢¤vpH-5”²‹°#Ui…Ø•*l)»…¤•0!ЋÂN/|< DB€3v"In@^¤Î\õrn„FRóA)»;#]¾A@ª°¥ì’{@ ÂN;#Z@€ ! 섹LGjŠ°3äÛq_RKï¤ [ÊnÄ%€kˆ†ÂN4©À@€ !€°ƒ°SH©ç¦Ô‹,%°HÙÍ»ŠˆÃ@؆#½@€ _R¿è÷õoèvRóA)»¬Øº‚"ía'ŒBiúq Q@؉*8@€²# %°HÙEØÉ®„Ý!ì ìRê„™„’„‹€@+OnED@b¤)»;b¥Ö°ÔÒ;©Â–²6«Xƒ@švÒÌ^CË pÝ9@ ^RóA)»;ñÖbžI¶”Ý,’F™ÂÎÈ€éBa'fŒ@˜‰€Ô|PÊ.ÂÎLeÂC¾¤ [Ê®/ÚA d;%gŸØ!„|rI$(€Ô)¶RóA)»;R•Vˆ]©Â–²[HZ ½ ìôÂÇÀ@$8c'’Dà àE@êÌU/çFh$5”²‹°3BÑå¤ Û×îåË—Õž={ÔáǧNïܹSÝwß}ê£ý¨:qâ„Ú¿¿Ú»w¯rµ­æzóæÍê‘GQ«V­r–é㦛nRÛ·oO¢Ttù¤:xð Z±bE>K9™*+»¶M½weØ%ö6açòŸÿ¹ºçŸV}ùØqµïc·«{ÿûNÝ9÷ä7•þsõî;¦?Ó¿òýTý"v_Û}Û ákÊñ÷å7ÔóCäa(_$ûyî¹çÔ¶mÛÔÙ³g§n9r$™ÿ·I²Ã6  ;a.Óñ‡];C´?©¥wR…ík7¤°sñâÅÉ^-¥òå÷Àjß¾}J‹];Í/wʬúNT»Æ.%ì¸üì{È!¿¯¯©Ç’u“­¾yˆ%Ž¾~ ìô%Èó€Â@ØAØ [qX…€Ô‹ì+° ´¯]„|ÅŠ¡kª­¿®âF[)ý}×ØÛ„¦Øû¬ØI]Øè+(tÍSJ5Ò×¾yé똶vƤK߀†' õ‹þá#ñëÑw>è×›+)»¬ØñÏQÒ-vÜ mÝt{uk+V×í*O=õ”Ú´iÓ5tæ̵qãFeÛ3ªÛºì/Ñ¿õ[¿¥¾ô¥/MVÿØíl;ëׯWGU>ú¨sÕMÕ'{UŽËíW]ܶo?þ¸:vìØt{[Ýö43É´¡úg>ñvéC÷}òäÉ ý1ñž?~Ùv×jª¾¬šž÷Õ5µõkÿ½‰Ë®y“›•+WN·$Ú9n«Ë¶:©{þÀ‡¶ªWûßÕ{î½Ï9®º¶bÙ?ûðÏ®S7Þ¾K}ò“Ÿœ>ß´ ®ÉÏ»ï¾{Yì7ß|ó²wµk=Ôý§žÛòÙ$(4½ ]â¯òÐÛPÍÇÄ ÿ[-k×®üU›ßuLÚêË~Î×F—1¡nu öØT̓OÍTøÄ>ƸZÍ¥½ÊŽÑ´óážâjÔ¤¿Ðá< xX¤ì"ìx•EúvâvÖ­[7Ý¢U­4[qM6l±EŸ±cÇŽÚbµ…›º¶ÆžîÄl³;ôv\‘ÉL]“/óœ™\´ÅÛ§cë½ï}¯zùå——¡ÿΞàôeÕö¼>©-VSŸ~]"Îc=6­§K0¨´/>u²k×®Úº¾iý¥>qíOª_ùÔ~g½¶ ;uE^'îø uýv©‡ºóµÚrÜ5Ÿö»Øö.Ô1º—c|©Š²æç]Æ’:>õežõa£s×ÆA‹èmy¨áuÿ.qÔ§f|DÓÆgÌ7mgWõ³>1êv¾ÜKv8<9ýïÇDäK@J`‘²‹°“o-/‹LjéTaûÚuÅŽxÑôeÞž€º&Köo—ÐQpÙ_¬ë&]®*ƺ/æ¾[7ìÉ’=Ù°n&¢®ßú»VHÙÏÖ­^ÒŒÍÊß>ìÕ)¶0fÛ3\\?ëÂÊ÷ù¦X]µæÛ¯~Ön«kàܹsË×m\ï©Áº|Ú5쪓¶çÿÇ_øyµÿ×ß8´ÜŽÓGØùüç?¯~îç~NéWŸýìg'â\ÛYPm[±lQ¯­žÚÞjÞ|êYÇÐÔ¯+O¾ï“-|ØœšúÔþ¸VÎò~Ty´ÕGW¾šòàGµßæ€}× ·¶šq½¿³Ä>Ô¸êã… ¦+Ûj½da‡ëÎ ùM˜€@’|çƒC'eagèLÒß2R…ík7agÙ„ÖqK—KØ©®Lp AÕ »™Ì?~|²bÃç·ñ³;Ußê~ûoâ®þV¸*ٓݺ׫k¦½=YrÕB—• .V¾ÏÛÛÁ|Ööí×ÜbVåSÍ}“°£™·ÝöÖ$츞o;c§MØùo¯[£>yè×ÕÕW_=) }k›Ï!ßmÂNUª¶ïòî4 ;¾ïHUôµE†ºÕsuïBaÇWèëZ‡6{Ìjª¯Ymt´o.ÁEÿ¼é&±¶šqÝ$è»KÀ­Ž›uìêâ÷± w„¥N:Å·?@ˆŒ€ï|ph·¥ì"ì IúCØùÁ¹ÕR¨ûòÛvºKØ©ž…P÷eÝ5™{à¦ç̸ÊÕž(Ì"ìøøÖ´mÌ%ì¸Î~èÓ‡kÒâšÌ6mí¨ .V¾ÏÛ¿wÅZÍ“o¿f+KµÆª6\±×Õ¥KxjLlÿõóïúáRÿákG{±óß=ø¥i·C ;U±¤W—w§IØ©ò÷ÍgÝYH>ïBaÇ9ÌxpèС+Ä3_¿]Û±|ë«‹ ¶Xâ3V¹ÆHŸ³ŽšVùÆÞÕWŸø›þ_aÇÚ…;ÂÂ_}!4Híà¢#%°HÙEØ‘ª´BìJ¶¯ÝYWìt=ØØ÷]è*ìØy³vmÁ«ûßYÛJ¯ê¸h÷cêËW`xöÙgWªg×Tãë+ì˜þÚÞ­.Ž︊°3ì—/ÎØ–'½Aã:suܨê{÷ퟔ]„¡3IËH¶¯]iaǶoO˜ÚÎØ©ûM³†_w{KÛV,Wéβb§m›‰9¼·î.aÇu.QÛ™(®pí‰g—­X¾¬Ú¶¡™~š&p.[¾ýšgÛn ò½¾ÙngçÏ·NìçõvªGŽQ+~äG®1Va§n+–Ï°ß”cß|6ÕhÛûÔUرۛCÆßþö·O¶%U5öp|¹êË—yß1AûW·¥Uÿ|÷îÝÓó¼†vl&®Ø›¶hVkÆw\õÑ·&«âœÏVRŸZ    OaÇ}™ÎФ}ç¡CÛEØšh¤ýI-½“*l_»1 ;fb¦KÈÖi 5Mþ‡'×qð°w9<Ùu¾½… ‹°cÿf½K¾+v\g\taåû|WaÇ·ßêdÕ¾ŠÞWÔª»y§mËž+϶Xðóoû õÛÿâU«®¹&aÇ5ÑöÅšrì›Ï&a§í]˜EØ©ÞžT]éåëw“8ÙV_®C|ûp˜e5ã ñõÚk¯ŽÑmÛ÷\gì´½æݲco;<Ùw\­ûE5F³mÛ¶M%wÝfÇ]òV¬H¿òá '„„^ H½È¾ËЈ}í†v\ç*èU$ú–¢¦kÊ}¶bi~®óôdàmo{›Òz× ‹*÷ê$«é VûÙº+„M›ºI‰+ï>ÂN›½¶>|…{2<+«º­$uL|ÎØñõË5‰³Å ÃÉõ4]G­í»nv3Œt¿M×ëv}nÅšu+–«ž]×}›8ºœ›Ô¶}§M¼ó©Wž|ß×Qwݹ‰¿Z®ÚôñÛõžw©/¾ÚòÐtxÓûÒT3Õø}co‹i–qµ*öV}³îúy„¡¿ÁÐ qHý¢œhÚ{õ¶÷Ô­…”]VìtËS²­vÜ mHaÇõ…ÚLŽí/Ðú·µ_þò—Õ§?ýiuøðaÕ&RØEéºmÅuè©Ë×vŠê¤î@ÐêdéäÉ“Óš]ýV'O¶ÀåúmµkBÙ§.Âάª¾Ö]ßn_/í3Ø´õ[ý-üÚìm×›f{à 7´®@°ý©æ£©N\“ïî_R×üÁï?<Ùåçg>óuï½÷.{×Ú&émÜ]¹k|ê¬n¼òyôö©®ñÛbPÓv«YxT7Õ—W›YÆÝKP±…Ì¡¶bµm‘c\5ÛèÚb4ùðÉ-ÂŽÏhM@M@J`‘²‹°ºÂ„ì!ìtv„Ò4ŠYßíT}ŒûLZûôϳyh»î¼)ÚsO~Sé?Wï¾cÚLÿŠr#ʸZ²°Ãáɹ½uÄäD@J`‘²‹°“Sõ6Ä"µôNª°}íº¶G5]Sk¹Ø€ºCTÇ<Ô2• H¬ù+Í/„Ò2N¼³ˆ}\õ]õ3Kì©<3777qõÔ©S©¸ŒŸ€Š!à;ˆ”]„¡3IËH¶¯Ý\„Wv"ÚÎÿè[¶±O@úÆÇóÃ@Ø–'½åI öqaG)„<ß=¢‚ò à;:Z)»;Cg’þvkÀufÃPW7…ûD0%˜v@Ø¡, ÐN öqaa§½ŠiÄD@j‡)EÊ.ÂŽT¥bWª°¥ì’V„@/;½ðñ0  Î؉$¸x:sÕ˹IÍ¥ì"ìŒPDtù©Â–²Kî!v;íŒh@€†$€°ã¾LgHƺ/©y(ÂÎЙŒ´?©¥wR…-e7Òôã¢"€°U:p€  ì ìPæù‡(õ"K ,Rvó¯$"„@;ýÒ @èB@êý]|²­Ô|PÊ.+v†¬žˆûBØ £ÐF\¸h ìD“  @YX¤ì"ìdYÆW…°ƒ°SH©fvH.B­8<¹ ˆX¤ì"숕ZXÃRKï¤ [Ênجb i@ØI3ox ,'0777ùÁ©S§@@‘šJÙE؉¬ssGª°¥ìæ–?âÀvÆ JŸ€@h;¡‰c€€?©ù ”]„ÿÚ å ¤ [Êî ˆxÅ@Ø).å , ìd™V‚‚@¶¤vpH•šJÙEØ‘ª´BìJ¶”ÝBÒJ˜èEa§>†"!À;‘$7 /Rg®z97B#©ù ”]„Šˆ.ß UØRvÉ= ÐNa§- @ syQ©×þs· ßtµR?¼ªÛ3º5¶Þ`– ÿ¾¬”úk¥^ûëüóµÊ›ŸÿàŸ?ôcJ­ø™ÆÚp ;¯>§Ô_ýY·šú¡¿©ÔÊ¿ÛíÝ:°­?þƒCêÍ«oR«úƒêôéÓjÍš0—éHÍCvº—d’OH-½“*l)»IǬNó%!½/ !¿@6|!ùοý=uáÿú·êïnÙ¬”zM©7½¦^û«7«¿ºüã­ÕxîÉo*ýçêÝwLÚ¾Pb¸ IDATý-ßUë~æ'[Ÿ[Ö ‘/$¹~Ñ ×wÿ×'3]>o^­Ô[þë.O¼Þ[o0K†áw¬ Se‚4™H½fýý÷•ºúo+µêw¯ ¿­Ô_¼Ôí¹7ÿm¥Vck †ÖûEm,{™¨Nµáv2føÿ]ü¿Õåú)„nÿ¢u줖ÞI ,­v/|]©¿ø»¥mæ/ZØš‚ž™!_Œsgø×ÿùoª¿¼ÔþÛŸª°sýOžW׬üó@ï2u˜{*ƨåïRÆ_ø[~jjžš¯û?(µ‘]m8ÑŸñ8°³´Ôí 2­“ €°SYz‡°ÓIáwyP†L¨sŸP#ìTÞ²Œ¿h1¡fBíþ ã|îã<Â)ã|í¤ );)†qaa' ¡¦«“;;ÃÿÏ4äJ$¾ðçþ…a‡/üÃQŒ¹ÒB×áɈ´ü"‰w™w¹æÿ¼_¼_u_.ü¶BØAØ骙$Ñž3vv†Ÿ4!ìðEk¸/Z;;ƒQœ{c}áçì eõ¥kãû¢”úJ½éoøýóª·^qX¨×uç v†q½QR³žÛÆY…o0Lå@ãg¾ö—þãÓ›®Jb>ÚÉÖ£9FrHÊ.‡'”Pº}€Ta·Úå _Hê^R¾h ôE«þ ɹ¯]}ûû-µñÎÿA½öÚ›”zMO¬ü>Õ3vôSZ¸æ@ 4/a'´S؃ ¸ç¡#åag$°t¹°C‚ 1\w.†Ã€À€v„IW€Àè¤vpŒXÖ_ô䘔]„‘J·;Ô à&€°Ce@9ð:c'‡@‰È‚€Ô™«Rð¤)»;R•Vˆ]©Â–²[HZ ½ ìôÂÇÀ @ 3„Ê™« ú= 5EØñËOò­¤–ÞI¶”Ýä … €ÂNȘ€ @„„^ˆ H½ÈR‹”Ý J… 0:„Ñc€ ,# õ‹~©4HÍ¥ì²bGªÒÛEØ £ÐN+æ $„$Ó†Ó€ @ R‹”]„dJ³Ÿ£;;ý*ˆ§!0„áXÒ G€Ã“åØc€@)EÊ.ÂN[Edò÷RKï¤ [Ên&åB•ÂΨxéD€ëÎÆ  HÍ¥ì"ìÌP$<âO@ª°¥ìú“¡%Ê%€°Snî‰9@ØÉ)›ÄäF@j>(ea'· Ž,©Â–²~Ü@”v¢L NA ìtFs@@”€Ô© ¥æƒRvv¤*­»R…-e·´&z@Øé…‡!HpÆN$‰À @À‹€Ô™«^ÎÐHj>(eag„"¢Ë7H¶”]r´@ØigD @€ 0$„0—éHÍCv†|["îKjéTaKÙ¸p Ñ@؉&8@€@!vv )õ¼Ã”z‘¥)»yWÑA`;Ãp¤@€ àK@êý¾þ ÝNj>(e—;CWP¤ý!ì„Qh#M?nA *;Q¥g @€@v¤)»;Ù•°; „„BJ0 €°“@’ph%ÀáÉ­ˆh@@Œ€”À"eaG¬Ô–Zz'UØRvÃfkH“ÂNšyÃk@`9®;§" ÄK@j>(ea'ÞZÌÂ3©Â–²›EÒ#@Ø0ÝCA ìÁŒ@3šJÙEØ™©LxÈ—€TaKÙõåB;”La§äì;ò!€°“O.‰%ÚÁ!ÅVj>(eaGªÒ ±+UØRv I+aB „^øxˆ„gìD’Ü€¼H¹êåܤæƒRvvF("º|ƒ€TaKÙ%÷€@;„vF´€ @C@Ø s™ŽÔ<agÈ·%⾤–ÞI¶”݈K×  „hR#€ Baa§RÏ;L©YJ`‘²›w†!€°3 Gz @¾¤~ÑïëßÐí¤æƒRvY±3tEÚÂN…6Òôã¢"€°U:p€ dG@J`‘²‹°“] »BØAØ)¤Ô 3; $ !VžÜŠˆ€ÄH ,RvvÄJ-¬a©¥wR…-e7lV±4 줙7¼†–àºs*€@¼¤æƒRvvâ­Å,<“*l)»Y$ 02„‘Ó= „ÂNÌ 0©ù ”]„™Ê„‡| H¶”]_.´ƒ@ÉvJÎ>±C ;ùä’H P©Rl¥æƒRvv¤*­»R…-e·´&z@Øé…‡!HpÆN$‰À @À‹€Ô™«^ÎÐHj>(eag„"¢Ë7H¶”]r´@ØigD @€ 0$„0—éHÍCv†|["îKjéTaKÙ¸p Ñ@؉&8@€@!vv )õ¼Ã”z‘¥)»yWÑA`;Ãp¤@€ àK@êý¾þ ÝNj>(e—;CWP¤ý!ì„Qh#M?nA *;Q¥g @€@v¤)»;Ù•°; „„BJ0 €°“@’ph%ÀáÉ­ˆh@@Œ€”À"eaG¬Ô–Zz'UØRvÃfkH“ÂNšyÃk@`9®;§" ÄK@j>(ea'ÞZÌÂ3©Â–²›EÒ#@Ø0ÝCA ìÁŒ@3šJÙEØ™©LxÈ—€TaKÙõåB;”La§äì;ò!€°“O.‰%ÚÁ!ÅVj>(eaGªÒ ±+UØRv I+aB „^ø¢øòåËjÏž=êïx‡Ú»w¯zꩧÔîÝ»ÕÑ£GÕÚµk£÷(Ÿ{î9õ¿ñêSŸú”Z±bEk·šÓ¦M›Ôúõë¬Ìß×utæ̵qãÆÉ_kÛÛ¶mS÷ÜsÚ¾}{«mÌF€3vfãÆS€€ ©3We¢UJj>(eaGªÒ ±+UØRv I+aB „^ø¢açõ8p@½øâ‹êàÁƒ^ÂŽn¯Å™—_~Ymݺu"ŠÙ#ìØŽùûGyDíرC9rd"ä ìDÿšà  àvÂ\¦#5EØ þJÉ”Zz'UØRve²‹U¤Ea'­|uõa§»°sñâʼn sÛm·©çŸ~²ÊI‹5«V­šâov sÝX IçÏŸgÅN×Â¥= Ì ì ìd^âe„'õ"K ,Rv˨&¢„@?;ýøý´ ><1åÚ¤W—ìÛ·oêÊæÍ›§BÄ,ÂNu›‘m³ÚŸ1ê:ÌJ•³gÏNšéžyæ™é¶0ý3#¢œ8qbÒfÿþý“Ú«kª ôßWWÊÔù|íµ×N¶¢~®g«9Ô"Îý÷ß?Ù‚¥?zÕƒ>8ÝZ¥Ö$ìè¿×91‚Ð… vÆ~Qè€@b¤~Ñ/…Ij>(e—;R•Ø.ÂN…6pZ1$ ìÄ›6#z¬Y³fº…Èô9Õ-FÕÕ":º.gì¸ móᇞˆE+W®\Ö_°cú1Û‘lqF‹7zk“~ì­NF¤Ú¹sç$æW_}u²zÆfPõ±Íg½ÒÆw+V?§½‹;ñ¾7x@ñX¤ì"ìÄWƒ£x„°ƒ°3JaÑ)f €°3´@Ø‚ŠÙd„‡›nºIÝx㵫IÌÉfÅŠïáÉ.›v¸>+vn¸á†‰øSCªBŽ½ªÅÄW³Ž?>]=cöl?ûØcM…'{»”í·¯°c|´WèTÅ4ݯÏ;fUgì„ya8<9 g¬@˜…€”À"eag–*Ið©¥wR…-e7ÁÒÀe'€°¹·A_1BwX·e««°coŸ2«mº ;ëÖ­›žQcßeD}[ÔÝwßí\ùc¯˜ùÌg>£î½÷Þ+"#¬ñJÿ·Þ.¥·|¹|ÖïËÒ%6¹V5ÝŠUÝ.‡°ã]ò½rÝy/|< @`TRóA)»;£–K¶”]2´@Øig$Ñ¢º%¨îŠnû|s¶Î³Ï>;½Ò¼«°ã‰ôϪ[ªÌ ÃÆ^Á²zõjç™2öjŸ]»vMÄ-òToœ2"Œvìóqì\¸Îþ±ÛÚ"°S=栗wû좶3vìgv¼A;a8c€À,¤æƒRvvf©žñ& UØRv½ÁÐ@؉3ù>ÂN¸ >ëV, #i¡ä–[ni=cgè;UÉ'c¶ÏZ@òv\[®\•£v|²¶ ÂNXÞXƒúÚÁÑÏëÙŸ–šJÙEØ™½VxÒƒ€TaKÙõ@BOa'Þ¨;ïƈïz×»Ô¯ýÚ¯Mnoªž?sìرÉÏgY±S%â³…JûºcÇŽÉmUCŸ±cn¶ÏÏi; ÈöY¯jvÚ„´êÙ?úv¯M›6]q;—«šX±æãŒ0œ± C@êÌÕa¼ïÞ‹Ô|PÊ.ÂN÷ቤ [Ên44…@±vâM½ëV,{¥ˆö\‹ ö–#ó÷f›RWaÇç àjû\s`pÓ­XÆߦ[±ÌÍY.U¡Ä×g—@d²ï#¾hqÈfú s„xß<ƒ ;„0—éHÍCvbòOjéTaKÙ(]t¬ ìÄÞºC‘Í ê!¾úšðÛo¿]-,,¨{î¹çŠ­Sö6-{•MÁ¬¾±fó³êÙ>zUŒÞþeß&e >ú9}þ^é¢oô2‡*WϵтŽù˜³wª ôßWýió¹íPhסÉÕÊ°…¨›o¾a'îWï DMaa'êÅ9?R/²”À"e×/´‚@ÙvÊÎÈ舣}Nëc_çnߨÒOl…#ð¹Ï}N½ÿýïWkÖ¬ gK€" õ‹~©Ð¥æƒRvY±#Uií"ì„Qh§sH’ÂN’i‹Úéºójª«b\«dš0Ž:hœ›‰€žØœ>}z²Âlqqg&Š<@ ~R‹”]„økrvv)$:Àv€HWp]®·‰ÂN‚€Ë€@1¤æƒRvvŠ)m™@¥ [Ê® e¬B -;iå o!:aga'§Ê äO@j‡Y©ù ”]„©J+Ä®TaKÙ-$­„ ^†v´3f‚ÕË1† P!ðÀ?ùUõ÷ÿâÏ{qÙüµÕëy† 0©3W‡ð}–>¤æƒRvvf©žñ& UØRv½ÁÐZØÑ×?ôÐC%t@À—€+^xá…ÆææÜùùyõñ¸A]ü×'Ô;7ÝäkbÚî»ÿñ¼ºô a§3:€F €°æ2©y(ÂÎ/MŒ]J-½“*l)»1æŸ ¡…=¾ñ àC i+–-è,--M®Bb¼BØñÉ m ± ì ìŒ]cô€€Ô‹,%°HÙ JL@ yCL”®Þ}Ç”ÂNò%AFÀ%ì¸ãÐãÂN°ôbh õ‹~©¤HÍ¥ì²bGªÒÛEØ £ÐN+æ $!&J;I¦§! NÀvš!…µ÷œÆ­Wñ ñ H ,RvvƯ©(, ì ìDQˆ8¥ÙÚ€°C)A³°…}†ŽÙrU×Bô,”y€€<)EÊ.ÂŽ|Íñ@jéTaKÙ ’LŒ@ qL”O îC aúû^5Ó&蘯N6®CEšJÙEØ)ºÜÇ^ª°¥ìŽO HŸ¥ôsHH•À¥K—Ô5×\ãí>ã•7*BˆŠ€Ô|PÊ.ÂNTå—Ÿ3R…-e7¿ †'ÀDix¦ôŒC€ñj®ô „' µƒ#|¤¯[”šJÙEØ‘ª´BìJ¶”ÝBÒJ˜èE€‰R/|< $Àx6¦ Q H¹:jP KÍ¥ì"ìHUZ!v¥ [Ên!i%Lô"ÀD©>†`¼ S€À¨vÂ\¦#5EØõõ‰§s©¥wR…-e7žŒã â%ÀD)ÞÜà °œãäBaa'—Z.:©YJ`‘²[t‘<< 0QòE3@@œã•x qàâÅ‹jûöíêĉêÌ™3jãƃô«;£ïçž{NmÛ¶mâãÑ£GÕÚµkó7TGPûöíSû÷ïW{÷îÅì#<¢žþùeýv[·nü\·Ù±cÇày% ‘;•úEÿÈaÕv/5”²ËŠ©J la'ŒB8­˜ƒ@’˜(%™6œ†@‘¯òHûâ‹!3Fß;íuW'™ŸÏ°|ûÛß>yV­ZÕÞ9-² %°HÙEØÉ¢lÛƒ@ØAØi¯Z@ &Ja8cèO€ñª?ÃzC|3.„vº.aÇ%â\¾|YíÙ³G>|˜U;íX³j!%°HÙEØɪ|냑Zz'UØRv )'„@/L”záãa@ Æ«fض`bZîܹS¡C 1ŒWõ «®±E™º§ª«@ªâ€½bÄ%U'ﶰS½íÊžøïÚµë aÇguˆÙ*ÖT¶MÂNÓV#½%èøñã“­CÕvU!â駟žnOsùR'^™¶MÛªªý5åD·­ãZ]%S×Î忉ÿ±Çsòpm¥ª ;MâMjg,5LJ¹:VÓ¥*Pa§z.QU¤Ð[Œôy5uŸ.ÂŽ}†‘«¿ª°ÓtöŽK0s‰I·Þzë²3’ªv«ÂN•ÂÎ0ƒ)ÂN˜Ët¤æ¡;ü'Ñ÷"µôNª°¥ìF_80QŠ ¸x`¼òÂ4iT=Ô¶*6˜Éºnkn)jº ;¾+FÌÊž¡Ôí#ì<óÌ3Wl±Ò|êV츦6²ýhbaBn;o¦í:ñ¦-VÚ§¶S®ÊªË ÂŽÿ{ØÔaag˜J¢QR/²”À"eW4ɇ@"˜(%’(Ü„ãU}¸¶JUÏϱWj¸ÛRØ™õŒaÕ7ý3û6ª¦W¡°ÓõŒ&_»ž±£;6+aGßÞeÚùž±ã: ¹zÆÎ,+v8c§}À–úE»gã´šJÙeÅÎ8u]¯;aÚèCˆ¥“‚K€€“ãU}aÔ];­Ÿ0çí\{íµÎŒL¯C ;.O@ðꫯ^qÆN“ÿCÝŠÕE pùïÃÑøÚv+–ÏWM¢‹ñ¯îl›¶[±ª×«›þ\fÏÂí7ó7Õïþîïr+cù”€”À"ea§âGØAØ)¤Ô 3L”H.BŒWí…P=«¦zˆrUpЫDÖ¬Y³l Ò¡C‡&gÈô=cçäÉ“Ó³hlq i5GÝÎ푿ޢϊ}»“þØ>è•/z%ÌîÝ»'g¯jòµêG Wõð誘â»bG÷W=kIçøù矿"§ÕvÕZé²ËUWÚ}uU˜«Þ¾å›[Ú¥M@J`‘²‹°“v½z{/µôNª°¥ìz'„†(˜¥‚“OèHŒãUÜ ã¶£7ò‹7¦·¿ýíJ‹9«V­Z&œUoN‹»º•úÜç>§Þÿþ÷O„P>Ý HÍ¥ì"ìt¯žè@@ª°¥ìv@CSK€‰R±©'p$G€ñ*î”)fÔmªñ½=kl’c²Û÷!û7+Œˆc¸h¶Ø3¤Í±úÒ¿˜?}ú´ZXXP‹‹‹<AKÍ¥ì"ìt,šw# UØRv»Ñ¡5Ê$ÀD©Ì¼5R$ÀxwÖÆ3vâÎ}wf»×Ö­['[ÚÌ6¬X¸.T°sÕUW©ïÿû½©]b²­Ô|PÊ.ÂÎÕC_W*l)»” ÐN€‰R;#Z@q`¼Š#x aÇÄÞWà‘:sU*wRóA)»;R•Vˆ]©Â–²[HZ ½0Qê…‡!€¯ÂÆ °Œ@UØé+ð 섹LGjŠ°SÈ"µôNª°¥ìRN„ ^˜(õÂÇÀ@@ŒWac ðvfxvvxÅ2 õ"K ,Rv3(B€Àè˜(Ž€À@†¯¶üËßȺ POàò«êïÿÅŸ÷B´ùkÿª×ó1=,5”²ËŠ˜ªoD_vÂ(´#¦®! !&JWï¾cÊC¯Hä@` CŒWÿâGß:uk‹ÇÈ}B O=ôzá…ƒ3çîÌÏÏ«ÿà êâ¿>¡Þ¹é¦Î@¾ûÏ«K/¼ v:£»â„þ é¡ÂÂ/b!0ÄD a'–lâò&Àx•w~‰1¨;cGûl :KKK“«Ð‡¯vúWÂN†ôÐ@€3vvxA  !¾x ìÄ’Mü€@Þ¯òÎ/ÑA f.aÇ%蘆¯vúWÂN†ô!©Â–²a p Ñâ‹ÂNtiÅ!dI€ñ*Ë´’ ` ;M‚ÎÂÎÚûNÙ¤¾uTj>(e—3v’x­ÓuRª°¥ì¦›)<‡@8L”±Æ ÐãU?~< ÌNÀvô:fËU]ŒWËÉHÍ¥ì"ìÌþ®ñ¤©Â–²ë„&(ž_<Š/@ ŒWɤ G!-ìèU3m‚Ž œñ açÔ©SêMJ©×'…ÃCX¤ìÅ~ 3¾xäœ]bƒ@^¯òÊ'Ñ@ %—.]R×\s·ËŒW;;Þ¯Kº 9<™Ã“Ó­^<Ï_œ‡@Q¯vvŠzåÃ+%°HÙ KkH“_<ÒÌ^C DŒW%f˜!&Æ+„„4ßÝ$¼–X¤ì&‘œ„€0¾x'ó€€7Æ+oT4„„ B@)ö IDAT0^!ì ì¿„!Ìsx2‡'‡¨3l@À‡_<|(шãU YÀ@À‡ãÂÂŽÏ›’x®;GØI¼„q?#|ñÈ(™„Ì 0^ež`ƒ@F¯vv2z¡ëBAØAØ) Ì 1|ñH$Q¸ (Æ+ŠH…ãÂÂN*ok?vvz”B`P|ñ'A#`¼.]Cƒ`¼BØAØô•Š³3ÎØA؉³2ñªD|ñ(1ëÄ 4 0^¥™7¼†@‰¯vvJ|óÅ,u;•”Ý@X1¤ ðÅ#éôá<Š"ÀxUTº I`¼BØAØIúŽÛy)EÊnÜÙÀ;ÄA€/qä/ vŒWíŒhÄA€ñ aa'Žw1K/¤)»Y&‘  00¾x ”î Ñ0^†–Ž! 0^!ì ì üRÑݤ)»äh'ÀvF´€â ÀxGðh'Àx…°ƒ°Óþž$߂Ó9<9ù"&€lðÅ#›T²'Àx•}Š Ù`¼BØAØÉæu®„ëÎv (sBL„_<InBŠñŠ"€R!Àx…°ƒ°“ÊÛÚÃO„„åã”_<ÅIg€Àˆ¯F„K×€À ¯vv}¥âì aa'ÎÊÄ« ðţĬ3Ò$Àx•fÞð%`¼BØAØ)àÍ猄ʜ!ÀD…›€[±¨@ |¿BØAØIæuMÏQ©Û©¤ì¦—!<†@x|ñÏ‹€Àl¯fãÆS€@xŒW;;áß»b,J ,Rv‹I,B ¾xô€Ç£€@PŒWAqc èA€ñ aa§Ç ģͤ)»Ô ÐN€/íŒhÄA€ñ*Ž<à ÐN€ñ aa§ý=¡ÅŒ¤)»3bâ1E€/E¥›`!4Æ«¤Ó‡ó(ŠãÂÂN¯<‡'sxreNˆ‰à‹G"‰ÂM@€Ã“©@ |¿BØAØIæuÝQ®;GØ™½zxÃà‹Ç°<é ãÕxlé–ãÂÂΰïT”½!ì ìDY˜8U$¾x™v‚†@’¯’LNC HŒW;;¼ú;;”9!&B€/‰$ 7!¶bQ€@2ø~…°ƒ°“Ìë:»£œ±ƒ°3{õð$†%ÀayÒ 0Æ«ñ؆èùòåËjqqQ}ä#Qk×® a²80Ž'åŒW;;ñ¼Ùy"u;•”ÝìH@_'Nœ˜üÑýÞzë­û>øàÄýsýÑýîÝ»·”nûâ‹/ªn¸aÒþ¬_¿þŠUDÏ=÷œÚ¶m›:{öì´¯#GŽLýlâPµÚ“.\XÆа5NtelX¸ø›¿ÓÛ·\ŒMîä«.M¯vvÒ|w“ðZJ`‘²›DRpÂøâ!œÌCÞ¯¼QÖÐ ¶h£áÀF,Ðg‚êj’‹/z ;‡ž HºO㇘´íU«VMf 0UZTÙ·oßDÔ±E'#$iɈ:Z4Ò‚•m¯*b¹8<üðÃSŸô³Ú¦ö×øi|0"”a¥clòkÆÆþ±cǦâ•á®cÓâŽm¿*$VHtÌx…°ƒ°SÀ‹ÎáÉž\@™b"øâ‘H¢pàV¬jÀ*¶P Œ›F ¹çž{&NaLj\B‹þ™(ÞñŽwÔ®Ú©Š,®çÌŠ{ÅJµï&ÕHU¡È僽JÉ,m?·nݺ C#Æø2®æÀ+vfal Vú"×Çg¥S¥—œ |¿BØAØIîµíî0gì ìt¯ž€À8øâq%W=ù8yòdãÍ*ãd£œ^Kf\·Ja–ì×m³0}ùLêf±+õ ã•ù7캄ºš®þÜWØÑ+ZÌa¿æV¬!WìØëÈlaÃÜÔUw–ÍÖ­['ÿoè²%­zèñ,+vÌ;Õó{Ú»‚Ö1Û?7gì°kØ÷‹ñ aagØwŠÞ,R·SIÙ%ù€@;¾x,gÄo.Ûk¦o‹Ò#ìÌ^AŒW³³êɶ-Hö0Õsk\"Cµ9¯ÇlcCØÑ·bÙ×¢»„¤sçÎMo±²oj:cÇÄgÄ[@Ñצ÷9cG MUH÷ïËؾ«zc™î‡3v†zCÞè‡ñ aagø÷Š@@J`‘²Kâ!v|ñ@Øi¯’a[ 츯xž…2+vü©{ò›Jÿ¹z÷Ó‡ô jŸÏW¾òõîw¿[­Y³Æ§yÖmš 6¢ŒP½ Üu­·ôUßú£ŸÑˆ¾IëèÑ£j aGoƒzË[Þ¢ôçúc_™®ÿÛöÓÄ¢E½’Ó¬öyöÙg•kk”w´cúþüç?¯>ùÉONo´šeÅNõÚõ.ŒW¬X1…Ìs¶°Uú˜<Ö Ë÷+„„±Þ.úUR‹”]R´Èñ‹‡ù’ª¯Œ=qâÄä™`T'ú¿ÍoQÍJ Ý^ÌoŒŸ~úé顚挂êõ¾ú˾þ²®'wÝu×äùo|ãêÑGÜ4rÝuש;v,ë×\i[Í’™|êSŸR_úÒ—&þëýEÜip‰*iG„÷±=^ÅÊÁø%5”²«ÏÔE؉½*öOª°¥ì&œ*\‡@09~ñ°'ÉöÄßLVõoÝQ½¦×õ›K×9%.aG_Ek ¶úæ-¤˜ŸÙËüë„{é~Õž&ì8ë–×kÅÅÁ>«¡zè§ùomLjöoã«b˜feìÚÂP]bì»D›auk‚+§ÕÉ]•-|™XíßÄWó©íëš:þüô66F†fvÝT·¬Ô ;ZP´¿jÍ»®yö¹Ê9Ø e 5^Ù‚Ž^©£ßÕ=\°éOaÇŸ-û5^õó2ÜÓRóA)»;ájKÔ‡'sx²hb¥Ð«“t½ ]궭ØÛ•+W^qÖ@aÇPêüh›4×ݬb?§ã±¯Ï5|í6f ƒlæZ_û–{Âc ½B Rúã—ªÓñãÇ'Ûª+Hڻą*?½rF‹gu«t´/>[ êµÙ­^½z·½’¥©®ª×$ÛÜ´_®¾°ãS—zÛ‡+vŸ­Xuù0¢ ‹k¬“ß±'J:!Gj[±Ö¶ ¬ŽI`ìñjLßÇè[J`‘²‹°3FEØ'×#ìDX–¸T(¿x¸&÷u“òê¤Ûužƒ¯°S=o¡Ndðv|ðÁ‰°`>u‚OuOÛ¡£v©Wσ0«aŒ°cûÐ&ì|æ3ŸQ÷Þ{ï¤{{kVãº~gD“º3-ª¯®9`Ôp¨Ûòf AfÛ˜yÆ;öª¦.ÂNSî¶lÙ2tßz«—ÞÖÖÆ̵°Êµi+–fb‹x.Ð܈ë6,ïXã‚N¡ÿ$lŒH`¬ñjD—GíZJ`‘²‹°3j9ÅÓ9ÂÂN<Õˆ'¥Èñ‹G“°cŸûbç¾I CØiZy¢Å}Ц>pÓž¬W'Üö1f«-~Ô:Z=ËÆœÝãZ±3‹°Ó•qÕŸê;銭I¬ÑÏW«ªÈÓÆ®°Ó–;—°ÓÆLŸÕT½¢ÙÄé²gêÚ}òÉ'…6ÍÒ®q_Mbìz¼zá…¦gè°åJ"£Ø„@¾†¯Rß*%°HÙEØÉ÷Ý^ÂÂN!¥N˜ Èñ‹G“°cV%Ô¥¦Ï;]Vì´ ;®mVöŠ}N«¦•&f׶'ýwC ;]û®i[éT—SÓÿK/½4Ù"váÂ…Vv}„¶Ü¹„6f.qq¨;Zر·§é›ì³›bƆ¯ô¡á=ôÒ5 S†ñùj¼2Dvf« „Ù¸ñ”'ÎØAØñ,šA`t9~ñ¨ÛU79¶î{ÆNõ°_—èдËGØi:§EoÍ©;;åرcµ××m'ªà;ËV,½ýJŸ±ã:¿§qݹöÏõY0®ë…}^ûÀ`}q»¾ÂN×3vÚ˜yÆŽ>{É® Ígï޽˶ú0ÑfˆñjáÌÿ9tôu×ßûÞ÷&n?ñÄÇ|NŸ>}ÅaÉ´®jã=Ë^uÞ‹ú÷bÇÏ®SúÏ{óðxîÉo*ýçêÝwLŸEØéŒqòÂÎlÜx*rR…-e7òtࢠ0ÄD)¶/u‚Šëö!×¹5U‘¡ÚÆ>ÓÆlcCØÑb!v IZ2·3é¶Õ[’ê¶béøŒøc¶4™­I}ÎØÑÂŽ¹Mʾy¬ ã­[·ND;sX²ÏŠ:áÊ^¥dD’&v}…¦ÜUW(u©K{;–ëæ²ê bβo3?«BmêǾA+ŠAÊrbˆñêÔO­¬Öyùå—•®S³bGÿ»ùhqK·±? “ö´Q“ëßáóFuP¼ºªïÅOþ§—ÔßùÎË;?xU¤æƒRvÙŠÛ7ˆÌü‘*l)»™¥p 0 !&J©;`õ aý³º ®ýwöUßúç?þøD1[hÆvvíÚ¥:¤ôµçúc ¶ða COÞo¿ýv¥'zň9˜×µÍÇ>cÆô­ÿiV²è¯n)j;<Ù˜<+ãêY;Õsq|„ºWÅŠj>«ììíRæV°.‡'7åεõ̇™ŽÍΛæcìTo!35aVJéøïºë®Éë•n:Äz”Ág†N‡¯{ì1õÕ¯~Uésvôo¹Ò|†œð à&0äx¥-°bg¶J“š‡"ìÌ–/žò$ UØRv=±Ð Eà‹G|鯻Ž;>Oñ¨J åÜ¥àûã•^y²´´„ÀÃë  JÀŒW}:íg}b‘šJÙEØéS-<ÛJ@ª°¥ì¶¡ 0ÚõÁ @ ìÙ£ËûÉ”sWwƒVLCØ1ñ!ðÄ”i|@ú^ùÖ*ýG.]º4ùÓõsÕ/l˜>Š®ô^o/5EØ™-_É=ÅáÉžœ\Ñâp¶Æœ(e mäÀRFF}÷)æÎœ¹óÙ:&ñ!Æ+žè_3„@rôù;úý¹óÎ;•Þö«ÿø|v|(]Ùag6n<åI€ëÎv‡GOÂø@èJaGnKÂN×j¥}';; †Æ‘€ÄDiÄpèȘãUÆÉ%4dLaagrJ?ŸüpÆÂN~UMD©`¢”jæðå`¼*/çD \ HÍ¥xJ­œ‘²ËáÉR•Vˆ]©Â–²[HZ ½0Qê…‡!€¯ÂÆ  HÍ¥ì"ì Xtß/^TÛ·oW'NœPgÎœQ7nÌ×P™°´½£Gªµk×nú©§žR?üðù(eagÈꉸ/©Ã²¤ [ÊnÄ%€kˆ†¥öT˜ ¶Ý²º}È*L»ýû÷«½{÷NþÓÞ¶óàƒ*=9×ÛôǬ²0vý3{‚nOð·nݪÞûÞ÷N]±ý¨̤ßåW{ô¯·¨öýꫯ.ÛJuòäÉÉŠýY¿~ý[“l†úïmv UŽ›7ožØ^µjÕ¤oÃH³½îºëÔŽ;jm^¾|YíÙ³G>|ØÙÆŽéöÛoŸ\e}öìÙI[;w†‘í—ÎíîÝ»'moŪ֋½Š¦*hÕq³mûºŸ-[¶L¸¿ôÒKËlæ ”ùÔ"ã•%Ú@)šJ±‘šJÙEØ‘ª´Àv¹îœ[±—æ PK€‰Rsq¸Dªxáuª"ŠvŒp`[ÕBÇÛÞö6õøã/sÆu>Ø~=Ùw ;uÏv]ÙÑ$ì¸Ú‚L›ÿFØ©cd E.ÁöoújʉiÓæ—-È4ÙõñÏEM~é8 ·C‡M…2;Ïú½2§*xÙ"VÎÛ±¯øŸ ©ù ?)EÊ.ÂŽT¥¶+õ"K¶”ÝÀiÅ’$ÀD©9mö ½BÞ@›ÉºìUæ9# œ?^mÛ¶m²"Ä5É7õ•+WNW™˜g?>]™b&í¶æÙÇ{lÒ®jSGhV”ØÏu9´¸IرE{uîݺuÓ•=.>Ú7Ýî†n˜Æm U¶¶ÀâqLì<ðÀD±mV·+µquåÎÅß;.\¸Bt1BŽYac:Ö«µš¸é䛄:—0W­Õ$¤§¯rÌ*1A LRóA)ÚRóA)»;R•خԋ,UØRv§sH’¥æ´UWu¸¶èØ=TÛ»Ä#H¸D"ÝWuBoˆêd¾z+ÓÓO?½LØyæ™g&BCݧ-W\Æ{+VÓV#݇ö¡º=«ºuÈ€\þVW²T}7¢n÷å/Y}úÓŸžnÁªög|1¼ª«_êøWÛÕñ¯ãmo¥ÒÂN·:a§I¼I錥YKÆ«YÉñ ©ù ©ù ”]„©J lWjO¥TaKÙ œVÌA IL”ÚÓV· ÇLÌ›¶Yù®žq­R©®Øé*ìØ+R\Q%ìÔ‘£n„ª(RvôJ³¢©¯°ó…/|A}⟘žaÔ&ìT¹Î*¬=úè£WlŸ²mW…&n;î÷’ñª}¼¢ ©ù ©ù ”]„©J+Ä®TaKÙ-$­„ ^˜(uÃgo5ª®±Å‹º­XÚšk[”°ã»bÄfÅNõ¹n¿Þºíðd-DèO`cÇíjg¯ØiÚ"VåºbÅŠ‰Ýº;MçÍÔ­p©þ¼ŽcÛŠ)ã›Í»îc×Ï]þ±bç«ê?|í¨zϽ÷u.ãsO~Sé?Wï¾cú¬žXñ ñ HÍ¥ì"ìŒ_SE[*l)»E'›à!àI@JØùÊW¾¢Þýîw+}l¬Ÿº­RFDÐ[z~ý×]---MV‡Áž¤¹bGsšåŒ}®Ë·¡ÎØiZy2Ë;ö šªÑõŒ×!Îm+¡ª‚Š}>’Ï;:OÕ¡°eŸ±3ËŠ¦íVœ±Ó<’ ìÄ:Òâ P©ù ”]„ªZ0F©Â–²+ˆÓH†@haG :Z9wîœZ\\œü{ÌŸêUᶯf+SÓIC ;.VÕ[ž\ÂHõ¹!oÅò(êòlž¯císë”îÛn×´=®‰—î§i¥Œ+c÷Úk¯]v½º«VfY±cúÑõvóÍ7s++vb2ñ €€“€Ô|PÊ.Â/¨¤ [Êî¨0é™%ìØ‚Ž^©£T¶A¸Dû|{e. ýw·ÞzëäÌýÑ[¯ôÇþïµk×.»aËg+–cn¿ývµ°°0¹]KlQ¥mk‘)Ù®¢ŽKì°Onvìçóàƒ*-ˆé•NöóUÖÕC—íU)F䨊:&N×Õâ]xU9Ùž^ ¤oIÛ½{÷4Ç:§úSúìZé"ì¸êj×®]“[ÆÌ-[Æfõö-óóL†©i¡Æ«Ü¸ iRóA)»;ÒȾÔaYR…-e7P:1¤ Œ=QJYЉ%±%Üvä˺„íFm, [ 4‚؆ ÔÁƒ•ë|Ÿ¶~Søû±Ç«à# ©ù =©ù ”]„©J lWêz;©Â–²8­˜ƒ@’Æš(!è Wc ;Õ!uqøòP4v”r‰8¦Fš‹*’ýŒ5^IÆ„m@ LRóA)ÚRóA)»;R•خԋ,UØRv§sH’ÀÐ¥^xaz†Nj[®bM ÂΙAØy…æpìرÉ6?s ³þ¹®•U«VÅZʽýz¼Je;hoptDG@j>(Bj>(eaGªÒÛ•z‘¥ [Ênà´bIj¢¤ÏJy衇&g è$Y 8 è 5^™@v¢O9B [RóA) RóA)»;R•خԞJ©Â–²8­˜ƒ@’úN”~ûøqõÙçþýDÐyË[Þ¢¾÷½ïM8<ñÄÇ|NŸ>}ÅaÉ´®jã=ËÆÞ‹ú÷bÇÏ®SúÏ{óxËuç‘ñ 0"©ùàˆ!5v-5”²‹°#Ui…Ø•*l)»…¤•0!Ћ@_aGO–NýÔÚÉj—_~Y­Y³fºbGÿ»ùèëÍuû£ow¢ÍëDàCm˜wƒ÷¢~ÜøÉÿô’ú;ßya§×¨ÏÀšJÙEØ _cEY”*l)»E%—`!0#!„«wß1±þØc©¯~õ«JŸ³£µ¸¸˜Ì•æ3âã1@ !Ç+í6[±&S€@ѤæƒRvvŠ.÷ñƒ—*l)»ãÅÒ'0ÆDI¯ÌYZZBàI¿<ˆQc¼Š*@œ )©ù ”]„L 9–°¤ [Ên,Üñ1s¢„Àsæñ és¼JCH‡€Ô|PÊ.ÂN:µÙËS©Ã²¤ [Ên¯$ñ0 !b¢„ÀSH1&F&b¼9º‡ 0! 5”Â/5”²‹°#UiíJ]o'UØRv§sH’@ȉO’%‚Óˆ†€¯êúƒï\Po[¹R½íGWÖúlÎ3«h‚Ã@ (RóA)ÈRóA)»;R•خԋ,UØRv§sH’@HaÇ2>‡GßÄ€€W¾õ‡JÿÑŸK—.MþØŸ_úgÿ³ºíÿ‘úð/ýbmwWý†éßqx²uÚ@cšŽ‹OŸRóA)»;>U‘A©Yª°¥ìfP*„Ñ H;£…@ {çÎSúý™››SóóóÞ‚1ÂNöeB€ˆ–€Ô|P ˆÔ|PÊ.ÂŽT¥¶+µ§Rª°¥ìN+æ $„$Ó†Ó(ž€KعóÎ;Õ–-[&|>;>”hŒA@j>8F,>}JÍ¥ì"ìøTmf& UØRvgŃ(ˆÂNAÉ&TdDÀ%ìt a§+1ÚC˜€Ô|PÊ.ÂÎluÂSž¤ [Ê®'šA h;E§Ÿà!,„dS‡ã€@¤æƒRvv ,ò!K¶”Ýl±T 줚9ü† @išJÙEØI£.“õRª°¥ì&›(‡@@;ac € H@j>(ea§"—:,Kª°¥ìRN„ ^vzáãa@ "]oÅŠÈu\ # 5”Â,5”²‹°#UiíJ]o'UØRv§sH’ÂN’iÃi@ÀA@¿Z\\TKKKðõÉ IDAT 5©ù ©ù ”]„©J lWêE–*l)»ÓŠ9$Ia'É´á4 €°C @ šJ!“šJÙEØ‘ª´Àv¥^d©Â–²8­˜ƒ@’v’LNC;Ô 0©ù 2©ù ”]„©J lWjO¥TaKÙ œVÌA I;I¦ §!©ïW$€@W¥WRóA)»;]ßÚw" UØRv;Á¡1 %€°Shâ € " 5”²‹°¨°J5#UØRvKÍ3qC  „.´h @€ Е€Ô|PÊ.ÂN× ¡}'R…-e·C P;…&ž°!@€@ RóA)»; «T3R…-e·Ô<7º@ØéB‹¶€ @] HÍ¥ì"ìt­DÛK–%UØRv-܆@P;Aqc ‘ÀÜÜœšŸŸW #Z¡k@ý HÍû{>[RóA)»;³ÕIrOI]o'UØRv“+ †€„蘄F! ¿_-..ª¥¥¥Qú§S@Cšå×~¤æƒRvvºVH¢í¥^d©Â–²›hyà6‚@Ø Šc€ÀˆvF„K×€À ¤æƒƒÑ¡3©ù ”]„Å‘rS©Yª°¥ì¦\#øPvB‘Æ 06„± Ó? 0©ùàPþwíGj>(ea§k…$Ú^jO¥TaKÙM´¼¬Ù‘#GÔöíÛ—ý¬êc/÷ÝwŸúèG?ªNœ81m²ÿ~µwïÞXJ? 0%€°C1@€ ŒIÀg>8†})»;cd“>§¤ ÛÇ®„°óÜsÏ©mÛ¶©³gÏ:«¤*Æ ìð2åHa'Ǭ @ˆ‡€Ï|p o¥ì"쌑MúDØ©©¡æÌ™3jãÆ“|Úë•B¬Øá¥K‰ÂNJÙÂW@€ )EÊ.ÂNz5š”ÇR…íc·n U×­U¾íí-XÕm[öJ{ÕŽoߦ(llÅJêU)ÊY„¢ÒM°€ @ 8ŸùàNIÙEØ#›ö)uX–TaûØ•vt‰Ø+sêJa'— —z@Øé HÌÍÍ©ùùyµ°°‰G¸@ÀM@j>(•Ÿùà¾IÙEØ#›ö)u½TaûØmvšÒèÚ.åsxòSO=¥6mÚtE×u«k|¶bÙ¾°b'——® €°CQ@¹àºó\2IÈŸ€Ô|PŠ¬Ï|p ߤì"쌑Íû”z‘¥ ÛÇ®„°£KCß¼µcÇg•TÅ!„_&\êMa§7B:€"!€°I"ph% 5lul¤>óÁ1LKÙEØ#›ö)õ"K¶])aÇ”‡Ïê„_&\êMa§7B:€"!€°I"ph% 5lul¤>óÁ1LKÙEØ#›ö)µ§Rª°}ì¶ ;>[«tª»žƒã*{ ÕæÍ›'«zV­ZÕ¹o¶bEøòáÒv( @ R߯ráG€@8¥W>óÁ1èKÙEØ#›ô9% UØ>vC ;ö #GŽ¨íÛ·/«ãÏúõëÕÑ£GÕÚµkvx—²$€°“eZ € DCÀg>8†³RvvÆÈ&}f!ì4¥Ñ%¾4µ×BΖ-[&bΉ'+Ä^)ä³Ëö…;¼|)@ØI!Kø@€Ò% %°HÙEØI·V“ð\ª°}춭ØZØÑ¢Nݹ:¶-×[¾¾ ì$ñZï$ÂNñ%@€ 0*ŸùàHÙEØ#›ô9% UØ>v%„ Æ_ìRqéÊ^¦ ìä˜Ub‚ @ñð™Žá­”]„1²aŸR‡eI¶Ý!=Ž-լ؉-#øã"€°C]@¹˜››Sóóójaa!—ˆÈ”€Ô|P §Ï|p ߤì"쌑Íû”ºÞNª°}ìVWÃØ·QE˜ÂF—.^¼xÅù=û÷ïW{÷îM-ü-€ÂNI&DB€ëÎ I4aB RóA)t>óÁ1|“²‹°3F6#ìSêE–*l»;*.Aa§ˆ4$Š €°SDš YšJÁó™Žá›”]„1²aŸR/²TaûØE؉°Pq©;E¤™ !P„"ÒLÈ‚€Ô|P žÏ|p ߤì"쌑Íû”ÚS)UØRv#L=.A :;Ñ¥‡  H}¿šÑ]ƒ &PÚx%5”²‹°SðË"t©Â–²‚)6 :„Ô3ˆÿ€ @ nRóA)»;q×còÞI¶”ÝäF@a'dL@€ ‚ HÍ¥ì"ì\ì!B—*lm— @€ ”I`õOPý­Ÿù:}ú´Z³fzç;ß9:©ù/ÂÎè©-Û€TaçÿùÚü¹sçÔ5×\3ùÃ'ðÊ·þP½òï¾¥Ölº©³3—ο .?¯®úïš>«ÿGÍ€ @6]õ³jåÿ=„Ê"R‡eI ;væB*´ùT ‘@`<lÅ-=Ca ÌÍÍ©ùùyµ°°Ö0Ö t$ 5ìèæ(ÍCΥ濬إtâëTêz;©ÂF؉¯ñ†ÂµäB€ëÎsÉ$q@ RóÁÈ"ìÄ|„€Ô‹‹°ÃV¬AʈN 0s¿óuuþw¾®6ìþDçþþè÷~OýÑ¿ù]õæÿÓé³6lèÜ@‚€Þ zçwNþð 3’Ç+}4gìÄ\øæM taÇ !Ñ ¼xê¤úã'Nªëw|¸³­—þà¬Ò~øœ>{ýõ×wî‡  A ä­ Cð£@ ÒÇ+„pµ†¥ H½È1¬Ø+]C3`+Ö Ðxˆ’€Ô÷«(aà 5Æ«0é‘šÿrÆN˜ükEª°‹NàH€ÂNIÂE@€ Τæ¿;SÅ]Hvi „%€°–7Ö @€Âšÿ"ì„Éo±V¤ »Xà ì$$\„ @èL@jþ‹°Ó9U<Ð…€Tawñ‘¶€@X;ayc € @ ©ù/ÂN˜üŠ[‘:,Kª°Åã PKa‡â€r!077§æççÕÂÂB.! )©ù`¦8kÃ’šÿ"ìRi%_w^HŠ É@ØI&U8 ´Ð߯ÕÒÒ¬ DM@j>5”œCØ*]¾A@êE–*lrÄKa'ÞÜà ÐÂN7^´†äHÍå"–±,5ÿeÅŽL¾ƒ[•z‘¥ ;8` BÞv¼QшœÂNä Â=@`J@j>XZ ¤æ¿;…TšÔžJ©Â.$­„ $ ì$™6œ†¤¾_‘ @] 0^u%6[{©ù/ÂÎlùâ)OR…íéÍ ;Ð1 @€ 0:©ù/ÂÎè©-Û€Ta—Mè!7„¸óƒw€ @³šÿ"ìÌ–/žò$ UØžîÑ  €°#“€ @£šÿ"쌞ڲ HvÙÔ‰q@؉;?x@€ 0©ù/ÂÎlùJî)©Ã²¤ ;¹á0 "€°SP² ™˜››Sóóójaa!óH H€Ô|0un]ý—šÿ"ìtÍT¢í¥®·“*ìDÓ„Û(‚ÂNi&HA€ë΋H3AB RóÁ,àuBjþ‹°Ó!I)7•z‘¥ ;å\á;r'€°“{†‰å@Ø)'×D Ô HÍSçÖÕ©ù/ÂN×L%Ú^êE–*ìDÓ„Û(‚ÂNi&HAa§ˆ4$² 5Ì^‡ ¤æ¿;’”rS©=•R…r®ð¹@ØÉ=ÃÄrH}¿*‡0‘BC`¼Šds?Ró_„0ù-ÖŠTa œÀ!„’„‹€ @ HÍv:§Šº*ì.>ÒKa',o¬A€ „! 5ÿEØ “ßb­Hv±À  @ØI I¸@€ Й€Ôüa§sªx  ©Âîâ#m!°vÂòÆ @€@Ró_„0ù·"uX–Ta‹Ç@ –ÂÅäB`nnNÍÏÏ«………\B"@ SRóÁLqÖ†%5ÿEØ)¤Ò¤®·“*ìBÒJ˜H’ÂN’iÃi@ÀA€ëÎ) @ RóÁTø å§Ôüag¨ FÞÔ‹,UØ‘§÷ P4„¢ÓOðÈŠÂNVé$dM@j>˜5TGpRó_„B*MêE–*ìBÒJ˜H’ÂN’iÃi@€;Ô 0©ù`ÂÈfr]jþ‹°3SºÒ{HjO¥Ta§—!<†@9vÊÉ5‘B wR߯rçJ|€Àð¯†gêêQjþ‹°&¿ÅZ‘*ìb8 €°“@’p€ @ 3©ù/ÂNçTñ@R…ÝÅGÚBa ì„å5@€ 0¤æ¿;aò[¬©Â.8C ; $ !@€:šÿ"ìtNt! UØ]|¤- –ÂNXÞXƒ @C@jþ‹°&¿âV¤Ë’*lqà8Ô@Ø¡8 \ÌÍÍ©ùùyµ°°KHÄdJ@j>˜)ÎÚ°¤æ¿;…TšÔõvR…]HZ I@ØI2m8 8èïW‹‹‹jii >€¢& 5ŒÊÎIÍvFHfŒ]J½ÈR…cð x•äBa'—Lò' 5ÌŸìò¥æ¿;…TšÔ‹,UØ…¤•0!$„$Ó†Ó€€ƒÂe¤B@j>˜ Ÿ¡ü”šÿ"ì •ÁÈû‘ÚS)UØ‘§÷ P4„¢ÓOðÈŠ€Ô÷«¬   „ãUÌJjþ‹°&¿ÅZ‘*ìb8 €°“@’p€ @ 3©ù/ÂNçTñ@R…ÝÅGÚBa ì„å5@€ 0¤æ¿;aò[¬©Â.8C ; $ !@€:šÿ"ìtNt!  {åÿ—jåÿ=uîÜ9uÍ5×LþðÊ%ðÊ·þP½òï¾¥Ölº©3„Kç_P—ΟWWýƒwMŸ]³fMç~x€ @C¸ðïÿ¥ZýÓTëg>¤NŸ>­ô÷Ôw¾óC›¹¢?„ÑÇa@ê°,-ìð @9XøŸN©÷ý£5ê¿ô_ä1A˜ÿg§Ôû‘ñ*DJvBP.ÔF ×Û…T, M3aC lÅJ"M8 xàºsH4¢ Ã|0 9ÿeÅNÀÄJšŠáEYØ’¬± 4@Ø¡B \ ìä’Iâ€@þb˜æOyy„!ç¿;…TW /rÈÂ.$­„ $ ì$™6œ†v( @ 1ÌSa5”Ÿ!ç¿;Ce-ò~¤Îر±èÂæ@àÅS'Õ?qR]¿ãÃa¼ôg•þóÃøàôÙ믿¾s?<@`wÞy§Ú²eËä@1`¼’ɇ'ËpÇêˆ^xá…{§k@ ç~çëêüï|]mØý‰Î.ÿÑïýžú£ó»êÍþ§Óg7lØй€ @€ÀØôÐo}ë[Ç6£X±3:b @€€M€­XÔ @€†#€°3Kz‚ ;h@€ O;ž h@Ã@؆#½@€ @@@Ø)¤b8<¹Ô„ ´@Ø¡D \ÌÍÍ©ùùyµ°°KHÄdJ€ù`¦‰ýAX;yçw×Û’h„@vH.B^¸îÜ  0Œ #º€°3"ܘºæEŽ)ø² 씢‡@NvrÊ&±@ oÌóÏï©S§Ô›”R¯-..ª¥¥¥¼#.4:^äBO؈ÂN„IÁ%@`&;3aã!@@€óAèM²b' lISì©”¤m@À&€°C=@¹àûU.™$äO€ñ*ï#ìä_¢ƒ „èR‚C€ @ @ØI8y¸@ E;)f Ÿ!@€b%€°kfð €@¦v2M,aA€ ˆ@ØÁŽQ@å@Ø)7÷D@€ 0<„á™FÙ#‡eE™œ‚@‘vŠL;AC Ksssj~~^-,,dAAù`>˜O.]‘ ìäßit\oWH¢  @ØI I¸xàºs/L4‚" À|0‚$ŒèÂΈpcêš9¦là Ê&€°Svþ‰9@ØÉ)›Ä¼ 0Ì?¿§NRoRJ½¶¸¸¨–––òŽ¸Ðèx‘ M¶!›ÂõäB€ïW¹d’8 ?Æ«¼sŒ°“w~‰€@tv¢K A€ $La'áäá:  줘5|† @ˆ•ÂN¬™Á/@™@ØÉ4±„@€ BaG;F!”Ka§ÜÜ9 @€Àðv†ge–eZp E@Ø)2í , ÌÍÍ©ùùyµ°°e|äC€ù`>¹tE‚°“w~§Ñq½]!‰&L$@a'$á" àE€ëν0ш€óÁ’0¢ ;#©k^䘲/(›ÂNÙù'zäDa'§l ò&À|0ÿüž:uJ½I)õÚââ¢ZZZÊ;âB£ãE.4ñ„  ìD˜\‚f"€°36‚0€Ð$+v–4ÅžJIú؆l;Ô ¾_å’Iâ€@þ¯òÎ1ÂNÞù%:@Ñ@؉.%8@€ 0„„“‡ë€R$€°“bÖð€ @ V;±f¿ dJa'ÓÄ @€€„ì… P.„rsOä€ @Ã@Øži”=rXV”iÁ)Ia§È´4²$077§æççÕÂÂB–ñ æƒùäÒ ÂNÞùFÇõv…$š0!„’„‹€€®;÷ÂD#@ Ì#Hˆ. ìŒ7¦®y‘cʾ@ l;eçŸè!„œ²I,È›óÁüó{êÔ)õ&¥Ôk‹‹‹jii)ïˆ ŽyøÄ?õÔSêäÉ“jïÞ½Ãw>PÏ=÷œÚ¶m›zðÁÕƽ^¼xQmß¾]ÝvÛm“õ¹|ù²ÒcÊG>òµvíZ5–¡üµûIÉ×1â»O„± Ó? ŠÂN(ÒØú`>Ø—`Üϳb'îü æ{*C9éH‹{öìQïxÇ;’v†%óFoZøÚ½{·:zôèDØI郰3n¶vÆåKï€@8|¿ ÇK€@?ŒWýøÅþ4ÂNì¿( ì´§a§Q©-vJÍúèdõÏu×]§vìر¬ß¶Õ.ÚžyF?¸ÿþé*"³k×®]êСCêìÙ³“¾í6u«Sšú5Pt<›6mš22ýºžÕ>˜-_7Þxãd‹Ø=÷ܳlûW•—îØÇ〉ק_‡}ûöM}×91¹«2q‰T¦~tºfV¬X1é«‹¿I¼(#8‰°3Tº„ @(–ÂN±©'p31×âml1fÍš5Ó »0L;׊=¡÷v´b$-Ø~9rd"t˜Ÿ;wn™PTÍš'Ž;6Ýîd ãûùóç'ŠtªÂ”›\ÂN]¿æ[À¨ë·*†Øvn¹å–‰pVEªÏøøa3©[\ý¾øâ‹ÓüVŸ{õÕW—;ä+ìtõ·Ô·a§ÔÌ7 @€ÀvÆ JŸIhZqqÿý÷_q6Œž´ë ¾pV®\yÅ;]„[Hò#ªPëF¶E¨Õ«W;WÆØB…ncž\·êÅîwݺuΗ͊$ÍâÙgŸ]vÆŽÏ*ÍØ.F”ª®¾©ŠlU..ÆÕoõ@é&&>Âάþ&ñ² ì$ÂÎÀ@é€ @ h;…¤ŸÃ²®L´kÕMÈ¢Ÿ¶'÷×^{íÌÂNõ@áºózÚΨ©û{[@1[žª"†ÝfË–-ËD-ʸ„-Ÿ~mÊM+v´Tzªÿíã‡ë¯¶~míÕRúçf[]UìòvŽ?ÞÊmÈ[ÇRºvRξC6¹¹95??¯@ jÌ£NOoçvz#L£®·ë&ìØgëØOš‰(aGoÛ²W÷ؾøZØùä'?©>ÿùÏ/»™ªMرÏì©’ÓÛÅôV¯&ߪBXÝuçöJšgžyfÙ ŸêY5.?ê„’¦~u?öù:æl{…ѬÂN7„׳ˆ°“Æÿ7ðh'ÀuçíŒhÄA€ù`yË „±ÈFÖ//r7a§íó>gìtY±Ó$žŒ¹b§zVP•^Ûv(_aÇŽA ­?úiýqmmó}­šú­ó½ïV,½b§›¯ÿ¹·CØÉ=ÃÄr 씓k"…@꘦žÁfÿvòÎï4:^d?a§IP°…ß3vÌy5[·nÞŠ5”°ãsNŸ3vªÛ·ì3}êÎرÛh–v¬®Cš«7“iQGЬ?ugÕýÜÎpS¿u+ìƒ}VìÔTÝÄÍÄVÈ°S&ÂNé@üȇÂN>¹$äN€ù`ÞFØÉ;¿ÓèØSé/ìT'ìúæ*—ˆbo÷qµ±Ïo1×€7ÕR]%ä³*Æ÷V,½¾’]o‰ªŠM>·b¹Î2[¥ª7…i[úúos˜°:Ú®U·¯7ÙªÆ×tR5ÃÆ¿j¿†«¹L?g~VwÆŽëv³Ë¾á¬¿… E“0vJÊ6±B o|¿Ê;¿Dœ0^å”Í+cAØÉ;¿½¢ÓWmêSŸRO<ñ„úö·¿Ý«¯®;´XûZ=TWÿ¬zÖ쿳¦þøãO®#7¢ÍÐÂŽ¶Q=‹Æ,Œˆ³k×.uèСɵçúc·i\LîŒ8e粯ÝÆf¨Åûî»O}ô£U·ÝvÛä°fó© MuM“®újê·ê·öïöÛoŸ~©oáª(­û7ýÙ Ÿþùé-^ZÜsåÃÅ-Æ÷!¤O;!ic € @ w;¹gx†øŒ óÐC©·¾õ­êÎ;ïœü¹æškfèGb'P'ìÄî7þ¥Ka'ÝÜá9 @€@|vâˉ˜G:bèE ûœY#ê Ƴ#€°“]J € @@ÂŽ üXL#èÄ’‰ð~¸Î‰ ïK#€°SZƉ€ @`L;cÒ¨o×aY:%W P„‚’M¨ÈœÀÜÜœšŸŸŸœÑÆ€@Ì8<9æìô÷ a§?Ã$z°¯·CÐI"e8 l ìd›Zƒ@q¸î¼¸”0’%ÀuçɦÎËq„/Lé7Ò‰Ö _ºtIéC‘äG~DmØ°AÝzë­êcûز¿ò•¯(-þ˜Ïš5k&¿²?´®jƒ÷ÂŒ ]Æ„W¾õ‡Jÿy÷¶_U·ýò?î4Àž{ò›Jÿ¹z÷Óçô(Q† IDATo ø@ €°#A›€À,vf¡–Î3;é䪗§:Ñ×_½úýßÿýI??öc?¦þìÏþLé Ñ©S§–õ­—ë+Î͇6ð1µ@mð^èZjLøonüyõøÿr¸Ó؆°Ó !‘ ìŒ ˜î!Á ì †2ÊŽv¢LËðN™=•úŸKKKJÿv]__®WñÜqÇ\e>aC@˜ÂŽp0@€ „¬ÒÙ=žîÌxèGa§?ž† @€€Ma‡z˜@à¡ PŒ°ÓLJ'÷¡Ç³€ @9@ØÉ)›Ä‚À3Dº€ ˜[±t#}SŸþÓõsÕ/l˜>­X]éÑ€ @ ';9e³!–®‡eUžoûÛ°\H­&BÐcþc>ú¶¾ûî»O}èCR7nôraÇ  èÛ"çççÕ½Ó% átg™žB@Ø A9³^o§'\úês¾°DD\€@†ªÂΟüÉŸ¨[n¹EíÙ³GýʯüŠWÄ;^˜hŒ@€ëÎG€J—€À(fŽâ Nagp¤qvÈ‹g^ð ¥@Ø)½ˆi@ØI;x’0Ì;Û;yçw/r!‰&L$Fa'±„á. °ŒÂ¤B€ù`*™šÍO„Ù¸%÷{*“KC Hú å÷¿ÿýê®»îòÞŠU$(‚†¢ À÷«(Ò€€€Æ+H 7AØI8y¸@€ @€@ÙvÊÎ?ÑC€ @€ 0„„“‡ë€ @€ ”Ma§ìü= @€ @ @ØI8y]\ç°¬.´h H0‡'ïÙ³G½ï}ï“r»€¼ÌÍÍ©ùùyµ°°àÕžF€¤0”"Æ.ÂNÎâV¸ÞN<8xøÎw¾£~â'~B:tH}ìcóx‚&€äpݹ{,CÝ0ìÆ+µÖ;©elFy‘gÇc€@P;Aqc èIa§'@‡‚`> µˆ!„ìáò"‡gŽE@ ;„îÌx#€°#ÇË€@7Ì»ñJ­5ÂNj›Ñ_öTÎŽÇ  8c'(nŒA= ðýª'@‡‚`¼ †ZÄÂŽvŒB€ @€ þvú3¤@€ @€ BaG;F!@€ @€@;ýÒ @€ @!€°#‚=¼QË Ï‹€@wßýîwÕûÞ÷>uÇw¨|àÝ;à @ ÌÍÍ©ùùyµ°°Ð*¦ t'À|°;³”ž@ØI)[=|åz»ðxFàâÅ‹jõêÕê‹_ü¢Ú½{w0»‚ 0 ®;Ÿ…Ï@˜JPga'kQK¼È¢ø1x@ØñE3@ ;Q¤' Ì= %Üa'áäuq¹ -ÚBRv¤Èc˜…ÂÎ,Ôx À|P‚z8›;áX‹ZbO¥(~ŒCž8cÇÍ (ðý*Š4à àA€ñÊRÂMvN®C€ @€ P6„²óOô€ @€ $La'áäá: @€ @e@Ø);ÿD@€ @€@ÂvN^×9,« -ÚBRþôOÿTýò/ÿ²ºãŽ;Ô>ð)7° @À‹ÀÜÜœšŸŸW ^íi@@ŠóA)òaì"ì„á,n…ëíÄS€€€W^yE­ZµJ}á _PÿøÇ=ž   9\w.ÇË€@7Ì»ñJ­5ÂNj›Ñ_^äÁñ ”ÂNP܃z@Øé Ç!`˜C-baG{x£¼Èá™cèNa§;3ž€ä ìȱÇ2 ÐóÁn¼Rk°“ZÆfô—=•3‚ã1@ (ÎØ Šc€@O|¿ê Ç!`¯‚¡1„°#‚£€ @€ @ ?„þ é€ @ÿ{÷jIuæ| ¤3 =‰sé7Cqdˆ29NDAébÚ³ÛØíCÔÑÖöJ„´ 1ä)! FBžò2­¦ÃÛŒ^†u†:îÞ½/Uµ«VÕªúhBNªÖúÖïÛ{sêŸ]« @€N;°›” @€ °¾€`g}C# @€ @€:ìtž~R›e¥77#ÕŠÍ“o¼ñÆ°{÷îê8ƒ 666Âææf˜L& g5ª ¸¬n–Ó‚œºµF­o·žS H&ðꫯ†|àáá‡ßøÆ7’Ík"Ôð¸ó:jÎ!@  ׃]¨§›S°“κә¼‘;å79%;%¡F€@/;½hƒ"(!àz°RƇv2n^•Ò½‘«h9–®;]É›—:‚:jÎ!@  ׃]¨§›S°“κәÜSÙ)¿É ()`’P#@ þ¾êEA€@ ŸW%2>D°“qó”N€ @€Œ[@°3îþ[= @€ ±€`'ãæ) @€·€`gÜý·z @€ @ cÁNÆÍ«RºÍ²ªh9–®þüç?‡K/½4Üpà áŠ+®èª ó @ ”ÀÆÆFØÜÜ “ɤÔñ"@€@W®»’O3¯`'sç³x¼]ç-P%^{íµ°sçÎðàƒ†}ûö•8Ã!èNÀãλ³73Õ\VóÊíhÁNn«Y¯7rM8§ T@°“”Ûd¬) ØYÐé$p=˜Œº“‰;°§ŸÔ9½¹ ¨. Ø©næ ºìtgofª ¸¬æ•ÛÑ‚Ü:V³^÷TÖ„sI챓”Ûd¬)àï«5N€@2ŸWɨ;™H°Ó »I  @€ @€ë vÖ74 @€ @ ÁN'ì&%@€ @€¬/ ØYßÐ @€ @€N;°§ŸÔfYéÍÍH€@u7Þx#\rÉ%á†nW\qEõœA€„ass3L&“„³šŠÕ\V7Ëé ÁNNÝZ£V·[Ï©$øÓŸþN?ýôpèС°ÿþdóšˆu<šsèBÀõ`êéæ줳ît&oäNùMN€@IÁNI(‡ Ð ÁN/Ú J¸,”ñ!‚Œ›W¥toä*ZŽ%@ +ÁNWòæ%@ Ž€`§ŽšsèBÀõ`êéæ줳ît&÷TvÊorJ Øc§$”ÃH"ðŸýsøýÏ]8×ÍO ~ì£áÂ]]xÌE?"I­&!@€À2׃Ã~}v†Ý_«#@€¨)P;;ïüÊ#¼ö_ÇÂñ—^ ‚ÊtN @€Š‚Š`'@€‡@ì|涕|ô7O‡øO°S™Î  PQ@°SÌá @€ã쌣ÏVI€Ü;¹wPý @€­4ìœõÀ¡íÚvíÚÕJ%@€q vFÒ›e¤Ñ–I sbóä½{÷†Ïþó™¯Fùä.ÐD°sêÞ}Û ñï1?èBÀõ`êéæ줳ît&·ë”ßä”xýõ×Ãi§¾ûÝnº©äY#@€@;‚v\J€@z׃éÍSÎ(ØI©Ýá\ÞÈ⛚Ò‚ÒT$@ €`'²)H"àz0 sg“v:£O;±7rZo³ PO@°SÏÍY´# ØiÇÕ¨¤p=˜Þ<匂”ÚÎåžÊñMM€d) ØɲmŠ&@`Ž€ëÁa¿,;Ãî¯Õ @€ÔìÔ„s$ì$å6 @€@.‚\:¥NŒ[@°3îþ[=#¸÷Þ{Ã\>ýéO×^ù‹/¾¾ð…/„ï}ï{ Çùãÿ®¼òÊ°gÏž­ÿlêç/ùK8xð`¸úê«ÃYgÚš§©z§ÇÉ©Ö6ÖoL9 vrìšš  0>ÁÎøznÅŒT L S†¦©qÊÌ5{ÌoûÛ°wïÞðè£n;9ývrê–Z ü¿€`Ç+rìäÐ¥j´YVˆ† ¹@SLSãÔáìÔQsu;uåœG€@ß\ö­#ÍÖ#ØiÖ³·£y¼]o[£° Ä[…nºé¦­o“üú׿Þúwíµ×†C‡m­4þo?øÁ¶W}äÈ‘íÛ™ŠÀäî»ïßÿþ÷·Î?÷ÜsO¸ãŽ;NúÙÏ~¾üå/oÿnú˜y5\vÙeá—¿üåöñEM;vì˜Ûeãu^ýõá‘G Ï?ÿüIu.úvʲq‹BbxsÞyç´¶yçÆŠ[¾Î=÷Ü­[ľùÍožpûWQK¼ý¬p,SGQ@±Þ2ãÆ[Ýî¼óÎíÚ/ºè¢çúà?xÒmcóBª¢wq€øš)úS¥Þ½¥,‡@§‚NùMN€@ƒ®ÄìáP‚6¥’¼‘ÛP5&“Š óÞL‡6E¸°k×®í ö"À(Ž+„”¿›JÄðàñÇß¾ivìéi^p´loœxîªñ;¶ ,«s^°³hÜyË¢õφ!ÓóìÞ½{+8› EfÏ)SÇtg…-óÆ}ùå—·û;{Þo¼q¾Ceƒªõz_ Ðœ€`§9K# Э€ëÁnýÛž]°Ó¶pOÆ÷FîI#”1xe߸xàNÚ&^´Ç üøŒW^yeî7N¦€8oãâéèœsΙp”¹…jÑ1ÓãŸqÆ+ëŒÇLož¼è[/Óã~ü㟻ár<&:E£ßýîw'ì±3 Í Kâ¹EàR„R³ß¾™ Ùf_¨eÇ Í¦Ï›5)ìÔ­wðo4 $H@°“Ú4´.àz°uâN'ìtÊŸnr÷T¦³6Ó¸Š`çÌ3ÏܾígQإʄ6Ó¡HüÆϼ̓ç}seº†8W™`gÑ6Óã·<͆ÓÇ\|ñÅ'„41”™l•wúµì;1Hš zfÿ{™:æ=ÅkÕ¸Ó5Nk+þþSŸúÔV W'ØùÅ/~±Ò­É§ŽûÝkõNìxU 0׃Céäüuv†Ý_«#@ ±À²`gzo鲊 ÿø»o¼1<üðÃ'<ñi:TˆÇ¬ HŠ[’ê;e‚쬪s^°3½'Ðl[~úÓŸ†ZŽu¦o›=nU°Ÿþ†Î3Ï.óÙ°‹=v–¹kké%mX£쌺ýO€l;Ù´J¡ä °(Ø™÷T¬Ù¥øïqqO–øÈôâwŸûÜ綿u²ê©Uñüøt¨Ù`gÑ#Èg]W_„ Ëê,óT¬y{·J·cÍ3t¬z¬úôãÆ‹uήoÙH³6E}³ãß؉·”·F¿[´Ç΢'žÅÛ¹¦G¿N½9¼gÔH Ï‚>wGm PvFòZ°YÖHm™ , vba³›êÆßÍ{ùõ×_y䑭ljǟ鰠Xàì^1ÓÇ,«!†18˜xLã-¿›–Õ¹*p)æºçž{¶«âwE 2ï˜iÃ~8p |ík_ {öìÙTâyó±eë›WǼÓ²qgëŽõ]wÝua2™„ø «Ù}‡¦ëœîõþð‡í§xíرc«ŒÙ~”­·ó7„d. ØɼÊ'@`[Àõà°_ ‚÷÷èѣᡇ Ï=÷\xòÉ'Ãi§.¸à‚pùå—‡ÍÍͯÜÒä)Pæ©U¹¬¬ì·ƒrY: †Àƒ>¸õwPܬ½Ì`§Œ’cÈAÀãÎsèRý;õíz}æ~ô£­§ÖÄÿwû­·ÞÚ®õ}ï{_xûí·CLlŸx≰sçÎ^¯CqÆ$0¤`gHkÓkÐZ ] þýóÔSOm}“îàÁƒ+ÁÎÐ_ÖG`<‚a÷Z°3ÀþÆPç+_ùÊÒ•Å€çŸøDxú駅;| XRžC CŠÛ½¦÷‰É³#ª&@`hE°sÊ)§lý_«ÁÎÐ^ÖC`¼‚a÷^°3°þÆÛ¯>ùÉO†×_½ÔÊöíÛâ×’ý @€†.P;Å:W<‚¡¿"¬Àxì±3ì^ vÖßýû÷‡øXÜéÛ¯V-ñÕW_õ­UHþw @ {Ù`gUÀ#Øɾå@€QvÖæE°,[æáÇ·öÜñC€²Àª¿“f¿ÁóÖ¿ýkøýÏ Ÿ¹í@e–£¿y:ħîÝ·}®¿·*3:JvJ åtHl¨ @€õvæÂ5gü`g}J# @€@‹‚q»úÃþpxùå—+M}Ùe—…³Ï>»Ò9&@€ä&0ñÒK/--»øÖÎææf˜œû·áŸúÁNnV/F& ØXÃ/¿üòð«_ý*¼óÎ;¥Wöì³Ï vJk9¶â­ ×]w]øâ¿ØöTÆ'@`dËnÅštîºë®­G¡Ûcgd/Ë%0`›'¸¹!ÁÎÀú[æQçÅ’cóã7|Ž;60Ë!@ W7ß|3ìر#|ç;ß 7ß|s®ËP7=˜ìÌ tŠò;=m¤²¨,àqç•É²:A°“U»Êÿh9räHxûí·Wž`ãä•D @ ¡€`'!¶©ŒP`:ØYèvFøâ°dì »Á‚ö÷øñãáüóÏ/¼ðÂÜÕŦ¿ûî»á‡?üa˜L&°$rìäÚ9uÈC`:؉{è·\-ªÞ7vòè«* X- ØYm”󂜻·¤öîÄ?Vzè¡“ŽúÈG>~üã{Äù@{oYr°ÇNîT?þ ÄÏ—¸wΪ@§X`§¿½TÕì±SÍ+·£;¹u¬b½1àyî¹ç“O>¹õ‡L|ú•'`UDt8 0øwÑÎ;K¯E°SšÊ С€`§C|S @€ôW@°ÓßÞ¨ŒÞìx5 @€˜# Øñ² @€;9tI @€É;ÉÉMH€5;5Ðr<ÅfY9vMÍÆ)?¯®¹æšð¥/}iœVM€@o;½i…BXSÀõàš€=?]°Óó5UžÇÛ5%iÚˆŸWßþö·Ã-·ÜÒöTÆ'@€ÀRÁŽCp=8”NÎ_‡`gØýÝ^7òHm™ Ø@-À@;i¤e \ûE Øv;#é¯e’€`gHÝ´y vòîŸê xO@°3ìWƒ`gØýÝ^{*GÒhË$0{ì  ‰–@` ‚4Ò2®‡ý"ì »¿VG€ PS@°SÎi T@°“”Ûd @€¹vré”:  0nÁθûoõ @€ ;^ ƒ€`'‡.©‘ @ ¹€`'9¹  @ †€`§ZŽ§Ø,+Ç®©™À8lž<ξ[5> vúØ5 PGÀõ`µ|ÎìäÓ«µ*õx»µøœL€@Bøyõ­o}+Üzë­ g5NìxU 0׃Céäüuv†ÝßíÕy#¤Ñ–I`‚4Ñ D@°3FZÁõà°_‚a÷W°3’þZ&! v†ÔMk!·€`'ïþ©ž÷;Ã~5v†ÝßíÕ¹§r$¶L°ÇΚh " ØH#-ƒàzpØ/Áΰûku @€5;5áœF€I;I¹MF€ ‹€`'—N©“ã쌻ÿVO€ °@@°ã¥A€9vrè’  @€’ v’“›jvj åxŠÍ²rìšš ŒS ~^}ýë_W^yå8¬šÞvzÓ … °¦€ëÁ5{~º`§ç jª<·kJÒ8´-?¯î¿ÿþpÛm·µ=•ñ  °T vôïÿýJøÐûß>ô×ï_8Ω{÷mÿoñÂʺp=Ø…zº9;é¬;ɹS~“ PA@°SË¡´*ð?ÿñBˆÿâÏñãÇ·þMÿ\xË?…=ŸýÇpÕ…Ÿ]XÇ)÷÷‚V»dpʸ,£”ï1‚|{W©roäJ\&@ CÁN‡ø¦&@`¡ÀÑ£GCü7ý³±±677Ãd2)%ç;¥˜D€@ ®[@íÑ‚5£ÍRÜSÙ¦®± hRÀ;Mj‹¦æ;û÷ï_|ñÖ¿2?‚2JŽ!@  ׃m¨ögLÁNz¡ @€ž Ì vª–*Ø©*æx(# Ø)£ä @€Q vFÝ~‹'@€@¯;½nâ @€ @€‹;^ @€ @€L;™6®jÙ6˪*æxºˆŸWW_}uسgOW%˜—¥ª>«Ô "@€@ ®[@íÑ‚5£ÍR<Þ®M]c Ф@ü¼ºï¾ûÂí·ßÞä°Æ"@€@ãñóêàÁƒá®»îj|l @ I׃Mjöo,ÁNÿzÒJEÞÈ­°”;- ’V;­°”\¶€Ú£!;=jF›¥x#·©klšì4©i,Úì´©klšp=ؤfÿÆìô¯'­TäžÊVX J€@ öØiÕ´"àï«VX J€@ >¯Z@íÑ‚5C) @€ @€*‚*ZŽ%@€ @€ôH@°Ó£f(… @€ PE@°SE˱ @€ @€ vzÔŒ6K±YV›ºÆ&@ I›'7©i,ÚØØØ›››a2™´9±  °¶€ëÁµ {=€`§×íi®8·kÎÒH´+?¯î½÷ÞpàÀv'2:Öð¸ó5N€@2׃ɨ;™H°Ó {úI½‘Ó››‘z‚znÎ"@ ½€`'½¹ ¨'àz°ž[.g vréÔšuz#¯ èt’ v’Q›ˆ5;k:d®“Qw2‘`§öô“º§2½¹ ¨'`znÎ"@ ½€¿¯Ò››‘z>¯ê¹år–`'—N©“ @€ 0# Øñ’ @€ @€d* ØÉ´qÊ&@€ @€v¼ @€ @€™ v2m\Õ²m–UUÌñt%?¯¾úÕ¯†«®ºª«ÌK€Rass3L&“RÇ;ˆ] ¸ìJ>ͼ‚4ÎÏâñv·@”ˆŸWwß}w¸óÎ;Kžá0t#àqçݸ›•ê®«›åt†`'§n­Q«7òxN%@ æ.ƒIDAT ©€`')·ÉXC@°³žS H*àz0)wòÉ;ÉÉ»™Ð¹w³ P]@°SÝÌt# ØéÆݬTp=XÝ,§3;9ukZÝS¹žS H*`¤Ü&#@` _­çT’ ø¼JÊ|2ÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€šì4ãh @€ @€@rÁNrr @€ @€š8!ØifH£ @€ @€¤8|øpø«Ã‡¿›jBó @€ @€4#pöÙg‡ÿ‰23»úy¦IEND®B`‚optuna-3.5.0/docs/make.bat000066400000000000000000000014561453453102400153710ustar00rootroot00000000000000@ECHO OFF pushd %~dp0 REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set SOURCEDIR=source set BUILDDIR=build set SPHINXPROJ=Optuna if "%1" == "" goto help %SPHINXBUILD% >NUL 2>NUL if errorlevel 9009 ( echo. echo.The 'sphinx-build' command was not found. Make sure you have Sphinx echo.installed, then set the SPHINXBUILD environment variable to point echo.to the full path of the 'sphinx-build' executable. Alternatively you echo.may add the Sphinx directory to PATH. echo. echo.If you don't have Sphinx installed, grab it from echo.http://sphinx-doc.org/ exit /b 1 ) %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% goto end :help %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% :end popd optuna-3.5.0/docs/source/000077500000000000000000000000001453453102400152565ustar00rootroot00000000000000optuna-3.5.0/docs/source/_static/000077500000000000000000000000001453453102400167045ustar00rootroot00000000000000optuna-3.5.0/docs/source/_static/css/000077500000000000000000000000001453453102400174745ustar00rootroot00000000000000optuna-3.5.0/docs/source/_static/css/custom.css000066400000000000000000000103521453453102400215210ustar00rootroot00000000000000/* WARNING: This style sheet may not be compatible with `sphinx-rtd-theme >= 0.5.0` */ /* Code-block for console */ .highlight-console pre { background: #d3d5d4; } /* Parameter names and colons after them in a signature */ em.sig-param > span:nth-child(1), em.sig-param > span:nth-child(2) { color: #555555; } /* Type hints and default values in a signature */ em.sig-param > span:not(:nth-child(1)):not(:nth-child(2)) { color: #2980b9; } /* Internal links in a signature */ .rst-content dl.class > dt a.reference.internal, .rst-content dl.method > dt a.reference.internal, .rst-content dl.function > dt a.reference.internal { color: #2980b9; } /* External links in a signature */ .rst-content dl.class > dt a.reference.external, .rst-content dl.method > dt a.reference.external, .rst-content dl.function > dt a.reference.external { color: #2980b9; } /* Containers for a signature */ .rst-content dl.class > dt, .rst-content dl.function > dt { background-color: #f0f0f0; } /* Containers for methods, properties, parameters, and returns */ .rst-content dl:not(.docutils) dl dt { border-left: solid 3px #6ab0de; } /* Main content */ .wy-nav-content { max-width: 1200px; } /* Sidebar header (and topbar for mobile) */ .wy-side-nav-search, .wy-nav-top { background: #f1f3f4; } .wy-side-nav-search div.version { color: #404040; } .wy-nav-top a { color: #404040; } .wy-nav-top i { color: #404040; } /* Sidebar */ .wy-nav-side { background: #f1f3f4; } /* A tag */ .wy-menu-vertical a { color: #707070; } a { color: #2ba9cd; } .wy-menu-vertical a:active { background-color: #2ba9cd; cursor: pointer; color: #f1f3f4; } .highlight { background: #f1f3f4; } .navbar { background: #ffffff; } @media only screen and (max-width: 896px) { .navbar { height: 0px; } } .navbar-nav { background: #ffffff; list-style: none; display: flex; flex-direction: row; align-items: center; justify-content: flex-end; padding: 20px; max-width: 1200px; margin-left: 300px; } .ml-auto { margin-left: auto !important; } .header_link { margin: 15px 2px; font-size: 16px; font-weight: 600; font-family: "Helvetica"; cursor: pointer; padding: 0.5rem 0.8rem 0.5rem 0.5rem; color: #636a73; } .navbar-nav a:focus, a:hover { color: #2ba9cd; text-decoration: none; } .navbar-nav a:visited { color: #636a73; text-decoration: none; } .wy-alert.wy-alert-info .wy-alert-title, .rst-content .note .wy-alert-title, .rst-content .wy-alert-info.attention .wy-alert-title, .rst-content .wy-alert-info.caution .wy-alert-title, .rst-content .wy-alert-info.danger .wy-alert-title, .rst-content .wy-alert-info.error .wy-alert-title, .rst-content .wy-alert-info.hint .wy-alert-title, .rst-content .wy-alert-info.important .wy-alert-title, .rst-content .wy-alert-info.tip .wy-alert-title, .rst-content .wy-alert-info.warning .wy-alert-title, .rst-content .seealso .wy-alert-title, .rst-content .wy-alert-info.admonition-todo .wy-alert-title, .rst-content .wy-alert-info.admonition .wy-alert-title, .wy-alert.wy-alert-info .rst-content .admonition-title, .rst-content .wy-alert.wy-alert-info .admonition-title, .rst-content .note .admonition-title, .rst-content .wy-alert-info.attention .admonition-title, .rst-content .wy-alert-info.caution .admonition-title, .rst-content .wy-alert-info.danger .admonition-title, .rst-content .wy-alert-info.error .admonition-title, .rst-content .wy-alert-info.hint .admonition-title, .rst-content .wy-alert-info.important .admonition-title, .rst-content .wy-alert-info.tip .admonition-title, .rst-content .wy-alert-info.warning .admonition-title, .rst-content .seealso .admonition-title, .rst-content .wy-alert-info.admonition-todo .admonition-title, .rst-content .wy-alert-info.admonition .admonition-title { background: #2ba9cd; } .wy-alert, .rst-content .note, .rst-content .attention, .rst-content .caution, .rst-content .danger, .rst-content .error, .rst-content .hint, .rst-content .important, .rst-content .tip, .rst-content .warning, .rst-content .seealso, .rst-content .admonition-todo, .rst-content .admonition { background: #f1f3f4; } /* This is the patch fix for tutorial page */ /* TODO (himkt): Remove here after resolving https://github.com/optuna/optuna/pull/3322 */ .sphx-glr-thumbcontainer { min-width: 182px; } optuna-3.5.0/docs/source/_templates/000077500000000000000000000000001453453102400174135ustar00rootroot00000000000000optuna-3.5.0/docs/source/_templates/autosummary/000077500000000000000000000000001453453102400220015ustar00rootroot00000000000000optuna-3.5.0/docs/source/_templates/autosummary/class.rst000066400000000000000000000006411453453102400236410ustar00rootroot00000000000000{% extends "!autosummary/class.rst" %} {# An autosummary template to exclude the class constructor (__init__) which doesn't contain any docstring in Optuna. #} {% block methods %} {% set methods = methods | select("ne", "__init__") | list %} {% if methods %} .. rubric:: Methods .. autosummary:: {% for item in methods %} ~{{ name }}.{{ item }} {%- endfor %} {% endif %} {% endblock %} optuna-3.5.0/docs/source/_templates/breadcrumbs.html000066400000000000000000000004531453453102400225740ustar00rootroot00000000000000 {%- extends "sphinx_rtd_theme/breadcrumbs.html" %} {% block breadcrumbs_aside %} {% endblock %} optuna-3.5.0/docs/source/_templates/footer.html000066400000000000000000000003001453453102400215700ustar00rootroot00000000000000{% extends "!footer.html" %} {% block extrafooter %} {% trans path=pathto('privacy') %}Privacy Policy{{ privacy }}.{% endtrans %} {{ super() }} {% endblock %} optuna-3.5.0/docs/source/_templates/layout.html000066400000000000000000000024461453453102400216240ustar00rootroot00000000000000{% extends "!layout.html" %} {%- block extrabody %}
{{ super() }} {% endblock %} optuna-3.5.0/docs/source/conf.py000066400000000000000000000156671453453102400165740ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Configuration file for the Sphinx documentation builder. # # This file does only contain a selection of the most common options. For a # full list see the documentation: # http://www.sphinx-doc.org/en/master/config # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) import warnings import plotly.io as pio from sklearn.exceptions import ConvergenceWarning from sphinx_gallery.sorting import FileNameSortKey import optuna # -- Project information ----------------------------------------------------- project = "Optuna" copyright = "2018, Optuna Contributors" author = "Optuna Contributors." # The short X.Y version version = optuna.version.__version__ # The full version, including alpha/beta/rc tags release = optuna.version.__version__ # -- General configuration --------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. # # needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ "sphinx.ext.autodoc", "sphinx.ext.autosummary", "sphinx.ext.doctest", "sphinx.ext.imgconverter", "sphinx.ext.intersphinx", "sphinx.ext.mathjax", "sphinx.ext.napoleon", "sphinx.ext.viewcode", "sphinx.ext.githubpages", "sphinx_copybutton", "sphinx_gallery.gen_gallery", "matplotlib.sphinxext.plot_directive", "sphinx_plotly_directive", ] # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] source_suffix = ".rst" # The master toctree document. master_doc = "index" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path . exclude_patterns = [] # The name of the Pygments (syntax highlighting) style to use. pygments_style = "sphinx" # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = "sphinx_rtd_theme" # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # html_theme_options = {"logo_only": True, "navigation_with_keys": True} html_favicon = "../image/favicon.ico" html_logo = "../image/optuna-logo.png" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ["_static"] html_css_files = ["css/custom.css"] # Custom sidebar templates, must be a dictionary that maps document names # to template names. # # The default sidebars (for documents that don't match any pattern) are # defined by theme itself. Builtin themes are using these templates by # default: ``['localtoc.html', 'relations.html', 'sourcelink.html', # 'searchbox.html']``. # # html_sidebars = {} # -- Options for HTMLHelp output --------------------------------------------- # Output file base name for HTML help builder. htmlhelp_basename = "Optunadoc" # -- Options for LaTeX output ------------------------------------------------ latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # # 'preamble': '', # Latex figure (float) alignment # # 'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, "Optuna.tex", "Optuna Documentation", "Optuna Contributors.", "manual"), ] # -- Options for manual page output ------------------------------------------ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [(master_doc, "optuna", "Optuna Documentation", [author], 1)] # -- Options for Texinfo output ---------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ( master_doc, "Optuna", "Optuna Documentation", author, "Optuna", "One line description of project.", "Miscellaneous", ), ] intersphinx_mapping = { "python": ("https://docs.python.org/3", None), "distributed": ("https://distributed.dask.org/en/stable", None), "lightgbm": ("https://lightgbm.readthedocs.io/en/latest", None), "matplotlib": ("https://matplotlib.org/stable", None), "numpy": ("https://numpy.org/doc/stable", None), "scipy": ("https://docs.scipy.org/doc/scipy", None), "sklearn": ("https://scikit-learn.org/stable", None), "torch": ("https://pytorch.org/docs/stable", None), "pandas": ("https://pandas.pydata.org/docs", None), "plotly": ("https://plotly.com/python-api-reference", None), } # -- Extension configuration ------------------------------------------------- autosummary_generate = True autodoc_typehints = "description" autodoc_default_options = { "members": True, "inherited-members": True, "exclude-members": "with_traceback", } # sphinx_copybutton option to not copy prompt. copybutton_prompt_text = "$ " # Sphinx Gallery pio.renderers.default = "sphinx_gallery" sphinx_gallery_conf = { "examples_dirs": [ "../../tutorial/10_key_features", "../../tutorial/20_recipes", ], "gallery_dirs": [ "tutorial/10_key_features", "tutorial/20_recipes", ], "within_subsection_order": FileNameSortKey, "filename_pattern": r"/*\.py", "first_notebook_cell": None, } # matplotlib plot directive plot_include_source = True plot_formats = [("png", 90)] plot_html_show_formats = False plot_html_show_source_link = False # sphinx plotly directive plotly_include_source = True plotly_formats = ["html"] plotly_html_show_formats = False plotly_html_show_source_link = False # Not showing common warning messages as in # https://sphinx-gallery.github.io/stable/configuration.html#removing-warnings. warnings.filterwarnings("ignore", category=ConvergenceWarning, module="sklearn") optuna-3.5.0/docs/source/faq.rst000066400000000000000000000721711453453102400165670ustar00rootroot00000000000000FAQ === .. contents:: :local: Can I use Optuna with X? (where X is your favorite ML library) -------------------------------------------------------------- Optuna is compatible with most ML libraries, and it's easy to use Optuna with those. Please refer to `examples `_. .. _objective-func-additional-args: How to define objective functions that have own arguments? ---------------------------------------------------------- There are two ways to realize it. First, callable classes can be used for that purpose as follows: .. code-block:: python import optuna class Objective: def __init__(self, min_x, max_x): # Hold this implementation specific arguments as the fields of the class. self.min_x = min_x self.max_x = max_x def __call__(self, trial): # Calculate an objective value by using the extra arguments. x = trial.suggest_float("x", self.min_x, self.max_x) return (x - 2) ** 2 # Execute an optimization by using an `Objective` instance. study = optuna.create_study() study.optimize(Objective(-100, 100), n_trials=100) Second, you can use ``lambda`` or ``functools.partial`` for creating functions (closures) that hold extra arguments. Below is an example that uses ``lambda``: .. code-block:: python import optuna # Objective function that takes three arguments. def objective(trial, min_x, max_x): x = trial.suggest_float("x", min_x, max_x) return (x - 2) ** 2 # Extra arguments. min_x = -100 max_x = 100 # Execute an optimization by using the above objective function wrapped by `lambda`. study = optuna.create_study() study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100) Please also refer to `sklearn_addtitional_args.py `_ example, which reuses the dataset instead of loading it in each trial execution. Can I use Optuna without remote RDB servers? -------------------------------------------- Yes, it's possible. In the simplest form, Optuna works with in-memory storage: .. code-block:: python study = optuna.create_study() study.optimize(objective) If you want to save and resume studies, it's handy to use SQLite as the local storage: .. code-block:: python study = optuna.create_study(study_name="foo_study", storage="sqlite:///example.db") study.optimize(objective) # The state of `study` will be persisted to the local SQLite file. Please see :ref:`rdb` for more details. How can I save and resume studies? ---------------------------------------------------- There are two ways of persisting studies, which depend if you are using in-memory storage (default) or remote databases (RDB). In-memory studies can be saved and loaded like usual Python objects using ``pickle`` or ``joblib``. For example, using ``joblib``: .. code-block:: python study = optuna.create_study() joblib.dump(study, "study.pkl") And to resume the study: .. code-block:: python study = joblib.load("study.pkl") print("Best trial until now:") print(" Value: ", study.best_trial.value) print(" Params: ") for key, value in study.best_trial.params.items(): print(f" {key}: {value}") Note that Optuna does not support saving/reloading across different Optuna versions with ``pickle``. To save/reload a study across different Optuna versions, please use RDBs and `upgrade storage schema `_ if necessary. If you are using RDBs, see :ref:`rdb` for more details. How to suppress log messages of Optuna? --------------------------------------- By default, Optuna shows log messages at the ``optuna.logging.INFO`` level. You can change logging levels by using :func:`optuna.logging.set_verbosity`. For instance, you can stop showing each trial result as follows: .. code-block:: python optuna.logging.set_verbosity(optuna.logging.WARNING) study = optuna.create_study() study.optimize(objective) # Logs like '[I 2020-07-21 13:41:45,627] Trial 0 finished with value:...' are disabled. Please refer to :class:`optuna.logging` for further details. How to save machine learning models trained in objective functions? ------------------------------------------------------------------- Optuna saves hyperparameter values with its corresponding objective value to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, please use features of the machine learning library you used. We recommend saving :obj:`optuna.trial.Trial.number` with a model in order to identify its corresponding trial. For example, you can save SVM models trained in the objective function as follows: .. code-block:: python def objective(trial): svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True) clf = sklearn.svm.SVC(C=svc_c) clf.fit(X_train, y_train) # Save a trained model to a file. with open("{}.pickle".format(trial.number), "wb") as fout: pickle.dump(clf, fout) return 1.0 - accuracy_score(y_valid, clf.predict(X_valid)) study = optuna.create_study() study.optimize(objective, n_trials=100) # Load the best model. with open("{}.pickle".format(study.best_trial.number), "rb") as fin: best_clf = pickle.load(fin) print(accuracy_score(y_valid, best_clf.predict(X_valid))) How can I obtain reproducible optimization results? --------------------------------------------------- To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via ``seed`` argument of an instance of :mod:`~optuna.samplers` as follows: .. code-block:: python sampler = TPESampler(seed=10) # Make the sampler behave in a deterministic way. study = optuna.create_study(sampler=sampler) study.optimize(objective) However, there are two caveats. First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result. Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it. How are exceptions from trials handled? --------------------------------------- Trials that raise exceptions without catching them will be treated as failures, i.e. with the :obj:`~optuna.trial.TrialState.FAIL` status. By default, all exceptions except :class:`~optuna.exceptions.TrialPruned` raised in objective functions are propagated to the caller of :func:`~optuna.study.Study.optimize`. In other words, studies are aborted when such exceptions are raised. It might be desirable to continue a study with the remaining trials. To do so, you can specify in :func:`~optuna.study.Study.optimize` which exception types to catch using the ``catch`` argument. Exceptions of these types are caught inside the study and will not propagate further. You can find the failed trials in log messages. .. code-block:: sh [W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \ the following error: ValueError('A sample error in objective.') You can also find the failed trials by checking the trial states as follows: .. code-block:: python study.trials_dataframe() .. csv-table:: number,state,value,...,params,system_attrs 0,TrialState.FAIL,,...,0,Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError('A test error in objective.') 1,TrialState.COMPLETE,1269,...,1, .. seealso:: The ``catch`` argument in :func:`~optuna.study.Study.optimize`. How are NaNs returned by trials handled? ---------------------------------------- Trials that return NaN (``float('nan')``) are treated as failures, but they will not abort studies. Trials which return NaN are shown as follows: .. code-block:: sh [W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \ objective function returned nan. What happens when I dynamically alter a search space? ----------------------------------------------------- Since parameters search spaces are specified in each call to the suggestion API, e.g. :func:`~optuna.trial.Trial.suggest_float` and :func:`~optuna.trial.Trial.suggest_int`, it is possible to, in a single study, alter the range by sampling parameters from different search spaces in different trials. The behavior when altered is defined by each sampler individually. .. note:: Discussion about the TPE sampler. https://github.com/optuna/optuna/issues/822 How can I use two GPUs for evaluating two trials simultaneously? ---------------------------------------------------------------- If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used in your script, ``main.py``, the easiest way is to set ``CUDA_VISIBLE_DEVICES`` environment variable: .. code-block:: bash # On a terminal. # # Specify to use the first GPU, and run an optimization. $ export CUDA_VISIBLE_DEVICES=0 $ python main.py # On another terminal. # # Specify to use the second GPU, and run another optimization. $ export CUDA_VISIBLE_DEVICES=1 $ python main.py Please refer to `CUDA C Programming Guide `_ for further details. How can I test my objective functions? -------------------------------------- When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use :class:`~optuna.trial.FixedTrial`, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of :math:`x` and :math:`y` to the objective function :math:`x + y` as follows: .. code-block:: python def objective(trial): x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_int("y", -5, 5) return x + y objective(FixedTrial({"x": 1.0, "y": -1})) # 0.0 objective(FixedTrial({"x": -1.0, "y": -4})) # -5.0 Using :class:`~optuna.trial.FixedTrial`, you can write unit tests as follows: .. code-block:: python # A test function of pytest def test_objective(): assert 1.0 == objective(FixedTrial({"x": 1.0, "y": 0})) assert -1.0 == objective(FixedTrial({"x": 0.0, "y": -1})) assert 0.0 == objective(FixedTrial({"x": -1.0, "y": 1})) .. _out-of-memory-gc-collect: How do I avoid running out of memory (OOM) when optimizing studies? ------------------------------------------------------------------- If the memory footprint increases as you run more trials, try to periodically run the garbage collector. Specify ``gc_after_trial`` to :obj:`True` when calling :func:`~optuna.study.Study.optimize` or call :func:`gc.collect` inside a callback. .. code-block:: python def objective(trial): x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_int("y", -5, 5) return x + y study = optuna.create_study() study.optimize(objective, n_trials=10, gc_after_trial=True) # `gc_after_trial=True` is more or less identical to the following. study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()]) There is a performance trade-off for running the garbage collector, which could be non-negligible depending on how fast your objective function otherwise is. Therefore, ``gc_after_trial`` is :obj:`False` by default. Note that the above examples are similar to running the garbage collector inside the objective function, except for the fact that :func:`gc.collect` is called even when errors, including :class:`~optuna.exceptions.TrialPruned` are raised. .. note:: :class:`~optuna.integration.ChainerMNStudy` does currently not provide ``gc_after_trial`` nor callbacks for :func:`~optuna.integration.ChainerMNStudy.optimize`. When using this class, you will have to call the garbage collector inside the objective function. How can I output a log only when the best value is updated? ----------------------------------------------------------- Here's how to replace the logging feature of optuna with your own logging callback function. The implemented callback can be passed to :func:`~optuna.study.Study.optimize`. Here's an example: .. code-block:: python import optuna # Turn off optuna log notes. optuna.logging.set_verbosity(optuna.logging.WARN) def objective(trial): x = trial.suggest_float("x", 0, 1) return x ** 2 def logging_callback(study, frozen_trial): previous_best_value = study.user_attrs.get("previous_best_value", None) if previous_best_value != study.best_value: study.set_user_attr("previous_best_value", study.best_value) print( "Trial {} finished with best value: {} and parameters: {}. ".format( frozen_trial.number, frozen_trial.value, frozen_trial.params, ) ) study = optuna.create_study() study.optimize(objective, n_trials=100, callbacks=[logging_callback]) Note that this callback may show incorrect values when you try to optimize an objective function with ``n_jobs!=1`` (or other forms of distributed optimization) due to its reads and writes to storage that are prone to race conditions. How do I suggest variables which represent the proportion, that is, are in accordance with Dirichlet distribution? ------------------------------------------------------------------------------------------------------------------ When you want to suggest :math:`n` variables which represent the proportion, that is, :math:`p[0], p[1], ..., p[n-1]` which satisfy :math:`0 \le p[k] \le 1` for any :math:`k` and :math:`p[0] + p[1] + ... + p[n-1] = 1`, try the below. For example, these variables can be used as weights when interpolating the loss functions. These variables are in accordance with the flat `Dirichlet distribution `_. .. code-block:: python import numpy as np import matplotlib.pyplot as plt import optuna def objective(trial): n = 5 x = [] for i in range(n): x.append(- np.log(trial.suggest_float(f"x_{i}", 0, 1))) p = [] for i in range(n): p.append(x[i] / sum(x)) for i in range(n): trial.set_user_attr(f"p_{i}", p[i]) return 0 study = optuna.create_study(sampler=optuna.samplers.RandomSampler()) study.optimize(objective, n_trials=1000) n = 5 p = [] for i in range(n): p.append([trial.user_attrs[f"p_{i}"] for trial in study.trials]) axes = plt.subplots(n, n, figsize=(20, 20))[1] for i in range(n): for j in range(n): axes[j][i].scatter(p[i], p[j], marker=".") axes[j][i].set_xlim(0, 1) axes[j][i].set_ylim(0, 1) axes[j][i].set_xlabel(f"p_{i}") axes[j][i].set_ylabel(f"p_{j}") plt.savefig("sampled_ps.png") This method is justified in the following way: First, if we apply the transformation :math:`x = - \log (u)` to the variable :math:`u` sampled from the uniform distribution :math:`Uni(0, 1)` in the interval :math:`[0, 1]`, the variable :math:`x` will follow the exponential distribution :math:`Exp(1)` with scale parameter :math:`1`. Furthermore, for :math:`n` variables :math:`x[0], ..., x[n-1]` that follow the exponential distribution of scale parameter :math:`1` independently, normalizing them with :math:`p[i] = x[i] / \sum_i x[i]`, the vector :math:`p` follows the Dirichlet distribution :math:`Dir(\alpha)` of scale parameter :math:`\alpha = (1, ..., 1)`. You can verify the transformation by calculating the elements of the Jacobian. How can I optimize a model with some constraints? ------------------------------------------------- When you want to optimize a model with constraints, you can use the following classes: :class:`~optuna.samplers.TPESampler`, :class:`~optuna.samplers.NSGAIISampler` or :class:`~optuna.integration.BoTorchSampler`. The following example is a benchmark of Binh and Korn function, a multi-objective optimization, with constraints using :class:`~optuna.samplers.NSGAIISampler`. This one has two constraints :math:`c_0 = (x-5)^2 + y^2 - 25 \le 0` and :math:`c_1 = -(x - 8)^2 - (y + 3)^2 + 7.7 \le 0` and finds the optimal solution satisfying these constraints. .. code-block:: python import optuna def objective(trial): # Binh and Korn function with constraints. x = trial.suggest_float("x", -15, 30) y = trial.suggest_float("y", -15, 30) # Constraints which are considered feasible if less than or equal to zero. # The feasible region is basically the intersection of a circle centered at (x=5, y=0) # and the complement to a circle centered at (x=8, y=-3). c0 = (x - 5) ** 2 + y ** 2 - 25 c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7 # Store the constraints as user attributes so that they can be restored after optimization. trial.set_user_attr("constraint", (c0, c1)) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 def constraints(trial): return trial.user_attrs["constraint"] sampler = optuna.samplers.NSGAIISampler(constraints_func=constraints) study = optuna.create_study( directions=["minimize", "minimize"], sampler=sampler, ) study.optimize(objective, n_trials=32, timeout=600) print("Number of finished trials: ", len(study.trials)) print("Pareto front:") trials = sorted(study.best_trials, key=lambda t: t.values) for trial in trials: print(" Trial#{}".format(trial.number)) print( " Values: Values={}, Constraint={}".format( trial.values, trial.user_attrs["constraint"][0] ) ) print(" Params: {}".format(trial.params)) If you are interested in an example for :class:`~optuna.integration.BoTorchSampler`, please refer to `this sample code `_. There are two kinds of constrained optimizations, one with soft constraints and the other with hard constraints. Soft constraints do not have to be satisfied, but an objective function is penalized if they are unsatisfied. On the other hand, hard constraints must be satisfied. Optuna is adopting the soft one and **DOES NOT** support the hard one. In other words, Optuna **DOES NOT** have built-in samplers for the hard constraints. How can I parallelize optimization? ----------------------------------- The variations of parallelization are in the following three cases. 1. Multi-threading parallelization with single node 2. Multi-processing parallelization with single node 3. Multi-processing parallelization with multiple nodes 1. Multi-threading parallelization with a single node ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Parallelization can be achieved by setting the argument ``n_jobs`` in :func:`optuna.study.Study.optimize`. However, the python code will not be faster due to GIL because :func:`optuna.study.Study.optimize` with ``n_jobs!=1`` uses multi-threading. While optimizing, it will be faster in limited situations, such as waiting for other server requests or C/C++ processing with numpy, etc., but it will not be faster in other cases. For more information about 1., see APIReference_. .. _APIReference: https://optuna.readthedocs.io/en/stable/reference/index.html 2. Multi-processing parallelization with single node ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This can be achieved by using :class:`~optuna.storages.JournalFileStorage` or client/server RDBs (such as PostgreSQL and MySQL). For more information about 2., see TutorialEasyParallelization_. .. _TutorialEasyParallelization: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html 3. Multi-processing parallelization with multiple nodes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This can be achieved by using client/server RDBs (such as PostgreSQL and MySQL). However, if you are in the environment where you can not install a client/server RDB, you can not run multi-processing parallelization with multiple nodes. For more information about 3., see TutorialEasyParallelization_. .. _sqlite_concurrency: How can I solve the error that occurs when performing parallel optimization with SQLite3? ----------------------------------------------------------------------------------------- We would never recommend SQLite3 for parallel optimization in the following reasons. - To concurrently evaluate trials enqueued by :func:`~optuna.study.Study.enqueue_trial`, :class:`~optuna.storages.RDBStorage` uses `SELECT ... FOR UPDATE` syntax, which is unsupported in `SQLite3 `_. - As described in `the SQLAlchemy's documentation `_, SQLite3 (and pysqlite driver) does not support a high level of concurrency. You may get a "database is locked" error, which occurs when one thread or process has an exclusive lock on a database connection (in reality a file handle) and another thread times out waiting for the lock to be released. You can increase the default `timeout `_ value like `optuna.storages.RDBStorage("sqlite:///example.db", engine_kwargs={"connect_args": {"timeout": 20.0}})` though. - For distributed optimization via NFS, SQLite3 does not work as described at `FAQ section of sqlite.org `_. If you want to use a file-based Optuna storage for these scenarios, please consider using :class:`~optuna.storages.JournalFileStorage` instead. .. code-block:: python import optuna from optuna.storages import JournalStorage, JournalFileStorage storage = JournalStorage(JournalFileStorage("optuna-journal.log")) study = optuna.create_study(storage=storage) ... See `the Medium blog post `_ for details. .. _heartbeat_monitoring: Can I monitor trials and make them failed automatically when they are killed unexpectedly? ------------------------------------------------------------------------------------------ .. note:: Heartbeat mechanism is experimental. API would change in the future. A process running a trial could be killed unexpectedly, typically by a job scheduler in a cluster environment. If trials are killed unexpectedly, they will be left on the storage with their states `RUNNING` until we remove them or update their state manually. For such a case, Optuna supports monitoring trials using `heartbeat `_ mechanism. Using heartbeat, if a process running a trial is killed unexpectedly, Optuna will automatically change the state of the trial that was running on that process to :obj:`~optuna.trial.TrialState.FAIL` from :obj:`~optuna.trial.TrialState.RUNNING`. .. code-block:: python import optuna def objective(trial): (Very time-consuming computation) # Recording heartbeats every 60 seconds. # Other processes' trials where more than 120 seconds have passed # since the last heartbeat was recorded will be automatically failed. storage = optuna.storages.RDBStorage(url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120) study = optuna.create_study(storage=storage) study.optimize(objective, n_trials=100) .. note:: The heartbeat is supposed to be used with :meth:`~optuna.study.Study.optimize`. If you use :meth:`~optuna.study.Study.ask` and :meth:`~optuna.study.Study.tell`, please change the state of the killed trials by calling :meth:`~optuna.study.Study.tell` explicitly. You can also execute a callback function to process the failed trial. Optuna provides a callback to retry failed trials as :class:`~optuna.storages.RetryFailedTrialCallback`. Note that a callback is invoked at a beginning of each trial, which means :class:`~optuna.storages.RetryFailedTrialCallback` will retry failed trials when a new trial starts to evaluate. .. code-block:: python import optuna from optuna.storages import RetryFailedTrialCallback storage = optuna.storages.RDBStorage( url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120, failed_trial_callback=RetryFailedTrialCallback(max_retry=3), ) study = optuna.create_study(storage=storage) How can I deal with permutation as a parameter? ----------------------------------------------- Although it is not straightforward to deal with combinatorial search spaces like permutations with existing API, there exists a convenient technique for handling them. It involves re-parametrization of permutation search space of :math:`n` items as an independent :math:`n`-dimensional integer search space. This technique is based on the concept of `Lehmer code `_. A Lehmer code of a sequence is the sequence of integers in the same size, whose :math:`i`-th entry denotes how many inversions the :math:`i`-th entry of the permutation has after itself. In other words, the :math:`i`-th entry of the Lehmer code represents the number of entries that are located after and are smaller than the :math:`i`-th entry of the original sequence. For instance, the Lehmer code of the permutation :math:`(3, 1, 4, 2, 0)` is :math:`(3, 1, 2, 1, 0)`. Not only does the Lehmer code provide a unique encoding of permutations into an integer space, but it also has some desirable properties. For example, the sum of Lehmer code entries is equal to the minimum number of adjacent transpositions necessary to transform the corresponding permutation into the identity permutation. Additionally, the lexicographical order of the encodings of two permutations is the same as that of the original sequence. Therefore, Lehmer code preserves "closeness" among permutations in some sense, which is important for the optimization algorithm. An Optuna implementation example to solve Euclid TSP is as follows: .. code-block:: python import numpy as np import optuna def decode(lehmer_code: list[int]) -> list[int]: """Decode Lehmer code to permutation. This function decodes Lehmer code represented as a list of integers to a permutation. """ all_indices = list(range(n)) output = [] for k in lehmer_code: value = all_indices[k] output.append(value) all_indices.remove(value) return output # Euclidean coordinates of cities for TSP. city_coordinates = np.array( [[0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [2.0, 2.0], [-1.0, -1.0]] ) n = len(city_coordinates) def objective(trial: optuna.Trial) -> float: # Suggest a permutation in the Lehmer code representation. lehmer_code = [trial.suggest_int(f"x{i}", 0, n - i - 1) for i in range(n)] permutation = decode(lehmer_code) # Calculate the total distance of the suggested path. total_distance = 0.0 for i in range(n): total_distance += np.linalg.norm( city_coordinates[permutation[i]] - city_coordinates[np.roll(permutation, 1)[i]] ) return total_distance study = optuna.create_study() study.optimize(objective, n_trials=10) lehmer_code = study.best_params.values() print(decode(lehmer_code)) How can I ignore duplicated samples? ------------------------------------ Optuna may sometimes suggest parameters evaluated in the past and if you would like to avoid this problem, you can try out the following workaround: .. code-block:: python import optuna from optuna.trial import TrialState def objective(trial): # Sample parameters. x = trial.suggest_int("x", -5, 5) y = trial.suggest_int("y", -5, 5) # Fetch all the trials to consider. # In this example, we use only completed trials, but users can specify other states # such as TrialState.PRUNED and TrialState.FAIL. states_to_consider = (TrialState.COMPLETE,) trials_to_consider = trial.study.get_trials(deepcopy=False, states=states_to_consider) # Check whether we already evaluated the sampled `(x, y)`. for t in reversed(trials_to_consider): if trial.params == t.params: # Use the existing value as trial duplicated the parameters. return t.value # Compute the objective function if the parameters are not duplicated. # We use the 2D sphere function in this example. return x ** 2 + y ** 2 study = optuna.create_study() study.optimize(objective, n_trials=100) optuna-3.5.0/docs/source/index.rst000066400000000000000000000106561453453102400171270ustar00rootroot00000000000000|optunalogo| Optuna: A hyperparameter optimization framework =============================================== *Optuna* is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, *define-by-run* style user API. Thanks to our *define-by-run* API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters. Key Features ------------ Optuna has modern functionalities as follows: - :doc:`Lightweight, versatile, and platform agnostic architecture ` - Handle a wide variety of tasks with a simple installation that has few requirements. - :doc:`Pythonic search spaces ` - Define search spaces using familiar Python syntax including conditionals and loops. - :doc:`Efficient optimization algorithms ` - Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials. - :doc:`Easy parallelization ` - Scale studies to tens or hundreds of workers with little or no changes to the code. - :doc:`Quick visualization ` - Inspect optimization histories from a variety of plotting functions. Basic Concepts -------------- We use the terms *study* and *trial* as follows: - Study: optimization based on an objective function - Trial: a single execution of the objective function Please refer to sample code below. The goal of a *study* is to find out the optimal set of hyperparameter values (e.g., ``classifier`` and ``svm_c``) through multiple *trials* (e.g., ``n_trials=100``). Optuna is a framework designed for the automation and the acceleration of the optimization *studies*. |Open in Colab| .. code:: python import ... # Define an objective function to be minimized. def objective(trial): # Invoke suggest methods of a Trial object to generate hyperparameters. regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest']) if regressor_name == 'SVR': svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True) regressor_obj = sklearn.svm.SVR(C=svr_c) else: rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32) regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth) X, y = sklearn.datasets.fetch_california_housing(return_X_y=True) X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0) regressor_obj.fit(X_train, y_train) y_pred = regressor_obj.predict(X_val) error = sklearn.metrics.mean_squared_error(y_val, y_pred) return error # An objective value linked with the Trial object. study = optuna.create_study() # Create a new study. study.optimize(objective, n_trials=100) # Invoke optimization of the objective function. Communication ------------- - `GitHub Discussions `__ for questions. - `GitHub Issues `__ for bug reports and feature requests. Contribution ------------ Any contributions to Optuna are welcome! When you send a pull request, please follow the `contribution guide `__. License ------- MIT License (see `LICENSE `__). Optuna uses the codes from SciPy and fdlibm projects (see :doc:`Third-party License `). Reference --------- Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (`arXiv `__). .. toctree:: :maxdepth: 2 :caption: Contents: installation tutorial/index reference/index faq Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. |optunalogo| image:: https://raw.githubusercontent.com/optuna/optuna/master/docs/image/optuna-logo.png :width: 800 :alt: OPTUNA .. |Open in Colab| image:: https://colab.research.google.com/assets/colab-badge.svg :target: http://colab.research.google.com/github/optuna/optuna-examples/blob/main/quickstart.ipynb optuna-3.5.0/docs/source/installation.rst000066400000000000000000000006621453453102400205150ustar00rootroot00000000000000Installation ============ Optuna supports Python 3.7 or newer. We recommend to install Optuna via pip: .. code-block:: bash $ pip install optuna You can also install the development version of Optuna from master branch of Git repository: .. code-block:: bash $ pip install git+https://github.com/optuna/optuna.git You can also install Optuna via conda: .. code-block:: bash $ conda install -c conda-forge optuna optuna-3.5.0/docs/source/license_thirdparty.rst000066400000000000000000000036001453453102400217030ustar00rootroot00000000000000:orphan: Third-party License =================== SciPy ----- The Optuna contains the codes from SciPy project. Copyright (c) 2001-2002 Enthought, Inc. 2003-2022, SciPy Developers. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. fdlibm ------ Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. Developed at SunPro, a Sun Microsystems, Inc. business. Permission to use, copy, modify, and distribute this software is freely granted, provided that this notice is preserved. optuna-3.5.0/docs/source/privacy.rst000066400000000000000000000011431453453102400174640ustar00rootroot00000000000000:orphan: Privacy Policy ============== Google Analytics ---------------- To collect information about how visitors use our website and to improve our services, we are using Google Analytics on this website. You can find out more about how Google Analytics works and about how information is collected on the Google Analytics terms of services and on Google's privacy policy. - Google Analytics Terms of Service: http://www.google.com/analytics/terms/us.html - Google Privacy Policy: https://policies.google.com/privacy?hl=en - Google Analytics Opt-out Add-on: https://tools.google.com/dlpage/gaoptout?hl=en optuna-3.5.0/docs/source/reference/000077500000000000000000000000001453453102400172145ustar00rootroot00000000000000optuna-3.5.0/docs/source/reference/artifacts.rst000066400000000000000000000006301453453102400217250ustar00rootroot00000000000000.. module:: optuna.artifacts optuna.artifacts ================ The :mod:`~optuna.artifacts` module provides the way to manage artifacts (output files) in Optuna. .. autosummary:: :toctree: generated/ :nosignatures: optuna.artifacts.FileSystemArtifactStore optuna.artifacts.Boto3ArtifactStore optuna.artifacts.GCSArtifactStore optuna.artifacts.Backoff optuna.artifacts.upload_artifactoptuna-3.5.0/docs/source/reference/cli.rst000066400000000000000000000004351453453102400205170ustar00rootroot00000000000000.. module:: optuna.cli optuna.cli ========== The :mod:`~optuna.cli` module implements Optuna's command-line functionality. For detail, please see the result of .. code-block:: console $ optuna --help .. seealso:: The :ref:`cli` tutorial provides use-cases with examples. optuna-3.5.0/docs/source/reference/distributions.rst000066400000000000000000000023631453453102400226540ustar00rootroot00000000000000.. module:: optuna.distributions optuna.distributions ==================== The :mod:`~optuna.distributions` module defines various classes representing probability distributions, mainly used to suggest initial hyperparameter values for an optimization trial. Distribution classes inherit from a library-internal :class:`~optuna.distributions.BaseDistribution`, and is initialized with specific parameters, such as the ``low`` and ``high`` endpoints for a :class:`~optuna.distributions.IntDistribution`. Optuna users should not use distribution classes directly, but instead use utility functions provided by :class:`~optuna.trial.Trial` such as :meth:`~optuna.trial.Trial.suggest_int`. .. autosummary:: :toctree: generated/ :nosignatures: optuna.distributions.FloatDistribution optuna.distributions.IntDistribution optuna.distributions.UniformDistribution optuna.distributions.LogUniformDistribution optuna.distributions.DiscreteUniformDistribution optuna.distributions.IntUniformDistribution optuna.distributions.IntLogUniformDistribution optuna.distributions.CategoricalDistribution optuna.distributions.distribution_to_json optuna.distributions.json_to_distribution optuna.distributions.check_distribution_compatibility optuna-3.5.0/docs/source/reference/exceptions.rst000066400000000000000000000012231453453102400221250ustar00rootroot00000000000000.. module:: optuna.exceptions optuna.exceptions ================= The :mod:`~optuna.exceptions` module defines Optuna-specific exceptions deriving from a base :class:`~optuna.exceptions.OptunaError` class. Of special importance for library users is the :class:`~optuna.exceptions.TrialPruned` exception to be raised if :func:`optuna.trial.Trial.should_prune` returns ``True`` for a trial that should be pruned. .. autosummary:: :toctree: generated/ :nosignatures: optuna.exceptions.OptunaError optuna.exceptions.TrialPruned optuna.exceptions.CLIUsageError optuna.exceptions.StorageInternalError optuna.exceptions.DuplicatedStudyError optuna-3.5.0/docs/source/reference/importance.rst000066400000000000000000000026271453453102400221160ustar00rootroot00000000000000.. module:: optuna.importance optuna.importance ================= The :mod:`~optuna.importance` module provides functionality for evaluating hyperparameter importances based on completed trials in a given study. The utility function :func:`~optuna.importance.get_param_importances` takes a :class:`~optuna.study.Study` and optional evaluator as two of its inputs. The evaluator must derive from :class:`~optuna.importance.BaseImportanceEvaluator`, and is initialized as a :class:`~optuna.importance.FanovaImportanceEvaluator` by default when not passed in. Users implementing custom evaluators should refer to either :class:`~optuna.importance.FanovaImportanceEvaluator` or :class:`~optuna.importance.MeanDecreaseImpurityImportanceEvaluator` as a guide, paying close attention to the format of the return value from the Evaluator's ``evaluate`` function. .. note:: :class:`~optuna.importance.FanovaImportanceEvaluator` takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `_ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. .. autosummary:: :toctree: generated/ :nosignatures: optuna.importance.get_param_importances optuna.importance.FanovaImportanceEvaluator optuna.importance.MeanDecreaseImpurityImportanceEvaluator optuna-3.5.0/docs/source/reference/index.rst000066400000000000000000000004411453453102400210540ustar00rootroot00000000000000API Reference ============= .. toctree:: :maxdepth: 1 optuna artifacts cli distributions exceptions importance integration logging pruners samplers/index search_space storages study terminator trial visualization/index optuna-3.5.0/docs/source/reference/integration.rst000066400000000000000000000366211453453102400223010ustar00rootroot00000000000000.. module:: optuna.integration optuna.integration ================== The :mod:`~optuna.integration` module contains classes used to integrate Optuna with external machine learning frameworks. .. note:: Optuna's integration modules for third-party libraries have started migrating from Optuna itself to a package called `optuna-integration`. Please check the `repository `_ and the `documentation `_. For most of the ML frameworks supported by Optuna, the corresponding Optuna integration class serves only to implement a callback object and functions, compliant with the framework's specific callback API, to be called with each intermediate step in the model training. The functionality implemented in these callbacks across the different ML frameworks includes: (1) Reporting intermediate model scores back to the Optuna trial using :func:`optuna.trial.Trial.report`, (2) According to the results of :func:`optuna.trial.Trial.should_prune`, pruning the current model by raising :func:`optuna.TrialPruned`, and (3) Reporting intermediate Optuna data such as the current trial number back to the framework, as done in :class:`~optuna.integration.MLflowCallback`. For scikit-learn, an integrated :class:`~optuna.integration.OptunaSearchCV` estimator is available that combines scikit-learn BaseEstimator functionality with access to a class-level ``Study`` object. BoTorch ------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.BoTorchSampler optuna.integration.botorch.ehvi_candidates_func optuna.integration.botorch.logei_candidates_func optuna.integration.botorch.qei_candidates_func optuna.integration.botorch.qnei_candidates_func optuna.integration.botorch.qehvi_candidates_func optuna.integration.botorch.qnehvi_candidates_func optuna.integration.botorch.qparego_candidates_func CatBoost -------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.CatBoostPruningCallback Dask ---- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.DaskStorage LightGBM -------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.LightGBMPruningCallback optuna.integration.lightgbm.train optuna.integration.lightgbm.LightGBMTuner optuna.integration.lightgbm.LightGBMTunerCV MLflow ------ .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.MLflowCallback Weights & Biases ---------------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.WeightsAndBiasesCallback pycma ----- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.PyCmaSampler optuna.integration.CmaEsSampler PyTorch ------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.PyTorchIgnitePruningHandler optuna.integration.PyTorchLightningPruningCallback optuna.integration.TorchDistributedTrial scikit-learn ------------ .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.OptunaSearchCV scikit-optimize --------------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.SkoptSampler TensorFlow ---------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.TensorBoardCallback XGBoost ------- .. autosummary:: :toctree: generated/ :nosignatures: optuna.integration.XGBoostPruningCallback Dependencies of each integration -------------------------------- We summarize the necessary dependencies for each integration. +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | Integration | Dependencies | +===================================================================================================================================================================================+====================================+ | `AllenNLP `_ | allennlp, torch, psutil, jsonnet | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `BoTorch `_ | botorch, gpytorch, torch | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Catalyst `_ | catalyst | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `CatBoost `_ | catboost | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `ChainerMN `_ | chainermn | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Chainer `_ | chainer | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `pycma `_ | cma | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Dask `_ | distributed | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | FastAI (`v1 `_, `v2 `_) | fastai | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Keras `_ | keras | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `LightGBMTuner `_ | lightgbm, scikit-learn | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `LightGBMPruningCallback `_ | lightgbm | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `MLflow `_ | mlflow | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `MXNet `_ | mxnet | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | PyTorch `Distributed `_ | torch | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | PyTorch (`Ignite `_) | pytorch-ignite | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | PyTorch (`Lightning `_) | pytorch-lightning | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `SHAP `_ | scikit-learn, shap | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Scikit-learn `_ | pandas, scipy, scikit-learn | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Scikit-optimize `_ | scikit-optimize | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `SKorch `_ | skorch | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `TensorBoard `_ | tensorboard, tensorflow | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `TensorFlow `_ | tensorflow, tensorflow-estimator | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `TensorFlow + Keras `_ | tensorflow | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `Weights & Biases `_ | wandb | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ | `XGBoost `_ | xgboost | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+ optuna-3.5.0/docs/source/reference/logging.rst000066400000000000000000000013651453453102400214010ustar00rootroot00000000000000.. module:: optuna.logging optuna.logging ============== The :mod:`~optuna.logging` module implements logging using the Python ``logging`` package. Library users may be especially interested in setting verbosity levels using :func:`~optuna.logging.set_verbosity` to one of ``optuna.logging.CRITICAL`` (aka ``optuna.logging.FATAL``), ``optuna.logging.ERROR``, ``optuna.logging.WARNING`` (aka ``optuna.logging.WARN``), ``optuna.logging.INFO``, or ``optuna.logging.DEBUG``. .. autosummary:: :toctree: generated/ :nosignatures: optuna.logging.get_verbosity optuna.logging.set_verbosity optuna.logging.disable_default_handler optuna.logging.enable_default_handler optuna.logging.disable_propagation optuna.logging.enable_propagation optuna-3.5.0/docs/source/reference/optuna.rst000066400000000000000000000011071453453102400212530ustar00rootroot00000000000000.. module:: optuna optuna ====== The :mod:`optuna` module is primarily used as an alias for basic Optuna functionality coded in other modules. Currently, two modules are aliased: (1) from :mod:`optuna.study`, functions regarding the Study lifecycle, and (2) from :mod:`optuna.exceptions`, the TrialPruned Exception raised when a trial is pruned. .. autosummary:: :toctree: generated/ :nosignatures: optuna.create_study optuna.load_study optuna.delete_study optuna.copy_study optuna.get_all_study_names optuna.get_all_study_summaries optuna.TrialPruned optuna-3.5.0/docs/source/reference/pruners.rst000066400000000000000000000023171453453102400214470ustar00rootroot00000000000000.. module:: optuna.pruners optuna.pruners ============== The :mod:`~optuna.pruners` module defines a :class:`~optuna.pruners.BasePruner` class characterized by an abstract :meth:`~optuna.pruners.BasePruner.prune` method, which, for a given trial and its associated study, returns a boolean value representing whether the trial should be pruned. This determination is made based on stored intermediate values of the objective function, as previously reported for the trial using :meth:`optuna.trial.Trial.report`. The remaining classes in this module represent child classes, inheriting from :class:`~optuna.pruners.BasePruner`, which implement different pruning strategies. .. seealso:: :ref:`pruning` tutorial explains the concept of the pruner classes and a minimal example. .. seealso:: :ref:`user_defined_pruner` tutorial could be helpful if you want to implement your own pruner classes. .. autosummary:: :toctree: generated/ :nosignatures: optuna.pruners.BasePruner optuna.pruners.MedianPruner optuna.pruners.NopPruner optuna.pruners.PatientPruner optuna.pruners.PercentilePruner optuna.pruners.SuccessiveHalvingPruner optuna.pruners.HyperbandPruner optuna.pruners.ThresholdPruner optuna-3.5.0/docs/source/reference/samplers/000077500000000000000000000000001453453102400210425ustar00rootroot00000000000000optuna-3.5.0/docs/source/reference/samplers/index.rst000066400000000000000000000347641453453102400227210ustar00rootroot00000000000000.. module:: optuna.samplers optuna.samplers =============== The :mod:`~optuna.samplers` module defines a base class for parameter sampling as described extensively in :class:`~optuna.samplers.BaseSampler`. The remaining classes in this module represent child classes, deriving from :class:`~optuna.samplers.BaseSampler`, which implement different sampling strategies. .. seealso:: :ref:`pruning` tutorial explains the overview of the sampler classes. .. seealso:: :ref:`user_defined_sampler` tutorial could be helpful if you want to implement your own sampler classes. +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | | RandomSampler | GridSampler | TPESampler | CmaEsSampler | NSGAIISampler | QMCSampler | BoTorchSampler | BruteForceSampler | +==================================+===============================+===============================+===============================+===============================+=============================================================================+===============================+===============================+===============================================================================+ | Float parameters |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark` (:math:`\color{red}\times` for infinite domain)| +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Integer parameters |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Categorical parameters |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Pruning |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{red}\times` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Multivariate optimization | :math:`\blacktriangle` | :math:`\blacktriangle` |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Conditional search space |:math:`\color{green}\checkmark`| :math:`\blacktriangle` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Multi-objective optimization |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{red}\times` |:math:`\color{green}\checkmark` (:math:`\blacktriangle` for single-objective)|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Batch optimization |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Distributed optimization |:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`|:math:`\color{green}\checkmark`| :math:`\color{green}\checkmark` |:math:`\color{green}\checkmark`| :math:`\blacktriangle` | :math:`\color{green}\checkmark` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Constrained optimization | :math:`\color{red}\times` | :math:`\color{red}\times` |:math:`\color{green}\checkmark`| :math:`\color{red}\times` | :math:`\color{green}\checkmark` | :math:`\color{red}\times` |:math:`\color{green}\checkmark`| :math:`\color{red}\times` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Time complexity (per trial) (*) | :math:`O(d)` | :math:`O(dn)` | :math:`O(dn \log n)` | :math:`O(d^3)` | :math:`O(mp^2)` (\*\*\*) | :math:`O(dn)` | :math:`O(n^3)` | :math:`O(d)` | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ | Recommended budgets (#trials) | as many as one likes | number of combinations | 100 – 1000 | 1000 – 10000 | 100 – 10000 | as many as one likes | 10 – 100 | number of combinations | | (**) | | | | | | | | | +----------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------------------------------------------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------------------------------+ .. note:: :math:`\color{green}\checkmark`: Supports this feature. :math:`\blacktriangle`: Works, but inefficiently. :math:`\color{red}\times`: Causes an error, or has no interface. (*): We assumes that :math:`d` is the dimension of the search space, :math:`n` is the number of finished trials, :math:`m` is the number of objectives, and :math:`p` is the population size (algorithm specific parameter). This table shows the time complexity of the sampling algorithms. We may omit other terms that depend on the implementation in Optuna, including :math:`O(d)` to call the sampling methods and :math:`O(n)` to collect the completed trials. This means that, for example, the actual time complexity of :class:`~optuna.samplers.RandomSampler` is :math:`O(d+n+d) = O(d+n)`. From another perspective, with the exception of :class:`~optuna.samplers.NSGAIISampler`, all time complexity is written for single-objective optimization. (**): (1) The budget depends on the number of parameters and the number of objectives. (2) This budget includes ``n_startup_trials`` if a sampler has ``n_startup_trials`` as one of its arguments. (\*\*\*): This time complexity assumes that the number of population size :math:`p` and the number of parallelization are regular. This means that the number of parallelization should not exceed the number of population size :math:`p`. .. note:: For float, integer, or categorical parameters, see :ref:`configurations` tutorial. For pruning, see :ref:`pruning` tutorial. For multivariate optimization, see :class:`~optuna.samplers.BaseSampler`. The multivariate optimization is implemented as :func:`~optuna.samplers.BaseSampler.sample_relative` in Optuna. Please check the concrete documents of samplers for more details. For conditional search space, see :ref:`configurations` tutorial and :class:`~optuna.samplers.TPESampler`. The ``group`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the conditional search space. For multi-objective optimization, see :ref:`multi_objective` tutorial. For batch optimization, see :ref:`Batch-Optimization` tutorial. Note that the ``constant_liar`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the batch optimization. For distributed optimization, see :ref:`distributed` tutorial. Note that the ``constant_liar`` option of :class:`~optuna.samplers.TPESampler` allows :class:`~optuna.samplers.TPESampler` to handle the distributed optimization. For constrained optimization, see an `example `_. .. autosummary:: :toctree: generated/ :nosignatures: optuna.samplers.BaseSampler optuna.samplers.GridSampler optuna.samplers.RandomSampler optuna.samplers.TPESampler optuna.samplers.CmaEsSampler optuna.samplers.PartialFixedSampler optuna.samplers.NSGAIISampler optuna.samplers.NSGAIIISampler optuna.samplers.QMCSampler optuna.samplers.BruteForceSampler optuna.samplers.IntersectionSearchSpace optuna.samplers.intersection_search_space .. note:: The following :mod:`optuna.samplers.nsgaii` module defines crossover operations used by :class:`~optuna.samplers.NSGAIISampler`. .. toctree:: :maxdepth: 1 nsgaii optuna-3.5.0/docs/source/reference/samplers/nsgaii.rst000066400000000000000000000010561453453102400230500ustar00rootroot00000000000000.. module:: optuna.samplers.nsgaii optuna.samplers.nsgaii ====================== The :mod:`~optuna.samplers.nsgaii` module defines crossover operations used by :class:`~optuna.samplers.NSGAIISampler`. .. autosummary:: :toctree: generated/ :nosignatures: optuna.samplers.nsgaii.BaseCrossover optuna.samplers.nsgaii.UniformCrossover optuna.samplers.nsgaii.BLXAlphaCrossover optuna.samplers.nsgaii.SPXCrossover optuna.samplers.nsgaii.SBXCrossover optuna.samplers.nsgaii.VSBXCrossover optuna.samplers.nsgaii.UNDXCrossover optuna-3.5.0/docs/source/reference/search_space.rst000066400000000000000000000005221453453102400223650ustar00rootroot00000000000000.. module:: optuna.search_space optuna.search_space =================== The :mod:`~optuna.search_space` module provides functionality for controlling search space of parameters. .. autosummary:: :toctree: generated/ :nosignatures: optuna.search_space.IntersectionSearchSpace optuna.search_space.intersection_search_space optuna-3.5.0/docs/source/reference/storages.rst000066400000000000000000000014761453453102400216050ustar00rootroot00000000000000.. module:: optuna.storages optuna.storages =============== The :mod:`~optuna.storages` module defines a :class:`~optuna.storages.BaseStorage` class which abstracts a backend database and provides library-internal interfaces to the read/write histories of the studies and trials. Library users who wish to use storage solutions other than the default in-memory storage should use one of the child classes of :class:`~optuna.storages.BaseStorage` documented below. .. autosummary:: :toctree: generated/ :nosignatures: optuna.storages.RDBStorage optuna.storages.RetryFailedTrialCallback optuna.storages.fail_stale_trials optuna.storages.JournalStorage optuna.storages.JournalFileStorage optuna.storages.JournalFileSymlinkLock optuna.storages.JournalFileOpenLock optuna.storages.JournalRedisStorage optuna-3.5.0/docs/source/reference/study.rst000066400000000000000000000014721453453102400211220ustar00rootroot00000000000000.. module:: optuna.study optuna.study ============ The :mod:`~optuna.study` module implements the :class:`~optuna.study.Study` object and related functions. A public constructor is available for the :class:`~optuna.study.Study` class, but direct use of this constructor is not recommended. Instead, library users should create and load a :class:`~optuna.study.Study` using :func:`~optuna.study.create_study` and :func:`~optuna.study.load_study` respectively. .. autosummary:: :toctree: generated/ :nosignatures: optuna.study.Study optuna.study.create_study optuna.study.load_study optuna.study.delete_study optuna.study.copy_study optuna.study.get_all_study_names optuna.study.get_all_study_summaries optuna.study.MaxTrialsCallback optuna.study.StudyDirection optuna.study.StudySummary optuna-3.5.0/docs/source/reference/terminator.rst000066400000000000000000000017201453453102400221320ustar00rootroot00000000000000.. module:: optuna.terminator optuna.terminator ================= The :mod:`~optuna.terminator` module implements a mechanism for automatically terminating the optimization process, accompanied by a callback class for the termination and evaluators for the estimated room for improvement in the optimization and statistical error of the objective function. The terminator stops the optimization process when the estimated potential improvement is smaller than the statistical error. .. autosummary:: :toctree: generated/ :nosignatures: optuna.terminator.BaseTerminator optuna.terminator.Terminator optuna.terminator.BaseImprovementEvaluator optuna.terminator.RegretBoundEvaluator optuna.terminator.BestValueStagnationEvaluator optuna.terminator.BaseErrorEvaluator optuna.terminator.CrossValidationErrorEvaluator optuna.terminator.StaticErrorEvaluator optuna.terminator.TerminatorCallback optuna.terminator.report_cross_validation_scoresoptuna-3.5.0/docs/source/reference/trial.rst000066400000000000000000000014161453453102400210630ustar00rootroot00000000000000.. module:: optuna.trial optuna.trial ============ The :mod:`~optuna.trial` module contains :class:`~optuna.trial.Trial` related classes and functions. A :class:`~optuna.trial.Trial` instance represents a process of evaluating an objective function. This instance is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial's state, and set/get user-defined attributes of the trial, so that Optuna users can define a custom objective function through the interfaces. Basically, Optuna users only use it in their custom objective functions. .. autosummary:: :toctree: generated/ :nosignatures: optuna.trial.Trial optuna.trial.FixedTrial optuna.trial.FrozenTrial optuna.trial.TrialState optuna.trial.create_trial optuna-3.5.0/docs/source/reference/visualization/000077500000000000000000000000001453453102400221155ustar00rootroot00000000000000optuna-3.5.0/docs/source/reference/visualization/index.rst000066400000000000000000000033421453453102400237600ustar00rootroot00000000000000.. module:: optuna.visualization optuna.visualization ==================== The :mod:`~optuna.visualization` module provides utility functions for plotting the optimization process using plotly and matplotlib. Plotting functions generally take a :class:`~optuna.study.Study` object and optional parameters are passed as a list to the ``params`` argument. .. note:: In the :mod:`optuna.visualization` module, the following functions use plotly to create figures, but `JupyterLab`_ cannot render them by default. Please follow this `installation guide`_ to show figures in `JupyterLab`_. .. note:: The :func:`~optuna.visualization.plot_param_importances` requires the Python package of `scikit-learn `_. .. _JupyterLab: https://github.com/jupyterlab/jupyterlab .. _installation guide: https://github.com/plotly/plotly.py#jupyterlab-support .. autosummary:: :toctree: generated/ :nosignatures: optuna.visualization.plot_contour optuna.visualization.plot_edf optuna.visualization.plot_hypervolume_history optuna.visualization.plot_intermediate_values optuna.visualization.plot_optimization_history optuna.visualization.plot_parallel_coordinate optuna.visualization.plot_param_importances optuna.visualization.plot_pareto_front optuna.visualization.plot_rank optuna.visualization.plot_slice optuna.visualization.plot_terminator_improvement optuna.visualization.plot_timeline optuna.visualization.is_available .. note:: The following :mod:`optuna.visualization.matplotlib` module uses Matplotlib as a backend. .. toctree:: :maxdepth: 1 matplotlib .. seealso:: The :ref:`visualization` tutorial provides use-cases with examples. optuna-3.5.0/docs/source/reference/visualization/matplotlib.rst000066400000000000000000000016641453453102400250250ustar00rootroot00000000000000.. module:: optuna.visualization.matplotlib optuna.visualization.matplotlib =============================== .. note:: The following functions use Matplotlib as a backend. .. autosummary:: :toctree: generated/ :nosignatures: optuna.visualization.matplotlib.plot_contour optuna.visualization.matplotlib.plot_edf optuna.visualization.matplotlib.plot_hypervolume_history optuna.visualization.matplotlib.plot_intermediate_values optuna.visualization.matplotlib.plot_optimization_history optuna.visualization.matplotlib.plot_parallel_coordinate optuna.visualization.matplotlib.plot_param_importances optuna.visualization.matplotlib.plot_pareto_front optuna.visualization.matplotlib.plot_rank optuna.visualization.matplotlib.plot_slice optuna.visualization.matplotlib.plot_terminator_improvement optuna.visualization.matplotlib.plot_timeline optuna.visualization.matplotlib.is_available optuna-3.5.0/docs/source/tutorial/000077500000000000000000000000001453453102400171215ustar00rootroot00000000000000optuna-3.5.0/docs/source/tutorial/index.rst000066400000000000000000000030551453453102400207650ustar00rootroot00000000000000:orphan: Tutorial ======== If you are new to Optuna or want a general introduction, we highly recommend the below video. .. raw:: html


Key Features ------------ Showcases Optuna's `Key Features `_. 1. :doc:`10_key_features/001_first` 2. :doc:`10_key_features/002_configurations` 3. :doc:`10_key_features/003_efficient_optimization_algorithms` 4. :doc:`10_key_features/004_distributed` 5. :doc:`10_key_features/005_visualization` Recipes ------- Showcases the recipes that might help you using Optuna with comfort. - :doc:`20_recipes/001_rdb` - :doc:`20_recipes/002_multi_objective` - :doc:`20_recipes/003_attributes` - :doc:`20_recipes/004_cli` - :doc:`20_recipes/005_user_defined_sampler` - :doc:`20_recipes/006_user_defined_pruner` - :doc:`20_recipes/007_optuna_callback` - :doc:`20_recipes/008_specify_params` - :doc:`20_recipes/009_ask_and_tell` - :doc:`20_recipes/010_reuse_best_trial` - :doc:`20_recipes/011_journal_storage` - `Human-in-the-loop Optimization with Optuna Dashboard `_ - :doc:`20_recipes/012_artifact_tutorial` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_ optuna-3.5.0/examples/000077500000000000000000000000001453453102400146445ustar00rootroot00000000000000optuna-3.5.0/examples/README.md000066400000000000000000000002121453453102400161160ustar00rootroot00000000000000Optuna Examples ================ Example files have been moved to [optuna/optuna-examples](https://github.com/optuna/optuna-examples/). optuna-3.5.0/formats.sh000077500000000000000000000045471453453102400150520ustar00rootroot00000000000000#!/bin/bash # As described in `CONTRIBUTING.md`, this script checks and formats Optuna's source codes by # `black`, `blackdoc`, and `isort`. If you pass `-n` as an option, this script checks codes # without updating codebase. missing_dependencies=() command -v black &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(black) fi command -v blackdoc &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(blackdoc) fi command -v flake8 &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(flake8) fi command -v isort &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(isort) fi command -v mypy &> /dev/null if [ $? -eq 1 ] ; then missing_dependencies+=(mypy) fi if [ ! ${#missing_dependencies[@]} -eq 0 ]; then echo "The following dependencies are missing:" "${missing_dependencies[@]}" read -p "Would you like to install the missing dependencies? (y/N): " yn case "$yn" in [yY]*) ;; *) echo "abort." ; exit ;; esac pip install "${missing_dependencies[@]}" fi update=1 while getopts "n" OPT do case $OPT in n) update=0 ;; *) ;; esac done target="optuna tests benchmarks tutorial" mypy_target="optuna tests benchmarks" res_all=0 res_black=$(black $target --check --diff 2>&1) if [ $? -eq 1 ] ; then if [ $update -eq 1 ] ; then echo "black failed. The code will be formatted by black." black $target else echo "$res_black" echo "black failed." res_all=1 fi else echo "black succeeded." fi res_blackdoc=$(blackdoc $target --check --diff 2>&1) if [ $? -eq 1 ] ; then if [ $update -eq 1 ] ; then echo "blackdoc failed. The docstrings will be formatted by blackdoc." blackdoc $target else echo "$res_blackdoc" echo "blackdoc failed." res_all=1 fi else echo "blackdoc succeeded." fi res_flake8=$(flake8 $target) if [ $? -eq 1 ] ; then echo "$res_flake8" echo "flake8 failed." res_all=1 else echo "flake8 succeeded." fi res_isort=$(isort $target --check 2>&1) if [ $? -eq 1 ] ; then if [ $update -eq 1 ] ; then echo "isort failed. The code will be formatted by isort." isort $target else echo "$res_isort" echo "isort failed." res_all=1 fi else echo "isort succeeded." fi res_mypy=$(mypy $mypy_target) if [ $? -eq 1 ] ; then echo "$res_mypy" echo "mypy failed." res_all=1 else echo "mypy succeeded." fi if [ $res_all -eq 1 ] ; then exit 1 fi optuna-3.5.0/optuna/000077500000000000000000000000001453453102400143345ustar00rootroot00000000000000optuna-3.5.0/optuna/__init__.py000066400000000000000000000026511453453102400164510ustar00rootroot00000000000000from optuna import distributions from optuna import exceptions from optuna import integration from optuna import logging from optuna import multi_objective from optuna import pruners from optuna import samplers from optuna import search_space from optuna import storages from optuna import study from optuna import trial from optuna import version from optuna._imports import _LazyImport from optuna.exceptions import TrialPruned from optuna.study import copy_study from optuna.study import create_study from optuna.study import delete_study from optuna.study import get_all_study_names from optuna.study import get_all_study_summaries from optuna.study import load_study from optuna.study import Study from optuna.trial import create_trial from optuna.trial import Trial from optuna.version import __version__ __all__ = [ "Study", "Trial", "TrialPruned", "__version__", "artifacts", "copy_study", "create_study", "create_trial", "delete_study", "distributions", "exceptions", "get_all_study_names", "get_all_study_summaries", "importance", "integration", "load_study", "logging", "multi_objective", "pruners", "samplers", "search_space", "storages", "study", "trial", "version", "visualization", ] artifacts = _LazyImport("optuna.artifacts") importance = _LazyImport("optuna.importance") visualization = _LazyImport("optuna.visualization") optuna-3.5.0/optuna/_callbacks.py000066400000000000000000000144341453453102400167720ustar00rootroot00000000000000from typing import Any from typing import Container from typing import Dict from typing import List from typing import Optional import optuna from optuna._experimental import experimental_class from optuna._experimental import experimental_func from optuna.trial import FrozenTrial from optuna.trial import TrialState class MaxTrialsCallback: """Set a maximum number of trials before ending the study. While the ``n_trials`` argument of :meth:`optuna.study.Study.optimize` sets the number of trials that will be run, you may want to continue running until you have a certain number of successfully completed trials or stop the study when you have a certain number of trials that fail. This ``MaxTrialsCallback`` class allows you to set a maximum number of trials for a particular :class:`~optuna.trial.TrialState` before stopping the study. Example: .. testcode:: import optuna from optuna.study import MaxTrialsCallback from optuna.trial import TrialState def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize( objective, callbacks=[MaxTrialsCallback(10, states=(TrialState.COMPLETE,))], ) Args: n_trials: The max number of trials. Must be set to an integer. states: Tuple of the :class:`~optuna.trial.TrialState` to be counted towards the max trials limit. Default value is ``(TrialState.COMPLETE,)``. If :obj:`None`, count all states. """ def __init__( self, n_trials: int, states: Optional[Container[TrialState]] = (TrialState.COMPLETE,) ) -> None: self._n_trials = n_trials self._states = states def __call__(self, study: "optuna.study.Study", trial: FrozenTrial) -> None: trials = study.get_trials(deepcopy=False, states=self._states) n_complete = len(trials) if n_complete >= self._n_trials: study.stop() @experimental_class("2.8.0") class RetryFailedTrialCallback: """Retry a failed trial up to a maximum number of times. When a trial fails, this callback can be used with a class in :mod:`optuna.storages` to recreate the trial in ``TrialState.WAITING`` to queue up the trial to be run again. The failed trial can be identified by the :func:`~optuna.storages.RetryFailedTrialCallback.retried_trial_number` function. Even if repetitive failure occurs (a retried trial fails again), this method returns the number of the original trial. To get a full list including the numbers of the retried trials as well as their original trial, call the :func:`~optuna.storages.RetryFailedTrialCallback.retry_history` function. This callback is helpful in environments where trials may fail due to external conditions, such as being preempted by other processes. Usage: .. testcode:: import optuna from optuna.storages import RetryFailedTrialCallback storage = optuna.storages.RDBStorage( url="sqlite:///:memory:", heartbeat_interval=60, grace_period=120, failed_trial_callback=RetryFailedTrialCallback(max_retry=3), ) study = optuna.create_study( storage=storage, ) .. seealso:: See :class:`~optuna.storages.RDBStorage`. Args: max_retry: The max number of times a trial can be retried. Must be set to :obj:`None` or an integer. If set to the default value of :obj:`None` will retry indefinitely. If set to an integer, will only retry that many times. inherit_intermediate_values: Option to inherit `trial.intermediate_values` reported by :func:`optuna.trial.Trial.report` from the failed trial. Default is :obj:`False`. """ def __init__( self, max_retry: Optional[int] = None, inherit_intermediate_values: bool = False ) -> None: self._max_retry = max_retry self._inherit_intermediate_values = inherit_intermediate_values def __call__(self, study: "optuna.study.Study", trial: FrozenTrial) -> None: system_attrs: Dict[str, Any] = { "failed_trial": trial.number, "retry_history": [], **trial.system_attrs, } system_attrs["retry_history"].append(trial.number) if self._max_retry is not None: if self._max_retry < len(system_attrs["retry_history"]): return study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.WAITING, params=trial.params, distributions=trial.distributions, user_attrs=trial.user_attrs, system_attrs=system_attrs, intermediate_values=( trial.intermediate_values if self._inherit_intermediate_values else None ), ) ) @staticmethod @experimental_func("2.8.0") def retried_trial_number(trial: FrozenTrial) -> Optional[int]: """Return the number of the original trial being retried. Args: trial: The trial object. Returns: The number of the first failed trial. If not retry of a previous trial, returns :obj:`None`. """ return trial.system_attrs.get("failed_trial", None) @staticmethod @experimental_func("3.0.0") def retry_history(trial: FrozenTrial) -> List[int]: """Return the list of retried trial numbers with respect to the specified trial. Args: trial: The trial object. Returns: A list of trial numbers in ascending order of the series of retried trials. The first item of the list indicates the original trial which is identical to the :func:`~optuna.storages.RetryFailedTrialCallback.retried_trial_number`, and the last item is the one right before the specified trial in the retry series. If the specified trial is not a retry of any trial, returns an empty list. """ return trial.system_attrs.get("retry_history", []) optuna-3.5.0/optuna/_convert_positional_args.py000066400000000000000000000062721453453102400220110ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from functools import wraps from inspect import Parameter from inspect import signature from typing import Any from typing import TYPE_CHECKING from typing import TypeVar import warnings if TYPE_CHECKING: from typing_extensions import ParamSpec _P = ParamSpec("_P") _T = TypeVar("_T") def _get_positional_arg_names(func: "Callable[_P, _T]") -> list[str]: params = signature(func).parameters positional_arg_names = [ name for name, p in params.items() if p.default == Parameter.empty and p.kind == p.POSITIONAL_OR_KEYWORD ] return positional_arg_names def _infer_kwargs(previous_positional_arg_names: Sequence[str], *args: Any) -> dict[str, Any]: inferred_kwargs = {arg_name: val for val, arg_name in zip(args, previous_positional_arg_names)} return inferred_kwargs def convert_positional_args( *, previous_positional_arg_names: Sequence[str], warning_stacklevel: int = 2, ) -> "Callable[[Callable[_P, _T]], Callable[_P, _T]]": """Convert positional arguments to keyword arguments. Args: previous_positional_arg_names: List of names previously given as positional arguments. warning_stacklevel: Level of the stack trace where decorated function locates. """ def converter_decorator(func: "Callable[_P, _T]") -> "Callable[_P, _T]": assert set(previous_positional_arg_names).issubset(set(signature(func).parameters)), ( f"{set(previous_positional_arg_names)} is not a subset of" f" {set(signature(func).parameters)}" ) @wraps(func) def converter_wrapper(*args: Any, **kwargs: Any) -> "_T": positional_arg_names = _get_positional_arg_names(func) inferred_kwargs = _infer_kwargs(previous_positional_arg_names, *args) if len(inferred_kwargs) > len(positional_arg_names): expected_kwds = set(inferred_kwargs) - set(positional_arg_names) warnings.warn( f"{func.__name__}() got {expected_kwds} as positional arguments " "but they were expected to be given as keyword arguments.", FutureWarning, stacklevel=warning_stacklevel, ) if len(args) > len(previous_positional_arg_names): raise TypeError( f"{func.__name__}() takes {len(previous_positional_arg_names)} positional" f" arguments but {len(args)} were given." ) duplicated_kwds = set(kwargs).intersection(inferred_kwargs) if len(duplicated_kwds): # When specifying positional arguments that are not located at the end of args as # keyword arguments, raise TypeError as follows by imitating the Python standard # behavior raise TypeError( f"{func.__name__}() got multiple values for arguments {duplicated_kwds}." ) kwargs.update(inferred_kwargs) return func(**kwargs) return converter_wrapper return converter_decorator optuna-3.5.0/optuna/_deprecated.py000066400000000000000000000154451453453102400171560ustar00rootroot00000000000000import functools import textwrap from typing import Any from typing import Callable from typing import Optional from typing import TYPE_CHECKING from typing import TypeVar import warnings from packaging import version from optuna._experimental import _get_docstring_indent from optuna._experimental import _validate_version if TYPE_CHECKING: from typing_extensions import ParamSpec FT = TypeVar("FT") FP = ParamSpec("FP") CT = TypeVar("CT") _DEPRECATION_NOTE_TEMPLATE = """ .. warning:: Deprecated in v{d_ver}. This feature will be removed in the future. The removal of this feature is currently scheduled for v{r_ver}, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v{d_ver}. """ _DEPRECATION_WARNING_TEMPLATE = ( "{name} has been deprecated in v{d_ver}. " "This feature will be removed in v{r_ver}. " "See https://github.com/optuna/optuna/releases/tag/v{d_ver}." ) def _validate_two_version(old_version: str, new_version: str) -> None: if version.parse(old_version) > version.parse(new_version): raise ValueError( "Invalid version relationship. The deprecated version must be smaller than " "the removed version, but (deprecated version, removed version) = ({}, {}) are " "specified.".format(old_version, new_version) ) def _format_text(text: str) -> str: return "\n\n" + textwrap.indent(text.strip(), " ") + "\n" def deprecated_func( deprecated_version: str, removed_version: str, name: Optional[str] = None, text: Optional[str] = None, ) -> "Callable[[Callable[FP, FT]], Callable[FP, FT]]": """Decorate function as deprecated. Args: deprecated_version: The version in which the target feature is deprecated. removed_version: The version in which the target feature will be removed. name: The name of the feature. Defaults to the function name. Optional. text: The additional text for the deprecation note. The default note is build using specified ``deprecated_version`` and ``removed_version``. If you want to provide additional information, please specify this argument yourself. .. note:: The default deprecation note is as follows: "Deprecated in v{d_ver}. This feature will be removed in the future. The removal of this feature is currently scheduled for v{r_ver}, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v{d_ver}." .. note:: The specified text is concatenated after the default deprecation note. """ _validate_version(deprecated_version) _validate_version(removed_version) _validate_two_version(deprecated_version, removed_version) def decorator(func: "Callable[FP, FT]") -> "Callable[FP, FT]": if func.__doc__ is None: func.__doc__ = "" note = _DEPRECATION_NOTE_TEMPLATE.format(d_ver=deprecated_version, r_ver=removed_version) if text is not None: note += _format_text(text) indent = _get_docstring_indent(func.__doc__) func.__doc__ = func.__doc__.strip() + textwrap.indent(note, indent) + indent @functools.wraps(func) def wrapper(*args: Any, **kwargs: Any) -> "FT": """Decorates a function as deprecated. This decorator is supposed to be applied to the deprecated function. """ message = _DEPRECATION_WARNING_TEMPLATE.format( name=(name if name is not None else func.__name__), d_ver=deprecated_version, r_ver=removed_version, ) if text is not None: message += " " + text warnings.warn(message, FutureWarning, stacklevel=2) return func(*args, **kwargs) return wrapper return decorator def deprecated_class( deprecated_version: str, removed_version: str, name: Optional[str] = None, text: Optional[str] = None, ) -> "Callable[[CT], CT]": """Decorate class as deprecated. Args: deprecated_version: The version in which the target feature is deprecated. removed_version: The version in which the target feature will be removed. name: The name of the feature. Defaults to the class name. Optional. text: The additional text for the deprecation note. The default note is build using specified ``deprecated_version`` and ``removed_version``. If you want to provide additional information, please specify this argument yourself. .. note:: The default deprecation note is as follows: "Deprecated in v{d_ver}. This feature will be removed in the future. The removal of this feature is currently scheduled for v{r_ver}, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v{d_ver}." .. note:: The specified text is concatenated after the default deprecation note. """ _validate_version(deprecated_version) _validate_version(removed_version) _validate_two_version(deprecated_version, removed_version) def decorator(cls: "CT") -> "CT": def wrapper(cls: "CT") -> "CT": """Decorates a class as deprecated. This decorator is supposed to be applied to the deprecated class. """ _original_init = getattr(cls, "__init__") _original_name = getattr(cls, "__name__") @functools.wraps(_original_init) def wrapped_init(self: Any, *args: Any, **kwargs: Any) -> None: message = _DEPRECATION_WARNING_TEMPLATE.format( name=(name if name is not None else _original_name), d_ver=deprecated_version, r_ver=removed_version, ) if text is not None: message += " " + text warnings.warn( message, FutureWarning, stacklevel=2, ) _original_init(self, *args, **kwargs) setattr(cls, "__init__", wrapped_init) if cls.__doc__ is None: cls.__doc__ = "" note = _DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) if text is not None: note += _format_text(text) indent = _get_docstring_indent(cls.__doc__) cls.__doc__ = cls.__doc__.strip() + textwrap.indent(note, indent) + indent return cls return wrapper(cls) return decorator optuna-3.5.0/optuna/_experimental.py000066400000000000000000000073671453453102400175570ustar00rootroot00000000000000import functools import textwrap from typing import Any from typing import Callable from typing import Optional from typing import TYPE_CHECKING from typing import TypeVar import warnings from optuna.exceptions import ExperimentalWarning if TYPE_CHECKING: from typing_extensions import ParamSpec FT = TypeVar("FT") FP = ParamSpec("FP") CT = TypeVar("CT") _EXPERIMENTAL_NOTE_TEMPLATE = """ .. note:: Added in v{ver} as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v{ver}. """ def _validate_version(version: str) -> None: if not isinstance(version, str) or len(version.split(".")) != 3: raise ValueError( "Invalid version specification. Must follow `x.y.z` format but `{}` is given".format( version ) ) def _get_docstring_indent(docstring: str) -> str: return docstring.split("\n")[-1] if "\n" in docstring else "" def experimental_func( version: str, name: Optional[str] = None, ) -> "Callable[[Callable[FP, FT]], Callable[FP, FT]]": """Decorate function as experimental. Args: version: The first version that supports the target feature. name: The name of the feature. Defaults to the function name. Optional. """ _validate_version(version) def decorator(func: "Callable[FP, FT]") -> "Callable[FP, FT]": if func.__doc__ is None: func.__doc__ = "" note = _EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) indent = _get_docstring_indent(func.__doc__) func.__doc__ = func.__doc__.strip() + textwrap.indent(note, indent) + indent @functools.wraps(func) def wrapper(*args: Any, **kwargs: Any) -> "FT": warnings.warn( "{} is experimental (supported from v{}). " "The interface can change in the future.".format( name if name is not None else func.__name__, version ), ExperimentalWarning, stacklevel=2, ) return func(*args, **kwargs) return wrapper return decorator def experimental_class( version: str, name: Optional[str] = None, ) -> "Callable[[CT], CT]": """Decorate class as experimental. Args: version: The first version that supports the target feature. name: The name of the feature. Defaults to the class name. Optional. """ _validate_version(version) def decorator(cls: "CT") -> "CT": def wrapper(cls: "CT") -> "CT": """Decorates a class as experimental. This decorator is supposed to be applied to the experimental class. """ _original_init = getattr(cls, "__init__") _original_name = getattr(cls, "__name__") @functools.wraps(_original_init) def wrapped_init(self: Any, *args: Any, **kwargs: Any) -> None: warnings.warn( "{} is experimental (supported from v{}). " "The interface can change in the future.".format( name if name is not None else _original_name, version ), ExperimentalWarning, stacklevel=2, ) _original_init(self, *args, **kwargs) setattr(cls, "__init__", wrapped_init) if cls.__doc__ is None: cls.__doc__ = "" note = _EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) indent = _get_docstring_indent(cls.__doc__) cls.__doc__ = cls.__doc__.strip() + textwrap.indent(note, indent) + indent return cls return wrapper(cls) return decorator optuna-3.5.0/optuna/_hypervolume/000077500000000000000000000000001453453102400170525ustar00rootroot00000000000000optuna-3.5.0/optuna/_hypervolume/__init__.py000066400000000000000000000005651453453102400211710ustar00rootroot00000000000000from optuna._hypervolume.base import BaseHypervolume from optuna._hypervolume.hssp import _solve_hssp from optuna._hypervolume.utils import _compute_2d from optuna._hypervolume.utils import _compute_2points_volume from optuna._hypervolume.wfg import WFG __all__ = [ "BaseHypervolume", "_compute_2d", "_compute_2points_volume", "_solve_hssp", "WFG", ] optuna-3.5.0/optuna/_hypervolume/base.py000066400000000000000000000054671453453102400203520ustar00rootroot00000000000000import abc import numpy as np class BaseHypervolume(abc.ABC): """Base class for hypervolume calculators. .. note:: This class is used for computing the hypervolumes of points in multi-objective space. Each coordinate of each point represents one value of the multi-objective function. .. note:: We check that each objective is to be minimized. Transform objective values that are to be maximized before calling this class's ``compute`` method. Example: .. testcode:: import numpy as np import optuna from optuna.multi_objective._hypervolume import WFG def objective(trial): return trial.suggest_float("x", 0, 1), trial.suggest_float("y", 0, 1) study = optuna.multi_objective.create_study(["maximize", "minimize"]) study.optimize(objective, n_trials=10) trials = study.get_pareto_front_trials() solution_sets = np.array([list(t.values) for t in trials]) # Normalize the solution set by negating. solution_sets = np.array([[-s[0], s[1]] for s in solution_sets]) # A reference point is dominated by all points. reference_point = np.max(solution_sets, axis=0) + 1 hypervolume = WFG().compute(solution_sets, reference_point) print("Hypervolume of the Pareto solutions is {}.".format(hypervolume)) """ def compute(self, solution_set: np.ndarray, reference_point: np.ndarray) -> float: """Compute the hypervolume for the given solution set and reference point. .. note:: We assume that all points in the solution set dominate or equal the reference point. In other words, for all points in the solution set and the coordinate ``i``, ``point[i] <= reference_point[i]``. Args: solution_set: The solution set which we want to compute the hypervolume. reference_point: The reference point to compute the hypervolume. """ self._validate(solution_set, reference_point) return self._compute(solution_set, reference_point) @staticmethod def _validate(solution_set: np.ndarray, reference_point: np.ndarray) -> None: # Validates that all points in the solution set dominate or equal the reference point. if not (solution_set <= reference_point).all(): raise ValueError( "All points must dominate or equal the reference point. " "That is, for all points in the solution_set and the coordinate `i`, " "`point[i] <= reference_point[i]`." ) @abc.abstractmethod def _compute(self, solution_set: np.ndarray, reference_point: np.ndarray) -> float: raise NotImplementedError optuna-3.5.0/optuna/_hypervolume/hssp.py000066400000000000000000000032031453453102400203770ustar00rootroot00000000000000from typing import List import numpy as np import optuna def _solve_hssp( rank_i_loss_vals: np.ndarray, rank_i_indices: np.ndarray, subset_size: int, reference_point: np.ndarray, ) -> np.ndarray: """Solve a hypervolume subset selection problem (HSSP) via a greedy algorithm. This method is a 1-1/e approximation algorithm to solve HSSP. For further information about algorithms to solve HSSP, please refer to the following paper: - `Greedy Hypervolume Subset Selection in Low Dimensions `_ """ selected_vecs: List[np.ndarray] = [] selected_indices: List[int] = [] contributions = [ optuna._hypervolume.WFG().compute(np.asarray([v]), reference_point) for v in rank_i_loss_vals ] hv_selected = 0.0 while len(selected_indices) < subset_size: max_index = int(np.argmax(contributions)) contributions[max_index] = -1 # mark as selected selected_index = rank_i_indices[max_index] selected_vec = rank_i_loss_vals[max_index] for j, v in enumerate(rank_i_loss_vals): if contributions[j] == -1: continue p = np.max([selected_vec, v], axis=0) contributions[j] -= ( optuna._hypervolume.WFG().compute(np.asarray(selected_vecs + [p]), reference_point) - hv_selected ) selected_vecs += [selected_vec] selected_indices += [selected_index] hv_selected = optuna._hypervolume.WFG().compute(np.asarray(selected_vecs), reference_point) return np.asarray(selected_indices, dtype=int) optuna-3.5.0/optuna/_hypervolume/utils.py000066400000000000000000000026641453453102400205740ustar00rootroot00000000000000import numpy as np def _compute_2points_volume(point1: np.ndarray, point2: np.ndarray) -> float: """Compute the hypervolume of the hypercube, whose diagonal endpoints are given 2 points. Args: point1: The first endpoint of the hypercube's diagonal. point2: The second endpoint of the hypercube's diagonal. """ return float(np.abs(np.prod(point1 - point2))) def _compute_2d(solution_set: np.ndarray, reference_point: np.ndarray) -> float: """Compute the hypervolume for the two-dimensional space. This algorithm divides a hypervolume into smaller rectangles and sum these areas. Args: solution_set: The solution set which we want to compute the hypervolume. reference_point: The reference point to compute the hypervolume. """ sorted_solution_set = solution_set[np.lexsort((-solution_set[:, 1], solution_set[:, 0]))] hypervolume = 0.0 for solution in sorted_solution_set: if reference_point[1] < solution[1]: continue # Compute an area of a rectangle with reference_point # and one point in solution_sets as diagonals. hypervolume += _compute_2points_volume(reference_point, solution) # Update a reference_point to create a new hypervolume that # exclude a rectangle from the current hypervolume reference_point[1] = solution[1] return hypervolume optuna-3.5.0/optuna/_hypervolume/wfg.py000066400000000000000000000100421453453102400202040ustar00rootroot00000000000000from typing import Optional import numpy as np from optuna._hypervolume import _compute_2d from optuna._hypervolume import _compute_2points_volume from optuna._hypervolume import BaseHypervolume class WFG(BaseHypervolume): """Hypervolume calculator for any dimension. This class exactly calculates the hypervolume for any dimension by using the WFG algorithm. For detail, see `While, Lyndon, Lucas Bradstreet, and Luigi Barone. "A fast way of calculating exact hypervolumes." Evolutionary Computation, IEEE Transactions on 16.1 (2012) : 86-95.`. """ def __init__(self) -> None: self._reference_point: Optional[np.ndarray] = None def _compute(self, solution_set: np.ndarray, reference_point: np.ndarray) -> float: self._reference_point = reference_point.astype(np.float64) return self._compute_rec(solution_set.astype(np.float64)) def _compute_rec(self, solution_set: np.ndarray) -> float: assert self._reference_point is not None n_points = solution_set.shape[0] if self._reference_point.shape[0] == 2: return _compute_2d(solution_set, self._reference_point) if n_points == 1: return _compute_2points_volume(solution_set[0], self._reference_point) elif n_points == 2: volume = 0.0 volume += _compute_2points_volume(solution_set[0], self._reference_point) volume += _compute_2points_volume(solution_set[1], self._reference_point) intersection = self._reference_point - np.maximum(solution_set[0], solution_set[1]) volume -= np.prod(intersection) return volume solution_set = solution_set[solution_set[:, 0].argsort()] # n_points >= 3 volume = 0.0 for i in range(n_points): volume += self._compute_exclusive_hv(solution_set[i], solution_set[i + 1 :]) return volume def _compute_exclusive_hv(self, point: np.ndarray, solution_set: np.ndarray) -> float: assert self._reference_point is not None volume = _compute_2points_volume(point, self._reference_point) limited_solution_set = self._limit(point, solution_set) n_points_of_s = limited_solution_set.shape[0] if n_points_of_s == 1: volume -= _compute_2points_volume(limited_solution_set[0], self._reference_point) elif n_points_of_s > 1: volume -= self._compute_rec(limited_solution_set) return volume @staticmethod def _limit(point: np.ndarray, solution_set: np.ndarray) -> np.ndarray: """Limit the points in the solution set for the given point. Let `S := solution set`, `p := point` and `d := dim(p)`. The returned solution set `S'` is `S' = Pareto({s' | for all i in [d], exists s in S, s'_i = max(s_i, p_i)})`, where `Pareto(T) = the points in T which are Pareto optimal`. """ n_points_of_s = solution_set.shape[0] limited_solution_set = np.maximum(solution_set, point) # Return almost Pareto optimal points for computational efficiency. # If the points in the solution set are completely sorted along all coordinates, # the following procedures return the complete Pareto optimal points. # For the computational efficiency, we do not completely sort the points, # but just sort the points according to its 0-th dimension. if n_points_of_s <= 1: return limited_solution_set else: # Assume limited_solution_set is sorted by its 0th dimension. # Therefore, we can simply scan the limited solution set from left to right. returned_limited_solution_set = [limited_solution_set[0]] left = 0 right = 1 while right < n_points_of_s: if (limited_solution_set[left] > limited_solution_set[right]).any(): left = right returned_limited_solution_set.append(limited_solution_set[left]) right += 1 return np.asarray(returned_limited_solution_set) optuna-3.5.0/optuna/_imports.py000066400000000000000000000077451453453102400165570ustar00rootroot00000000000000import importlib import types from types import TracebackType from typing import Any from typing import Optional from typing import Tuple from typing import Type class _DeferredImportExceptionContextManager: """Context manager to defer exceptions from imports. Catches :exc:`ImportError` and :exc:`SyntaxError`. If any exception is caught, this class raises an :exc:`ImportError` when being checked. """ def __init__(self) -> None: self._deferred: Optional[Tuple[Exception, str]] = None def __enter__(self) -> "_DeferredImportExceptionContextManager": """Enter the context manager. Returns: Itself. """ return self def __exit__( self, exc_type: Optional[Type[Exception]], exc_value: Optional[Exception], traceback: Optional[TracebackType], ) -> Optional[bool]: """Exit the context manager. Args: exc_type: Raised exception type. :obj:`None` if nothing is raised. exc_value: Raised exception object. :obj:`None` if nothing is raised. traceback: Associated traceback. :obj:`None` if nothing is raised. Returns: :obj:`None` if nothing is deferred, otherwise :obj:`True`. :obj:`True` will suppress any exceptions avoiding them from propagating. """ if isinstance(exc_value, (ImportError, SyntaxError)): if isinstance(exc_value, ImportError): message = ( "Tried to import '{}' but failed. Please make sure that the package is " "installed correctly to use this feature. Actual error: {}." ).format(exc_value.name, exc_value) elif isinstance(exc_value, SyntaxError): message = ( "Tried to import a package but failed due to a syntax error in {}. Please " "make sure that the Python version is correct to use this feature. Actual " "error: {}." ).format(exc_value.filename, exc_value) else: assert False self._deferred = (exc_value, message) return True return None def is_successful(self) -> bool: """Return whether the context manager has caught any exceptions. Returns: :obj:`True` if no exceptions are caught, :obj:`False` otherwise. """ return self._deferred is None def check(self) -> None: """Check whether the context manager has caught any exceptions. Raises: :exc:`ImportError`: If any exception was caught from the caught exception. """ if self._deferred is not None: exc_value, message = self._deferred raise ImportError(message) from exc_value def try_import() -> _DeferredImportExceptionContextManager: """Create a context manager that can wrap imports of optional packages to defer exceptions. Returns: Deferred import context manager. """ return _DeferredImportExceptionContextManager() class _LazyImport(types.ModuleType): """Module wrapper for lazy import. This class wraps the specified modules and lazily imports them only when accessed. Otherwise, `import optuna` is slowed down by importing all submodules and dependencies even if not required. Within this project's usage, importlib override this module's attribute on the first access and the imported submodule is directly accessed from the second access. Args: name: Name of module to apply lazy import. """ def __init__(self, name: str) -> None: super().__init__(name) self._name = name def _load(self) -> types.ModuleType: module = importlib.import_module(self._name) self.__dict__.update(module.__dict__) return module def __getattr__(self, item: str) -> Any: return getattr(self._load(), item) optuna-3.5.0/optuna/_transform.py000066400000000000000000000275571453453102400171000ustar00rootroot00000000000000import math from typing import Any from typing import Dict from typing import List from typing import Tuple from typing import Union import numpy from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution class _SearchSpaceTransform: """Transform a search space and parameter configurations to continuous space. The search space bounds and parameter configurations are represented as ``numpy.ndarray``s and transformed into continuous space. Bounds and parameters associated with categorical distributions are one-hot encoded. Parameter configurations in this space can additionally be untransformed, or mapped back to the original space. This type of transformation/untransformation is useful for e.g. implementing samplers without having to condition on distribution types before sampling parameter values. Args: search_space: The search space. If any transformations are to be applied, parameter configurations are assumed to hold parameter values for all of the distributions defined in this search space. Otherwise, assertion failures will be raised. transform_log: If :obj:`True`, apply log/exp operations to the bounds and parameters with corresponding distributions in log space during transformation/untransformation. Should always be :obj:`True` if any parameters are going to be sampled from the transformed space. transform_step: If :obj:`True`, offset the lower and higher bounds by a half step each, increasing the space by one step. This allows fair sampling for values close to the bounds. Should always be :obj:`True` if any parameters are going to be sampled from the transformed space. transform_0_1: If :obj:`True`, apply a linear transformation to the bounds and parameters so that they are in the unit cube. Attributes: bounds: Constructed bounds from the given search space. column_to_encoded_columns: Constructed mapping from original parameter column index to encoded column indices. encoded_column_to_column: Constructed mapping from encoded column index to original parameter column index. Note: Parameter values are not scaled to the unit cube. Note: ``transform_log`` and ``transform_step`` are useful for constructing bounds and parameters without any actual transformations by setting those arguments to :obj:`False`. This is needed for e.g. the hyperparameter importance assessments. """ def __init__( self, search_space: Dict[str, BaseDistribution], transform_log: bool = True, transform_step: bool = True, transform_0_1: bool = False, ) -> None: bounds, column_to_encoded_columns, encoded_column_to_column = _transform_search_space( search_space, transform_log, transform_step ) self._raw_bounds = bounds self._column_to_encoded_columns = column_to_encoded_columns self._encoded_column_to_column = encoded_column_to_column self._search_space = search_space self._transform_log = transform_log self._transform_0_1 = transform_0_1 @property def bounds(self) -> numpy.ndarray: if self._transform_0_1: return numpy.array([[0.0, 1.0]] * self._raw_bounds.shape[0]) else: return self._raw_bounds @property def column_to_encoded_columns(self) -> List[numpy.ndarray]: return self._column_to_encoded_columns @property def encoded_column_to_column(self) -> numpy.ndarray: return self._encoded_column_to_column def transform(self, params: Dict[str, Any]) -> numpy.ndarray: """Transform a parameter configuration from actual values to continuous space. Args: params: A parameter configuration to transform. Returns: A 1-dimensional ``numpy.ndarray`` holding the transformed parameters in the configuration. """ trans_params = numpy.zeros(self._raw_bounds.shape[0], dtype=numpy.float64) bound_idx = 0 for name, distribution in self._search_space.items(): assert name in params, "Parameter configuration must contain all distributions." param = params[name] if isinstance(distribution, CategoricalDistribution): choice_idx = distribution.to_internal_repr(param) trans_params[bound_idx + choice_idx] = 1 bound_idx += len(distribution.choices) else: trans_params[bound_idx] = _transform_numerical_param( param, distribution, self._transform_log ) bound_idx += 1 if self._transform_0_1: single_mask = self._raw_bounds[:, 0] == self._raw_bounds[:, 1] trans_params[single_mask] = 0.5 trans_params[~single_mask] = ( trans_params[~single_mask] - self._raw_bounds[~single_mask, 0] ) / (self._raw_bounds[~single_mask, 1] - self._raw_bounds[~single_mask, 0]) return trans_params def untransform(self, trans_params: numpy.ndarray) -> Dict[str, Any]: """Untransform a parameter configuration from continuous space to actual values. Args: trans_params: A 1-dimensional ``numpy.ndarray`` in the transformed space corresponding to a parameter configuration. Returns: A dictionary of an untransformed parameter configuration. Keys are parameter names. Values are untransformed parameter values. """ assert trans_params.shape == (self._raw_bounds.shape[0],) if self._transform_0_1: trans_params = self._raw_bounds[:, 0] + trans_params * ( self._raw_bounds[:, 1] - self._raw_bounds[:, 0] ) params = {} for (name, distribution), encoded_columns in zip( self._search_space.items(), self.column_to_encoded_columns ): trans_param = trans_params[encoded_columns] if isinstance(distribution, CategoricalDistribution): # Select the highest rated one-hot encoding. param = distribution.to_external_repr(trans_param.argmax()) else: param = _untransform_numerical_param( trans_param.item(), distribution, self._transform_log ) params[name] = param return params def _transform_search_space( search_space: Dict[str, BaseDistribution], transform_log: bool, transform_step: bool ) -> Tuple[numpy.ndarray, List[numpy.ndarray], numpy.ndarray]: assert len(search_space) > 0, "Cannot transform if no distributions are given." n_bounds = sum( len(d.choices) if isinstance(d, CategoricalDistribution) else 1 for d in search_space.values() ) bounds = numpy.empty((n_bounds, 2), dtype=numpy.float64) column_to_encoded_columns: List[numpy.ndarray] = [] encoded_column_to_column = numpy.empty(n_bounds, dtype=numpy.int64) bound_idx = 0 for distribution in search_space.values(): d = distribution if isinstance(d, CategoricalDistribution): n_choices = len(d.choices) bounds[bound_idx : bound_idx + n_choices] = (0, 1) # Broadcast across all choices. encoded_columns = numpy.arange(bound_idx, bound_idx + n_choices) encoded_column_to_column[encoded_columns] = len(column_to_encoded_columns) column_to_encoded_columns.append(encoded_columns) bound_idx += n_choices elif isinstance( d, ( FloatDistribution, IntDistribution, ), ): if isinstance(d, FloatDistribution): if d.step is not None: half_step = 0.5 * d.step if transform_step else 0.0 bds = ( _transform_numerical_param(d.low, d, transform_log) - half_step, _transform_numerical_param(d.high, d, transform_log) + half_step, ) else: bds = ( _transform_numerical_param(d.low, d, transform_log), _transform_numerical_param(d.high, d, transform_log), ) elif isinstance(d, IntDistribution): half_step = 0.5 * d.step if transform_step else 0.0 if d.log: bds = ( _transform_numerical_param(d.low - half_step, d, transform_log), _transform_numerical_param(d.high + half_step, d, transform_log), ) else: bds = ( _transform_numerical_param(d.low, d, transform_log) - half_step, _transform_numerical_param(d.high, d, transform_log) + half_step, ) else: assert False, "Should not reach. Unexpected distribution." bounds[bound_idx] = bds encoded_column = numpy.atleast_1d(bound_idx) encoded_column_to_column[encoded_column] = len(column_to_encoded_columns) column_to_encoded_columns.append(encoded_column) bound_idx += 1 else: assert False, "Should not reach. Unexpected distribution." assert bound_idx == n_bounds return bounds, column_to_encoded_columns, encoded_column_to_column def _transform_numerical_param( param: Union[int, float], distribution: BaseDistribution, transform_log: bool ) -> float: d = distribution if isinstance(d, CategoricalDistribution): assert False, "Should not reach. Should be one-hot encoded." elif isinstance(d, FloatDistribution): if d.log: trans_param = math.log(param) if transform_log else float(param) else: trans_param = float(param) elif isinstance(d, IntDistribution): if d.log: trans_param = math.log(param) if transform_log else float(param) else: trans_param = float(param) else: assert False, "Should not reach. Unexpected distribution." return trans_param def _untransform_numerical_param( trans_param: float, distribution: BaseDistribution, transform_log: bool ) -> Union[int, float]: d = distribution if isinstance(d, CategoricalDistribution): assert False, "Should not reach. Should be one-hot encoded." elif isinstance(d, FloatDistribution): if d.log: param = math.exp(trans_param) if transform_log else trans_param if d.single(): pass else: param = min(param, numpy.nextafter(d.high, d.high - 1)) elif d.step is not None: param = float( numpy.clip( numpy.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high ) ) else: if d.single(): param = trans_param else: param = min(trans_param, numpy.nextafter(d.high, d.high - 1)) elif isinstance(d, IntDistribution): if d.log: if transform_log: param = int(numpy.clip(numpy.round(math.exp(trans_param)), d.low, d.high)) else: param = int(trans_param) else: param = int( numpy.clip( numpy.round((trans_param - d.low) / d.step) * d.step + d.low, d.low, d.high ) ) else: assert False, "Should not reach. Unexpected distribution." return param optuna-3.5.0/optuna/_typing.py000066400000000000000000000004531453453102400163610ustar00rootroot00000000000000from __future__ import annotations from typing import Mapping from typing import Sequence from typing import Union JSONSerializable = Union[ Mapping[str, "JSONSerializable"], Sequence["JSONSerializable"], str, int, float, bool, None, ] __all__ = ["JSONSerializable"] optuna-3.5.0/optuna/artifacts/000077500000000000000000000000001453453102400163145ustar00rootroot00000000000000optuna-3.5.0/optuna/artifacts/__init__.py000066400000000000000000000006251453453102400204300ustar00rootroot00000000000000from optuna.artifacts._backoff import Backoff from optuna.artifacts._boto3 import Boto3ArtifactStore from optuna.artifacts._filesystem import FileSystemArtifactStore from optuna.artifacts._gcs import GCSArtifactStore from optuna.artifacts._upload import upload_artifact __all__ = [ "FileSystemArtifactStore", "Boto3ArtifactStore", "GCSArtifactStore", "Backoff", "upload_artifact", ] optuna-3.5.0/optuna/artifacts/_backoff.py000066400000000000000000000067621453453102400204330ustar00rootroot00000000000000from __future__ import annotations import logging import time from typing import TYPE_CHECKING from optuna.artifacts.exceptions import ArtifactNotFound _logger = logging.getLogger(__name__) if TYPE_CHECKING: from typing import BinaryIO from ._protocol import ArtifactStore class Backoff: """An artifact store's middleware for exponential backoff. Example: .. code-block:: python import optuna from optuna.artifacts import upload_artifact from optuna.artifacts import Boto3ArtifactStore from optuna.artifacts import Backoff artifact_store = Backoff(Boto3ArtifactStore("my-bucket")) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact(trial, file_path, artifact_store) return ... """ def __init__( self, backend: ArtifactStore, *, max_retries: int = 10, multiplier: float = 2, min_delay: float = 0.1, max_delay: float = 30, ) -> None: # Default sleep seconds: # 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8, 25.6, 30 self._backend = backend assert max_retries > 0 assert multiplier > 0 assert min_delay > 0 assert max_delay > min_delay self._max_retries = max_retries self._multiplier = multiplier self._min_delay = min_delay self._max_delay = max_delay def _get_sleep_secs(self, n_retry: int) -> float: return min(self._min_delay * self._multiplier**n_retry, self._max_delay) def open_reader(self, artifact_id: str) -> BinaryIO: for i in range(self._max_retries): try: return self._backend.open_reader(artifact_id) except ArtifactNotFound: raise except Exception as e: if i == self._max_retries - 1: raise else: _logger.error(f"Failed to open artifact={artifact_id} n_retry={i}", exc_info=e) time.sleep(self._get_sleep_secs(i)) assert False, "must not reach here" def write(self, artifact_id: str, content_body: BinaryIO) -> None: for i in range(self._max_retries): try: self._backend.write(artifact_id, content_body) break except ArtifactNotFound: raise except Exception as e: if i == self._max_retries - 1: raise else: _logger.error(f"Failed to open artifact={artifact_id} n_retry={i}", exc_info=e) content_body.seek(0) time.sleep(self._get_sleep_secs(i)) def remove(self, artifact_id: str) -> None: for i in range(self._max_retries): try: self._backend.remove(artifact_id) except ArtifactNotFound: raise except Exception as e: if i == self._max_retries - 1: raise else: _logger.error(f"Failed to delete artifact={artifact_id}", exc_info=e) time.sleep(self._get_sleep_secs(i)) if TYPE_CHECKING: # A mypy-runtime assertion to ensure that the Backoff middleware implements # all abstract methods in ArtifactStore. from . import FileSystemArtifactStore _: ArtifactStore = Backoff(FileSystemArtifactStore(".")) optuna-3.5.0/optuna/artifacts/_boto3.py000066400000000000000000000065631453453102400200650ustar00rootroot00000000000000from __future__ import annotations import io import shutil from typing import TYPE_CHECKING from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO from mypy_boto3_s3 import S3Client with try_import(): import boto3 from botocore.exceptions import ClientError @experimental_class("3.3.0") class Boto3ArtifactStore: """An artifact backend for Boto3. Args: bucket_name: The name of the bucket to store artifacts. client: A Boto3 client to use for storage operations. If not specified, a new client will be created. avoid_buf_copy: If True, skip procedure to copy the content of the source file object to a buffer before uploading it to S3 ins. This is default to False because using upload_fileobj() method of Boto3 client might close the source file object. Example: .. code-block:: python import optuna from optuna.artifacts import upload_artifact from optuna.artifacts import Boto3ArtifactStore artifact_store = Boto3ArtifactStore("my-bucket") def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact(trial, file_path, artifact_store) return ... """ def __init__( self, bucket_name: str, client: S3Client | None = None, *, avoid_buf_copy: bool = False ) -> None: self.bucket = bucket_name self.client = client or boto3.client("s3") # This flag is added to avoid that upload_fileobj() method of Boto3 client may close the # source file object. See https://github.com/boto/boto3/issues/929. self._avoid_buf_copy = avoid_buf_copy def open_reader(self, artifact_id: str) -> BinaryIO: try: obj = self.client.get_object(Bucket=self.bucket, Key=artifact_id) except ClientError as e: if _is_not_found_error(e): raise ArtifactNotFound( f"Artifact storage with bucket: {self.bucket}, artifact_id: {artifact_id} was" " not found" ) from e raise body = obj.get("Body") assert body is not None return body def write(self, artifact_id: str, content_body: BinaryIO) -> None: fsrc: BinaryIO = content_body if not self._avoid_buf_copy: buf = io.BytesIO() shutil.copyfileobj(content_body, buf) buf.seek(0) fsrc = buf self.client.upload_fileobj(fsrc, self.bucket, artifact_id) def remove(self, artifact_id: str) -> None: self.client.delete_object(Bucket=self.bucket, Key=artifact_id) def _is_not_found_error(e: ClientError) -> bool: error_code = e.response.get("Error", {}).get("Code") http_status_code = e.response.get("ResponseMetadata", {}).get("HTTPStatusCode") return error_code == "NoSuchKey" or http_status_code == 404 if TYPE_CHECKING: # A mypy-runtime assertion to ensure that Boto3ArtifactStore implements all abstract methods # in ArtifactStore. from ._protocol import ArtifactStore _: ArtifactStore = Boto3ArtifactStore("") optuna-3.5.0/optuna/artifacts/_filesystem.py000066400000000000000000000044721453453102400212200ustar00rootroot00000000000000from __future__ import annotations import os from pathlib import Path import shutil from typing import TYPE_CHECKING from optuna._experimental import experimental_class from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO @experimental_class("3.3.0") class FileSystemArtifactStore: """An artifact store for file systems. Args: base_path: The base path to a directory to store artifacts. Example: .. code-block:: python import os import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = FileSystemArtifactStore(base_path=base_path) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact(trial, file_path, artifact_store) return ... """ def __init__(self, base_path: str | Path) -> None: if isinstance(base_path, str): base_path = Path(base_path) # TODO(Shinichi): Check if the base_path is valid directory. self._base_path = base_path def open_reader(self, artifact_id: str) -> BinaryIO: filepath = os.path.join(self._base_path, artifact_id) try: f = open(filepath, "rb") except FileNotFoundError as e: raise ArtifactNotFound("not found") from e return f def write(self, artifact_id: str, content_body: BinaryIO) -> None: filepath = os.path.join(self._base_path, artifact_id) with open(filepath, "wb") as f: shutil.copyfileobj(content_body, f) def remove(self, artifact_id: str) -> None: filepath = os.path.join(self._base_path, artifact_id) try: os.remove(filepath) except FileNotFoundError as e: raise ArtifactNotFound("not found") from e if TYPE_CHECKING: # A mypy-runtime assertion to ensure that LocalArtifactBackend # implements all abstract methods in ArtifactBackendProtocol. from ._protocol import ArtifactStore _: ArtifactStore = FileSystemArtifactStore("") optuna-3.5.0/optuna/artifacts/_gcs.py000066400000000000000000000051161453453102400176040ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import TYPE_CHECKING from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO with try_import(): import google.cloud.storage @experimental_class("3.4.0") class GCSArtifactStore: """An artifact backend for Google Cloud Storage (GCS). Args: bucket_name: The name of the bucket to store artifacts. client: A google-cloud-storage `Client` to use for storage operations. If not specified, a new client will be created with default settings. Example: .. code-block:: python import optuna from optuna.artifacts import GCSArtifactStore, upload_artifact artifact_backend = GCSArtifactStore("my-bucket") def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) file_path = generate_example(...) upload_artifact(trial, file_path, artifact_backend) return ... Before running this code, you will have to install `gcloud` and run .. code-block:: bash gcloud auth application-default login so that the Cloud Storage library can automatically find the credential. """ def __init__( self, bucket_name: str, client: google.cloud.storage.Client | None = None, ) -> None: self.bucket_name = bucket_name self.client = client or google.cloud.storage.Client() self.bucket_obj = self.client.bucket(bucket_name) def open_reader(self, artifact_id: str) -> "BinaryIO": blob = self.bucket_obj.get_blob(artifact_id) if blob is None: raise ArtifactNotFound( f"Artifact storage with bucket: {self.bucket_name}, artifact_id: {artifact_id} was" " not found" ) body = blob.download_as_bytes() return BytesIO(body) def write(self, artifact_id: str, content_body: "BinaryIO") -> None: blob = self.bucket_obj.blob(artifact_id) data = content_body.read() blob.upload_from_string(data) def remove(self, artifact_id: str) -> None: self.bucket_obj.delete_blob(artifact_id) if TYPE_CHECKING: # A mypy-runtime assertion to ensure that GCS3ArtifactStore implements all abstract methods # in ArtifactStore. from ._protocol import ArtifactStore _: ArtifactStore = GCSArtifactStore("") optuna-3.5.0/optuna/artifacts/_protocol.py000066400000000000000000000034021453453102400206650ustar00rootroot00000000000000from __future__ import annotations from typing import TYPE_CHECKING try: from typing import Protocol except ImportError: from typing_extensions import Protocol # type: ignore if TYPE_CHECKING: from typing import BinaryIO class ArtifactStore(Protocol): """A protocol defining the interface for an artifact backend. An artifact backend is responsible for managing the storage and retrieval of artifact data. The backend should provide methods for opening, writing and removing artifacts. """ def open_reader(self, artifact_id: str) -> BinaryIO: """Open the artifact identified by the artifact_id. This method should return a binary file-like object in read mode, similar to ``open(..., mode="rb")``. If the artifact does not exist, an :exc:`~optuna.artifacts.exceptions.ArtifactNotFound` exception should be raised. Args: artifact_id: The identifier of the artifact to open. Returns: BinaryIO: A binary file-like object that can be read from. """ ... def write(self, artifact_id: str, content_body: BinaryIO) -> None: """Save the content to the backend. Args: artifact_id: The identifier of the artifact to write to. content_body: The content to write to the artifact. """ ... def remove(self, artifact_id: str) -> None: """Remove the artifact identified by the artifact_id. This method should delete the artifact from the backend. If the artifact does not exist, an :exc:`~optuna.artifacts.exceptions.ArtifactNotFound` exception may be raised. Args: artifact_id: The identifier of the artifact to remove. """ ... optuna-3.5.0/optuna/artifacts/_upload.py000066400000000000000000000055261453453102400203210ustar00rootroot00000000000000from __future__ import annotations from dataclasses import asdict from dataclasses import dataclass import json import mimetypes import os import uuid from optuna._experimental import experimental_func from optuna.artifacts._protocol import ArtifactStore from optuna.storages import BaseStorage from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import Trial ARTIFACTS_ATTR_PREFIX = "artifacts:" DEFAULT_MIME_TYPE = "application/octet-stream" @dataclass class ArtifactMeta: artifact_id: str filename: str mimetype: str encoding: str | None @experimental_func("3.3.0") def upload_artifact( study_or_trial: Trial | FrozenTrial | Study, file_path: str, artifact_store: ArtifactStore, *, storage: BaseStorage | None = None, mimetype: str | None = None, encoding: str | None = None, ) -> str: """Upload an artifact to the artifact store. Args: study_or_trial: A :class:`~optuna.trial.Trial` object, a :class:`~optuna.trial.FrozenTrial`, or a :class:`~optuna.study.Study` object. file_path: A path to the file to be uploaded. artifact_store: An artifact store. storage: A storage object. If trial is not a :class:`~optuna.trial.Trial` object, this argument is required. mimetype: A MIME type of the artifact. If not specified, the MIME type is guessed from the file extension. encoding: An encoding of the artifact, which is suitable for use as a ``Content-Encoding`` header (e.g. gzip). If not specified, the encoding is guessed from the file extension. Returns: An artifact ID. """ filename = os.path.basename(file_path) if isinstance(study_or_trial, Trial) and storage is None: storage = study_or_trial.storage elif isinstance(study_or_trial, Study) and storage is None: storage = study_or_trial._storage if storage is None: raise ValueError("storage is required for FrozenTrial.") artifact_id = str(uuid.uuid4()) guess_mimetype, guess_encoding = mimetypes.guess_type(filename) artifact = ArtifactMeta( artifact_id=artifact_id, filename=filename, mimetype=mimetype or guess_mimetype or DEFAULT_MIME_TYPE, encoding=encoding or guess_encoding, ) attr_key = ARTIFACTS_ATTR_PREFIX + artifact_id if isinstance(study_or_trial, (Trial, FrozenTrial)): trial_id = study_or_trial._trial_id storage.set_trial_system_attr(trial_id, attr_key, json.dumps(asdict(artifact))) else: study_id = study_or_trial._study_id storage.set_study_system_attr(study_id, attr_key, json.dumps(asdict(artifact))) with open(file_path, "rb") as f: artifact_store.write(artifact_id, f) return artifact_id optuna-3.5.0/optuna/artifacts/exceptions.py000066400000000000000000000005161453453102400210510ustar00rootroot00000000000000from optuna.exceptions import OptunaError class ArtifactNotFound(OptunaError): """Exception raised when an artifact is not found. It is typically raised while calling :meth:`~optuna.artifacts._protocol.ArtifactStore.open_reader` or :meth:`~optuna.artifacts._protocol.ArtifactStore.remove` methods. """ ... optuna-3.5.0/optuna/cli.py000066400000000000000000001154241453453102400154640ustar00rootroot00000000000000"""Optuna CLI module. If you want to add a new command, you also need to update the constant `_COMMANDS` """ from argparse import ArgumentParser from argparse import Namespace import datetime from enum import Enum from importlib.machinery import SourceFileLoader import inspect import json import logging import os import sys import types from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple from typing import Type from typing import Union import warnings import sqlalchemy.exc import yaml import optuna from optuna._imports import _LazyImport from optuna.exceptions import CLIUsageError from optuna.exceptions import ExperimentalWarning from optuna.storages import BaseStorage from optuna.storages import JournalFileStorage from optuna.storages import JournalRedisStorage from optuna.storages import JournalStorage from optuna.storages import RDBStorage from optuna.trial import TrialState _dataframe = _LazyImport("optuna.study._dataframe") _DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S" def _check_storage_url(storage_url: Optional[str]) -> str: if storage_url is not None: return storage_url env_storage = os.environ.get("OPTUNA_STORAGE") if env_storage is not None: warnings.warn( "Specifying the storage url via 'OPTUNA_STORAGE' environment variable" " is an experimental feature. The interface can change in the future.", ExperimentalWarning, ) return env_storage raise CLIUsageError("Storage URL is not specified.") def _get_storage(storage_url: Optional[str], storage_class: Optional[str]) -> BaseStorage: storage_url = _check_storage_url(storage_url) if storage_class: if storage_class == JournalRedisStorage.__name__: return JournalStorage(JournalRedisStorage(storage_url)) if storage_class == JournalFileStorage.__name__: return JournalStorage(JournalFileStorage(storage_url)) if storage_class == RDBStorage.__name__: return RDBStorage(storage_url) raise CLIUsageError("Unsupported storage class") if storage_url.startswith("redis"): return JournalStorage(JournalRedisStorage(storage_url)) if os.path.isfile(storage_url): return JournalStorage(JournalFileStorage(storage_url)) try: return RDBStorage(storage_url) except sqlalchemy.exc.ArgumentError: raise CLIUsageError("Failed to guess storage class from storage_url") def _format_value(value: Any) -> Any: # Format value that can be serialized to JSON or YAML. if value is None or isinstance(value, (int, float)): return value elif isinstance(value, datetime.datetime): return value.strftime(_DATETIME_FORMAT) elif isinstance(value, list): return list(_format_value(v) for v in value) elif isinstance(value, tuple): return tuple(_format_value(v) for v in value) elif isinstance(value, dict): return {_format_value(k): _format_value(v) for k, v in value.items()} else: return str(value) def _convert_to_dict( records: List[Dict[Tuple[str, str], Any]], columns: List[Tuple[str, str]], flatten: bool ) -> Tuple[List[Dict[str, Any]], List[str]]: header = [] ret = [] if flatten: for column in columns: if column[1] != "": header.append(f"{column[0]}_{column[1]}") elif any(isinstance(record.get(column), (list, tuple)) for record in records): max_length = 0 for record in records: if column in record: max_length = max(max_length, len(record[column])) for i in range(max_length): header.append(f"{column[0]}_{i}") else: header.append(column[0]) for record in records: row = {} for column in columns: if column not in record: continue value = _format_value(record[column]) if column[1] != "": row[f"{column[0]}_{column[1]}"] = value elif any(isinstance(record.get(column), (list, tuple)) for record in records): for i, v in enumerate(value): row[f"{column[0]}_{i}"] = v else: row[f"{column[0]}"] = value ret.append(row) else: for column in columns: if column[0] not in header: header.append(column[0]) for record in records: attrs: Dict[str, Any] = {column_name: {} for column_name in header} for column in columns: if column not in record: continue value = _format_value(record[column]) if isinstance(column[1], int): # Reconstruct list of values. `_dataframe._create_records_and_aggregate_column` # returns indices of list as the second key of column. if attrs[column[0]] == {}: attrs[column[0]] = [] attrs[column[0]] += [None] * max(column[1] + 1 - len(attrs[column[0]]), 0) attrs[column[0]][column[1]] = value elif column[1] != "": attrs[column[0]][column[1]] = value else: attrs[column[0]] = value ret.append(attrs) return ret, header class ValueType(Enum): NONE = 0 NUMERIC = 1 STRING = 2 class CellValue: def __init__(self, value: Any) -> None: self.value = value if value is None: self.value_type = ValueType.NONE elif isinstance(value, (int, float)): self.value_type = ValueType.NUMERIC else: self.value_type = ValueType.STRING def __str__(self) -> str: if isinstance(self.value, datetime.datetime): return self.value.strftime(_DATETIME_FORMAT) else: return str(self.value) def width(self) -> int: return len(str(self.value)) def get_string(self, value_type: ValueType, width: int) -> str: value = str(self.value) if self.value is None: return " " * width elif value_type == ValueType.NUMERIC: return f"{value:>{width}}" else: return f"{value:<{width}}" def _dump_value(records: List[Dict[str, Any]], header: List[str]) -> str: values = [] for record in records: row = [] for column_name in header: row.append(str(record.get(column_name, ""))) values.append(" ".join(row)) return "\n".join(values) def _dump_table(records: List[Dict[str, Any]], header: List[str]) -> str: rows = [] for record in records: row = [] for column_name in header: row.append(CellValue(record.get(column_name))) rows.append(row) separator = "+" header_string = "|" rows_string = ["|" for _ in rows] for column in range(len(header)): value_types = [row[column].value_type for row in rows] value_type = ValueType.NUMERIC for t in value_types: if t == ValueType.STRING: value_type = ValueType.STRING max_width = max(len(header[column]), max(row[column].width() for row in rows)) separator += "-" * (max_width + 2) + "+" if value_type == ValueType.NUMERIC: header_string += f" {header[column]:>{max_width}} |" else: header_string += f" {header[column]:<{max_width}} |" for i, row in enumerate(rows): rows_string[i] += " " + row[column].get_string(value_type, max_width) + " |" ret = "" ret += separator + "\n" ret += header_string + "\n" ret += separator + "\n" ret += "\n".join(rows_string) + "\n" ret += separator + "\n" return ret def _format_output( records: Union[List[Dict[Tuple[str, str], Any]], Dict[Tuple[str, str], Any]], columns: List[Tuple[str, str]], output_format: str, flatten: bool, ) -> str: if isinstance(records, list): values, header = _convert_to_dict(records, columns, flatten) else: values, header = _convert_to_dict([records], columns, flatten) if output_format == "value": if isinstance(records, list): return _dump_value(values, header).strip() else: return str(values[0]).strip() elif output_format == "table": return _dump_table(values, header).strip() elif output_format == "json": if isinstance(records, list): return json.dumps(values).strip() else: return json.dumps(values[0]).strip() elif output_format == "yaml": if isinstance(records, list): return yaml.safe_dump(values).strip() else: return yaml.safe_dump(values[0]).strip() else: raise CLIUsageError(f"Optuna CLI does not supported the {output_format} format.") class _BaseCommand: """Base class for commands. Note that commands class are not supposed to be called by library users. They are used only in this file to manage optuna CLI commands. """ def __init__(self) -> None: self.logger = optuna.logging.get_logger(__name__) def add_arguments(self, parser: ArgumentParser) -> None: """Add arguments required for each command. Args: parser: `ArgumentParser` object to add arguments """ pass def take_action(self, parsed_args: Namespace) -> int: """Define action if the command is called. Args: parsed_args: `Namespace` object including arguments specified by user. Returns: Running status of the action. 0 if this method finishes normally, otherwise 1. """ raise NotImplementedError class _CreateStudy(_BaseCommand): """Create a new study.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", default=None, help="A human-readable name of a study to distinguish it from others.", ) parser.add_argument( "--direction", default=None, type=str, choices=("minimize", "maximize"), help="Set direction of optimization to a new study. Set 'minimize' " "for minimization and 'maximize' for maximization.", ) parser.add_argument( "--skip-if-exists", default=False, action="store_true", help="If specified, the creation of the study is skipped " "without any error when the study name is duplicated.", ) parser.add_argument( "--directions", type=str, default=None, choices=("minimize", "maximize"), help="Set directions of optimization to a new study." " Put whitespace between directions. Each direction should be" ' either "minimize" or "maximize".', nargs="+", ) def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study_name = optuna.create_study( storage=storage, study_name=parsed_args.study_name, direction=parsed_args.direction, directions=parsed_args.directions, load_if_exists=parsed_args.skip_if_exists, ).study_name print(study_name) return 0 class _DeleteStudy(_BaseCommand): """Delete a specified study.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument("--study-name", default=None, help="The name of the study to delete.") def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study_id = storage.get_study_id_from_name(parsed_args.study_name) storage.delete_study(study_id) return 0 class _StudySetUserAttribute(_BaseCommand): """Set a user attribute to a study.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study", default=None, help="This argument is deprecated. Use --study-name instead." ) parser.add_argument( "--study-name", default=None, help="The name of the study to set the user attribute to.", ) parser.add_argument("--key", "-k", required=True, help="Key of the user attribute.") parser.add_argument("--value", required=True, help="Value to be set.") def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) if parsed_args.study and parsed_args.study_name: raise ValueError( "Both `--study-name` and the deprecated `--study` was specified. " "Please remove the `--study` flag." ) elif parsed_args.study: message = "The use of `--study` is deprecated. Please use `--study-name` instead." warnings.warn(message, FutureWarning) study = optuna.load_study(storage=storage, study_name=parsed_args.study) elif parsed_args.study_name: study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) else: raise ValueError("Missing study name. Please use `--study-name`.") study.set_user_attr(parsed_args.key, parsed_args.value) self.logger.info("Attribute successfully written.") return 0 class _StudyNames(_BaseCommand): """Get all study names stored in a specified storage""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "-f", "--format", type=str, choices=("value", "json", "table", "yaml"), default="value", help="Output format.", ) def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) all_study_names = optuna.get_all_study_names(storage) records = [] record_key = ("name", "") for study_name in all_study_names: records.append({record_key: study_name}) print(_format_output(records, [record_key], parsed_args.format, flatten=False)) return 0 class _Studies(_BaseCommand): """Show a list of studies.""" _study_list_header = [ ("name", ""), ("direction", ""), ("n_trials", ""), ("datetime_start", ""), ] def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as directions.", ) def take_action(self, parsed_args: Namespace) -> int: storage = _get_storage(parsed_args.storage, parsed_args.storage_class) summaries = optuna.get_all_study_summaries(storage, include_best_trial=False) records = [] for s in summaries: start = ( s.datetime_start.strftime(_DATETIME_FORMAT) if s.datetime_start is not None else None ) record: Dict[Tuple[str, str], Any] = {} record[("name", "")] = s.study_name record[("direction", "")] = tuple(d.name for d in s.directions) record[("n_trials", "")] = s.n_trials record[("datetime_start", "")] = start record[("user_attrs", "")] = s.user_attrs records.append(record) if any(r[("user_attrs", "")] != {} for r in records): self._study_list_header.append(("user_attrs", "")) print( _format_output( records, self._study_list_header, parsed_args.format, parsed_args.flatten ) ) return 0 class _Trials(_BaseCommand): """Show a list of trials.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", type=str, required=True, help="The name of the study which includes trials.", ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params and user_attrs.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'trials' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) attrs = ( "number", "value" if not study._is_multi_objective() else "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) records, columns = _dataframe._create_records_and_aggregate_column(study, attrs) print(_format_output(records, columns, parsed_args.format, parsed_args.flatten)) return 0 class _BestTrial(_BaseCommand): """Show the best trial.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", type=str, required=True, help="The name of the study to get the best trial.", ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params and user_attrs.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'best-trial' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) attrs = ( "number", "value" if not study._is_multi_objective() else "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) records, columns = _dataframe._create_records_and_aggregate_column(study, attrs) print( _format_output( records[study.best_trial.number], columns, parsed_args.format, parsed_args.flatten ) ) return 0 class _BestTrials(_BaseCommand): """Show a list of trials located at the Pareto front.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--study-name", type=str, required=True, help="The name of the study to get the best trials (trials at the Pareto front).", ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="table", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params and user_attrs.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'best-trials' is an experimental CLI command. The interface can change in the " "future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) best_trials = [trial.number for trial in study.best_trials] attrs = ( "number", "value" if not study._is_multi_objective() else "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) records, columns = _dataframe._create_records_and_aggregate_column(study, attrs) best_records = list(filter(lambda record: record[("number", "")] in best_trials, records)) print(_format_output(best_records, columns, parsed_args.format, parsed_args.flatten)) return 0 class _StudyOptimize(_BaseCommand): """Start optimization of a study. Deprecated since version 2.0.0.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument( "--n-trials", type=int, help="The number of trials. If this argument is not given, as many " "trials run as possible.", ) parser.add_argument( "--timeout", type=float, help="Stop study after the given number of second(s). If this argument" " is not given, as many trials run as possible.", ) parser.add_argument( "--n-jobs", type=int, default=1, help="The number of parallel jobs. If this argument is set to -1, the " "number is set to CPU counts.", ) parser.add_argument( "--study", default=None, help="This argument is deprecated. Use --study-name instead." ) parser.add_argument( "--study-name", default=None, help="The name of the study to start optimization on." ) parser.add_argument( "file", help="Python script file where the objective function resides.", ) parser.add_argument( "method", help="The method name of the objective function.", ) def take_action(self, parsed_args: Namespace) -> int: message = ( "The use of the `study optimize` command is deprecated. Please execute your Python " "script directly instead." ) warnings.warn(message, FutureWarning) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) if parsed_args.study and parsed_args.study_name: raise ValueError( "Both `--study-name` and the deprecated `--study` was specified. " "Please remove the `--study` flag." ) elif parsed_args.study: message = "The use of `--study` is deprecated. Please use `--study-name` instead." warnings.warn(message, FutureWarning) study = optuna.load_study(storage=storage, study_name=parsed_args.study) elif parsed_args.study_name: study = optuna.load_study(storage=storage, study_name=parsed_args.study_name) else: raise ValueError("Missing study name. Please use `--study-name`.") # We force enabling the debug flag. As we are going to execute user codes, we want to show # exception stack traces by default. parsed_args.debug = True module_name = "optuna_target_module" target_module = types.ModuleType(module_name) loader = SourceFileLoader(module_name, parsed_args.file) loader.exec_module(target_module) try: target_method = getattr(target_module, parsed_args.method) except AttributeError: self.logger.error( "Method {} not found in file {}.".format(parsed_args.method, parsed_args.file) ) return 1 study.optimize( target_method, n_trials=parsed_args.n_trials, timeout=parsed_args.timeout, n_jobs=parsed_args.n_jobs, ) return 0 class _StorageUpgrade(_BaseCommand): """Upgrade the schema of an RDB storage.""" def take_action(self, parsed_args: Namespace) -> int: storage_url = _check_storage_url(parsed_args.storage) try: storage = RDBStorage( storage_url, skip_compatibility_check=True, skip_table_creation=True ) except sqlalchemy.exc.ArgumentError: self.logger.error("Invalid RDBStorage URL.") return 1 current_version = storage.get_current_version() head_version = storage.get_head_version() known_versions = storage.get_all_versions() if current_version == head_version: self.logger.info("This storage is up-to-date.") elif current_version in known_versions: self.logger.info("Upgrading the storage schema to the latest version.") storage.upgrade() self.logger.info("Completed to upgrade the storage.") else: warnings.warn( "Your optuna version seems outdated against the storage version. " "Please try updating optuna to the latest version by " "`$ pip install -U optuna`." ) return 0 class _Ask(_BaseCommand): """Create a new trial and suggest parameters.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument("--study-name", type=str, help="Name of study.") parser.add_argument( "--direction", type=str, choices=("minimize", "maximize"), help=( "Direction of optimization. This argument is deprecated." " Please create a study in advance." ), ) parser.add_argument( "--directions", type=str, nargs="+", choices=("minimize", "maximize"), help=( "Directions of optimization, if there are multiple objectives." " This argument is deprecated. Please create a study in advance." ), ) parser.add_argument("--sampler", type=str, help="Class name of sampler object to create.") parser.add_argument( "--sampler-kwargs", type=str, help="Sampler object initialization keyword arguments as JSON.", ) parser.add_argument( "--search-space", type=str, help=( "Search space as JSON. Keys are names and values are outputs from " ":func:`~optuna.distributions.distribution_to_json`." ), ) parser.add_argument( "-f", "--format", type=str, choices=("json", "table", "yaml"), default="json", help="Output format.", ) parser.add_argument( "--flatten", default=False, action="store_true", help="Flatten nested columns such as params.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'ask' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) create_study_kwargs = { "storage": storage, "study_name": parsed_args.study_name, "direction": parsed_args.direction, "directions": parsed_args.directions, "load_if_exists": True, } if parsed_args.direction is not None or parsed_args.directions is not None: message = ( "The `direction` and `directions` arguments of the `study ask` command are" " deprecated because the command will no longer create a study when you specify" " the arguments. Please create a study in advance." ) warnings.warn(message, FutureWarning) if parsed_args.sampler is not None: if parsed_args.sampler_kwargs is not None: sampler_kwargs = json.loads(parsed_args.sampler_kwargs) else: sampler_kwargs = {} sampler_cls = getattr(optuna.samplers, parsed_args.sampler) sampler = sampler_cls(**sampler_kwargs) create_study_kwargs["sampler"] = sampler else: if parsed_args.sampler_kwargs is not None: raise ValueError( "`--sampler_kwargs` is set without `--sampler`. Please specify `--sampler` as" " well or omit `--sampler-kwargs`." ) if parsed_args.search_space is not None: # The search space is expected to be a JSON serialized string, e.g. # '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, # "y": ...}'. search_space = { name: optuna.distributions.json_to_distribution(json.dumps(dist)) for name, dist in json.loads(parsed_args.search_space).items() } else: search_space = {} try: study = optuna.load_study( study_name=create_study_kwargs["study_name"], storage=create_study_kwargs["storage"], sampler=create_study_kwargs.get("sampler"), ) directions = None if ( create_study_kwargs["direction"] is not None and create_study_kwargs["directions"] is not None ): raise ValueError("Specify only one of `direction` and `directions`.") if create_study_kwargs["direction"] is not None: directions = [ optuna.study.StudyDirection[create_study_kwargs["direction"].upper()] ] if create_study_kwargs["directions"] is not None: directions = [ optuna.study.StudyDirection[d.upper()] for d in create_study_kwargs["directions"] ] if directions is not None and study.directions != directions: raise ValueError( f"Cannot overwrite study direction from {study.directions} to {directions}." ) except KeyError: study = optuna.create_study(**create_study_kwargs) trial = study.ask(fixed_distributions=search_space) self.logger.info(f"Asked trial {trial.number} with parameters {trial.params}.") record: Dict[Tuple[str, str], Any] = {("number", ""): trial.number} columns = [("number", "")] if len(trial.params) == 0 and not parsed_args.flatten: record[("params", "")] = {} columns.append(("params", "")) else: for param_name, param_value in trial.params.items(): record[("params", param_name)] = param_value columns.append(("params", param_name)) print(_format_output(record, columns, parsed_args.format, parsed_args.flatten)) return 0 class _Tell(_BaseCommand): """Finish a trial, which was created by the ask command.""" def add_arguments(self, parser: ArgumentParser) -> None: parser.add_argument("--study-name", type=str, help="Name of study.") parser.add_argument("--trial-number", type=int, help="Trial number.") parser.add_argument("--values", type=float, nargs="+", help="Objective values.") parser.add_argument( "--state", type=str, help="Trial state.", choices=("complete", "pruned", "fail"), ) parser.add_argument( "--skip-if-finished", default=False, action="store_true", help="If specified, tell is skipped without any error when the trial is already " "finished.", ) def take_action(self, parsed_args: Namespace) -> int: warnings.warn( "'tell' is an experimental CLI command. The interface can change in the future.", ExperimentalWarning, ) storage = _get_storage(parsed_args.storage, parsed_args.storage_class) study = optuna.load_study( storage=storage, study_name=parsed_args.study_name, ) if parsed_args.state is not None: state: Optional[TrialState] = TrialState[parsed_args.state.upper()] else: state = None trial_number = parsed_args.trial_number values = parsed_args.values study.tell( trial=trial_number, values=values, state=state, skip_if_finished=parsed_args.skip_if_finished, ) self.logger.info(f"Told trial {trial_number} with values {values} and state {state}.") return 0 _COMMANDS: Dict[str, Type[_BaseCommand]] = { "create-study": _CreateStudy, "delete-study": _DeleteStudy, "study set-user-attr": _StudySetUserAttribute, "study-names": _StudyNames, "studies": _Studies, "trials": _Trials, "best-trial": _BestTrial, "best-trials": _BestTrials, "study optimize": _StudyOptimize, "storage upgrade": _StorageUpgrade, "ask": _Ask, "tell": _Tell, } def _add_common_arguments(parser: ArgumentParser) -> ArgumentParser: parser.add_argument( "--storage", default=None, help=( "DB URL. (e.g. sqlite:///example.db) " "Also can be specified via OPTUNA_STORAGE environment variable." ), ) parser.add_argument( "--storage-class", help="Storage class hint (e.g. JournalFileStorage)", default=None, choices=[ RDBStorage.__name__, JournalFileStorage.__name__, JournalRedisStorage.__name__, ], ) verbose_group = parser.add_mutually_exclusive_group() verbose_group.add_argument( "-v", "--verbose", action="count", dest="verbose_level", default=1, help="Increase verbosity of output. Can be repeated.", ) verbose_group.add_argument( "-q", "--quiet", action="store_const", dest="verbose_level", const=0, help="Suppress output except warnings and errors.", ) parser.add_argument( "--log-file", action="store", default=None, help="Specify a file to log output. Disabled by default.", ) parser.add_argument( "--debug", default=False, action="store_true", help="Show tracebacks on errors.", ) return parser def _add_commands( main_parser: ArgumentParser, parent_parser: ArgumentParser ) -> Dict[str, ArgumentParser]: subparsers = main_parser.add_subparsers() command_name_to_subparser = {} for command_name, command_type in _COMMANDS.items(): command = command_type() subparser = subparsers.add_parser( command_name, parents=[parent_parser], help=inspect.getdoc(command_type) ) command.add_arguments(subparser) subparser.set_defaults(handler=command.take_action) command_name_to_subparser[command_name] = subparser def _print_help(args: Namespace) -> None: main_parser.print_help() subparsers.add_parser("help", help="Show help message and exit.").set_defaults( handler=_print_help ) return command_name_to_subparser def _get_parser(description: str = "") -> Tuple[ArgumentParser, Dict[str, ArgumentParser]]: # Use `parent_parser` is necessary to avoid namespace conflict for -h/--help # between `main_parser` and `subparser`. parent_parser = ArgumentParser(add_help=False) parent_parser = _add_common_arguments(parent_parser) main_parser = ArgumentParser(description=description, parents=[parent_parser]) main_parser.add_argument( "--version", action="version", version="{0} {1}".format("optuna", optuna.__version__) ) command_name_to_subparser = _add_commands(main_parser, parent_parser) return main_parser, command_name_to_subparser def _preprocess_argv(argv: List[str]) -> List[str]: # Some preprocess is necessary for argv because some subcommand includes space # (e.g. optuna study optimize, optuna storage upgrade, ...). argv = argv[1:] if len(argv) > 1 else ["help"] for i in range(len(argv)): for j in range(i, i + 2): # Commands consist of one or two words. command_candidate = " ".join(argv[i : j + 1]) if command_candidate in _COMMANDS: options = argv[:i] + argv[j + 1 :] return [command_candidate] + options # No subcommand is found. return argv def _set_verbosity(args: Namespace) -> None: root_logger = logging.getLogger() root_logger.setLevel(logging.DEBUG) stream_handler = logging.StreamHandler(sys.stderr) logging_level = { 0: logging.WARNING, 1: logging.INFO, 2: logging.DEBUG, }.get(args.verbose_level, logging.DEBUG) stream_handler.setLevel(logging_level) stream_handler.setFormatter(optuna.logging.create_default_formatter()) root_logger.addHandler(stream_handler) optuna.logging.set_verbosity(logging_level) def _set_log_file(args: Namespace) -> None: if args.log_file is None: return root_logger = logging.getLogger() root_logger.setLevel(logging.DEBUG) file_handler = logging.FileHandler( filename=args.log_file, ) file_handler.setFormatter(optuna.logging.create_default_formatter()) root_logger.addHandler(file_handler) def main() -> int: main_parser, command_name_to_subparser = _get_parser() argv = sys.argv preprocessed_argv = _preprocess_argv(argv) args = main_parser.parse_args(preprocessed_argv) _set_verbosity(args) _set_log_file(args) logger = logging.getLogger("optuna") try: return args.handler(args) except CLIUsageError as e: if args.debug: logger.exception(e) else: logger.error(e) # This code is required to show help for each subcommand. # NOTE: the first element of `preprocessed_argv` is command name. command_name_to_subparser[preprocessed_argv[0]].print_help() return 1 except AttributeError: # Exception for the case -v/--verbose/-q/--quiet/--log-file/--debug # without any subcommand. argv_str = " ".join(argv[1:]) logger.error(f"'{argv_str}' is not an optuna command. see 'optuna --help'") main_parser.print_help() return 1 optuna-3.5.0/optuna/distributions.py000066400000000000000000000667541453453102400176320ustar00rootroot00000000000000import abc import copy import decimal import json from numbers import Real from typing import Any from typing import cast from typing import Dict from typing import Sequence from typing import Union import warnings import numpy as np from optuna._deprecated import deprecated_class CategoricalChoiceType = Union[None, bool, int, float, str] _float_distribution_deprecated_msg = ( "Use :class:`~optuna.distributions.FloatDistribution` instead." ) _int_distribution_deprecated_msg = "Use :class:`~optuna.distributions.IntDistribution` instead." class BaseDistribution(abc.ABC): """Base class for distributions. Note that distribution classes are not supposed to be called by library users. They are used by :class:`~optuna.trial.Trial` and :class:`~optuna.samplers` internally. """ def to_external_repr(self, param_value_in_internal_repr: float) -> Any: """Convert internal representation of a parameter value into external representation. Args: param_value_in_internal_repr: Optuna's internal representation of a parameter value. Returns: Optuna's external representation of a parameter value. """ return param_value_in_internal_repr @abc.abstractmethod def to_internal_repr(self, param_value_in_external_repr: Any) -> float: """Convert external representation of a parameter value into internal representation. Args: param_value_in_external_repr: Optuna's external representation of a parameter value. Returns: Optuna's internal representation of a parameter value. """ raise NotImplementedError @abc.abstractmethod def single(self) -> bool: """Test whether the range of this distribution contains just a single value. Returns: :obj:`True` if the range of this distribution contains just a single value, otherwise :obj:`False`. """ raise NotImplementedError @abc.abstractmethod def _contains(self, param_value_in_internal_repr: float) -> bool: """Test if a parameter value is contained in the range of this distribution. Args: param_value_in_internal_repr: Optuna's internal representation of a parameter value. Returns: :obj:`True` if the parameter value is contained in the range of this distribution, otherwise :obj:`False`. """ raise NotImplementedError def _asdict(self) -> Dict: return self.__dict__ def __eq__(self, other: Any) -> bool: if not isinstance(other, BaseDistribution): return NotImplemented if not type(self) is type(other): return False return self.__dict__ == other.__dict__ def __hash__(self) -> int: return hash((self.__class__,) + tuple(sorted(self.__dict__.items()))) def __repr__(self) -> str: kwargs = ", ".join("{}={}".format(k, v) for k, v in sorted(self._asdict().items())) return "{}({})".format(self.__class__.__name__, kwargs) class FloatDistribution(BaseDistribution): """A distribution on floats. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float`, and passed to :mod:`~optuna.samplers` in general. .. note:: When ``step`` is not :obj:`None`, if the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`\\mathsf{step}`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k \\times \\mathsf{step} + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than 0. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. log: If ``log`` is :obj:`True`, this distribution is in log-scaled domain. In this case, all parameters enqueued to the distribution must be positive values. This parameter must be :obj:`False` when the parameter ``step`` is not :obj:`None`. step: A discretization step. ``step`` must be larger than 0. This parameter must be :obj:`None` when the parameter ``log`` is :obj:`True`. """ def __init__( self, low: float, high: float, log: bool = False, step: Union[None, float] = None ) -> None: if log and step is not None: raise ValueError("The parameter `step` is not supported when `log` is true.") if low > high: raise ValueError( "The `low` value must be smaller than or equal to the `high` value " "(low={}, high={}).".format(low, high) ) if log and low <= 0.0: raise ValueError( "The `low` value must be larger than 0 for a log distribution " "(low={}, high={}).".format(low, high) ) if step is not None and step <= 0: raise ValueError( "The `step` value must be non-zero positive value, " "but step={}.".format(step) ) self.step = None if step is not None: high = _adjust_discrete_uniform_high(low, high, step) self.step = float(step) self.low = float(low) self.high = float(high) self.log = log def single(self) -> bool: if self.step is None: return self.low == self.high else: if self.low == self.high: return True high = decimal.Decimal(str(self.high)) low = decimal.Decimal(str(self.low)) step = decimal.Decimal(str(self.step)) return (high - low) < step def _contains(self, param_value_in_internal_repr: float) -> bool: value = param_value_in_internal_repr if self.step is None: return self.low <= value <= self.high else: k = (value - self.low) / self.step return self.low <= value <= self.high and abs(k - round(k)) < 1.0e-8 def to_internal_repr(self, param_value_in_external_repr: float) -> float: try: internal_repr = float(param_value_in_external_repr) except (ValueError, TypeError) as e: raise ValueError( f"'{param_value_in_external_repr}' is not a valid type. " "float-castable value is expected." ) from e if np.isnan(internal_repr): raise ValueError(f"`{param_value_in_external_repr}` is invalid value.") if self.log and internal_repr <= 0.0: raise ValueError( f"`{param_value_in_external_repr}` is invalid value for the case log=True." ) return internal_repr @deprecated_class("3.0.0", "6.0.0", text=_float_distribution_deprecated_msg) class UniformDistribution(FloatDistribution): """A uniform distribution in the linear domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float`, and passed to :mod:`~optuna.samplers` in general. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. """ def __init__(self, low: float, high: float) -> None: super().__init__(low=low, high=high, log=False, step=None) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") d.pop("step") return d @deprecated_class("3.0.0", "6.0.0", text=_float_distribution_deprecated_msg) class LogUniformDistribution(FloatDistribution): """A uniform distribution in the log domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float` with ``log=True``, and passed to :mod:`~optuna.samplers` in general. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be larger than 0. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. """ def __init__(self, low: float, high: float) -> None: super().__init__(low=low, high=high, log=True, step=None) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") d.pop("step") return d @deprecated_class("3.0.0", "6.0.0", text=_float_distribution_deprecated_msg) class DiscreteUniformDistribution(FloatDistribution): """A discretized uniform distribution in the linear domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_float` with ``step`` argument, and passed to :mod:`~optuna.samplers` in general. .. note:: If the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`q`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k q + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Args: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. q: A discretization step. ``q`` must be larger than 0. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. """ def __init__(self, low: float, high: float, q: float) -> None: super().__init__(low=low, high=high, step=q) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") step = d.pop("step") d["q"] = step return d @property def q(self) -> float: """Discretization step. :class:`~optuna.distributions.DiscreteUniformDistribution` is a subtype of :class:`~optuna.distributions.FloatDistribution`. This property is a proxy for its ``step`` attribute. """ return cast(float, self.step) @q.setter def q(self, v: float) -> None: self.step = v class IntDistribution(BaseDistribution): """A distribution on integers. This object is instantiated by :func:`~optuna.trial.Trial.suggest_int`, and passed to :mod:`~optuna.samplers` in general. .. note:: When ``step`` is not :obj:`None`, if the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`\\mathsf{step}`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k \\times \\mathsf{step} + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than or equal to 1. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. log: If ``log`` is :obj:`True`, this distribution is in log-scaled domain. In this case, all parameters enqueued to the distribution must be positive values. This parameter must be :obj:`False` when the parameter ``step`` is not 1. step: A discretization step. ``step`` must be a positive integer. This parameter must be 1 when the parameter ``log`` is :obj:`True`. """ def __init__(self, low: int, high: int, log: bool = False, step: int = 1) -> None: if log and step != 1: raise ValueError( "Samplers and other components in Optuna only accept step is 1 " "when `log` argument is True." ) if low > high: raise ValueError( "The `low` value must be smaller than or equal to the `high` value " "(low={}, high={}).".format(low, high) ) if log and low < 1: raise ValueError( "The `low` value must be equal to or greater than 1 for a log distribution " "(low={}, high={}).".format(low, high) ) if step <= 0: raise ValueError( "The `step` value must be non-zero positive value, but step={}.".format(step) ) self.log = log self.step = int(step) self.low = int(low) high = int(high) self.high = _adjust_int_uniform_high(self.low, high, self.step) def to_external_repr(self, param_value_in_internal_repr: float) -> int: return int(param_value_in_internal_repr) def to_internal_repr(self, param_value_in_external_repr: int) -> float: try: internal_repr = float(param_value_in_external_repr) except (ValueError, TypeError) as e: raise ValueError( f"'{param_value_in_external_repr}' is not a valid type. " "float-castable value is expected." ) from e if np.isnan(internal_repr): raise ValueError(f"`{param_value_in_external_repr}` is invalid value.") if self.log and internal_repr <= 0.0: raise ValueError( f"`{param_value_in_external_repr}` is invalid value for the case log=True." ) return internal_repr def single(self) -> bool: if self.log: return self.low == self.high if self.low == self.high: return True return (self.high - self.low) < self.step def _contains(self, param_value_in_internal_repr: float) -> bool: value = param_value_in_internal_repr return self.low <= value <= self.high and (value - self.low) % self.step == 0 @deprecated_class("3.0.0", "6.0.0", text=_int_distribution_deprecated_msg) class IntUniformDistribution(IntDistribution): """A uniform distribution on integers. This object is instantiated by :func:`~optuna.trial.Trial.suggest_int`, and passed to :mod:`~optuna.samplers` in general. .. note:: If the range :math:`[\\mathsf{low}, \\mathsf{high}]` is not divisible by :math:`\\mathsf{step}`, :math:`\\mathsf{high}` will be replaced with the maximum of :math:`k \\times \\mathsf{step} + \\mathsf{low} < \\mathsf{high}`, where :math:`k` is an integer. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A discretization step. ``step`` must be a positive integer. """ def __init__(self, low: int, high: int, step: int = 1) -> None: super().__init__(low=low, high=high, log=False, step=step) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") return d @deprecated_class("3.0.0", "6.0.0", text=_int_distribution_deprecated_msg) class IntLogUniformDistribution(IntDistribution): """A uniform distribution on integers in the log domain. This object is instantiated by :func:`~optuna.trial.Trial.suggest_int`, and passed to :mod:`~optuna.samplers` in general. Attributes: low: Lower endpoint of the range of the distribution. ``low`` is included in the range and must be larger than or equal to 1. ``low`` must be less than or equal to ``high``. high: Upper endpoint of the range of the distribution. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A discretization step. ``step`` must be a positive integer. .. warning:: Deprecated in v2.0.0. ``step`` argument will be removed in the future. The removal of this feature is currently scheduled for v4.0.0, but this schedule is subject to change. Samplers and other components in Optuna relying on this distribution will ignore this value and assume that ``step`` is always 1. User-defined samplers may continue to use other values besides 1 during the deprecation. """ def __init__(self, low: int, high: int, step: int = 1) -> None: super().__init__(low=low, high=high, log=True, step=step) def _asdict(self) -> Dict: d = copy.deepcopy(self.__dict__) d.pop("log") return d def _categorical_choice_equal( value1: CategoricalChoiceType, value2: CategoricalChoiceType ) -> bool: """A function to check two choices equal considering NaN. This function can handle NaNs like np.float32("nan") other than float. """ value1_is_nan = isinstance(value1, Real) and np.isnan(float(value1)) value2_is_nan = isinstance(value2, Real) and np.isnan(float(value2)) return (value1 == value2) or (value1_is_nan and value2_is_nan) class CategoricalDistribution(BaseDistribution): """A categorical distribution. This object is instantiated by :func:`~optuna.trial.Trial.suggest_categorical`, and passed to :mod:`~optuna.samplers` in general. Args: choices: Parameter value candidates. ``choices`` must have one element at least. .. note:: Not all types are guaranteed to be compatible with all storages. It is recommended to restrict the types of the choices to :obj:`None`, :class:`bool`, :class:`int`, :class:`float` and :class:`str`. Attributes: choices: Parameter value candidates. """ def __init__(self, choices: Sequence[CategoricalChoiceType]) -> None: if len(choices) == 0: raise ValueError("The `choices` must contain one or more elements.") for choice in choices: if choice is not None and not isinstance(choice, (bool, int, float, str)): message = ( "Choices for a categorical distribution should be a tuple of None, bool, " "int, float and str for persistent storage but contains {} which is of type " "{}.".format(choice, type(choice).__name__) ) warnings.warn(message) self.choices = tuple(choices) def to_external_repr(self, param_value_in_internal_repr: float) -> CategoricalChoiceType: return self.choices[int(param_value_in_internal_repr)] def to_internal_repr(self, param_value_in_external_repr: CategoricalChoiceType) -> float: for index, choice in enumerate(self.choices): if _categorical_choice_equal(param_value_in_external_repr, choice): return index raise ValueError(f"'{param_value_in_external_repr}' not in {self.choices}.") def single(self) -> bool: return len(self.choices) == 1 def _contains(self, param_value_in_internal_repr: float) -> bool: index = int(param_value_in_internal_repr) return 0 <= index < len(self.choices) def __eq__(self, other: Any) -> bool: if not isinstance(other, BaseDistribution): return NotImplemented if not isinstance(other, self.__class__): return False if self.__dict__.keys() != other.__dict__.keys(): return False for key, value in self.__dict__.items(): if key == "choices": if len(value) != len(getattr(other, key)): return False for choice, other_choice in zip(value, getattr(other, key)): if not _categorical_choice_equal(choice, other_choice): return False else: if value != getattr(other, key): return False return True __hash__ = BaseDistribution.__hash__ DISTRIBUTION_CLASSES = ( IntDistribution, IntLogUniformDistribution, IntUniformDistribution, FloatDistribution, UniformDistribution, LogUniformDistribution, DiscreteUniformDistribution, CategoricalDistribution, ) def json_to_distribution(json_str: str) -> BaseDistribution: """Deserialize a distribution in JSON format. Args: json_str: A JSON-serialized distribution. Returns: A deserialized distribution. """ json_dict = json.loads(json_str) if "name" in json_dict: if json_dict["name"] == CategoricalDistribution.__name__: json_dict["attributes"]["choices"] = tuple(json_dict["attributes"]["choices"]) for cls in DISTRIBUTION_CLASSES: if json_dict["name"] == cls.__name__: return cls(**json_dict["attributes"]) raise ValueError("Unknown distribution class: {}".format(json_dict["name"])) else: # Deserialize a distribution from an abbreviated format. if json_dict["type"] == "categorical": return CategoricalDistribution(json_dict["choices"]) elif json_dict["type"] in ("float", "int"): low = json_dict["low"] high = json_dict["high"] step = json_dict.get("step") log = json_dict.get("log", False) if json_dict["type"] == "float": return FloatDistribution(low, high, log=log, step=step) else: if step is None: step = 1 return IntDistribution(low=low, high=high, log=log, step=step) raise ValueError("Unknown distribution type: {}".format(json_dict["type"])) def distribution_to_json(dist: BaseDistribution) -> str: """Serialize a distribution to JSON format. Args: dist: A distribution to be serialized. Returns: A JSON string of a given distribution. """ return json.dumps({"name": dist.__class__.__name__, "attributes": dist._asdict()}) def check_distribution_compatibility( dist_old: BaseDistribution, dist_new: BaseDistribution ) -> None: """A function to check compatibility of two distributions. It checks whether ``dist_old`` and ``dist_new`` are the same kind of distributions. If ``dist_old`` is :class:`~optuna.distributions.CategoricalDistribution`, it further checks ``choices`` are the same between ``dist_old`` and ``dist_new``. Note that this method is not supposed to be called by library users. Args: dist_old: A distribution previously recorded in storage. dist_new: A distribution newly added to storage. """ if dist_old.__class__ != dist_new.__class__: raise ValueError("Cannot set different distribution kind to the same parameter name.") if isinstance(dist_old, (FloatDistribution, IntDistribution)): # For mypy. assert isinstance(dist_new, (FloatDistribution, IntDistribution)) if dist_old.log != dist_new.log: raise ValueError("Cannot set different log configuration to the same parameter name.") if not isinstance(dist_old, CategoricalDistribution): return if not isinstance(dist_new, CategoricalDistribution): return if dist_old != dist_new: raise ValueError( CategoricalDistribution.__name__ + " does not support dynamic value space." ) def _adjust_discrete_uniform_high(low: float, high: float, step: float) -> float: d_high = decimal.Decimal(str(high)) d_low = decimal.Decimal(str(low)) d_step = decimal.Decimal(str(step)) d_r = d_high - d_low if d_r % d_step != decimal.Decimal("0"): old_high = high high = float((d_r // d_step) * d_step + d_low) warnings.warn( "The distribution is specified by [{low}, {old_high}] and step={step}, but the range " "is not divisible by `step`. It will be replaced by [{low}, {high}].".format( low=low, old_high=old_high, high=high, step=step ) ) return high def _adjust_int_uniform_high(low: int, high: int, step: int) -> int: r = high - low if r % step != 0: old_high = high high = r // step * step + low warnings.warn( "The distribution is specified by [{low}, {old_high}] and step={step}, but the range " "is not divisible by `step`. It will be replaced by [{low}, {high}].".format( low=low, old_high=old_high, high=high, step=step ) ) return high def _get_single_value(distribution: BaseDistribution) -> Union[int, float, CategoricalChoiceType]: assert distribution.single() if isinstance( distribution, ( FloatDistribution, IntDistribution, ), ): return distribution.low elif isinstance(distribution, CategoricalDistribution): return distribution.choices[0] assert False # TODO(himkt): Remove this method with the deletion of deprecated distributions. # https://github.com/optuna/optuna/issues/2941 def _convert_old_distribution_to_new_distribution( distribution: BaseDistribution, suppress_warning: bool = False, ) -> BaseDistribution: new_distribution: BaseDistribution # Float distributions. if isinstance(distribution, UniformDistribution): new_distribution = FloatDistribution( low=distribution.low, high=distribution.high, log=False, step=None, ) elif isinstance(distribution, LogUniformDistribution): new_distribution = FloatDistribution( low=distribution.low, high=distribution.high, log=True, step=None, ) elif isinstance(distribution, DiscreteUniformDistribution): new_distribution = FloatDistribution( low=distribution.low, high=distribution.high, log=False, step=distribution.q, ) # Integer distributions. elif isinstance(distribution, IntUniformDistribution): new_distribution = IntDistribution( low=distribution.low, high=distribution.high, log=False, step=distribution.step, ) elif isinstance(distribution, IntLogUniformDistribution): new_distribution = IntDistribution( low=distribution.low, high=distribution.high, log=True, step=distribution.step, ) # Categorical distribution. else: new_distribution = distribution if new_distribution != distribution and not suppress_warning: message = ( f"{distribution} is deprecated and internally converted to" f" {new_distribution}. See https://github.com/optuna/optuna/issues/2941." ) warnings.warn(message, FutureWarning) return new_distribution def _is_distribution_log(distribution: BaseDistribution) -> bool: if isinstance(distribution, (FloatDistribution, IntDistribution)): return distribution.log return False optuna-3.5.0/optuna/exceptions.py000066400000000000000000000046121453453102400170720ustar00rootroot00000000000000class OptunaError(Exception): """Base class for Optuna specific errors.""" pass class TrialPruned(OptunaError): """Exception for pruned trials. This error tells a trainer that the current :class:`~optuna.trial.Trial` was pruned. It is supposed to be raised after :func:`optuna.trial.Trial.should_prune` as shown in the following example. See also: :class:`optuna.TrialPruned` is an alias of :class:`optuna.exceptions.TrialPruned`. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=20) """ pass class CLIUsageError(OptunaError): """Exception for CLI. CLI raises this exception when it receives invalid configuration. """ pass class StorageInternalError(OptunaError): """Exception for storage operation. This error is raised when an operation failed in backend DB of storage. """ pass class DuplicatedStudyError(OptunaError): """Exception for a duplicated study name. This error is raised when a specified study name already exists in the storage. """ pass class ExperimentalWarning(Warning): """Experimental Warning class. This implementation exists here because the policy of `FutureWarning` has been changed since Python 3.7 was released. See the details in https://docs.python.org/3/library/warnings.html#warning-categories. """ pass optuna-3.5.0/optuna/importance/000077500000000000000000000000001453453102400164755ustar00rootroot00000000000000optuna-3.5.0/optuna/importance/__init__.py000066400000000000000000000116031453453102400206070ustar00rootroot00000000000000from typing import Callable from typing import Dict from typing import List from typing import Optional import warnings from optuna.exceptions import ExperimentalWarning from optuna.importance._base import BaseImportanceEvaluator from optuna.importance._fanova import FanovaImportanceEvaluator from optuna.importance._mean_decrease_impurity import MeanDecreaseImpurityImportanceEvaluator from optuna.study import Study from optuna.trial import FrozenTrial __all__ = [ "BaseImportanceEvaluator", "FanovaImportanceEvaluator", "MeanDecreaseImpurityImportanceEvaluator", "get_param_importances", ] def get_param_importances( study: Study, *, evaluator: Optional[BaseImportanceEvaluator] = None, params: Optional[List[str]] = None, target: Optional[Callable[[FrozenTrial], float]] = None, normalize: bool = True, ) -> Dict[str, float]: """Evaluate parameter importances based on completed trials in the given study. The parameter importances are returned as a dictionary where the keys consist of parameter names and their values importances. The importances are represented by non-negative floating point numbers, where higher values mean that the parameters are more important. The returned dictionary is ordered by its values in a descending order. By default, the sum of the importance values are normalized to 1.0. If ``params`` is :obj:`None`, all parameter that are present in all of the completed trials are assessed. This implies that conditional parameters will be excluded from the evaluation. To assess the importances of conditional parameters, a :obj:`list` of parameter names can be specified via ``params``. If specified, only completed trials that contain all of the parameters will be considered. If no such trials are found, an error will be raised. If the given study does not contain completed trials, an error will be raised. .. note:: If ``params`` is specified as an empty list, an empty dictionary is returned. .. seealso:: See :func:`~optuna.visualization.plot_param_importances` to plot importances. Args: study: An optimized study. evaluator: An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to :class:`~optuna.importance.FanovaImportanceEvaluator`. .. note:: :class:`~optuna.importance.FanovaImportanceEvaluator` takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `_ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to evaluate importances. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are used. ``target`` must be specified if ``study`` is being used for multi-objective optimization. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. For example, to get the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. normalize: A boolean option to specify whether the sum of the importance values should be normalized to 1.0. Defaults to :obj:`True`. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Returns: A :obj:`dict` where the keys are parameter names and the values are assessed importances. """ if evaluator is None: evaluator = FanovaImportanceEvaluator() if not isinstance(evaluator, BaseImportanceEvaluator): raise TypeError("Evaluator must be a subclass of BaseImportanceEvaluator.") res = evaluator.evaluate(study, params=params, target=target) if normalize: s = sum(res.values()) if s == 0.0: n_params = len(res) return dict((param, 1.0 / n_params) for param in res.keys()) else: return dict((param, value / s) for (param, value) in res.items()) else: warnings.warn( "`normalize` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) return res optuna-3.5.0/optuna/importance/_base.py000066400000000000000000000150111453453102400201160ustar00rootroot00000000000000import abc from typing import Callable from typing import cast from typing import Collection from typing import Dict from typing import List from typing import Optional from typing import Union import numpy from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.search_space import intersection_search_space from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState class BaseImportanceEvaluator(abc.ABC): """Abstract parameter importance evaluator.""" @abc.abstractmethod def evaluate( self, study: Study, params: Optional[List[str]] = None, *, target: Optional[Callable[[FrozenTrial], float]] = None, ) -> Dict[str, float]: """Evaluate parameter importances based on completed trials in the given study. .. note:: This method is not meant to be called by library users. .. seealso:: Please refer to :func:`~optuna.importance.get_param_importances` for how a concrete evaluator should implement this method. Args: study: An optimized study. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to evaluate importances. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are used. Can also be used for other trial attributes, such as the duration, like ``target=lambda t: t.duration.total_seconds()``. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. For example, to get the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. Returns: A :obj:`dict` where the keys are parameter names and the values are assessed importances. """ # TODO(hvy): Reconsider the interface as logic might violate DRY among multiple evaluators. raise NotImplementedError def _get_distributions(study: Study, params: Optional[List[str]]) -> Dict[str, BaseDistribution]: completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) _check_evaluate_args(completed_trials, params) if params is None: return intersection_search_space(study.get_trials(deepcopy=False)) # New temporary required to pass mypy. Seems like a bug. params_not_none = params assert params_not_none is not None # Compute the search space based on the subset of trials containing all parameters. distributions = None for trial in completed_trials: trial_distributions = trial.distributions if not all(name in trial_distributions for name in params_not_none): continue if distributions is None: distributions = dict( filter( lambda name_and_distribution: name_and_distribution[0] in params_not_none, trial_distributions.items(), ) ) continue if any( trial_distributions[name] != distribution for name, distribution in distributions.items() ): raise ValueError( "Parameters importances cannot be assessed with dynamic search spaces if " "parameters are specified. Specified parameters: {}.".format(params) ) assert distributions is not None # Required to pass mypy. distributions = dict( sorted(distributions.items(), key=lambda name_and_distribution: name_and_distribution[0]) ) return distributions def _check_evaluate_args(completed_trials: List[FrozenTrial], params: Optional[List[str]]) -> None: if len(completed_trials) == 0: raise ValueError("Cannot evaluate parameter importances without completed trials.") if len(completed_trials) == 1: raise ValueError("Cannot evaluate parameter importances with only a single trial.") if params is not None: if not isinstance(params, (list, tuple)): raise TypeError( "Parameters must be specified as a list. Actual parameters: {}.".format(params) ) if any(not isinstance(p, str) for p in params): raise TypeError( "Parameters must be specified by their names with strings. Actual parameters: " "{}.".format(params) ) if len(params) > 0: at_least_one_trial = False for trial in completed_trials: if all(p in trial.distributions for p in params): at_least_one_trial = True break if not at_least_one_trial: raise ValueError( "Study must contain completed trials with all specified parameters. " "Specified parameters: {}.".format(params) ) def _get_filtered_trials( study: Study, params: Collection[str], target: Optional[Callable[[FrozenTrial], float]] ) -> List[FrozenTrial]: trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) return [ trial for trial in trials if set(params) <= set(trial.params) and numpy.isfinite(target(trial) if target is not None else cast(float, trial.value)) ] def _param_importances_to_dict( params: Collection[str], param_importances: Union[numpy.ndarray, float] ) -> Dict[str, float]: return { name: value for name, value in zip(params, numpy.broadcast_to(param_importances, (len(params),))) } def _get_trans_params(trials: List[FrozenTrial], trans: _SearchSpaceTransform) -> numpy.ndarray: return numpy.array([trans.transform(trial.params) for trial in trials]) def _get_target_values( trials: List[FrozenTrial], target: Optional[Callable[[FrozenTrial], float]] ) -> numpy.ndarray: return numpy.array([target(trial) if target is not None else trial.value for trial in trials]) def _sort_dict_by_importance(param_importances: Dict[str, float]) -> Dict[str, float]: return dict( reversed( sorted( param_importances.items(), key=lambda name_and_importance: name_and_importance[1] ) ) ) optuna-3.5.0/optuna/importance/_fanova/000077500000000000000000000000001453453102400201065ustar00rootroot00000000000000optuna-3.5.0/optuna/importance/_fanova/__init__.py000066400000000000000000000001651453453102400222210ustar00rootroot00000000000000from optuna.importance._fanova._evaluator import FanovaImportanceEvaluator __all__ = ["FanovaImportanceEvaluator"] optuna-3.5.0/optuna/importance/_fanova/_evaluator.py000066400000000000000000000121421453453102400226210ustar00rootroot00000000000000from typing import Callable from typing import Dict from typing import List from typing import Optional import numpy from optuna._transform import _SearchSpaceTransform from optuna.importance._base import _get_distributions from optuna.importance._base import _get_filtered_trials from optuna.importance._base import _get_target_values from optuna.importance._base import _get_trans_params from optuna.importance._base import _param_importances_to_dict from optuna.importance._base import _sort_dict_by_importance from optuna.importance._base import BaseImportanceEvaluator from optuna.importance._fanova._fanova import _Fanova from optuna.study import Study from optuna.trial import FrozenTrial class FanovaImportanceEvaluator(BaseImportanceEvaluator): """fANOVA importance evaluator. Implements the fANOVA hyperparameter importance evaluation algorithm in `An Efficient Approach for Assessing Hyperparameter Importance `_. fANOVA fits a random forest regression model that predicts the objective values of :class:`~optuna.trial.TrialState.COMPLETE` trials given their parameter configurations. The more accurate this model is, the more reliable the importances assessed by this class are. .. note:: This class takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `_ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. .. note:: Requires the `sklearn `_ Python package. .. note:: The performance of fANOVA depends on the prediction performance of the underlying random forest model. In order to obtain high prediction performance, it is necessary to cover a wide range of the hyperparameter search space. It is recommended to use an exploration-oriented sampler such as :class:`~optuna.samplers.RandomSampler`. .. note:: For how to cite the original work, please refer to https://automl.github.io/fanova/cite.html. Args: n_trees: The number of trees in the forest. max_depth: The maximum depth of the trees in the forest. seed: Controls the randomness of the forest. For deterministic behavior, specify a value other than :obj:`None`. """ def __init__( self, *, n_trees: int = 64, max_depth: int = 64, seed: Optional[int] = None ) -> None: self._evaluator = _Fanova( n_trees=n_trees, max_depth=max_depth, min_samples_split=2, min_samples_leaf=1, seed=seed, ) def evaluate( self, study: Study, params: Optional[List[str]] = None, *, target: Optional[Callable[[FrozenTrial], float]] = None, ) -> Dict[str, float]: if target is None and study._is_multi_objective(): raise ValueError( "If the `study` is being used for multi-objective optimization, " "please specify the `target`. For example, use " "`target=lambda t: t.values[0]` for the first objective value." ) distributions = _get_distributions(study, params=params) if params is None: params = list(distributions.keys()) assert params is not None # fANOVA does not support parameter distributions with a single value. # However, there is no reason to calculate parameter importance in such case anyway, # since it will always be 0 as the parameter is constant in the objective function. non_single_distributions = { name: dist for name, dist in distributions.items() if not dist.single() } single_distributions = { name: dist for name, dist in distributions.items() if dist.single() } if len(non_single_distributions) == 0: return {} trials: List[FrozenTrial] = _get_filtered_trials(study, params=params, target=target) trans = _SearchSpaceTransform( non_single_distributions, transform_log=False, transform_step=False ) trans_params: numpy.ndarray = _get_trans_params(trials, trans) target_values: numpy.ndarray = _get_target_values(trials, target) evaluator = self._evaluator evaluator.fit( X=trans_params, y=target_values, search_spaces=trans.bounds, column_to_encoded_columns=trans.column_to_encoded_columns, ) param_importances = numpy.array( [evaluator.get_importance(i)[0] for i in range(len(non_single_distributions))] ) param_importances /= numpy.sum(param_importances) return _sort_dict_by_importance( { **_param_importances_to_dict(non_single_distributions.keys(), param_importances), **_param_importances_to_dict(single_distributions.keys(), 0.0), } ) optuna-3.5.0/optuna/importance/_fanova/_fanova.py000066400000000000000000000101571453453102400220750ustar00rootroot00000000000000"""An implementation of `An Efficient Approach for Assessing Hyperparameter Importance`. See http://proceedings.mlr.press/v32/hutter14.pdf and https://automl.github.io/fanova/cite.html for how to cite the original work. This implementation is inspired by the efficient algorithm in `fanova` (https://github.com/automl/fanova) and `pyrfr` (https://github.com/automl/random_forest_run) by the original authors. Differences include relying on scikit-learn to fit random forests (`sklearn.ensemble.RandomForestRegressor`) and that it is otherwise written entirely in Python. This stands in contrast to the original implementation which is partially written in C++. Since Python runtime overhead may become noticeable, included are instead several optimizations, e.g. vectorized NumPy functions to compute the marginals, instead of keeping all running statistics. Known cases include assessing categorical features with a larger number of choices since each choice is given a unique one-hot encoded raw feature. """ from typing import Dict from typing import List from typing import Optional from typing import Tuple from typing import Union import numpy from optuna._imports import try_import from optuna.importance._fanova._tree import _FanovaTree with try_import() as _imports: from sklearn.ensemble import RandomForestRegressor class _Fanova: def __init__( self, n_trees: int, max_depth: int, min_samples_split: Union[int, float], min_samples_leaf: Union[int, float], seed: Optional[int], ) -> None: _imports.check() self._forest = RandomForestRegressor( n_estimators=n_trees, max_depth=max_depth, min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf, random_state=seed, ) self._trees: Optional[List[_FanovaTree]] = None self._variances: Optional[Dict[int, numpy.ndarray]] = None self._column_to_encoded_columns: Optional[List[numpy.ndarray]] = None def fit( self, X: numpy.ndarray, y: numpy.ndarray, search_spaces: numpy.ndarray, column_to_encoded_columns: List[numpy.ndarray], ) -> None: assert X.shape[0] == y.shape[0] assert X.shape[1] == search_spaces.shape[0] assert search_spaces.shape[1] == 2 self._forest.fit(X, y) self._trees = [_FanovaTree(e.tree_, search_spaces) for e in self._forest.estimators_] self._column_to_encoded_columns = column_to_encoded_columns self._variances = {} if all(tree.variance == 0 for tree in self._trees): # If all trees have 0 variance, we cannot assess any importances. # This could occur if for instance `X.shape[0] == 1`. raise RuntimeError("Encountered zero total variance in all trees.") def get_importance(self, feature: int) -> Tuple[float, float]: # Assert that `fit` has been called. assert self._trees is not None assert self._variances is not None self._compute_variances(feature) fractions: Union[List[float], numpy.ndarray] = [] for tree_index, tree in enumerate(self._trees): tree_variance = tree.variance if tree_variance > 0.0: fraction = self._variances[feature][tree_index] / tree_variance fractions = numpy.append(fractions, fraction) fractions = numpy.asarray(fractions) return float(fractions.mean()), float(fractions.std()) def _compute_variances(self, feature: int) -> None: assert self._trees is not None assert self._variances is not None assert self._column_to_encoded_columns is not None if feature in self._variances: return raw_features = self._column_to_encoded_columns[feature] variances = numpy.empty(len(self._trees), dtype=numpy.float64) for tree_index, tree in enumerate(self._trees): marginal_variance = tree.get_marginal_variance(raw_features) variances[tree_index] = numpy.clip(marginal_variance, 0.0, None) self._variances[feature] = variances optuna-3.5.0/optuna/importance/_fanova/_tree.py000066400000000000000000000313031453453102400215560ustar00rootroot00000000000000from __future__ import annotations from functools import lru_cache import itertools from typing import List from typing import Set from typing import Tuple from typing import TYPE_CHECKING from typing import Union import numpy if TYPE_CHECKING: import sklearn.tree class _FanovaTree: def __init__(self, tree: "sklearn.tree._tree.Tree", search_spaces: numpy.ndarray) -> None: assert search_spaces.shape[0] == tree.n_features assert search_spaces.shape[1] == 2 self._tree = tree self._search_spaces = search_spaces statistics = self._precompute_statistics() split_midpoints, split_sizes = self._precompute_split_midpoints_and_sizes() subtree_active_features = self._precompute_subtree_active_features() self._statistics = statistics self._split_midpoints = split_midpoints self._split_sizes = split_sizes self._subtree_active_features = subtree_active_features self._variance = None # Computed lazily and requires `self._statistics`. @property def variance(self) -> float: if self._variance is None: leaf_node_indices = numpy.nonzero(numpy.array(self._tree.feature) < 0)[0] statistics = self._statistics[leaf_node_indices] values = statistics[:, 0] weights = statistics[:, 1] average_values = numpy.average(values, weights=weights) variance = numpy.average((values - average_values) ** 2, weights=weights) self._variance = variance assert self._variance is not None return self._variance def get_marginal_variance(self, features: numpy.ndarray) -> float: assert features.size > 0 # For each midpoint along the given dimensions, traverse this tree to compute the # marginal predictions. selected_midpoints = [self._split_midpoints[f] for f in features] selected_sizes = [self._split_sizes[f] for f in features] product_midpoints = itertools.product(*selected_midpoints) product_sizes = itertools.product(*selected_sizes) sample = numpy.full(self._n_features, fill_value=numpy.nan, dtype=numpy.float64) values: Union[List[float], numpy.ndarray] = [] weights: Union[List[float], numpy.ndarray] = [] for midpoints, sizes in zip(product_midpoints, product_sizes): sample[features] = numpy.array(midpoints) value, weight = self._get_marginalized_statistics(sample) weight *= float(numpy.prod(sizes)) values = numpy.append(values, value) weights = numpy.append(weights, weight) weights = numpy.asarray(weights) values = numpy.asarray(values) average_values = numpy.average(values, weights=weights) variance = numpy.average((values - average_values) ** 2, weights=weights) assert variance >= 0.0 return variance def _get_marginalized_statistics(self, feature_vector: numpy.ndarray) -> Tuple[float, float]: assert feature_vector.size == self._n_features marginalized_features = numpy.isnan(feature_vector) active_features = ~marginalized_features # Reduce search space cardinalities to 1 for non-active features. search_spaces = self._search_spaces.copy() search_spaces[marginalized_features] = [0.0, 1.0] # Start from the root and traverse towards the leafs. active_nodes = [0] active_search_spaces = [search_spaces] node_indices = [] active_leaf_search_spaces = [] while len(active_nodes) > 0: node_index = active_nodes.pop() search_spaces = active_search_spaces.pop() feature = self._get_node_split_feature(node_index) if feature >= 0: # Not leaf. Avoid unnecessary call to `_is_node_leaf`. # If node splits on an active feature, push the child node that we end up in. response = feature_vector[feature] if not numpy.isnan(response): if response <= self._get_node_split_threshold(node_index): next_node_index = self._get_node_left_child(node_index) next_subspace = self._get_node_left_child_subspaces( node_index, search_spaces ) else: next_node_index = self._get_node_right_child(node_index) next_subspace = self._get_node_right_child_subspaces( node_index, search_spaces ) active_nodes.append(next_node_index) active_search_spaces.append(next_subspace) continue # If subtree starting from node splits on an active feature, push both child nodes. # Here, we use `any` for list because `ndarray.any` is slow. if any(self._subtree_active_features[node_index][active_features].tolist()): for child_node_index in self._get_node_children(node_index): active_nodes.append(child_node_index) active_search_spaces.append(search_spaces) continue # If node is a leaf or the subtree does not split on any of the active features. node_indices.append(node_index) active_leaf_search_spaces.append(search_spaces) statistics = self._statistics[node_indices] values = statistics[:, 0] weights = statistics[:, 1] active_features_cardinalities = _get_cardinality_batched(active_leaf_search_spaces) weights = weights / active_features_cardinalities value = numpy.average(values, weights=weights) weight = weights.sum() return value, weight def _precompute_statistics(self) -> numpy.ndarray: n_nodes = self._n_nodes # Holds for each node, its weighted average value and the sum of weights. statistics = numpy.empty((n_nodes, 2), dtype=numpy.float64) subspaces = numpy.array([None for _ in range(n_nodes)]) subspaces[0] = self._search_spaces # Compute marginals for leaf nodes. for node_index in range(n_nodes): subspace = subspaces[node_index] if self._is_node_leaf(node_index): value = self._get_node_value(node_index) weight = _get_cardinality(subspace) statistics[node_index] = [value, weight] else: for child_node_index, child_subspace in zip( self._get_node_children(node_index), self._get_node_children_subspaces(node_index, subspace), ): assert subspaces[child_node_index] is None subspaces[child_node_index] = child_subspace # Compute marginals for internal nodes. for node_index in reversed(range(n_nodes)): if not self._is_node_leaf(node_index): child_values = [] child_weights = [] for child_node_index in self._get_node_children(node_index): child_values.append(statistics[child_node_index, 0]) child_weights.append(statistics[child_node_index, 1]) value = numpy.average(child_values, weights=child_weights) weight = float(numpy.sum(child_weights)) statistics[node_index] = [value, weight] return statistics def _precompute_split_midpoints_and_sizes( self, ) -> Tuple[List[numpy.ndarray], List[numpy.ndarray]]: midpoints = [] sizes = [] search_spaces = self._search_spaces for feature, feature_split_values in enumerate(self._compute_features_split_values()): feature_split_values = numpy.concatenate( ( numpy.atleast_1d(search_spaces[feature, 0]), feature_split_values, numpy.atleast_1d(search_spaces[feature, 1]), ) ) midpoint = 0.5 * (feature_split_values[1:] + feature_split_values[:-1]) size = feature_split_values[1:] - feature_split_values[:-1] midpoints.append(midpoint) sizes.append(size) return midpoints, sizes def _compute_features_split_values(self) -> List[numpy.ndarray]: all_split_values: List[Set[float]] = [set() for _ in range(self._n_features)] for node_index in range(self._n_nodes): feature = self._get_node_split_feature(node_index) if feature >= 0: # Not leaf. Avoid unnecessary call to `_is_node_leaf`. threshold = self._get_node_split_threshold(node_index) all_split_values[feature].add(threshold) sorted_all_split_values: List[numpy.ndarray] = [] for split_values in all_split_values: split_values_array = numpy.array(list(split_values), dtype=numpy.float64) split_values_array.sort() sorted_all_split_values.append(split_values_array) return sorted_all_split_values def _precompute_subtree_active_features(self) -> numpy.ndarray: subtree_active_features = numpy.full((self._n_nodes, self._n_features), fill_value=False) for node_index in reversed(range(self._n_nodes)): feature = self._get_node_split_feature(node_index) if feature >= 0: # Not leaf. Avoid unnecessary call to `_is_node_leaf`. subtree_active_features[node_index, feature] = True for child_node_index in self._get_node_children(node_index): subtree_active_features[node_index] |= subtree_active_features[ child_node_index ] return subtree_active_features @property def _n_features(self) -> int: return len(self._search_spaces) @property def _n_nodes(self) -> int: return self._tree.node_count @lru_cache(maxsize=None) def _is_node_leaf(self, node_index: int) -> bool: return self._tree.feature[node_index] < 0 @lru_cache(maxsize=None) def _get_node_left_child(self, node_index: int) -> int: return self._tree.children_left[node_index] @lru_cache(maxsize=None) def _get_node_right_child(self, node_index: int) -> int: return self._tree.children_right[node_index] @lru_cache(maxsize=None) def _get_node_children(self, node_index: int) -> Tuple[int, int]: return self._get_node_left_child(node_index), self._get_node_right_child(node_index) @lru_cache(maxsize=None) def _get_node_value(self, node_index: int) -> float: # self._tree.value: sklearn.tree._tree.Tree.value has # the shape (node_count, n_outputs, max_n_classes) return float(self._tree.value[node_index].reshape(-1)[0]) @lru_cache(maxsize=None) def _get_node_split_threshold(self, node_index: int) -> float: return self._tree.threshold[node_index] @lru_cache(maxsize=None) def _get_node_split_feature(self, node_index: int) -> int: return self._tree.feature[node_index] def _get_node_left_child_subspaces( self, node_index: int, search_spaces: numpy.ndarray ) -> numpy.ndarray: return _get_subspaces( search_spaces, search_spaces_column=1, feature=self._get_node_split_feature(node_index), threshold=self._get_node_split_threshold(node_index), ) def _get_node_right_child_subspaces( self, node_index: int, search_spaces: numpy.ndarray ) -> numpy.ndarray: return _get_subspaces( search_spaces, search_spaces_column=0, feature=self._get_node_split_feature(node_index), threshold=self._get_node_split_threshold(node_index), ) def _get_node_children_subspaces( self, node_index: int, search_spaces: numpy.ndarray ) -> Tuple[numpy.ndarray, numpy.ndarray]: return ( self._get_node_left_child_subspaces(node_index, search_spaces), self._get_node_right_child_subspaces(node_index, search_spaces), ) def _get_cardinality(search_spaces: numpy.ndarray) -> float: return numpy.prod(search_spaces[:, 1] - search_spaces[:, 0]) def _get_cardinality_batched(search_spaces_list: list[numpy.ndarray]) -> float: search_spaces = numpy.asarray(search_spaces_list) return numpy.prod(search_spaces[:, :, 1] - search_spaces[:, :, 0], axis=1) def _get_subspaces( search_spaces: numpy.ndarray, *, search_spaces_column: int, feature: int, threshold: float ) -> numpy.ndarray: search_spaces_subspace = numpy.copy(search_spaces) search_spaces_subspace[feature, search_spaces_column] = threshold return search_spaces_subspace optuna-3.5.0/optuna/importance/_mean_decrease_impurity.py000066400000000000000000000074021453453102400237260ustar00rootroot00000000000000from typing import Callable from typing import Dict from typing import List from typing import Optional import numpy from optuna._imports import try_import from optuna._transform import _SearchSpaceTransform from optuna.importance._base import _get_distributions from optuna.importance._base import _get_filtered_trials from optuna.importance._base import _get_target_values from optuna.importance._base import _get_trans_params from optuna.importance._base import _param_importances_to_dict from optuna.importance._base import _sort_dict_by_importance from optuna.importance._base import BaseImportanceEvaluator from optuna.study import Study from optuna.trial import FrozenTrial with try_import() as _imports: from sklearn.ensemble import RandomForestRegressor class MeanDecreaseImpurityImportanceEvaluator(BaseImportanceEvaluator): """Mean Decrease Impurity (MDI) parameter importance evaluator. This evaluator fits fits a random forest regression model that predicts the objective values of :class:`~optuna.trial.TrialState.COMPLETE` trials given their parameter configurations. Feature importances are then computed using MDI. .. note:: This evaluator requires the `sklearn `_ Python package and is based on `sklearn.ensemble.RandomForestClassifier.feature_importances_ `_. Args: n_trees: Number of trees in the random forest. max_depth: The maximum depth of each tree in the random forest. seed: Seed for the random forest. """ def __init__( self, *, n_trees: int = 64, max_depth: int = 64, seed: Optional[int] = None ) -> None: _imports.check() self._forest = RandomForestRegressor( n_estimators=n_trees, max_depth=max_depth, min_samples_split=2, min_samples_leaf=1, random_state=seed, ) self._trans_params = numpy.empty(0) self._trans_values = numpy.empty(0) self._param_names: List[str] = list() def evaluate( self, study: Study, params: Optional[List[str]] = None, *, target: Optional[Callable[[FrozenTrial], float]] = None, ) -> Dict[str, float]: if target is None and study._is_multi_objective(): raise ValueError( "If the `study` is being used for multi-objective optimization, " "please specify the `target`. For example, use " "`target=lambda t: t.values[0]` for the first objective value." ) distributions = _get_distributions(study, params=params) if params is None: params = list(distributions.keys()) assert params is not None if len(params) == 0: return {} trials: List[FrozenTrial] = _get_filtered_trials(study, params=params, target=target) trans = _SearchSpaceTransform(distributions, transform_log=False, transform_step=False) trans_params: numpy.ndarray = _get_trans_params(trials, trans) target_values: numpy.ndarray = _get_target_values(trials, target) forest = self._forest forest.fit(X=trans_params, y=target_values) feature_importances = forest.feature_importances_ # Untransform feature importances to param importances # by adding up relevant feature importances. param_importances = numpy.zeros(len(params)) numpy.add.at(param_importances, trans.encoded_column_to_column, feature_importances) return _sort_dict_by_importance(_param_importances_to_dict(params, param_importances)) optuna-3.5.0/optuna/integration/000077500000000000000000000000001453453102400166575ustar00rootroot00000000000000optuna-3.5.0/optuna/integration/__init__.py000066400000000000000000000140441453453102400207730ustar00rootroot00000000000000import os import sys from types import ModuleType from typing import Any from typing import TYPE_CHECKING _import_structure = { "allennlp": ["AllenNLPExecutor", "AllenNLPPruningCallback"], "botorch": ["BoTorchSampler"], "catalyst": ["CatalystPruningCallback"], "catboost": ["CatBoostPruningCallback"], "chainer": ["ChainerPruningExtension"], "chainermn": ["ChainerMNStudy"], "cma": ["CmaEsSampler", "PyCmaSampler"], "dask": ["DaskStorage"], "mlflow": ["MLflowCallback"], "wandb": ["WeightsAndBiasesCallback"], "keras": ["KerasPruningCallback"], "lightgbm": ["LightGBMPruningCallback", "LightGBMTuner", "LightGBMTunerCV"], "pytorch_distributed": ["TorchDistributedTrial"], "pytorch_ignite": ["PyTorchIgnitePruningHandler"], "pytorch_lightning": ["PyTorchLightningPruningCallback"], "sklearn": ["OptunaSearchCV"], "shap": ["ShapleyImportanceEvaluator"], "skorch": ["SkorchPruningCallback"], "mxnet": ["MXNetPruningCallback"], "skopt": ["SkoptSampler"], "tensorboard": ["TensorBoardCallback"], "tensorflow": ["TensorFlowPruningHook"], "tfkeras": ["TFKerasPruningCallback"], "xgboost": ["XGBoostPruningCallback"], "fastaiv1": ["FastAIV1PruningCallback"], "fastaiv2": ["FastAIV2PruningCallback", "FastAIPruningCallback"], } if TYPE_CHECKING: from optuna.integration.allennlp import AllenNLPExecutor from optuna.integration.allennlp import AllenNLPPruningCallback from optuna.integration.botorch import BoTorchSampler from optuna.integration.catalyst import CatalystPruningCallback from optuna.integration.catboost import CatBoostPruningCallback from optuna.integration.chainer import ChainerPruningExtension from optuna.integration.chainermn import ChainerMNStudy from optuna.integration.cma import CmaEsSampler from optuna.integration.cma import PyCmaSampler from optuna.integration.dask import DaskStorage from optuna.integration.fastaiv1 import FastAIV1PruningCallback from optuna.integration.fastaiv2 import FastAIPruningCallback from optuna.integration.fastaiv2 import FastAIV2PruningCallback from optuna.integration.keras import KerasPruningCallback from optuna.integration.lightgbm import LightGBMPruningCallback from optuna.integration.lightgbm import LightGBMTuner from optuna.integration.lightgbm import LightGBMTunerCV from optuna.integration.mlflow import MLflowCallback from optuna.integration.mxnet import MXNetPruningCallback from optuna.integration.pytorch_distributed import TorchDistributedTrial from optuna.integration.pytorch_ignite import PyTorchIgnitePruningHandler from optuna.integration.pytorch_lightning import PyTorchLightningPruningCallback from optuna.integration.shap import ShapleyImportanceEvaluator from optuna.integration.sklearn import OptunaSearchCV from optuna.integration.skopt import SkoptSampler from optuna.integration.skorch import SkorchPruningCallback from optuna.integration.tensorboard import TensorBoardCallback from optuna.integration.tensorflow import TensorFlowPruningHook from optuna.integration.tfkeras import TFKerasPruningCallback from optuna.integration.wandb import WeightsAndBiasesCallback from optuna.integration.xgboost import XGBoostPruningCallback else: class _IntegrationModule(ModuleType): """Module class that implements `optuna.integration` package. This class applies lazy import under `optuna.integration`, where submodules are imported when they are actually accessed. Otherwise, `import optuna` becomes much slower because it imports all submodules and their dependencies (e.g., chainer, keras, lightgbm) all at once. """ __file__ = globals()["__file__"] __path__ = [os.path.dirname(__file__)] _modules = set(_import_structure.keys()) _class_to_module = {} for key, values in _import_structure.items(): for value in values: _class_to_module[value] = key def __getattr__(self, name: str) -> Any: if name in self._modules: value = self._get_module(name) elif name in self._class_to_module.keys(): module = self._get_module(self._class_to_module[name]) value = getattr(module, name) else: raise AttributeError("module {} has no attribute {}".format(self.__name__, name)) setattr(self, name, value) return value def _get_module(self, module_name: str) -> ModuleType: import importlib try: return importlib.import_module("." + module_name, self.__name__) except ModuleNotFoundError: raise ModuleNotFoundError( "Optuna's integration modules for third-party libraries have started " "migrating from Optuna itself to a package called `optuna-integration`. " "The module you are trying to use has already been migrated to " "`optuna-integration`. Please install the package by running " "`pip install optuna-integration`." ) sys.modules[__name__] = _IntegrationModule(__name__) __all__ = [ "AllenNLPExecutor", "AllenNLPPruningCallback", "BoTorchSampler", "CatalystPruningCallback", "CatBoostPruningCallback", "ChainerPruningExtension", "ChainerMNStudy", "CmaEsSampler", "PyCmaSampler", "DaskStorage", "MLflowCallback", "WeightsAndBiasesCallback", "KerasPruningCallback", "LightGBMPruningCallback", "LightGBMTuner", "LightGBMTunerCV", "TorchDistributedTrial", "PyTorchIgnitePruningHandler", "PyTorchLightningPruningCallback", "OptunaSearchCV", "ShapleyImportanceEvaluator", "SkorchPruningCallback", "MXNetPruningCallback", "SkoptSampler", "TensorBoardCallback", "TensorFlowPruningHook", "TFKerasPruningCallback", "XGBoostPruningCallback", "FastAIV1PruningCallback", "FastAIV2PruningCallback", "FastAIPruningCallback", ] optuna-3.5.0/optuna/integration/_lightgbm_tuner/000077500000000000000000000000001453453102400220305ustar00rootroot00000000000000optuna-3.5.0/optuna/integration/_lightgbm_tuner/__init__.py000066400000000000000000000012061453453102400241400ustar00rootroot00000000000000from optuna.integration._lightgbm_tuner._train import train from optuna.integration._lightgbm_tuner.optimize import _imports from optuna.integration._lightgbm_tuner.optimize import LightGBMTuner from optuna.integration._lightgbm_tuner.optimize import LightGBMTunerCV if _imports.is_successful(): from optuna.integration._lightgbm_tuner.sklearn import LGBMClassifier from optuna.integration._lightgbm_tuner.sklearn import LGBMModel from optuna.integration._lightgbm_tuner.sklearn import LGBMRegressor __all__ = [ "LightGBMTuner", "LightGBMTunerCV", "LGBMClassifier", "LGBMModel", "LGBMRegressor", "train", ] optuna-3.5.0/optuna/integration/_lightgbm_tuner/_train.py000066400000000000000000000136641453453102400236700ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from typing import Any from optuna._imports import try_import from optuna.integration._lightgbm_tuner.optimize import _imports from optuna.integration._lightgbm_tuner.optimize import LightGBMTuner from optuna.study import Study from optuna.trial import FrozenTrial with try_import(): import lightgbm as lgb def train( params: dict[str, Any], train_set: "lgb.Dataset", num_boost_round: int = 1000, valid_sets: list["lgb.Dataset"] | tuple["lgb.Dataset", ...] | "lgb.Dataset" | None = None, valid_names: Any | None = None, feval: Callable[..., Any] | None = None, feature_name: str = "auto", categorical_feature: str = "auto", keep_training_booster: bool = False, callbacks: list[Callable[..., Any]] | None = None, time_budget: int | None = None, sample_size: int | None = None, study: Study | None = None, optuna_callbacks: list[Callable[[Study, FrozenTrial], None]] | None = None, model_dir: str | None = None, verbosity: int | None = None, show_progress_bar: bool = True, *, optuna_seed: int | None = None, ) -> "lgb.Booster": """Wrapper of LightGBM Training API to tune hyperparameters. It optimizes the following hyperparameters in a stepwise manner: ``lambda_l1``, ``lambda_l2``, ``num_leaves``, ``feature_fraction``, ``bagging_fraction``, ``bagging_freq`` and ``min_child_samples``. It is a drop-in replacement for `lightgbm.train()`_. See `a simple example of LightGBM Tuner `_ which optimizes the validation log loss of cancer detection. :func:`~optuna.integration.lightgbm.train` is a wrapper function of :class:`~optuna.integration.lightgbm.LightGBMTuner`. To use feature in Optuna such as suspended/resumed optimization and/or parallelization, refer to :class:`~optuna.integration.lightgbm.LightGBMTuner` instead of this function. .. note:: Arguments and keyword arguments for `lightgbm.train()`_ can be passed. For ``params``, please check `the official documentation for LightGBM `_. Args: time_budget: A time budget for parameter tuning in seconds. study: A :class:`~optuna.study.Study` instance to store optimization results. The :class:`~optuna.trial.Trial` instances in it has the following user attributes: ``elapsed_secs`` is the elapsed time since the optimization starts. ``average_iteration_time`` is the average time of iteration to train the booster model in the trial. ``lgbm_params`` is a JSON-serialized dictionary of LightGBM parameters used in the trial. optuna_callbacks: List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. Please note that this is not a ``callbacks`` argument of `lightgbm.train()`_ . model_dir: A directory to save boosters. By default, it is set to :obj:`None` and no boosters are saved. Please set shared directory (e.g., directories on NFS) if you want to access :meth:`~optuna.integration.lightgbm.LightGBMTuner.get_best_booster` in distributed environments. Otherwise, it may raise :obj:`ValueError`. If the directory does not exist, it will be created. The filenames of the boosters will be ``{model_dir}/{trial_number}.pkl`` (e.g., ``./boosters/0.pkl``). verbosity: A verbosity level to change Optuna's logging level. The level is aligned to `LightGBM's verbosity`_ . .. warning:: Deprecated in v2.0.0. ``verbosity`` argument will be removed in the future. The removal of this feature is currently scheduled for v4.0.0, but this schedule is subject to change. Please use :func:`~optuna.logging.set_verbosity` instead. show_progress_bar: Flag to show progress bars or not. To disable progress bar, set this :obj:`False`. .. note:: Progress bars will be fragmented by logging messages of LightGBM and Optuna. Please suppress such messages to show the progress bars properly. optuna_seed: ``seed`` of :class:`~optuna.samplers.TPESampler` for random number generator that affects sampling for ``num_leaves``, ``bagging_fraction``, ``bagging_freq``, ``lambda_l1``, and ``lambda_l2``. .. note:: The `deterministic`_ parameter of LightGBM makes training reproducible. Please enable it when you use this argument. .. _lightgbm.train(): https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.train.html .. _LightGBM's verbosity: https://lightgbm.readthedocs.io/en/latest/Parameters.html#verbosity .. _deterministic: https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic """ _imports.check() auto_booster = LightGBMTuner( params=params, train_set=train_set, num_boost_round=num_boost_round, valid_sets=valid_sets, valid_names=valid_names, feval=feval, feature_name=feature_name, categorical_feature=categorical_feature, keep_training_booster=keep_training_booster, callbacks=callbacks, time_budget=time_budget, sample_size=sample_size, study=study, optuna_callbacks=optuna_callbacks, model_dir=model_dir, verbosity=verbosity, show_progress_bar=show_progress_bar, optuna_seed=optuna_seed, ) auto_booster.run() return auto_booster.get_best_booster() optuna-3.5.0/optuna/integration/_lightgbm_tuner/alias.py000066400000000000000000000111201453453102400234660ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Iterable from typing import Any import warnings _ALIAS_GROUP_LIST: list[dict[str, Any]] = [ {"param_name": "bagging_fraction", "alias_names": ["sub_row", "subsample", "bagging"]}, {"param_name": "learning_rate", "alias_names": ["shrinkage_rate", "eta"]}, { "param_name": "min_child_samples", "alias_names": ["min_data_per_leaf", "min_data", "min_data_in_leaf", "min_samples_leaf"], }, { "param_name": "min_sum_hessian_in_leaf", "alias_names": [ "min_sum_hessian_per_leaf", "min_sum_hessian", "min_hessian", "min_child_weight", ], }, { "param_name": "num_leaves", "alias_names": [ "num_leaf", "max_leaves", "max_leaf", "max_leaf_nodes", ], }, {"param_name": "bagging_freq", "alias_names": ["subsample_freq"]}, {"param_name": "feature_fraction", "alias_names": ["sub_feature", "colsample_bytree"]}, {"param_name": "lambda_l1", "alias_names": ["reg_alpha", "l1_regularization"]}, {"param_name": "lambda_l2", "alias_names": ["reg_lambda", "lambda", "l2_regularization"]}, {"param_name": "min_gain_to_split", "alias_names": ["min_split_gain"]}, ] def _handling_alias_parameters(lgbm_params: dict[str, Any]) -> None: """Handling alias parameters.""" for alias_group in _ALIAS_GROUP_LIST: param_name = alias_group["param_name"] alias_names = alias_group["alias_names"] duplicated_alias: dict[str, Any] = {} for alias_name in alias_names: if alias_name in lgbm_params: duplicated_alias[alias_name] = lgbm_params[alias_name] lgbm_params[param_name] = lgbm_params[alias_name] del lgbm_params[alias_name] if len(duplicated_alias) > 1: msg = ( f"{param_name} in param detected multiple identical aliases {duplicated_alias}, " f"but we use {param_name}={lgbm_params[param_name]}." ) warnings.warn(msg) _ALIAS_METRIC_LIST: list[dict[str, Any]] = [ # The list `alias_names` do not include the `metric_name` itself. { "metric_name": "ndcg", "alias_names": [ "lambdarank", "rank_xendcg", "xendcg", "xe_ndcg", "xe_ndcg_mart", "xendcg_mart", ], }, {"metric_name": "map", "alias_names": ["mean_average_precision"]}, { "metric_name": "l2", "alias_names": ["regression", "regression_l2", "mean_squared_error", "mse"], }, { "metric_name": "l1", "alias_names": ["regression_l1", "mean_absolute_error", "mae"], }, { "metric_name": "binary_logloss", "alias_names": ["binary"], }, { "metric_name": "multi_logloss", "alias_names": [ "multiclass", "softmax", "multiclassova", "multiclass_ova", "ova", "ovr", ], }, { "metric_name": "cross_entropy", "alias_names": ["xentropy"], }, { "metric_name": "cross_entropy_lambda", "alias_names": ["xentlambda"], }, { "metric_name": "kullback_leibler", "alias_names": ["kldiv"], }, { "metric_name": "mape", "alias_names": ["mean_absolute_percentage_error"], }, { "metric_name": "custom", "alias_names": ["none", "null", "na"], }, { "metric_name": "rmse", "alias_names": ["l2_root", "root_mean_squared_error"], }, ] _ALIAS_METRIC_MAP: dict[str, str] = { alias_name: canonical_metric["metric_name"] for canonical_metric in _ALIAS_METRIC_LIST for alias_name in canonical_metric["alias_names"] } def _handling_alias_metrics(lgbm_params: dict[str, Any]) -> None: """Handling alias metrics.""" if "metric" not in lgbm_params.keys(): return if not isinstance(lgbm_params["metric"], (str, Iterable)): raise ValueError( "The `metric` parameter is expected to be a string or an iterable object, but got " f"{type(lgbm_params['metric'])}." ) if isinstance(lgbm_params["metric"], str): lgbm_params["metric"] = ( _ALIAS_METRIC_MAP.get(lgbm_params["metric"]) or lgbm_params["metric"] ) return canonical_metrics = [] for metric in lgbm_params["metric"]: canonical_metrics.append(_ALIAS_METRIC_MAP.get(metric) or metric) lgbm_params["metric"] = canonical_metrics optuna-3.5.0/optuna/integration/_lightgbm_tuner/optimize.py000066400000000000000000001244711453453102400242530ustar00rootroot00000000000000from __future__ import annotations import abc from collections.abc import Callable from collections.abc import Container from collections.abc import Generator from collections.abc import Iterable from collections.abc import Iterator from collections.abc import Sequence import copy import json import os import pickle import time from typing import Any from typing import cast import warnings import numpy as np import tqdm import optuna from optuna._imports import try_import from optuna.integration._lightgbm_tuner.alias import _handling_alias_metrics from optuna.integration._lightgbm_tuner.alias import _handling_alias_parameters from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState with try_import() as _imports: import lightgbm as lgb from sklearn.model_selection import BaseCrossValidator # Define key names of `Trial.system_attrs`. _ELAPSED_SECS_KEY = "lightgbm_tuner:elapsed_secs" _AVERAGE_ITERATION_TIME_KEY = "lightgbm_tuner:average_iteration_time" _STEP_NAME_KEY = "lightgbm_tuner:step_name" _LGBM_PARAMS_KEY = "lightgbm_tuner:lgbm_params" # EPS is used to ensure that a sampled parameter value is in pre-defined value range. _EPS = 1e-12 # Default value of tree_depth, used for upper bound of num_leaves. _DEFAULT_TUNER_TREE_DEPTH = 8 # Default parameter values described in the official webpage. _DEFAULT_LIGHTGBM_PARAMETERS = { "lambda_l1": 0.0, "lambda_l2": 0.0, "num_leaves": 31, "feature_fraction": 1.0, "bagging_fraction": 1.0, "bagging_freq": 0, "min_child_samples": 20, } _logger = optuna.logging.get_logger(__name__) class _BaseTuner: def __init__( self, lgbm_params: dict[str, Any] | None = None, lgbm_kwargs: dict[str, Any] | None = None, ) -> None: # Handling alias metrics. if lgbm_params is not None: _handling_alias_metrics(lgbm_params) self.lgbm_params = lgbm_params or {} self.lgbm_kwargs = lgbm_kwargs or {} def _get_metric_for_objective(self) -> str: metric = self.lgbm_params.get("metric", "binary_logloss") # todo (smly): This implementation is different logic from the LightGBM's python bindings. if isinstance(metric, str): pass elif isinstance(metric, Sequence): metric = metric[-1] elif isinstance(metric, Iterable): metric = list(metric)[-1] else: raise NotImplementedError metric = self._metric_with_eval_at(metric) return metric def _get_booster_best_score(self, booster: "lgb.Booster") -> float: metric = self._get_metric_for_objective() valid_sets: list["lgb.Dataset"] | tuple[ "lgb.Dataset", ... ] | "lgb.Dataset" | None = self.lgbm_kwargs.get("valid_sets") if self.lgbm_kwargs.get("valid_names") is not None: if isinstance(self.lgbm_kwargs["valid_names"], str): valid_name = self.lgbm_kwargs["valid_names"] elif isinstance(self.lgbm_kwargs["valid_names"], Sequence): valid_name = self.lgbm_kwargs["valid_names"][-1] else: raise NotImplementedError elif isinstance(valid_sets, lgb.Dataset): valid_name = "valid_0" elif isinstance(valid_sets, Sequence) and len(valid_sets) > 0: valid_set_idx = len(valid_sets) - 1 valid_name = f"valid_{valid_set_idx}" else: raise NotImplementedError val_score = booster.best_score[valid_name][metric] return val_score def _metric_with_eval_at(self, metric: str) -> str: # The parameter eval_at is only available when the metric is ndcg or map if metric not in ["ndcg", "map"]: return metric eval_at = ( self.lgbm_params.get("eval_at") or self.lgbm_params.get(f"{metric}_at") or self.lgbm_params.get(f"{metric}_eval_at") # Set default value of LightGBM when no possible key is absent. # See https://lightgbm.readthedocs.io/en/latest/Parameters.html#eval_at. or [1, 2, 3, 4, 5] ) # Optuna can handle only a single metric. Choose first one. if isinstance(eval_at, (list, tuple)): return f"{metric}@{eval_at[0]}" if isinstance(eval_at, int): return f"{metric}@{eval_at}" raise ValueError( f"The value of eval_at is expected to be int or a list/tuple of int. '{eval_at}' is " "specified." ) def higher_is_better(self) -> bool: metric_name = self.lgbm_params.get("metric", "binary_logloss") return metric_name in ("auc", "auc_mu", "ndcg", "map", "average_precision") def compare_validation_metrics(self, val_score: float, best_score: float) -> bool: if self.higher_is_better(): return val_score > best_score else: return val_score < best_score class _OptunaObjective(_BaseTuner): """Objective for hyperparameter-tuning with Optuna.""" def __init__( self, target_param_names: list[str], lgbm_params: dict[str, Any], train_set: "lgb.Dataset", lgbm_kwargs: dict[str, Any], best_score: float, step_name: str, model_dir: str | None, pbar: tqdm.tqdm | None = None, ): self.target_param_names = target_param_names self.pbar = pbar self.lgbm_params = lgbm_params self.lgbm_kwargs = lgbm_kwargs self.train_set = train_set self.trial_count = 0 self.best_score = best_score self.best_booster_with_trial_number: tuple[ "lgb.Booster" | "lgb.CVBooster", int ] | None = None self.step_name = step_name self.model_dir = model_dir self._check_target_names_supported() self.pbar_fmt = "{}, val_score: {:.6f}" def _check_target_names_supported(self) -> None: for target_param_name in self.target_param_names: if target_param_name in _DEFAULT_LIGHTGBM_PARAMETERS: continue raise NotImplementedError( f"Parameter `{target_param_name}` is not supported for tuning." ) def _preprocess(self, trial: optuna.trial.Trial) -> None: if self.pbar is not None: self.pbar.set_description(self.pbar_fmt.format(self.step_name, self.best_score)) if "lambda_l1" in self.target_param_names: self.lgbm_params["lambda_l1"] = trial.suggest_float("lambda_l1", 1e-8, 10.0, log=True) if "lambda_l2" in self.target_param_names: self.lgbm_params["lambda_l2"] = trial.suggest_float("lambda_l2", 1e-8, 10.0, log=True) if "num_leaves" in self.target_param_names: tree_depth = self.lgbm_params.get("max_depth", _DEFAULT_TUNER_TREE_DEPTH) max_num_leaves = 2**tree_depth if tree_depth > 0 else 2**_DEFAULT_TUNER_TREE_DEPTH self.lgbm_params["num_leaves"] = trial.suggest_int("num_leaves", 2, max_num_leaves) if "feature_fraction" in self.target_param_names: # `GridSampler` is used for sampling feature_fraction value. # The value 1.0 for the hyperparameter is always sampled. param_value = min(trial.suggest_float("feature_fraction", 0.4, 1.0 + _EPS), 1.0) self.lgbm_params["feature_fraction"] = param_value if "bagging_fraction" in self.target_param_names: # `TPESampler` is used for sampling bagging_fraction value. # The value 1.0 for the hyperparameter might by sampled. param_value = min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + _EPS), 1.0) self.lgbm_params["bagging_fraction"] = param_value if "bagging_freq" in self.target_param_names: self.lgbm_params["bagging_freq"] = trial.suggest_int("bagging_freq", 1, 7) if "min_child_samples" in self.target_param_names: # `GridSampler` is used for sampling min_child_samples value. # The value 1.0 for the hyperparameter is always sampled. param_value = trial.suggest_int("min_child_samples", 5, 100) self.lgbm_params["min_child_samples"] = param_value def _copy_valid_sets( self, valid_sets: list["lgb.Dataset"] | tuple["lgb.Dataset", ...] | "lgb.Dataset" ) -> list["lgb.Dataset"] | tuple["lgb.Dataset", ...] | "lgb.Dataset": if isinstance(valid_sets, list): return [copy.copy(d) for d in valid_sets] if isinstance(valid_sets, tuple): return tuple([copy.copy(d) for d in valid_sets]) return copy.copy(valid_sets) def __call__(self, trial: optuna.trial.Trial) -> float: self._preprocess(trial) start_time = time.time() train_set = copy.copy(self.train_set) kwargs = copy.copy(self.lgbm_kwargs) kwargs["valid_sets"] = self._copy_valid_sets(kwargs["valid_sets"]) booster = lgb.train(self.lgbm_params, train_set, **kwargs) val_score = self._get_booster_best_score(booster) elapsed_secs = time.time() - start_time average_iteration_time = elapsed_secs / booster.current_iteration() if self.model_dir is not None: path = os.path.join(self.model_dir, f"{trial.number}.pkl") with open(path, "wb") as fout: pickle.dump(booster, fout) _logger.info(f"The booster of trial#{trial.number} was saved as {path}.") if self.compare_validation_metrics(val_score, self.best_score): self.best_score = val_score self.best_booster_with_trial_number = (booster, trial.number) self._postprocess(trial, elapsed_secs, average_iteration_time) return val_score def _postprocess( self, trial: optuna.trial.Trial, elapsed_secs: float, average_iteration_time: float ) -> None: if self.pbar is not None: self.pbar.set_description(self.pbar_fmt.format(self.step_name, self.best_score)) self.pbar.update(1) trial.storage.set_trial_system_attr(trial._trial_id, _ELAPSED_SECS_KEY, elapsed_secs) trial.storage.set_trial_system_attr( trial._trial_id, _AVERAGE_ITERATION_TIME_KEY, average_iteration_time ) trial.storage.set_trial_system_attr(trial._trial_id, _STEP_NAME_KEY, self.step_name) trial.storage.set_trial_system_attr( trial._trial_id, _LGBM_PARAMS_KEY, json.dumps(self.lgbm_params) ) self.trial_count += 1 class _OptunaObjectiveCV(_OptunaObjective): def __init__( self, target_param_names: list[str], lgbm_params: dict[str, Any], train_set: "lgb.Dataset", lgbm_kwargs: dict[str, Any], best_score: float, step_name: str, model_dir: str | None, pbar: tqdm.tqdm | None = None, ): super().__init__( target_param_names, lgbm_params, train_set, lgbm_kwargs, best_score, step_name, model_dir, pbar=pbar, ) def _get_cv_scores(self, cv_results: dict[str, list[float] | "lgb.CVBooster"]) -> list[float]: metric = self._get_metric_for_objective() metric_key = f"{metric}-mean" # The prefix "valid " is added to metric name since LightGBM v4.0.0. val_scores = ( cv_results[metric_key] if metric_key in cv_results else cv_results["valid " + metric_key] ) assert not isinstance(val_scores, lgb.CVBooster) return val_scores def __call__(self, trial: optuna.trial.Trial) -> float: self._preprocess(trial) start_time = time.time() train_set = copy.copy(self.train_set) cv_results = lgb.cv(self.lgbm_params, train_set, **self.lgbm_kwargs) val_scores = self._get_cv_scores(cv_results) val_score = val_scores[-1] elapsed_secs = time.time() - start_time average_iteration_time = elapsed_secs / len(val_scores) if self.model_dir is not None and self.lgbm_kwargs.get("return_cvbooster"): path = os.path.join(self.model_dir, f"{trial.number}.pkl") with open(path, "wb") as fout: # At version `lightgbm==3.0.0`, :class:`lightgbm.CVBooster` does not # have `__getstate__` which is required for pickle serialization. cvbooster = cv_results["cvbooster"] assert isinstance(cvbooster, lgb.CVBooster) pickle.dump((cvbooster.boosters, cvbooster.best_iteration), fout) _logger.info(f"The booster of trial#{trial.number} was saved as {path}.") if self.compare_validation_metrics(val_score, self.best_score): self.best_score = val_score if self.lgbm_kwargs.get("return_cvbooster"): assert not isinstance(cv_results["cvbooster"], list) self.best_booster_with_trial_number = (cv_results["cvbooster"], trial.number) self._postprocess(trial, elapsed_secs, average_iteration_time) return val_score class _LightGBMBaseTuner(_BaseTuner): """Base class of LightGBM Tuners. This class has common attributes and methods of :class:`~optuna.integration.lightgbm.LightGBMTuner` and :class:`~optuna.integration.lightgbm.LightGBMTunerCV`. """ def __init__( self, params: dict[str, Any], train_set: "lgb.Dataset", callbacks: list[Callable[..., Any]] | None = None, num_boost_round: int = 1000, feval: Callable[..., Any] | None = None, feature_name: str = "auto", categorical_feature: str = "auto", time_budget: int | None = None, sample_size: int | None = None, study: optuna.study.Study | None = None, optuna_callbacks: list[Callable[[Study, FrozenTrial], None]] | None = None, verbosity: int | None = None, show_progress_bar: bool = True, model_dir: str | None = None, *, optuna_seed: int | None = None, ) -> None: _imports.check() params = copy.deepcopy(params) # Handling alias metrics. _handling_alias_metrics(params) args = [params, train_set] kwargs: dict[str, Any] = dict( num_boost_round=num_boost_round, feval=feval, feature_name=feature_name, categorical_feature=categorical_feature, callbacks=callbacks, time_budget=time_budget, sample_size=sample_size, verbosity=verbosity, show_progress_bar=show_progress_bar, ) self._parse_args(*args, **kwargs) self._start_time: float | None = None self._optuna_callbacks = optuna_callbacks self._best_booster_with_trial_number: tuple[lgb.Booster | lgb.CVBooster, int] | None = None self._model_dir = model_dir self._optuna_seed = optuna_seed # Should not alter data since `min_child_samples` is tuned. # https://lightgbm.readthedocs.io/en/latest/Parameters.html#feature_pre_filter if self.lgbm_params.get("feature_pre_filter", False): warnings.warn( "feature_pre_filter is given as True but will be set to False. This is required " "for the tuner to tune min_child_samples." ) self.lgbm_params["feature_pre_filter"] = False if study is None: self.study = optuna.create_study( direction="maximize" if self.higher_is_better() else "minimize" ) else: self.study = study if self.higher_is_better(): if self.study.direction != optuna.study.StudyDirection.MAXIMIZE: metric_name = self.lgbm_params.get("metric", "binary_logloss") raise ValueError( f"Study direction is inconsistent with the metric {metric_name}. " "Please set 'maximize' as the direction." ) else: if self.study.direction != optuna.study.StudyDirection.MINIMIZE: metric_name = self.lgbm_params.get("metric", "binary_logloss") raise ValueError( f"Study direction is inconsistent with the metric {metric_name}. " "Please set 'minimize' as the direction." ) if verbosity is not None: warnings.warn( "`verbosity` argument is deprecated and will be removed in the future. " "The removal of this feature is currently scheduled for v4.0.0, " "but this schedule is subject to change. Please use optuna.logging.set_verbosity()" " instead.", FutureWarning, ) if self._model_dir is not None and not os.path.exists(self._model_dir): os.mkdir(self._model_dir) @property def best_score(self) -> float: """Return the score of the best booster.""" try: return self.study.best_value except ValueError: # Return the default score because no trials have completed. return -np.inf if self.higher_is_better() else np.inf @property def best_params(self) -> dict[str, Any]: """Return parameters of the best booster.""" try: return json.loads(self.study.best_trial.system_attrs[_LGBM_PARAMS_KEY]) except ValueError: # Return the default score because no trials have completed. params = copy.deepcopy(_DEFAULT_LIGHTGBM_PARAMETERS) # self.lgbm_params may contain parameters given by users. params.update(self.lgbm_params) return params def _parse_args(self, *args: Any, **kwargs: Any) -> None: self.auto_options = { option_name: kwargs.get(option_name) for option_name in ["time_budget", "sample_size", "verbosity", "show_progress_bar"] } # Split options. for option_name in self.auto_options.keys(): if option_name in kwargs: del kwargs[option_name] self.lgbm_params = args[0] self.train_set = args[1] self.train_subset = None # Use for sampling. self.lgbm_kwargs = kwargs def run(self) -> None: """Perform the hyperparameter-tuning with given parameters.""" verbosity = self.auto_options["verbosity"] if verbosity is not None: if verbosity > 1: optuna.logging.set_verbosity(optuna.logging.DEBUG) elif verbosity == 1: optuna.logging.set_verbosity(optuna.logging.INFO) elif verbosity == 0: optuna.logging.set_verbosity(optuna.logging.WARNING) else: optuna.logging.set_verbosity(optuna.logging.CRITICAL) # Handling aliases. _handling_alias_parameters(self.lgbm_params) # Sampling. self.sample_train_set() self.tune_feature_fraction() self.tune_num_leaves() self.tune_bagging() self.tune_feature_fraction_stage2() self.tune_regularization_factors() self.tune_min_data_in_leaf() def sample_train_set(self) -> None: """Make subset of `self.train_set` Dataset object.""" if self.auto_options["sample_size"] is None: return self.train_set.construct() n_train_instance = self.train_set.get_label().shape[0] if n_train_instance > self.auto_options["sample_size"]: offset = n_train_instance - self.auto_options["sample_size"] idx_list = offset + np.arange(self.auto_options["sample_size"]) self.train_subset = self.train_set.subset(idx_list) def tune_feature_fraction(self, n_trials: int = 7) -> None: param_name = "feature_fraction" param_values = np.linspace(0.4, 1.0, n_trials).tolist() sampler = optuna.samplers.GridSampler({param_name: param_values}, seed=self._optuna_seed) self._tune_params([param_name], len(param_values), sampler, "feature_fraction") def tune_num_leaves(self, n_trials: int = 20) -> None: self._tune_params( ["num_leaves"], n_trials, optuna.samplers.TPESampler(seed=self._optuna_seed), "num_leaves", ) def tune_bagging(self, n_trials: int = 10) -> None: self._tune_params( ["bagging_fraction", "bagging_freq"], n_trials, optuna.samplers.TPESampler(seed=self._optuna_seed), "bagging", ) def tune_feature_fraction_stage2(self, n_trials: int = 6) -> None: param_name = "feature_fraction" best_feature_fraction = self.best_params[param_name] param_values = np.linspace( best_feature_fraction - 0.08, best_feature_fraction + 0.08, n_trials ).tolist() param_values = [val for val in param_values if val >= 0.4 and val <= 1.0] sampler = optuna.samplers.GridSampler({param_name: param_values}, seed=self._optuna_seed) self._tune_params([param_name], len(param_values), sampler, "feature_fraction_stage2") def tune_regularization_factors(self, n_trials: int = 20) -> None: self._tune_params( ["lambda_l1", "lambda_l2"], n_trials, optuna.samplers.TPESampler(seed=self._optuna_seed), "regularization_factors", ) def tune_min_data_in_leaf(self) -> None: param_name = "min_child_samples" param_values = [5, 10, 25, 50, 100] sampler = optuna.samplers.GridSampler({param_name: param_values}, seed=self._optuna_seed) self._tune_params([param_name], len(param_values), sampler, "min_child_samples") def _tune_params( self, target_param_names: list[str], n_trials: int, sampler: optuna.samplers.BaseSampler, step_name: str, ) -> _OptunaObjective: pbar = ( tqdm.tqdm(total=n_trials, ascii=True) if self.auto_options["show_progress_bar"] else None ) # Set current best parameters. self.lgbm_params.update(self.best_params) train_set = self.train_set if self.train_subset is not None: train_set = self.train_subset objective = self._create_objective(target_param_names, train_set, step_name, pbar) study = self._create_stepwise_study(self.study, step_name) study.sampler = sampler complete_trials = study.get_trials( deepcopy=True, states=(optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED), ) _n_trials = n_trials - len(complete_trials) if self._start_time is None: self._start_time = time.time() if self.auto_options["time_budget"] is not None: _timeout = self.auto_options["time_budget"] - (time.time() - self._start_time) else: _timeout = None if _n_trials > 0: study.optimize( objective, n_trials=_n_trials, timeout=_timeout, catch=(), callbacks=self._optuna_callbacks, ) if pbar: pbar.close() del pbar if objective.best_booster_with_trial_number is not None: self._best_booster_with_trial_number = objective.best_booster_with_trial_number return objective @abc.abstractmethod def _create_objective( self, target_param_names: list[str], train_set: "lgb.Dataset", step_name: str, pbar: tqdm.tqdm | None, ) -> _OptunaObjective: raise NotImplementedError def _create_stepwise_study( self, study: optuna.study.Study, step_name: str ) -> optuna.study.Study: # This class is assumed to be passed to a sampler and a pruner corresponding to the step. class _StepwiseStudy(optuna.study.Study): def __init__(self, study: optuna.study.Study, step_name: str) -> None: super().__init__( study_name=study.study_name, storage=study._storage, sampler=study.sampler, pruner=study.pruner, ) self._step_name = step_name def get_trials( self, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list[optuna.trial.FrozenTrial]: trials = super()._get_trials(deepcopy=deepcopy, states=states) return [t for t in trials if t.system_attrs.get(_STEP_NAME_KEY) == self._step_name] @property def best_trial(self) -> optuna.trial.FrozenTrial: """Return the best trial in the study. Returns: A :class:`~optuna.trial.FrozenTrial` object of the best trial. """ trials = self.get_trials(deepcopy=False) trials = [t for t in trials if t.state is optuna.trial.TrialState.COMPLETE] if len(trials) == 0: raise ValueError("No trials are completed yet.") if self.direction == optuna.study.StudyDirection.MINIMIZE: best_trial = min(trials, key=lambda t: cast(float, t.value)) else: best_trial = max(trials, key=lambda t: cast(float, t.value)) return copy.deepcopy(best_trial) return _StepwiseStudy(study, step_name) class LightGBMTuner(_LightGBMBaseTuner): """Hyperparameter tuner for LightGBM. It optimizes the following hyperparameters in a stepwise manner: ``lambda_l1``, ``lambda_l2``, ``num_leaves``, ``feature_fraction``, ``bagging_fraction``, ``bagging_freq`` and ``min_child_samples``. You can find the details of the algorithm and benchmark results in `this blog article `_ by `Kohei Ozaki `_, a Kaggle Grandmaster. .. note:: Arguments and keyword arguments for `lightgbm.train() `_ can be passed. For ``params``, please check `the official documentation for LightGBM `_. The arguments that only :class:`~optuna.integration.lightgbm.LightGBMTuner` has are listed below: Args: time_budget: A time budget for parameter tuning in seconds. study: A :class:`~optuna.study.Study` instance to store optimization results. The :class:`~optuna.trial.Trial` instances in it has the following user attributes: ``elapsed_secs`` is the elapsed time since the optimization starts. ``average_iteration_time`` is the average time of iteration to train the booster model in the trial. ``lgbm_params`` is a JSON-serialized dictionary of LightGBM parameters used in the trial. optuna_callbacks: List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. Please note that this is not a ``callbacks`` argument of `lightgbm.train()`_ . model_dir: A directory to save boosters. By default, it is set to :obj:`None` and no boosters are saved. Please set shared directory (e.g., directories on NFS) if you want to access :meth:`~optuna.integration.lightgbm.LightGBMTuner.get_best_booster` in distributed environments. Otherwise, it may raise :obj:`ValueError`. If the directory does not exist, it will be created. The filenames of the boosters will be ``{model_dir}/{trial_number}.pkl`` (e.g., ``./boosters/0.pkl``). verbosity: A verbosity level to change Optuna's logging level. The level is aligned to `LightGBM's verbosity`_ . .. warning:: Deprecated in v2.0.0. ``verbosity`` argument will be removed in the future. The removal of this feature is currently scheduled for v4.0.0, but this schedule is subject to change. Please use :func:`~optuna.logging.set_verbosity` instead. show_progress_bar: Flag to show progress bars or not. To disable progress bar, set this :obj:`False`. .. note:: Progress bars will be fragmented by logging messages of LightGBM and Optuna. Please suppress such messages to show the progress bars properly. optuna_seed: ``seed`` of :class:`~optuna.samplers.TPESampler` for random number generator that affects sampling for ``num_leaves``, ``bagging_fraction``, ``bagging_freq``, ``lambda_l1``, and ``lambda_l2``. .. note:: The `deterministic`_ parameter of LightGBM makes training reproducible. Please enable it when you use this argument. .. _lightgbm.train(): https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.train.html .. _LightGBM's verbosity: https://lightgbm.readthedocs.io/en/latest/Parameters.html#verbosity .. _deterministic: https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic """ def __init__( self, params: dict[str, Any], train_set: "lgb.Dataset", num_boost_round: int = 1000, valid_sets: list["lgb.Dataset"] | tuple["lgb.Dataset", ...] | "lgb.Dataset" | None = None, valid_names: Any | None = None, feval: Callable[..., Any] | None = None, feature_name: str = "auto", categorical_feature: str = "auto", keep_training_booster: bool = False, callbacks: list[Callable[..., Any]] | None = None, time_budget: int | None = None, sample_size: int | None = None, study: optuna.study.Study | None = None, optuna_callbacks: list[Callable[[Study, FrozenTrial], None]] | None = None, model_dir: str | None = None, verbosity: int | None = None, show_progress_bar: bool = True, *, optuna_seed: int | None = None, ) -> None: super().__init__( params, train_set, callbacks=callbacks, num_boost_round=num_boost_round, feval=feval, feature_name=feature_name, categorical_feature=categorical_feature, time_budget=time_budget, sample_size=sample_size, study=study, optuna_callbacks=optuna_callbacks, verbosity=verbosity, show_progress_bar=show_progress_bar, model_dir=model_dir, optuna_seed=optuna_seed, ) self.lgbm_kwargs["valid_sets"] = valid_sets self.lgbm_kwargs["valid_names"] = valid_names self.lgbm_kwargs["keep_training_booster"] = keep_training_booster self._best_booster_with_trial_number: tuple[lgb.Booster, int] | None = None if valid_sets is None: raise ValueError("`valid_sets` is required.") def _create_objective( self, target_param_names: list[str], train_set: "lgb.Dataset", step_name: str, pbar: tqdm.tqdm | None, ) -> _OptunaObjective: return _OptunaObjective( target_param_names, self.lgbm_params, train_set, self.lgbm_kwargs, self.best_score, step_name=step_name, model_dir=self._model_dir, pbar=pbar, ) def get_best_booster(self) -> "lgb.Booster": """Return the best booster. If the best booster cannot be found, :class:`ValueError` will be raised. To prevent the errors, please save boosters by specifying the ``model_dir`` argument of :meth:`~optuna.integration.lightgbm.LightGBMTuner.__init__`, when you resume tuning or you run tuning in parallel. """ if self._best_booster_with_trial_number is not None: if self._best_booster_with_trial_number[1] == self.study.best_trial.number: return self._best_booster_with_trial_number[0] if len(self.study.trials) == 0: raise ValueError("The best booster is not available because no trials completed.") # The best booster exists, but this instance does not have it. # This may be due to resuming or parallelization. if self._model_dir is None: raise ValueError( "The best booster cannot be found. It may be found in the other processes due to " "resuming or distributed computing. Please set the `model_dir` argument of " "`LightGBMTuner.__init__` and make sure that boosters are shared with all " "processes." ) best_trial = self.study.best_trial path = os.path.join(self._model_dir, f"{best_trial.number}.pkl") if not os.path.exists(path): raise ValueError( f"The best booster cannot be found in {self._model_dir}. If you execute " "`LightGBMTuner` in distributed environment, please use network file system " "(e.g., NFS) to share models with multiple workers." ) with open(path, "rb") as fin: booster = pickle.load(fin) return booster class LightGBMTunerCV(_LightGBMBaseTuner): """Hyperparameter tuner for LightGBM with cross-validation. It employs the same stepwise approach as :class:`~optuna.integration.lightgbm.LightGBMTuner`. :class:`~optuna.integration.lightgbm.LightGBMTunerCV` invokes `lightgbm.cv()`_ to train and validate boosters while :class:`~optuna.integration.lightgbm.LightGBMTuner` invokes `lightgbm.train()`_. See `a simple example `_ which optimizes the validation log loss of cancer detection. .. note:: Arguments and keyword arguments for `lightgbm.cv()`_ can be passed except ``metrics``, ``init_model`` and ``eval_train_metric``. For ``params``, please check `the official documentation for LightGBM `_. The arguments that only :class:`~optuna.integration.lightgbm.LightGBMTunerCV` has are listed below: Args: time_budget: A time budget for parameter tuning in seconds. study: A :class:`~optuna.study.Study` instance to store optimization results. The :class:`~optuna.trial.Trial` instances in it has the following user attributes: ``elapsed_secs`` is the elapsed time since the optimization starts. ``average_iteration_time`` is the average time of iteration to train the booster model in the trial. ``lgbm_params`` is a JSON-serialized dictionary of LightGBM parameters used in the trial. optuna_callbacks: List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. Please note that this is not a ``callbacks`` argument of `lightgbm.train()`_ . model_dir: A directory to save boosters. By default, it is set to :obj:`None` and no boosters are saved. Please set shared directory (e.g., directories on NFS) if you want to access :meth:`~optuna.integration.lightgbm.LightGBMTunerCV.get_best_booster` in distributed environments. Otherwise, it may raise :obj:`ValueError`. If the directory does not exist, it will be created. The filenames of the boosters will be ``{model_dir}/{trial_number}.pkl`` (e.g., ``./boosters/0.pkl``). verbosity: A verbosity level to change Optuna's logging level. The level is aligned to `LightGBM's verbosity`_ . .. warning:: Deprecated in v2.0.0. ``verbosity`` argument will be removed in the future. The removal of this feature is currently scheduled for v4.0.0, but this schedule is subject to change. Please use :func:`~optuna.logging.set_verbosity` instead. show_progress_bar: Flag to show progress bars or not. To disable progress bar, set this :obj:`False`. .. note:: Progress bars will be fragmented by logging messages of LightGBM and Optuna. Please suppress such messages to show the progress bars properly. return_cvbooster: Flag to enable :meth:`~optuna.integration.lightgbm.LightGBMTunerCV.get_best_booster`. optuna_seed: ``seed`` of :class:`~optuna.samplers.TPESampler` for random number generator that affects sampling for ``num_leaves``, ``bagging_fraction``, ``bagging_freq``, ``lambda_l1``, and ``lambda_l2``. .. note:: The `deterministic`_ parameter of LightGBM makes training reproducible. Please enable it when you use this argument. .. _lightgbm.train(): https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.train.html .. _lightgbm.cv(): https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.cv.html .. _LightGBM's verbosity: https://lightgbm.readthedocs.io/en/latest/Parameters.html#verbosity .. _deterministic: https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic """ def __init__( self, params: dict[str, Any], train_set: "lgb.Dataset", num_boost_round: int = 1000, folds: Generator[tuple[int, int], None, None] | Iterator[tuple[int, int]] | "BaseCrossValidator" | None = None, nfold: int = 5, stratified: bool = True, shuffle: bool = True, feval: Callable[..., Any] | None = None, feature_name: str = "auto", categorical_feature: str = "auto", fpreproc: Callable[..., Any] | None = None, seed: int = 0, callbacks: list[Callable[..., Any]] | None = None, time_budget: int | None = None, sample_size: int | None = None, study: optuna.study.Study | None = None, optuna_callbacks: list[Callable[[Study, FrozenTrial], None]] | None = None, verbosity: int | None = None, show_progress_bar: bool = True, model_dir: str | None = None, return_cvbooster: bool = False, *, optuna_seed: int | None = None, ) -> None: super().__init__( params, train_set, callbacks=callbacks, num_boost_round=num_boost_round, feval=feval, feature_name=feature_name, categorical_feature=categorical_feature, time_budget=time_budget, sample_size=sample_size, study=study, optuna_callbacks=optuna_callbacks, verbosity=verbosity, show_progress_bar=show_progress_bar, model_dir=model_dir, optuna_seed=optuna_seed, ) self.lgbm_kwargs["folds"] = folds self.lgbm_kwargs["nfold"] = nfold self.lgbm_kwargs["stratified"] = stratified self.lgbm_kwargs["shuffle"] = shuffle self.lgbm_kwargs["seed"] = seed self.lgbm_kwargs["fpreproc"] = fpreproc self.lgbm_kwargs["return_cvbooster"] = return_cvbooster def _create_objective( self, target_param_names: list[str], train_set: "lgb.Dataset", step_name: str, pbar: tqdm.tqdm | None, ) -> _OptunaObjective: return _OptunaObjectiveCV( target_param_names, self.lgbm_params, train_set, self.lgbm_kwargs, self.best_score, step_name=step_name, model_dir=self._model_dir, pbar=pbar, ) def get_best_booster(self) -> "lgb.CVBooster": """Return the best cvbooster. If the best booster cannot be found, :class:`ValueError` will be raised. To prevent the errors, please save boosters by specifying both of the ``model_dir`` and the ``return_cvbooster`` arguments of :meth:`~optuna.integration.lightgbm.LightGBMTunerCV.__init__`, when you resume tuning or you run tuning in parallel. """ if self.lgbm_kwargs.get("return_cvbooster") is not True: raise ValueError( "LightGBMTunerCV requires `return_cvbooster=True` for method `get_best_booster()`." ) if self._best_booster_with_trial_number is not None: if self._best_booster_with_trial_number[1] == self.study.best_trial.number: assert isinstance(self._best_booster_with_trial_number[0], lgb.CVBooster) return self._best_booster_with_trial_number[0] if len(self.study.trials) == 0: raise ValueError("The best booster is not available because no trials completed.") # The best booster exists, but this instance does not have it. # This may be due to resuming or parallelization. if self._model_dir is None: raise ValueError( "The best booster cannot be found. It may be found in the other processes due to " "resuming or distributed computing. Please set the `model_dir` argument of " "`LightGBMTunerCV.__init__` and make sure that boosters are shared with all " "processes." ) best_trial = self.study.best_trial path = os.path.join(self._model_dir, f"{best_trial.number}.pkl") if not os.path.exists(path): raise ValueError( f"The best booster cannot be found in {self._model_dir}. If you execute " "`LightGBMTunerCV` in distributed environment, please use network file system " "(e.g., NFS) to share models with multiple workers." ) with open(path, "rb") as fin: boosters, best_iteration = pickle.load(fin) # At version `lightgbm==3.0.0`, :class:`lightgbm.CVBooster` does not # have `__getstate__` which is required for pickle serialization. cvbooster = lgb.CVBooster() cvbooster.boosters = boosters cvbooster.best_iteration = best_iteration return cvbooster optuna-3.5.0/optuna/integration/_lightgbm_tuner/sklearn.py000066400000000000000000000023241453453102400240420ustar00rootroot00000000000000from typing import Any import warnings import lightgbm as lgb class LGBMModel(lgb.LGBMModel): """Proxy of lightgbm.LGBMModel. See: `pydoc lightgbm.LGBMModel` """ def __init__(self, *args: Any, **kwargs: Any) -> None: warnings.warn( "LightGBMTuner doesn't support sklearn API. " "Use `train()` or `LightGBMTuner` for hyperparameter tuning." ) super().__init__(*args, **kwargs) class LGBMClassifier(lgb.LGBMClassifier): """Proxy of lightgbm.LGBMClassifier. See: `pydoc lightgbm.LGBMClassifier` """ def __init__(self, *args: Any, **kwargs: Any) -> None: warnings.warn( "LightGBMTuner doesn't support sklearn API. " "Use `train()` or `LightGBMTuner` for hyperparameter tuning." ) super().__init__(*args, **kwargs) class LGBMRegressor(lgb.LGBMRegressor): """Proxy of LGBMRegressor. See: `pydoc lightgbm.LGBMRegressor` """ def __init__(self, *args: Any, **kwargs: Any) -> None: warnings.warn( "LightGBMTuner doesn't support sklearn API. " "Use `train()` or `LightGBMTuner` for hyperparameter tuning." ) super().__init__(*args, **kwargs) optuna-3.5.0/optuna/integration/allennlp/000077500000000000000000000000001453453102400204645ustar00rootroot00000000000000optuna-3.5.0/optuna/integration/allennlp/__init__.py000066400000000000000000000004461453453102400226010ustar00rootroot00000000000000from optuna_integration.allennlp._dump_best_config import dump_best_config from optuna_integration.allennlp._executor import AllenNLPExecutor from optuna_integration.allennlp._pruner import AllenNLPPruningCallback __all__ = ["dump_best_config", "AllenNLPExecutor", "AllenNLPPruningCallback"] optuna-3.5.0/optuna/integration/botorch.py000066400000000000000000001110441453453102400206720ustar00rootroot00000000000000from typing import Any from typing import Callable from typing import Dict from typing import Optional from typing import Sequence from typing import Union import warnings import numpy from packaging import version from optuna import logging from optuna._experimental import experimental_class from optuna._experimental import experimental_func from optuna._imports import try_import from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.samplers import BaseSampler from optuna.samplers import RandomSampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._base import _process_constraints_after_trial from optuna.search_space import IntersectionSearchSpace from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState with try_import() as _imports: from botorch.acquisition.monte_carlo import qExpectedImprovement from botorch.acquisition.monte_carlo import qNoisyExpectedImprovement from botorch.acquisition.multi_objective import monte_carlo from botorch.acquisition.multi_objective.analytic import ExpectedHypervolumeImprovement from botorch.acquisition.multi_objective.objective import IdentityMCMultiOutputObjective from botorch.acquisition.objective import ConstrainedMCObjective from botorch.acquisition.objective import GenericMCObjective from botorch.models import SingleTaskGP from botorch.models.transforms.outcome import Standardize from botorch.optim import optimize_acqf from botorch.sampling import SobolQMCNormalSampler import botorch.version if version.parse(botorch.version.version) < version.parse("0.8.0"): from botorch.fit import fit_gpytorch_model as fit_gpytorch_mll def _get_sobol_qmc_normal_sampler(num_samples: int) -> SobolQMCNormalSampler: return SobolQMCNormalSampler(num_samples) else: from botorch.fit import fit_gpytorch_mll def _get_sobol_qmc_normal_sampler(num_samples: int) -> SobolQMCNormalSampler: return SobolQMCNormalSampler(torch.Size((num_samples,))) from botorch.utils.multi_objective.box_decompositions import NondominatedPartitioning from botorch.utils.multi_objective.scalarization import get_chebyshev_scalarization from botorch.utils.sampling import manual_seed from botorch.utils.sampling import sample_simplex from botorch.utils.transforms import normalize from botorch.utils.transforms import unnormalize from gpytorch.mlls import ExactMarginalLogLikelihood import torch _logger = logging.get_logger(__name__) with try_import() as _imports_logei: from botorch.acquisition.analytic import LogConstrainedExpectedImprovement from botorch.acquisition.analytic import LogExpectedImprovement @experimental_func("3.3.0") def logei_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Log Expected Improvement (LogEI). The default value of ``candidates_func`` in :class:`~optuna.integration.BoTorchSampler` with single-objective optimization. Args: train_x: Previous parameter configurations. A ``torch.Tensor`` of shape ``(n_trials, n_params)``. ``n_trials`` is the number of already observed trials and ``n_params`` is the number of parameters. ``n_params`` may be larger than the actual number of parameters if categorical parameters are included in the search space, since these parameters are one-hot encoded. Values are not normalized. train_obj: Previously observed objectives. A ``torch.Tensor`` of shape ``(n_trials, n_objectives)``. ``n_trials`` is identical to that of ``train_x``. ``n_objectives`` is the number of objectives. Observations are not normalized. train_con: Objective constraints. A ``torch.Tensor`` of shape ``(n_trials, n_constraints)``. ``n_trials`` is identical to that of ``train_x``. ``n_constraints`` is the number of constraints. A constraint is violated if strictly larger than 0. If no constraints are involved in the optimization, this argument will be :obj:`None`. bounds: Search space bounds. A ``torch.Tensor`` of shape ``(2, n_params)``. ``n_params`` is identical to that of ``train_x``. The first and the second rows correspond to the lower and upper bounds for each parameter respectively. pending_x: Pending parameter configurations. A ``torch.Tensor`` of shape ``(n_pending, n_params)``. ``n_pending`` is the number of the trials which are already suggested all their parameters but have not completed their evaluation, and ``n_params`` is identical to that of ``train_x``. Returns: Next set of candidates. Usually the return value of BoTorch's ``optimize_acqf``. """ # We need botorch >=0.8.1 for LogExpectedImprovement. if not _imports_logei.is_successful(): raise ImportError( "logei_candidates_func requires botorch >=0.8.1. " "Please upgrade botorch or use qei_candidates_func as candidates_func instead." ) if train_obj.size(-1) != 1: raise ValueError("Objective may only contain single values with logEI.") n_constraints = train_con.size(1) if train_con is not None else 0 if n_constraints > 0: assert train_con is not None train_y = torch.cat([train_obj, train_con], dim=-1) is_feas = (train_con <= 0).all(dim=-1) train_obj_feas = train_obj[is_feas] if train_obj_feas.numel() == 0: _logger.warning( "No objective values are feasible. Using 0 as the best objective in logEI." ) best_f = train_obj.min() else: best_f = train_obj_feas.max() else: train_y = train_obj best_f = train_obj.max() train_x = normalize(train_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.size(-1))) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) if n_constraints > 0: acqf = LogConstrainedExpectedImprovement( model=model, best_f=best_f, objective_index=0, constraints={i: (None, 0.0) for i in range(1, n_constraints + 1)}, ) else: acqf = LogExpectedImprovement( model=model, best_f=best_f, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=10, raw_samples=512, options={"batch_limit": 5, "maxiter": 200}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates @experimental_func("2.4.0") def qei_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Quasi MC-based batch Expected Improvement (qEI). Args: train_x: Previous parameter configurations. A ``torch.Tensor`` of shape ``(n_trials, n_params)``. ``n_trials`` is the number of already observed trials and ``n_params`` is the number of parameters. ``n_params`` may be larger than the actual number of parameters if categorical parameters are included in the search space, since these parameters are one-hot encoded. Values are not normalized. train_obj: Previously observed objectives. A ``torch.Tensor`` of shape ``(n_trials, n_objectives)``. ``n_trials`` is identical to that of ``train_x``. ``n_objectives`` is the number of objectives. Observations are not normalized. train_con: Objective constraints. A ``torch.Tensor`` of shape ``(n_trials, n_constraints)``. ``n_trials`` is identical to that of ``train_x``. ``n_constraints`` is the number of constraints. A constraint is violated if strictly larger than 0. If no constraints are involved in the optimization, this argument will be :obj:`None`. bounds: Search space bounds. A ``torch.Tensor`` of shape ``(2, n_params)``. ``n_params`` is identical to that of ``train_x``. The first and the second rows correspond to the lower and upper bounds for each parameter respectively. pending_x: Pending parameter configurations. A ``torch.Tensor`` of shape ``(n_pending, n_params)``. ``n_pending`` is the number of the trials which are already suggested all their parameters but have not completed their evaluation, and ``n_params`` is identical to that of ``train_x``. Returns: Next set of candidates. Usually the return value of BoTorch's ``optimize_acqf``. """ if train_obj.size(-1) != 1: raise ValueError("Objective may only contain single values with qEI.") if train_con is not None: train_y = torch.cat([train_obj, train_con], dim=-1) is_feas = (train_con <= 0).all(dim=-1) train_obj_feas = train_obj[is_feas] if train_obj_feas.numel() == 0: # TODO(hvy): Do not use 0 as the best observation. _logger.warning( "No objective values are feasible. Using 0 as the best objective in qEI." ) best_f = torch.zeros(()) else: best_f = train_obj_feas.max() n_constraints = train_con.size(1) objective = ConstrainedMCObjective( objective=lambda Z: Z[..., 0], constraints=[ (lambda Z, i=i: Z[..., -n_constraints + i]) for i in range(n_constraints) ], ) else: train_y = train_obj best_f = train_obj.max() objective = None # Using the default identity objective. train_x = normalize(train_x, bounds=bounds) if pending_x is not None: pending_x = normalize(pending_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.size(-1))) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) acqf = qExpectedImprovement( model=model, best_f=best_f, sampler=_get_sobol_qmc_normal_sampler(256), objective=objective, X_pending=pending_x, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=10, raw_samples=512, options={"batch_limit": 5, "maxiter": 200}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates @experimental_func("3.3.0") def qnei_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Quasi MC-based batch Noisy Expected Improvement (qNEI). This function may perform better than qEI (`qei_candidates_func`) when the evaluated values of objective function are noisy. .. seealso:: :func:`~optuna.integration.botorch.qei_candidates_func` for argument and return value descriptions. """ if train_obj.size(-1) != 1: raise ValueError("Objective may only contain single values with qNEI.") if train_con is not None: train_y = torch.cat([train_obj, train_con], dim=-1) n_constraints = train_con.size(1) objective = ConstrainedMCObjective( objective=lambda Z: Z[..., 0], constraints=[ (lambda Z, i=i: Z[..., -n_constraints + i]) for i in range(n_constraints) ], ) else: train_y = train_obj objective = None # Using the default identity objective. train_x = normalize(train_x, bounds=bounds) if pending_x is not None: pending_x = normalize(pending_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.size(-1))) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) acqf = qNoisyExpectedImprovement( model=model, X_baseline=train_x, sampler=_get_sobol_qmc_normal_sampler(256), objective=objective, X_pending=pending_x, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=10, raw_samples=512, options={"batch_limit": 5, "maxiter": 200}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates @experimental_func("2.4.0") def qehvi_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Quasi MC-based batch Expected Hypervolume Improvement (qEHVI). The default value of ``candidates_func`` in :class:`~optuna.integration.BoTorchSampler` with multi-objective optimization when the number of objectives is three or less. .. seealso:: :func:`~optuna.integration.botorch.qei_candidates_func` for argument and return value descriptions. """ n_objectives = train_obj.size(-1) if train_con is not None: train_y = torch.cat([train_obj, train_con], dim=-1) is_feas = (train_con <= 0).all(dim=-1) train_obj_feas = train_obj[is_feas] n_constraints = train_con.size(1) additional_qehvi_kwargs = { "objective": IdentityMCMultiOutputObjective(outcomes=list(range(n_objectives))), "constraints": [ (lambda Z, i=i: Z[..., -n_constraints + i]) for i in range(n_constraints) ], } else: train_y = train_obj train_obj_feas = train_obj additional_qehvi_kwargs = {} train_x = normalize(train_x, bounds=bounds) if pending_x is not None: pending_x = normalize(pending_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.shape[-1])) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) # Approximate box decomposition similar to Ax when the number of objectives is large. # https://github.com/pytorch/botorch/blob/36d09a4297c2a0ff385077b7fcdd5a9d308e40cc/botorch/acquisition/multi_objective/utils.py#L46-L63 if n_objectives > 4: alpha = 10 ** (-8 + n_objectives) else: alpha = 0.0 ref_point = train_obj.min(dim=0).values - 1e-8 partitioning = NondominatedPartitioning(ref_point=ref_point, Y=train_obj_feas, alpha=alpha) ref_point_list = ref_point.tolist() acqf = monte_carlo.qExpectedHypervolumeImprovement( model=model, ref_point=ref_point_list, partitioning=partitioning, sampler=_get_sobol_qmc_normal_sampler(256), X_pending=pending_x, **additional_qehvi_kwargs, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=20, raw_samples=1024, options={"batch_limit": 5, "maxiter": 200, "nonnegative": True}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates @experimental_func("3.5.0") def ehvi_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Expected Hypervolume Improvement (EHVI). The default value of ``candidates_func`` in :class:`~optuna.integration.BoTorchSampler` with multi-objective optimization without constraints. .. seealso:: :func:`~optuna.integration.botorch.qei_candidates_func` for argument and return value descriptions. """ n_objectives = train_obj.size(-1) if train_con is not None: raise ValueError("Constraints are not supported with ehvi_candidates_func.") train_y = train_obj train_x = normalize(train_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.size(-1))) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) # Approximate box decomposition similar to Ax when the number of objectives is large. # https://github.com/pytorch/botorch/blob/36d09a4297c2a0ff385077b7fcdd5a9d308e40cc/botorch/acquisition/multi_objective/utils.py#L46-L63 if n_objectives > 4: alpha = 10 ** (-8 + n_objectives) else: alpha = 0.0 ref_point = train_obj.min(dim=0).values - 1e-8 partitioning = NondominatedPartitioning(ref_point=ref_point, Y=train_y, alpha=alpha) ref_point_list = ref_point.tolist() acqf = ExpectedHypervolumeImprovement( model=model, ref_point=ref_point_list, partitioning=partitioning, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=20, raw_samples=1024, options={"batch_limit": 5, "maxiter": 200}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates @experimental_func("3.1.0") def qnehvi_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Quasi MC-based batch Noisy Expected Hypervolume Improvement (qNEHVI). According to Botorch/Ax documentation, this function may perform better than qEHVI (`qehvi_candidates_func`). (cf. https://botorch.org/tutorials/constrained_multi_objective_bo ) .. seealso:: :func:`~optuna.integration.botorch.qei_candidates_func` for argument and return value descriptions. """ n_objectives = train_obj.size(-1) if train_con is not None: train_y = torch.cat([train_obj, train_con], dim=-1) n_constraints = train_con.size(1) additional_qnehvi_kwargs = { "objective": IdentityMCMultiOutputObjective(outcomes=list(range(n_objectives))), "constraints": [ (lambda Z, i=i: Z[..., -n_constraints + i]) for i in range(n_constraints) ], } else: train_y = train_obj additional_qnehvi_kwargs = {} train_x = normalize(train_x, bounds=bounds) if pending_x is not None: pending_x = normalize(pending_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.shape[-1])) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) # Approximate box decomposition similar to Ax when the number of objectives is large. # https://github.com/pytorch/botorch/blob/36d09a4297c2a0ff385077b7fcdd5a9d308e40cc/botorch/acquisition/multi_objective/utils.py#L46-L63 if n_objectives > 4: alpha = 10 ** (-8 + n_objectives) else: alpha = 0.0 ref_point = train_obj.min(dim=0).values - 1e-8 ref_point_list = ref_point.tolist() # prune_baseline=True is generally recommended by the documentation of BoTorch. # cf. https://botorch.org/api/acquisition.html (accessed on 2022/11/18) acqf = monte_carlo.qNoisyExpectedHypervolumeImprovement( model=model, ref_point=ref_point_list, X_baseline=train_x, alpha=alpha, prune_baseline=True, sampler=_get_sobol_qmc_normal_sampler(256), X_pending=pending_x, **additional_qnehvi_kwargs, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=20, raw_samples=1024, options={"batch_limit": 5, "maxiter": 200, "nonnegative": True}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates @experimental_func("2.4.0") def qparego_candidates_func( train_x: "torch.Tensor", train_obj: "torch.Tensor", train_con: Optional["torch.Tensor"], bounds: "torch.Tensor", pending_x: Optional["torch.Tensor"], ) -> "torch.Tensor": """Quasi MC-based extended ParEGO (qParEGO) for constrained multi-objective optimization. The default value of ``candidates_func`` in :class:`~optuna.integration.BoTorchSampler` with multi-objective optimization when the number of objectives is larger than three. .. seealso:: :func:`~optuna.integration.botorch.qei_candidates_func` for argument and return value descriptions. """ n_objectives = train_obj.size(-1) weights = sample_simplex(n_objectives).squeeze() scalarization = get_chebyshev_scalarization(weights=weights, Y=train_obj) if train_con is not None: train_y = torch.cat([train_obj, train_con], dim=-1) n_constraints = train_con.size(1) objective = ConstrainedMCObjective( objective=lambda Z: scalarization(Z[..., :n_objectives]), constraints=[ (lambda Z, i=i: Z[..., -n_constraints + i]) for i in range(n_constraints) ], ) else: train_y = train_obj objective = GenericMCObjective(scalarization) train_x = normalize(train_x, bounds=bounds) if pending_x is not None: pending_x = normalize(pending_x, bounds=bounds) model = SingleTaskGP(train_x, train_y, outcome_transform=Standardize(m=train_y.size(-1))) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_mll(mll) acqf = qExpectedImprovement( model=model, best_f=objective(train_y).max(), sampler=_get_sobol_qmc_normal_sampler(256), objective=objective, X_pending=pending_x, ) standard_bounds = torch.zeros_like(bounds) standard_bounds[1] = 1 candidates, _ = optimize_acqf( acq_function=acqf, bounds=standard_bounds, q=1, num_restarts=20, raw_samples=1024, options={"batch_limit": 5, "maxiter": 200}, sequential=True, ) candidates = unnormalize(candidates.detach(), bounds=bounds) return candidates def _get_default_candidates_func( n_objectives: int, has_constraint: bool, consider_running_trials: bool, ) -> Callable[ [ "torch.Tensor", "torch.Tensor", Optional["torch.Tensor"], "torch.Tensor", Optional["torch.Tensor"], ], "torch.Tensor", ]: if n_objectives > 3 and not has_constraint and not consider_running_trials: return ehvi_candidates_func elif n_objectives > 3: return qparego_candidates_func elif n_objectives > 1: return qehvi_candidates_func elif consider_running_trials: return qei_candidates_func else: return logei_candidates_func @experimental_class("2.4.0") class BoTorchSampler(BaseSampler): """A sampler that uses BoTorch, a Bayesian optimization library built on top of PyTorch. This sampler allows using BoTorch's optimization algorithms from Optuna to suggest parameter configurations. Parameters are transformed to continuous space and passed to BoTorch, and then transformed back to Optuna's representations. Categorical parameters are one-hot encoded. .. seealso:: See an `example `_ how to use the sampler. .. seealso:: See the `BoTorch `_ homepage for details and for how to implement your own ``candidates_func``. .. note:: An instance of this sampler *should not be used with different studies* when used with constraints. Instead, a new instance should be created for each new study. The reason for this is that the sampler is stateful keeping all the computed constraints. Args: candidates_func: An optional function that suggests the next candidates. It must take the training data, the objectives, the constraints, the search space bounds and return the next candidates. The arguments are of type ``torch.Tensor``. The return value must be a ``torch.Tensor``. However, if ``constraints_func`` is omitted, constraints will be :obj:`None`. For any constraints that failed to compute, the tensor will contain NaN. If omitted, it is determined automatically based on the number of objectives and whether a constraint is specified. If the number of objectives is one and no constraint is specified, log-Expected Improvement is used. If constraints are specified, quasi MC-based batch Expected Improvement (qEI) is used. If the number of objectives is either two or three, Quasi MC-based batch Expected Hypervolume Improvement (qEHVI) is used. Otherwise, for a larger number of objectives, analytic Expected Hypervolume Improvement is used if no constraints are specified, or the faster Quasi MC-based extended ParEGO (qParEGO) is used if constraints are present. The function should assume *maximization* of the objective. .. seealso:: See :func:`optuna.integration.botorch.qei_candidates_func` for an example. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraint is violated. A value equal to or smaller than 0 is considered feasible. If omitted, no constraints will be passed to ``candidates_func`` nor taken into account during suggestion. n_startup_trials: Number of initial trials, that is the number of trials to resort to independent sampling. consider_running_trials: If True, the acquisition function takes into consideration the running parameters whose evaluation has not completed. Enabling this option is considered to improve the performance of parallel optimization. .. note:: Added in v3.2.0 as an experimental argument. independent_sampler: An independent sampler to use for the initial trials and for parameters that are conditional. seed: Seed for random number generator. device: A ``torch.device`` to store input and output data of BoTorch. Please set a CUDA device if you fasten sampling. """ def __init__( self, *, candidates_func: Optional[ Callable[ [ "torch.Tensor", "torch.Tensor", Optional["torch.Tensor"], "torch.Tensor", Optional["torch.Tensor"], ], "torch.Tensor", ] ] = None, constraints_func: Optional[Callable[[FrozenTrial], Sequence[float]]] = None, n_startup_trials: int = 10, consider_running_trials: bool = False, independent_sampler: Optional[BaseSampler] = None, seed: Optional[int] = None, device: Optional["torch.device"] = None, ): _imports.check() self._candidates_func = candidates_func self._constraints_func = constraints_func self._consider_running_trials = consider_running_trials self._independent_sampler = independent_sampler or RandomSampler(seed=seed) self._n_startup_trials = n_startup_trials self._seed = seed self._study_id: Optional[int] = None self._search_space = IntersectionSearchSpace() self._device = device or torch.device("cpu") def infer_relative_search_space( self, study: Study, trial: FrozenTrial, ) -> Dict[str, BaseDistribution]: if self._study_id is None: self._study_id = study._study_id if self._study_id != study._study_id: # Note that the check below is meaningless when `InMemoryStorage` is used # because `InMemoryStorage.create_new_study` always returns the same study ID. raise RuntimeError("BoTorchSampler cannot handle multiple studies.") search_space: Dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # built-in `candidates_func` cannot handle distributions that contain just a # single value, so we skip them. Note that the parameter values for such # distributions are sampled in `Trial`. continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: assert isinstance(search_space, dict) if len(search_space) == 0: return {} completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) running_trials = [ t for t in study.get_trials(deepcopy=False, states=(TrialState.RUNNING,)) if t != trial ] trials = completed_trials + running_trials n_trials = len(trials) n_completed_trials = len(completed_trials) if n_trials < self._n_startup_trials: return {} trans = _SearchSpaceTransform(search_space) n_objectives = len(study.directions) values: Union[numpy.ndarray, torch.Tensor] = numpy.empty( (n_trials, n_objectives), dtype=numpy.float64 ) params: Union[numpy.ndarray, torch.Tensor] con: Optional[Union[numpy.ndarray, torch.Tensor]] = None bounds: Union[numpy.ndarray, torch.Tensor] = trans.bounds params = numpy.empty((n_trials, trans.bounds.shape[0]), dtype=numpy.float64) for trial_idx, trial in enumerate(trials): if trial.state == TrialState.COMPLETE: params[trial_idx] = trans.transform(trial.params) assert len(study.directions) == len(trial.values) for obj_idx, (direction, value) in enumerate(zip(study.directions, trial.values)): assert value is not None if ( direction == StudyDirection.MINIMIZE ): # BoTorch always assumes maximization. value *= -1 values[trial_idx, obj_idx] = value if self._constraints_func is not None: constraints = study._storage.get_trial_system_attrs(trial._trial_id).get( _CONSTRAINTS_KEY ) if constraints is not None: n_constraints = len(constraints) if con is None: con = numpy.full( (n_completed_trials, n_constraints), numpy.nan, dtype=numpy.float64 ) elif n_constraints != con.shape[1]: raise RuntimeError( f"Expected {con.shape[1]} constraints " f"but received {n_constraints}." ) con[trial_idx] = constraints elif trial.state == TrialState.RUNNING: if all(p in trial.params for p in search_space): params[trial_idx] = trans.transform(trial.params) else: params[trial_idx] = numpy.nan else: assert False, "trail.state must be TrialState.COMPLETE or TrialState.RUNNING." if self._constraints_func is not None: if con is None: warnings.warn( "`constraints_func` was given but no call to it correctly computed " "constraints. Constraints passed to `candidates_func` will be `None`." ) elif numpy.isnan(con).any(): warnings.warn( "`constraints_func` was given but some calls to it did not correctly compute " "constraints. Constraints passed to `candidates_func` will contain NaN." ) values = torch.from_numpy(values).to(self._device) params = torch.from_numpy(params).to(self._device) if con is not None: con = torch.from_numpy(con).to(self._device) bounds = torch.from_numpy(bounds).to(self._device) if con is not None: if con.dim() == 1: con.unsqueeze_(-1) bounds.transpose_(0, 1) if self._candidates_func is None: self._candidates_func = _get_default_candidates_func( n_objectives=n_objectives, has_constraint=con is not None, consider_running_trials=self._consider_running_trials, ) completed_values = values[:n_completed_trials] completed_params = params[:n_completed_trials] if self._consider_running_trials: running_params = params[n_completed_trials:] running_params = running_params[~torch.isnan(running_params).any(dim=1)] else: running_params = None with manual_seed(self._seed): # `manual_seed` makes the default candidates functions reproducible. # `SobolQMCNormalSampler`'s constructor has a `seed` argument, but its behavior is # deterministic when the BoTorch's seed is fixed. candidates = self._candidates_func( completed_params, completed_values, con, bounds, running_params ) if self._seed is not None: self._seed += 1 if not isinstance(candidates, torch.Tensor): raise TypeError("Candidates must be a torch.Tensor.") if candidates.dim() == 2: if candidates.size(0) != 1: raise ValueError( "Candidates batch optimization is not supported and the first dimension must " "have size 1 if candidates is a two-dimensional tensor. Actual: " f"{candidates.size()}." ) # Batch size is one. Get rid of the batch dimension. candidates = candidates.squeeze(0) if candidates.dim() != 1: raise ValueError("Candidates must be one or two-dimensional.") if candidates.size(0) != bounds.size(1): raise ValueError( "Candidates size must match with the given bounds. Actual candidates: " f"{candidates.size(0)}, bounds: {bounds.size(1)}." ) return trans.untransform(candidates.cpu().numpy()) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def reseed_rng(self) -> None: self._independent_sampler.reseed_rng() if self._seed is not None: self._seed = numpy.random.RandomState().randint(numpy.iinfo(numpy.int32).max) def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: if self._constraints_func is not None: _process_constraints_after_trial(self._constraints_func, study, trial, state) self._independent_sampler.after_trial(study, trial, state, values) optuna-3.5.0/optuna/integration/catalyst.py000066400000000000000000000001501453453102400210510ustar00rootroot00000000000000from optuna_integration.catalyst import CatalystPruningCallback __all__ = ["CatalystPruningCallback"] optuna-3.5.0/optuna/integration/catboost.py000066400000000000000000000106761453453102400210610ustar00rootroot00000000000000from typing import Any from typing import Optional from packaging import version import optuna from optuna._experimental import experimental_class from optuna._imports import try_import with try_import() as _imports: import catboost as cb if version.parse(cb.__version__) < version.parse("0.26"): raise ImportError(f"You don't have CatBoost>=0.26! CatBoost version: {cb.__version__}") @experimental_class("3.0.0") class CatBoostPruningCallback: """Callback for catboost to prune unpromising trials. See `the example `__ if you want to add a pruning callback which observes validation accuracy of a CatBoost model. .. note:: :class:`optuna.TrialPruned` cannot be raised in :meth:`~optuna.integration.CatBoostPruningCallback.after_iteration` that is called in CatBoost via ``CatBoostPruningCallback``. You must call :meth:`~optuna.integration.CatBoostPruningCallback.check_pruned` after training manually unlike other pruning callbacks to raise :class:`optuna.TrialPruned`. .. note:: This callback cannot be used with CatBoost on GPUs because CatBoost doesn't support a user-defined callback for GPU. Please refer to `CatBoost issue `_. Args: trial: A :class:`~optuna.trial.Trial` corresponding to the current evaluation of the objective function. metric: An evaluation metric for pruning, e.g., ``Logloss`` and ``AUC``. Please refer to `CatBoost reference `_ for further details. eval_set_index: The index of the target validation dataset. If you set only one ``eval_set``, ``eval_set_index`` is None. If you set multiple datasets as ``eval_set``, the index of ``eval_set`` must be ``eval_set_index``, e.g., ``0`` or ``1`` when ``eval_set`` contains two datasets. """ def __init__( self, trial: optuna.trial.Trial, metric: str, eval_set_index: Optional[int] = None ) -> None: default_valid_name = "validation" self._trial = trial self._metric = metric if eval_set_index is None: self._valid_name = default_valid_name else: self._valid_name = default_valid_name + "_" + str(eval_set_index) self._pruned = False self._message = "" def after_iteration(self, info: Any) -> bool: """Report an evaluation metric value for Optuna pruning after each CatBoost's iteration. This method is called by CatBoost. Args: info: A ``SimpleNamespace`` containing iteraion, ``validation_name``, ``metric_name`` and history of losses. For example ``SimpleNamespace(iteration=2, metrics={ 'learn': {'Logloss': [0.6, 0.5]}, 'validation': {'Logloss': [0.7, 0.6], 'AUC': [0.8, 0.9]} })``. Returns: A boolean value. If :obj:`False`, CatBoost internally stops the optimization with Optuna's pruning logic without raising :class:`optuna.TrialPruned`. Otherwise, the optimization continues. """ step = info.iteration - 1 if self._valid_name not in info.metrics: raise ValueError( 'The entry associated with the validation name "{}" ' "is not found in the evaluation result list {}.".format(self._valid_name, info) ) metrics = info.metrics[self._valid_name] if self._metric not in metrics: raise ValueError( 'The entry associated with the metric name "{}" ' "is not found in the evaluation result list {}.".format(self._metric, info) ) current_score = metrics[self._metric][-1] self._trial.report(current_score, step=step) if self._trial.should_prune(): self._message = "Trial was pruned at iteration {}.".format(step) self._pruned = True return False return True def check_pruned(self) -> None: """Raise :class:`optuna.TrialPruned` manually if the CatBoost optimization is pruned.""" if self._pruned: raise optuna.TrialPruned(self._message) optuna-3.5.0/optuna/integration/chainer.py000066400000000000000000000001471453453102400206440ustar00rootroot00000000000000from optuna_integration.chainer import ChainerPruningExtension __all__ = ["ChainerPruningExtension"] optuna-3.5.0/optuna/integration/chainermn.py000066400000000000000000000001271453453102400211750ustar00rootroot00000000000000from optuna_integration.chainermn import ChainerMNStudy __all__ = ["ChainerMNStudy"] optuna-3.5.0/optuna/integration/cma.py000066400000000000000000000503461453453102400200010ustar00rootroot00000000000000import math import random from typing import Any from typing import Container from typing import Dict from typing import List from typing import Optional from typing import Sequence import numpy import optuna from optuna import distributions from optuna import logging from optuna._deprecated import deprecated_class from optuna._imports import try_import from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.search_space import IntersectionSearchSpace from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState with try_import() as _imports: import cma _logger = logging.get_logger(__name__) _EPS = 1e-10 _cma_deprecated_msg = "This class is renamed to :class:`~optuna.integration.PyCmaSampler`." class PyCmaSampler(BaseSampler): """A Sampler using cma library as the backend. Example: Optimize a simple quadratic function by using :class:`~optuna.integration.PyCmaSampler`. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y sampler = optuna.integration.PyCmaSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=20) Note that parallel execution of trials may affect the optimization performance of CMA-ES, especially if the number of trials running in parallel exceeds the population size. .. note:: :class:`~optuna.integration.CmaEsSampler` is deprecated and renamed to :class:`~optuna.integration.PyCmaSampler` in v2.0.0. Please use :class:`~optuna.integration.PyCmaSampler` instead of :class:`~optuna.integration.CmaEsSampler`. Args: x0: A dictionary of an initial parameter values for CMA-ES. By default, the mean of ``low`` and ``high`` for each distribution is used. Please refer to cma.CMAEvolutionStrategy_ for further details of ``x0``. sigma0: Initial standard deviation of CMA-ES. By default, ``sigma0`` is set to ``min_range / 6``, where ``min_range`` denotes the minimum range of the distributions in the search space. If distribution is categorical, ``min_range`` is ``len(choices) - 1``. Please refer to cma.CMAEvolutionStrategy_ for further details of ``sigma0``. cma_stds: A dictionary of multipliers of sigma0 for each parameters. The default value is 1.0. Please refer to cma.CMAEvolutionStrategy_ for further details of ``cma_stds``. seed: A random seed for CMA-ES. cma_opts: Options passed to the constructor of cma.CMAEvolutionStrategy_ class. Note that default option is cma_default_options_, but ``BoundaryHandler``, ``bounds``, ``CMA_stds`` and ``seed`` arguments in ``cma_opts`` will be ignored because it is added by :class:`~optuna.integration.PyCmaSampler` automatically. n_startup_trials: The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study. independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for :class:`~optuna.integration.PyCmaSampler` is determined by :func:`~optuna.search_space.intersection_search_space()`. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. .. _cma.CMAEvolutionStrategy: https://cma-es.github.io/apidocs-pycma/\ cma.evolution_strategy.CMAEvolutionStrategy.html .. _cma_default_options: https://cma-es.github.io/apidocs-pycma/\ cma.evolution_strategy.html#cma_default_options_ """ def __init__( self, x0: Optional[Dict[str, Any]] = None, sigma0: Optional[float] = None, cma_stds: Optional[Dict[str, float]] = None, seed: Optional[int] = None, cma_opts: Optional[Dict[str, Any]] = None, n_startup_trials: int = 1, independent_sampler: Optional[BaseSampler] = None, warn_independent_sampling: bool = True, ) -> None: _imports.check() self._x0 = x0 self._sigma0 = sigma0 self._cma_stds = cma_stds if seed is None: seed = random.randint(1, 2**32) self._cma_opts = cma_opts or {} self._cma_opts["seed"] = seed self._cma_opts.setdefault("verbose", -2) self._n_startup_trials = n_startup_trials self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) self._warn_independent_sampling = warn_independent_sampling self._search_space = IntersectionSearchSpace() def reseed_rng(self) -> None: self._cma_opts["seed"] = random.randint(1, 2**32) self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: search_space = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # `cma` cannot handle distributions that contain just a single value, so we skip # them. Note that the parameter values for such distributions are sampled in # `Trial`. continue search_space[name] = distribution return search_space def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> float: self._raise_error_if_multi_objective(study) if self._warn_independent_sampling: complete_trials = study._get_trials( deepcopy=False, states=(TrialState.COMPLETE,), use_cache=True ) if len(complete_trials) >= self._n_startup_trials: self._log_independent_sampling(trial, param_name) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, float]: self._raise_error_if_multi_objective(study) if len(search_space) == 0: return {} if len(search_space) == 1: _logger.info( "`PyCmaSampler` does not support optimization of 1-D search space. " "`{}` is used instead of `PyCmaSampler`.".format( self._independent_sampler.__class__.__name__ ) ) self._warn_independent_sampling = False return {} complete_trials = study._get_trials( deepcopy=False, states=(TrialState.COMPLETE,), use_cache=True ) if len(complete_trials) < self._n_startup_trials: return {} if self._x0 is None: self._x0 = self._initialize_x0(search_space) if self._sigma0 is None: sigma0 = self._initialize_sigma0(search_space) else: sigma0 = self._sigma0 # Avoid ZeroDivisionError in cma.CMAEvolutionStrategy. sigma0 = max(sigma0, _EPS) optimizer = _Optimizer(search_space, self._x0, sigma0, self._cma_stds, self._cma_opts) trials = study.trials last_told_trial_number = optimizer.tell(trials, study.direction) return optimizer.ask(trials, last_told_trial_number) @staticmethod def _initialize_x0(search_space: Dict[str, BaseDistribution]) -> Dict[str, Any]: x0: Dict[str, Any] = {} for name, distribution in search_space.items(): if isinstance(distribution, FloatDistribution): if distribution.log: log_high = math.log(distribution.high) log_low = math.log(distribution.low) x0[name] = math.exp(numpy.mean([log_high, log_low])) else: x0[name] = numpy.mean([distribution.high, distribution.low]) elif isinstance(distribution, CategoricalDistribution): index = (len(distribution.choices) - 1) // 2 x0[name] = distribution.choices[index] elif isinstance(distribution, IntDistribution): if distribution.log: log_high = math.log(distribution.high) log_low = math.log(distribution.low) x0[name] = math.exp(numpy.mean([log_high, log_low])) else: x0[name] = int(numpy.mean([distribution.high, distribution.low])) else: raise NotImplementedError( "The distribution {} is not implemented.".format(distribution) ) return x0 @staticmethod def _initialize_sigma0(search_space: Dict[str, BaseDistribution]) -> float: sigma0s = [] for name, distribution in search_space.items(): if isinstance(distribution, (IntDistribution, FloatDistribution)): if distribution.log: log_high = math.log(distribution.high) log_low = math.log(distribution.low) sigma0s.append((log_high - log_low) / 6) else: sigma0s.append((distribution.high - distribution.low) / 6) elif isinstance(distribution, CategoricalDistribution): sigma0s.append((len(distribution.choices) - 1) / 6) else: raise NotImplementedError( "The distribution {} is not implemented.".format(distribution) ) return min(sigma0s) def _log_independent_sampling(self, trial: FrozenTrial, param_name: str) -> None: _logger.warning( "The parameter '{}' in trial#{} is sampled independently " "by using `{}` instead of `PyCmaSampler` " "(optimization performance may be degraded). " "`PyCmaSampler` does not support dynamic search space or `CategoricalDistribution`. " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `PyCmaSampler`, " "if this independent sampling is intended behavior.".format( param_name, trial.number, self._independent_sampler.__class__.__name__ ) ) def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: self._independent_sampler.after_trial(study, trial, state, values) class _Optimizer: def __init__( self, search_space: Dict[str, BaseDistribution], x0: Dict[str, Any], sigma0: float, cma_stds: Optional[Dict[str, float]], cma_opts: Dict[str, Any], ) -> None: self._search_space = search_space self._param_names = list(sorted(self._search_space.keys())) lows = [] highs = [] for param_name in self._param_names: dist = self._search_space[param_name] if isinstance(dist, CategoricalDistribution): # Handle categorical values by ordinal representation. # TODO(Yanase): Support one-hot representation. lows.append(-0.5) highs.append(len(dist.choices) - 0.5) elif isinstance(dist, FloatDistribution): if dist.step is not None: r = dist.high - dist.low lows.append(0 - 0.5 * dist.step) highs.append(r + 0.5 * dist.step) else: lows.append(self._to_cma_params(search_space, param_name, dist.low)) highs.append(self._to_cma_params(search_space, param_name, dist.high) - _EPS) elif isinstance(dist, IntDistribution): if dist.log: lows.append(self._to_cma_params(search_space, param_name, dist.low - 0.5)) highs.append(self._to_cma_params(search_space, param_name, dist.high + 0.5)) else: lows.append(dist.low - 0.5 * dist.step) highs.append(dist.high + 0.5 * dist.step) else: raise NotImplementedError("The distribution {} is not implemented.".format(dist)) # Set initial params. initial_cma_params = [] for param_name in self._param_names: initial_cma_params.append( self._to_cma_params(self._search_space, param_name, x0[param_name]) ) cma_option = { "BoundaryHandler": cma.BoundTransform, "bounds": [lows, highs], } if cma_stds: cma_option["CMA_stds"] = [cma_stds.get(name, 1.0) for name in self._param_names] cma_opts.update(cma_option) self._es = cma.CMAEvolutionStrategy(initial_cma_params, sigma0, cma_opts) def tell(self, trials: List[FrozenTrial], study_direction: StudyDirection) -> int: complete_trials = self._collect_target_trials(trials, target_states={TrialState.COMPLETE}) popsize = self._es.popsize generation = len(complete_trials) // popsize last_told_trial_number = -1 for i in range(generation): xs = [] ys = [] for t in complete_trials[i * popsize : (i + 1) * popsize]: x = [ self._to_cma_params(self._search_space, name, t.params[name]) for name in self._param_names ] xs.append(x) ys.append(t.value) last_told_trial_number = t.number if study_direction == StudyDirection.MAXIMIZE: ys = [-1 * y if y is not None else y for y in ys] # Calling `ask` is required to avoid RuntimeError which claims that `tell` should only # be called once per iteration. self._es.ask() self._es.tell(xs, ys) return last_told_trial_number def ask(self, trials: List[FrozenTrial], last_told_trial_number: int) -> Dict[str, Any]: individual_index = len(self._collect_target_trials(trials, last_told_trial_number)) popsize = self._es.popsize # individual_index may exceed the population size due to the parallel execution of multiple # trials. In such cases, `cma.cma.CMAEvolutionStrategy.ask` is called multiple times in an # iteration, and that may affect the optimization performance of CMA-ES. # In addition, please note that some trials may suggest the same parameters when multiple # samplers invoke this method simultaneously. while individual_index >= popsize: individual_index -= popsize self._es.ask() cma_params = self._es.ask()[individual_index] ret_val = {} for param_name, value in zip(self._param_names, cma_params): ret_val[param_name] = self._to_optuna_params(self._search_space, param_name, value) return ret_val def _is_compatible(self, trial: FrozenTrial) -> bool: # Thanks to `intersection_search_space()` function, in sequential optimization, # the parameters of complete trials are always compatible with the search space. # # However, in distributed optimization, incompatible trials may complete on a worker # just after an intersection search space is calculated on another worker. for name, distribution in self._search_space.items(): if name not in trial.params: return False distributions.check_distribution_compatibility(distribution, trial.distributions[name]) param_value = trial.params[name] param_internal_value = distribution.to_internal_repr(param_value) if not distribution._contains(param_internal_value): return False return True def _collect_target_trials( self, trials: List[FrozenTrial], last_told: int = -1, target_states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: target_trials = [t for t in trials if t.number > last_told] target_trials = [t for t in target_trials if self._is_compatible(t)] if target_states is not None: target_trials = [t for t in target_trials if t.state in target_states] return target_trials @staticmethod def _to_cma_params( search_space: Dict[str, BaseDistribution], param_name: str, optuna_param_value: Any ) -> float: dist = search_space[param_name] if isinstance(dist, IntDistribution): if dist.log: return math.log(optuna_param_value) elif isinstance(dist, FloatDistribution): if dist.log: return math.log(optuna_param_value) elif dist.step is not None: return optuna_param_value - dist.low elif isinstance(dist, CategoricalDistribution): return dist.choices.index(optuna_param_value) return optuna_param_value @staticmethod def _to_optuna_params( search_space: Dict[str, BaseDistribution], param_name: str, cma_param_value: float ) -> Any: dist = search_space[param_name] if isinstance(dist, FloatDistribution): if dist.log: return math.exp(cma_param_value) elif dist.step is not None: v = numpy.round(cma_param_value / dist.step) * dist.step + dist.low return float(min(max(v, dist.low), dist.high)) else: return float(cma_param_value) elif isinstance(dist, IntDistribution): if dist.log: exp_value = math.exp(cma_param_value) v = numpy.round(exp_value) return int(min(max(v, dist.low), dist.high)) else: r = numpy.round((cma_param_value - dist.low) / dist.step) v = r * dist.step + dist.low return int(v) elif isinstance(dist, CategoricalDistribution): v = int(numpy.round(cma_param_value)) return dist.choices[v] return cma_param_value @deprecated_class("2.0.0", "4.0.0", text=_cma_deprecated_msg) class CmaEsSampler(PyCmaSampler): """Wrapper class of PyCmaSampler for backward compatibility.""" def __init__( self, x0: Optional[Dict[str, Any]] = None, sigma0: Optional[float] = None, cma_stds: Optional[Dict[str, float]] = None, seed: Optional[int] = None, cma_opts: Optional[Dict[str, Any]] = None, n_startup_trials: int = 1, independent_sampler: Optional[BaseSampler] = None, warn_independent_sampling: bool = True, ) -> None: super().__init__( x0=x0, sigma0=sigma0, cma_stds=cma_stds, seed=seed, cma_opts=cma_opts, n_startup_trials=n_startup_trials, independent_sampler=independent_sampler, warn_independent_sampling=warn_independent_sampling, ) optuna-3.5.0/optuna/integration/dask.py000066400000000000000000000655111453453102400201630ustar00rootroot00000000000000import asyncio from datetime import datetime from typing import Any from typing import Container from typing import Dict from typing import Generator from typing import Iterable from typing import List from typing import Optional from typing import Sequence from typing import Tuple from typing import Union import uuid import optuna from optuna._experimental import experimental_class from optuna._imports import try_import from optuna._typing import JSONSerializable from optuna.distributions import BaseDistribution from optuna.distributions import distribution_to_json from optuna.distributions import json_to_distribution from optuna.storages import BaseStorage from optuna.study import StudyDirection from optuna.study._frozen import FrozenStudy from optuna.trial import FrozenTrial from optuna.trial import TrialState with try_import() as _imports: import distributed from distributed.protocol.pickle import dumps from distributed.protocol.pickle import loads from distributed.utils import thread_state # type: ignore[attr-defined] from distributed.worker import get_client def _serialize_frozentrial(trial: FrozenTrial) -> dict: data = trial.__dict__.copy() data["state"] = data["state"].name attrs = [a for a in data.keys() if a.startswith("_")] for attr in attrs: data[attr[1:]] = data.pop(attr) data["system_attrs"] = ( dumps(data["system_attrs"]) # type: ignore[no-untyped-call] if data["system_attrs"] else {} ) data["user_attrs"] = ( dumps(data["user_attrs"]) if data["user_attrs"] else {} # type: ignore[no-untyped-call] ) data["distributions"] = {k: distribution_to_json(v) for k, v in data["distributions"].items()} if data["datetime_start"] is not None: data["datetime_start"] = data["datetime_start"].isoformat(timespec="microseconds") if data["datetime_complete"] is not None: data["datetime_complete"] = data["datetime_complete"].isoformat(timespec="microseconds") data["value"] = None return data def _deserialize_frozentrial(data: dict) -> FrozenTrial: data["state"] = TrialState[data["state"]] data["distributions"] = {k: json_to_distribution(v) for k, v in data["distributions"].items()} if data["datetime_start"] is not None: data["datetime_start"] = datetime.fromisoformat(data["datetime_start"]) if data["datetime_complete"] is not None: data["datetime_complete"] = datetime.fromisoformat(data["datetime_complete"]) data["system_attrs"] = ( loads(data["system_attrs"]) # type: ignore[no-untyped-call] if data["system_attrs"] else {} ) data["user_attrs"] = ( loads(data["user_attrs"]) if data["user_attrs"] else {} # type: ignore[no-untyped-call] ) return FrozenTrial(**data) def _serialize_frozenstudy(study: FrozenStudy) -> dict: data = { "directions": [d.name for d in study._directions], "study_id": study._study_id, "study_name": study.study_name, "user_attrs": dumps(study.user_attrs) # type: ignore[no-untyped-call] if study.user_attrs else {}, "system_attrs": dumps(study.system_attrs) # type: ignore[no-untyped-call] if study.system_attrs else {}, } return data def _deserialize_frozenstudy(data: dict) -> FrozenStudy: data["directions"] = [StudyDirection[d] for d in data["directions"]] data["direction"] = None data["system_attrs"] = ( loads(data["system_attrs"]) # type: ignore[no-untyped-call] if data["system_attrs"] else {} ) data["user_attrs"] = ( loads(data["user_attrs"]) if data["user_attrs"] else {} # type: ignore[no-untyped-call] ) return FrozenStudy(**data) class _OptunaSchedulerExtension: def __init__(self, scheduler: "distributed.Scheduler"): self.scheduler = scheduler self.storages: Dict[str, BaseStorage] = {} methods = [ "create_new_study", "delete_study", "set_study_user_attr", "set_study_system_attr", "get_study_id_from_name", "get_study_name_from_id", "get_study_directions", "get_study_user_attrs", "get_study_system_attrs", "get_all_studies", "create_new_trial", "set_trial_param", "get_trial_id_from_study_id_trial_number", "get_trial_number_from_id", "get_trial_param", "set_trial_state_values", "set_trial_intermediate_value", "set_trial_user_attr", "set_trial_system_attr", "get_trial", "get_all_trials", "get_n_trials", ] handlers = {f"optuna_{method}": getattr(self, method) for method in methods} self.scheduler.handlers.update(handlers) self.scheduler.extensions["optuna"] = self def get_storage(self, name: str) -> BaseStorage: return self.storages[name] def create_new_study( self, comm: "distributed.comm.tcp.TCP", storage_name: str, directions: List[str], study_name: Optional[str] = None, ) -> int: return self.get_storage(storage_name).create_new_study( directions=[StudyDirection[direction] for direction in directions], study_name=study_name, ) def delete_study( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, ) -> None: return self.get_storage(storage_name).delete_study(study_id=study_id) def set_study_user_attr( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, key: str, value: Any, ) -> None: return self.get_storage(storage_name).set_study_user_attr( study_id=study_id, key=key, value=loads(value) # type: ignore[no-untyped-call] ) def set_study_system_attr( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, key: str, value: Any, ) -> None: return self.get_storage(storage_name).set_study_system_attr( study_id=study_id, key=key, value=loads(value), # type: ignore[no-untyped-call] ) def get_study_id_from_name( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_name: str, ) -> int: return self.get_storage(storage_name).get_study_id_from_name(study_name=study_name) def get_study_name_from_id( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, ) -> str: return self.get_storage(storage_name).get_study_name_from_id(study_id=study_id) def get_study_directions( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, ) -> List[str]: directions = self.get_storage(storage_name).get_study_directions(study_id=study_id) return [direction.name for direction in directions] def get_study_user_attrs( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, ) -> Dict[str, Any]: return dumps( self.get_storage(storage_name).get_study_user_attrs( # type: ignore[no-untyped-call] study_id=study_id ) ) def get_study_system_attrs( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, ) -> Dict[str, Any]: return dumps( self.get_storage(storage_name).get_study_system_attrs( # type: ignore[no-untyped-call] study_id=study_id ) ) def get_all_studies(self, comm: "distributed.comm.tcp.TCP", storage_name: str) -> List[dict]: studies = self.get_storage(storage_name).get_all_studies() return [_serialize_frozenstudy(s) for s in studies] def create_new_trial( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, template_trial: Optional[dict] = None, ) -> int: deserialized_template_trial = None if template_trial is not None: deserialized_template_trial = _deserialize_frozentrial(template_trial) return self.get_storage(storage_name).create_new_trial( study_id=study_id, template_trial=deserialized_template_trial, ) def set_trial_param( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, param_name: str, param_value_internal: float, distribution: str, ) -> None: return self.get_storage(storage_name).set_trial_param( trial_id=trial_id, param_name=param_name, param_value_internal=param_value_internal, distribution=json_to_distribution(distribution), ) def get_trial_id_from_study_id_trial_number( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, trial_number: int ) -> int: return self.get_storage(storage_name).get_trial_id_from_study_id_trial_number( study_id=study_id, trial_number=trial_number, ) def get_trial_number_from_id( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, ) -> int: return self.get_storage(storage_name).get_trial_number_from_id(trial_id=trial_id) def get_trial_param( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, param_name: str, ) -> float: return self.get_storage(storage_name).get_trial_param( trial_id=trial_id, param_name=param_name, ) def set_trial_state_values( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, state: str, values: Optional[Sequence[float]] = None, ) -> bool: return self.get_storage(storage_name).set_trial_state_values( trial_id=trial_id, state=TrialState[state], values=values, ) def set_trial_intermediate_value( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, step: int, intermediate_value: float, ) -> None: return self.get_storage(storage_name).set_trial_intermediate_value( trial_id=trial_id, step=step, intermediate_value=intermediate_value, ) def set_trial_user_attr( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, key: str, value: Any, ) -> None: return self.get_storage(storage_name).set_trial_user_attr( trial_id=trial_id, key=key, value=loads(value), # type: ignore[no-untyped-call] ) def set_trial_system_attr( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, key: str, value: JSONSerializable, ) -> None: return self.get_storage(storage_name).set_trial_system_attr( trial_id=trial_id, key=key, value=loads(value), # type: ignore[no-untyped-call] ) def get_trial( self, comm: "distributed.comm.tcp.TCP", storage_name: str, trial_id: int, ) -> dict: trial = self.get_storage(storage_name).get_trial(trial_id=trial_id) return _serialize_frozentrial(trial) def get_all_trials( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, deepcopy: bool = True, states: Optional[Tuple[str, ...]] = None, ) -> List[dict]: deserialized_states = None if states is not None: deserialized_states = tuple(TrialState[s] for s in states) trials = self.get_storage(storage_name).get_all_trials( study_id=study_id, deepcopy=deepcopy, states=deserialized_states, ) return [_serialize_frozentrial(t) for t in trials] def get_n_trials( self, comm: "distributed.comm.tcp.TCP", storage_name: str, study_id: int, state: Optional[Union[Tuple[str, ...], str]] = None, ) -> int: deserialized_state: Optional[Union[Tuple[TrialState, ...], TrialState]] = None if state is not None: if isinstance(state, str): deserialized_state = TrialState[state] else: deserialized_state = tuple(TrialState[s] for s in state) return self.get_storage(storage_name).get_n_trials( study_id=study_id, state=deserialized_state, ) def _register_with_scheduler( dask_scheduler: "distributed.Scheduler", storage: Union[None, str, BaseStorage], name: str ) -> None: if "optuna" not in dask_scheduler.extensions: ext = _OptunaSchedulerExtension(dask_scheduler) else: ext = dask_scheduler.extensions["optuna"] if name not in ext.storages: ext.storages[name] = optuna.storages.get_storage(storage) @experimental_class("3.1.0") class DaskStorage(BaseStorage): """Dask-compatible storage class. This storage class wraps a Optuna storage class (e.g. Optuna’s in-memory or sqlite storage) and is used to run optimization trials in parallel on a Dask cluster. The underlying Optuna storage object lives on the cluster’s scheduler and any method calls on the :obj:`DaskStorage` instance results in the same method being called on the underlying Optuna storage object. See `this example `_ or the following YouTube video for how to use :obj:`DaskStorage` to extend Optuna's in-memory storage class to run across multiple processes. .. raw:: html

Args: storage: Optuna storage url to use for underlying Optuna storage class to wrap (e.g. :obj:`None` for in-memory storage, ``sqlite:///example.db`` for SQLite storage). Defaults to :obj:`None`. name: Unique identifier for the Dask storage class. Specifying a custom name can sometimes be useful for logging or debugging. If :obj:`None` is provided, a random name will be automatically generated. client: Dask ``Client`` to connect to. If not provided, will attempt to find an existing ``Client``. register: Whether or not to register this storage instance with the cluster scheduler. Most common usage of this storage class will not need to specify this argument. Defaults to ``True``. """ def __init__( self, storage: Union[None, str, BaseStorage] = None, name: Optional[str] = None, client: Optional["distributed.Client"] = None, register: bool = True, ): _imports.check() self.name = name or f"dask-storage-{uuid.uuid4().hex}" self._client = client if register: if self.client.asynchronous or getattr(thread_state, "on_event_loop_thread", False): async def _register() -> DaskStorage: await self.client.run_on_scheduler( # type: ignore[no-untyped-call] _register_with_scheduler, storage=storage, name=self.name ) return self self._started = asyncio.ensure_future(_register()) else: self.client.run_on_scheduler( # type: ignore[no-untyped-call] _register_with_scheduler, storage=storage, name=self.name ) @property def client(self) -> "distributed.Client": if not self._client: self._client = get_client() return self._client def __await__(self) -> Generator[Any, None, "DaskStorage"]: if hasattr(self, "_started"): return self._started.__await__() else: async def _() -> DaskStorage: return self return _().__await__() def __reduce__(self) -> tuple: # We don't have a reference to underlying Optuna storage instance which lives # on the scheduler. This is okay since this DaskStorage instance has already been # registered with the scheduler, and ``storage`` is only ever needed during the # scheduler registration process. We use ``storage=None`` below by convention. return (DaskStorage, (None, self.name, None, False)) def get_base_storage(self) -> BaseStorage: """Retrieve underlying Optuna storage instance from the scheduler. This is a convenience method to extract the Optuna storage instance stored on the Dask scheduler process to the local Python process. """ def _get_base_storage(dask_scheduler: distributed.Scheduler, name: str) -> BaseStorage: return dask_scheduler.extensions["optuna"].storages[name] return self.client.run_on_scheduler( # type: ignore[no-untyped-call] _get_base_storage, name=self.name ) def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_create_new_study, # type: ignore[union-attr] storage_name=self.name, study_name=study_name, directions=[direction.name for direction in directions], ) def delete_study(self, study_id: int) -> None: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_delete_study, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, ) def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_set_study_user_attr, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, key=key, value=dumps(value), # type: ignore[no-untyped-call] ) def set_study_system_attr(self, study_id: int, key: str, value: Any) -> None: return self.client.sync( self.client.scheduler.optuna_set_study_system_attr, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, key=key, value=dumps(value), # type: ignore[no-untyped-call] ) # Basic study access def get_study_id_from_name(self, study_name: str) -> int: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_study_id_from_name, # type: ignore[union-attr] study_name=study_name, storage_name=self.name, ) def get_study_name_from_id(self, study_id: int) -> str: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_study_name_from_id, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, ) def get_study_directions(self, study_id: int) -> List[StudyDirection]: directions = self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_study_directions, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, ) return [StudyDirection[direction] for direction in directions] def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: return loads( # type: ignore[no-untyped-call] self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_study_user_attrs, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, ) ) def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: return loads( # type: ignore[no-untyped-call] self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_study_system_attrs, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, ) ) def get_all_studies(self) -> List[FrozenStudy]: results = self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_all_studies, # type: ignore[union-attr] storage_name=self.name, ) return [_deserialize_frozenstudy(i) for i in results] # Basic trial manipulation def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: serialized_template_trial = None if template_trial is not None: serialized_template_trial = _serialize_frozentrial(template_trial) return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_create_new_trial, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, template_trial=serialized_template_trial, ) def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: BaseDistribution, ) -> None: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_set_trial_param, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, param_name=param_name, param_value_internal=param_value_internal, distribution=distribution_to_json(distribution), ) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_trial_id_from_study_id_trial_number, # type: ignore[union-attr] # NOQA: E501 storage_name=self.name, study_id=study_id, trial_number=trial_number, ) def get_trial_number_from_id(self, trial_id: int) -> int: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_trial_number_from_id, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, ) def get_trial_param(self, trial_id: int, param_name: str) -> float: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_trial_param, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, param_name=param_name, ) def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_set_trial_state_values, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, state=state.name, values=values, ) def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_set_trial_intermediate_value, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, step=step, intermediate_value=intermediate_value, ) def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_set_trial_user_attr, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, key=key, value=dumps(value), # type: ignore[no-untyped-call] ) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_set_trial_system_attr, # type: ignore[union-attr] storage_name=self.name, trial_id=trial_id, key=key, value=dumps(value), # type: ignore[no-untyped-call] ) # Basic trial access async def _get_trial(self, trial_id: int) -> FrozenTrial: serialized_trial = await self.client.scheduler.optuna_get_trial( # type: ignore[union-attr] # NOQA: E501 trial_id=trial_id, storage_name=self.name ) return _deserialize_frozentrial(serialized_trial) def get_trial(self, trial_id: int) -> FrozenTrial: return self.client.sync( # type: ignore[no-untyped-call] self._get_trial, trial_id=trial_id ) async def _get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Iterable[TrialState]] = None ) -> List[FrozenTrial]: serialized_states = None if states is not None: serialized_states = tuple(s.name for s in states) serialized_trials = await self.client.scheduler.optuna_get_all_trials( # type: ignore[union-attr] # NOQA: E501 storage_name=self.name, study_id=study_id, deepcopy=deepcopy, states=serialized_states, ) return [_deserialize_frozentrial(t) for t in serialized_trials] def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None ) -> List[FrozenTrial]: return self.client.sync( # type: ignore[no-untyped-call] self._get_all_trials, study_id=study_id, deepcopy=deepcopy, states=states, ) def get_n_trials( self, study_id: int, state: Optional[Union[Tuple[TrialState, ...], TrialState]] = None ) -> int: serialized_state: Optional[Union[Tuple[str, ...], str]] = None if state is not None: if isinstance(state, TrialState): serialized_state = state.name else: serialized_state = tuple(s.name for s in state) return self.client.sync( # type: ignore[no-untyped-call] self.client.scheduler.optuna_get_n_trials, # type: ignore[union-attr] storage_name=self.name, study_id=study_id, state=serialized_state, ) optuna-3.5.0/optuna/integration/fastaiv1.py000066400000000000000000000001501453453102400207430ustar00rootroot00000000000000from optuna_integration.fastaiv1 import FastAIV1PruningCallback __all__ = ["FastAIV1PruningCallback"] optuna-3.5.0/optuna/integration/fastaiv2.py000066400000000000000000000002771453453102400207560ustar00rootroot00000000000000from optuna_integration.fastaiv2 import FastAIPruningCallback from optuna_integration.fastaiv2 import FastAIV2PruningCallback __all__ = ["FastAIV2PruningCallback", "FastAIPruningCallback"] optuna-3.5.0/optuna/integration/keras.py000066400000000000000000000001371453453102400203370ustar00rootroot00000000000000from optuna_integration.keras import KerasPruningCallback __all__ = ["KerasPruningCallback"] optuna-3.5.0/optuna/integration/lightgbm.py000066400000000000000000000147461453453102400210420ustar00rootroot00000000000000from __future__ import annotations import sys from typing import TYPE_CHECKING import optuna from optuna._imports import try_import from optuna.integration import _lightgbm_tuner as tuner if TYPE_CHECKING: from lightgbm.basic import _LGBM_BoosterEvalMethodResultType from lightgbm.callback import CallbackEnv with try_import() as _imports: import lightgbm as lgb # Attach lightgbm API. if _imports.is_successful(): # To pass tests/integration_tests/lightgbm_tuner_tests/test_optimize.py. from lightgbm import Dataset from optuna.integration._lightgbm_tuner import LightGBMTuner from optuna.integration._lightgbm_tuner import LightGBMTunerCV _names_from_tuners = ["train", "LGBMModel", "LGBMClassifier", "LGBMRegressor"] # API from lightgbm. for api_name in lgb.__dict__["__all__"]: if api_name in _names_from_tuners: continue setattr(sys.modules[__name__], api_name, lgb.__dict__[api_name]) # API from lightgbm_tuner. for api_name in _names_from_tuners: setattr(sys.modules[__name__], api_name, tuner.__dict__[api_name]) else: # To create docstring of train. setattr(sys.modules[__name__], "train", tuner.__dict__["train"]) setattr(sys.modules[__name__], "LightGBMTuner", tuner.__dict__["LightGBMTuner"]) setattr(sys.modules[__name__], "LightGBMTunerCV", tuner.__dict__["LightGBMTunerCV"]) __all__ = ["Dataset", "LightGBMTuner", "LightGBMTunerCV"] class LightGBMPruningCallback: """Callback for LightGBM to prune unpromising trials. See `the example `__ if you want to add a pruning callback which observes accuracy of a LightGBM model. Args: trial: A :class:`~optuna.trial.Trial` corresponding to the current evaluation of the objective function. metric: An evaluation metric for pruning, e.g., ``binary_error`` and ``multi_error``. Please refer to `LightGBM reference `_ for further details. valid_name: The name of the target validation. Validation names are specified by ``valid_names`` option of `train method `_. If omitted, ``valid_0`` is used which is the default name of the first validation. Note that this argument will be ignored if you are calling `cv method `_ instead of train method. report_interval: Check if the trial should report intermediate values for pruning every n-th boosting iteration. By default ``report_interval=1`` and reporting is performed after every iteration. Note that the pruning itself is performed according to the interval definition of the pruner. """ def __init__( self, trial: optuna.trial.Trial, metric: str, valid_name: str = "valid_0", report_interval: int = 1, ) -> None: _imports.check() self._trial = trial self._valid_name = valid_name self._metric = metric self._report_interval = report_interval def _find_evaluation_result( self, target_valid_name: str, env: CallbackEnv ) -> _LGBM_BoosterEvalMethodResultType | None: evaluation_result_list = env.evaluation_result_list if evaluation_result_list is None: return None for evaluation_result in evaluation_result_list: valid_name, metric, current_score, is_higher_better = evaluation_result[:4] # The prefix "valid " is added to metric name since LightGBM v4.0.0. if valid_name != target_valid_name or ( metric != "valid " + self._metric and metric != self._metric ): continue return evaluation_result return None def __call__(self, env: CallbackEnv) -> None: if (env.iteration + 1) % self._report_interval == 0: # If this callback has been passed to `lightgbm.cv` function, # the value of `is_cv` becomes `True`. See also: # https://github.com/microsoft/LightGBM/blob/v4.1.0/python-package/lightgbm/engine.py#L533 # Note that `5` is not the number of folds but the length of sequence. evaluation_result_list = env.evaluation_result_list is_cv = ( evaluation_result_list is not None and len(evaluation_result_list) > 0 and len(evaluation_result_list[0]) == 5 ) if is_cv: target_valid_name = "cv_agg" else: target_valid_name = self._valid_name evaluation_result = self._find_evaluation_result(target_valid_name, env) if evaluation_result is None: raise ValueError( 'The entry associated with the validation name "{}" and the metric name "{}" ' "is not found in the evaluation result list {}.".format( target_valid_name, self._metric, str(env.evaluation_result_list) ) ) valid_name, metric, current_score, is_higher_better = evaluation_result[:4] if is_higher_better: if self._trial.study.direction != optuna.study.StudyDirection.MAXIMIZE: raise ValueError( "The intermediate values are inconsistent with the objective values" "in terms of study directions. Please specify a metric to be minimized" "for LightGBMPruningCallback." ) else: if self._trial.study.direction != optuna.study.StudyDirection.MINIMIZE: raise ValueError( "The intermediate values are inconsistent with the objective values" "in terms of study directions. Please specify a metric to be" "maximized for LightGBMPruningCallback." ) self._trial.report(current_score, step=env.iteration) if self._trial.should_prune(): message = "Trial was pruned at iteration {}.".format(env.iteration) raise optuna.TrialPruned(message) optuna-3.5.0/optuna/integration/mlflow.py000066400000000000000000000270721453453102400205410ustar00rootroot00000000000000import functools import threading from typing import Any from typing import Callable from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import Union import optuna from optuna._experimental import experimental_class from optuna._experimental import experimental_func from optuna._imports import try_import from optuna.study.study import ObjectiveFuncType with try_import() as _imports: import mlflow RUN_ID_ATTRIBUTE_KEY = "mlflow_run_id" @experimental_class("1.4.0") class MLflowCallback: """Callback to track Optuna trials with MLflow. This callback adds relevant information that is tracked by Optuna to MLflow. Example: Add MLflow callback to Optuna optimization. .. testsetup:: import pathlib import tempfile tempdir = tempfile.mkdtemp() YOUR_TRACKING_URI = pathlib.Path(tempdir).as_uri() .. testcode:: import optuna from optuna.integration.mlflow import MLflowCallback def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 mlflc = MLflowCallback( tracking_uri=YOUR_TRACKING_URI, metric_name="my metric score", ) study = optuna.create_study(study_name="my_study") study.optimize(objective, n_trials=10, callbacks=[mlflc]) .. testcleanup:: import shutil shutil.rmtree(tempdir) Args: tracking_uri: The URI of the MLflow tracking server. Please refer to `mlflow.set_tracking_uri `_ for more details. metric_name: Name assigned to optimized metric. In case of multi-objective optimization, list of names can be passed. Those names will be assigned to metrics in the order returned by objective function. If single name is provided, or this argument is left to default value, it will be broadcasted to each objective with a number suffix in order returned by objective function e.g. two objectives and default metric name will be logged as ``value_0`` and ``value_1``. The number of metrics must be the same as the number of values an objective function returns. create_experiment: When :obj:`True`, new MLflow experiment will be created for each optimization run, named after the Optuna study. Setting this argument to :obj:`False` lets user run optimization under existing experiment, set via `mlflow.set_experiment `_, by passing ``experiment_id`` as one of ``mlflow_kwargs`` or under default MLflow experiment, when no additional arguments are passed. Note that this argument must be set to :obj:`False` when using Optuna with this callback within Databricks Notebook. mlflow_kwargs: Set of arguments passed when initializing MLflow run. Please refer to `MLflow API documentation `_ for more details. .. note:: ``nest_trials`` argument added in v2.3.0 is a part of ``mlflow_kwargs`` since v3.0.0. Anyone using ``nest_trials=True`` should migrate to ``mlflow_kwargs={"nested": True}`` to avoid raising :exc:`TypeError`. tag_study_user_attrs: Flag indicating whether or not to add the study's user attrs to the mlflow trial as tags. Please note that when this flag is set, key value pairs in :attr:`~optuna.study.Study.user_attrs` will supersede existing tags. tag_trial_user_attrs: Flag indicating whether or not to add the trial's user attrs to the mlflow trial as tags. Please note that when both trial and study user attributes are logged, the latter will supersede the former in case of a collision. """ def __init__( self, tracking_uri: Optional[str] = None, metric_name: Union[str, Sequence[str]] = "value", create_experiment: bool = True, mlflow_kwargs: Optional[Dict[str, Any]] = None, tag_study_user_attrs: bool = False, tag_trial_user_attrs: bool = True, ) -> None: _imports.check() if not isinstance(metric_name, Sequence): raise TypeError( "Expected metric_name to be string or sequence of strings, got {}.".format( type(metric_name) ) ) self._tracking_uri = tracking_uri self._metric_name = metric_name self._create_experiment = create_experiment self._mlflow_kwargs = mlflow_kwargs or {} self._tag_study_user_attrs = tag_study_user_attrs self._tag_trial_user_attrs = tag_trial_user_attrs self._lock = threading.Lock() def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None: with self._lock: self._initialize_experiment(study) with mlflow.start_run( run_id=trial.system_attrs.get(RUN_ID_ATTRIBUTE_KEY), experiment_id=self._mlflow_kwargs.get("experiment_id"), run_name=self._mlflow_kwargs.get("run_name") or str(trial.number), nested=self._mlflow_kwargs.get("nested") or False, tags=self._mlflow_kwargs.get("tags"), ): # This sets the metrics for MLflow. self._log_metrics(trial.values) # This sets the params for MLflow. self._log_params(trial.params) # This sets the tags for MLflow. self._set_tags(trial, study) @experimental_func("2.9.0") def track_in_mlflow(self) -> Callable: """Decorator for using MLflow logging in the objective function. This decorator enables the extension of MLflow logging provided by the callback. All information logged in the decorated objective function will be added to the MLflow run for the trial created by the callback. Example: Add additional logging to MLflow. .. testcode:: import optuna import mlflow from optuna.integration.mlflow import MLflowCallback mlflc = MLflowCallback( tracking_uri=YOUR_TRACKING_URI, metric_name="my metric score", ) @mlflc.track_in_mlflow() def objective(trial): x = trial.suggest_float("x", -10, 10) mlflow.log_param("power", 2) mlflow.log_metric("base of metric", x - 2) return (x - 2) ** 2 study = optuna.create_study(study_name="my_other_study") study.optimize(objective, n_trials=10, callbacks=[mlflc]) Returns: Objective function with tracking to MLflow enabled. """ def decorator(func: ObjectiveFuncType) -> ObjectiveFuncType: @functools.wraps(func) def wrapper(trial: optuna.trial.Trial) -> Union[float, Sequence[float]]: with self._lock: study = trial.study self._initialize_experiment(study) nested = self._mlflow_kwargs.get("nested") with mlflow.start_run(run_name=str(trial.number), nested=nested) as run: trial.storage.set_trial_system_attr( trial._trial_id, RUN_ID_ATTRIBUTE_KEY, run.info.run_id ) return func(trial) return wrapper return decorator def _initialize_experiment(self, study: optuna.study.Study) -> None: """Initialize an MLflow experiment with the study name. If a tracking uri has been provided, MLflow will be initialized to use it. Args: study: Study to be tracked in MLflow. """ # This sets the `tracking_uri` for MLflow. if self._tracking_uri is not None: mlflow.set_tracking_uri(self._tracking_uri) if self._create_experiment: mlflow.set_experiment(study.study_name) def _set_tags(self, trial: optuna.trial.FrozenTrial, study: optuna.study.Study) -> None: """Sets the Optuna tags for the current MLflow run. Args: trial: Trial to be tracked. study: Study to be tracked. """ tags: Dict[str, Union[str, List[str]]] = {} tags["number"] = str(trial.number) tags["datetime_start"] = str(trial.datetime_start) tags["datetime_complete"] = str(trial.datetime_complete) # Set trial state. if trial.state.is_finished(): tags["state"] = trial.state.name # Set study directions. directions = [d.name for d in study.directions] tags["direction"] = directions if len(directions) != 1 else directions[0] distributions = {(k + "_distribution"): str(v) for (k, v) in trial.distributions.items()} tags.update(distributions) if self._tag_trial_user_attrs: tags.update(trial.user_attrs) if self._tag_study_user_attrs: tags.update(study.user_attrs) # This is a temporary fix on Optuna side. It avoids an error with user # attributes that are too long. It should be fixed on MLflow side later. # When it is fixed on MLflow side this codeblock can be removed. # see https://github.com/optuna/optuna/issues/1340 # see https://github.com/mlflow/mlflow/issues/2931 for key, value in tags.items(): value = str(value) # make sure it is a string max_val_length = mlflow.utils.validation.MAX_TAG_VAL_LENGTH if len(value) > max_val_length: tags[key] = "{}...".format(value[: max_val_length - 3]) mlflow.set_tags(tags) def _log_metrics(self, values: Optional[List[float]]) -> None: """Log the trial results as metrics to MLflow. Args: values: Results of a trial. """ if values is None: return if isinstance(self._metric_name, str): if len(values) > 1: # Broadcast default name for multi-objective optimization. names = ["{}_{}".format(self._metric_name, i) for i in range(len(values))] else: names = [self._metric_name] else: if len(self._metric_name) != len(values): raise ValueError( "Running multi-objective optimization " "with {} objective values, but {} names specified. " "Match objective values and names, or use default broadcasting.".format( len(values), len(self._metric_name) ) ) else: names = [*self._metric_name] metrics = {name: val for name, val in zip(names, values)} mlflow.log_metrics(metrics) @staticmethod def _log_params(params: Dict[str, Any]) -> None: """Log the parameters of the trial to MLflow. Args: params: Trial params. """ mlflow.log_params(params) optuna-3.5.0/optuna/integration/mxnet.py000066400000000000000000000001371453453102400203650ustar00rootroot00000000000000from optuna_integration.mxnet import MXNetPruningCallback __all__ = ["MXNetPruningCallback"] optuna-3.5.0/optuna/integration/pytorch_distributed.py000066400000000000000000000277451453453102400233420ustar00rootroot00000000000000from datetime import datetime import functools import pickle from typing import Any from typing import Callable from typing import Dict from typing import Optional from typing import overload from typing import Sequence from typing import TYPE_CHECKING from typing import TypeVar import optuna from optuna._deprecated import deprecated_func from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType with try_import() as _imports: import torch import torch.distributed as dist from torch.distributed import ProcessGroup # type: ignore[attr-defined] if TYPE_CHECKING: from typing_extensions import ParamSpec _T = TypeVar("_T") _P = ParamSpec("_P") _suggest_deprecated_msg = "Use suggest_float{args} instead." _g_pg: Optional["ProcessGroup"] = None def broadcast_properties(f: "Callable[_P, _T]") -> "Callable[_P, _T]": """Method decorator to fetch updated trial properties from rank 0 after ``f`` is run. This decorator ensures trial properties (params, distributions, etc.) on all distributed processes are up-to-date with the wrapped trial stored on rank 0. It should be applied to all :class:`~optuna.integration.TorchDistributedTrial` methods that update property values. """ @functools.wraps(f) def wrapped(*args: "_P.args", **kwargs: "_P.kwargs") -> "_T": # TODO(nlgranger): Remove type ignore after mypy includes # https://github.com/python/mypy/pull/12668 self: TorchDistributedTrial = args[0] # type: ignore[assignment] def fetch_properties() -> Sequence: assert self._delegate is not None return ( self._delegate.number, self._delegate.params, self._delegate.distributions, self._delegate.user_attrs, self._delegate.system_attrs, self._delegate.datetime_start, ) try: return f(*args, **kwargs) finally: ( self._number, self._params, self._distributions, self._user_attrs, self._system_attrs, self._datetime_start, ) = self._call_and_communicate_obj(fetch_properties) return wrapped @experimental_class("2.6.0") class TorchDistributedTrial(optuna.trial.BaseTrial): """A wrapper of :class:`~optuna.trial.Trial` to incorporate Optuna with PyTorch distributed. .. seealso:: :class:`~optuna.integration.TorchDistributedTrial` provides the same interface as :class:`~optuna.trial.Trial`. Please refer to :class:`optuna.trial.Trial` for further details. See `the example `__ if you want to optimize an objective function that trains neural network written with PyTorch distributed data parallel. Args: trial: A :class:`~optuna.trial.Trial` object or :obj:`None`. Please set trial object in rank-0 node and set :obj:`None` in the other rank node. group: A `torch.distributed.ProcessGroup` to communicate with the other nodes. TorchDistributedTrial use CPU tensors to communicate, make sure the group supports CPU tensors communications. Use `gloo` backend when group is None. Create a global `gloo` backend when group is None and WORLD is nccl. .. note:: The methods of :class:`~optuna.integration.TorchDistributedTrial` are expected to be called by all workers at once. They invoke synchronous data transmission to share processing results and synchronize timing. """ def __init__( self, trial: Optional[optuna.trial.BaseTrial], group: Optional["ProcessGroup"] = None, ) -> None: _imports.check() global _g_pg if group is not None: self._group: "ProcessGroup" = group else: if _g_pg is None: if dist.group.WORLD is None: raise RuntimeError("torch distributed is not initialized.") default_pg: "ProcessGroup" = dist.group.WORLD if dist.get_backend(default_pg) == "nccl": new_group: "ProcessGroup" = dist.new_group(backend="gloo") _g_pg = new_group else: _g_pg = default_pg self._group = _g_pg if dist.get_rank(self._group) == 0: if not isinstance(trial, optuna.trial.BaseTrial): raise ValueError( "Rank 0 node expects an optuna.trial.Trial instance as the trial argument." ) else: if trial is not None: raise ValueError( "Non-rank 0 node is supposed to receive None as the trial argument." ) assert trial is None, "error message" self._delegate = trial self._number = self._broadcast(getattr(self._delegate, "number", None)) self._params = self._broadcast(getattr(self._delegate, "params", None)) self._distributions = self._broadcast(getattr(self._delegate, "distributions", None)) self._user_attrs = self._broadcast(getattr(self._delegate, "user_attrs", None)) self._system_attrs = self._broadcast(getattr(self._delegate, "system_attrs", None)) self._datetime_start = self._broadcast(getattr(self._delegate, "datetime_start", None)) @broadcast_properties def suggest_float( self, name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False, ) -> float: def func() -> float: assert self._delegate is not None return self._delegate.suggest_float(name, low, high, step=step, log=log) return self._call_and_communicate(func, torch.float) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: return self.suggest_float(name, low, high, step=q) @broadcast_properties def suggest_int(self, name: str, low: int, high: int, step: int = 1, log: bool = False) -> int: def func() -> float: assert self._delegate is not None return self._delegate.suggest_int(name, low, high, step=step, log=log) return self._call_and_communicate(func, torch.int) @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... @broadcast_properties def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: def func() -> CategoricalChoiceType: assert self._delegate is not None return self._delegate.suggest_categorical(name, choices) return self._call_and_communicate_obj(func) @broadcast_properties def report(self, value: float, step: int) -> None: err = None if dist.get_rank(self._group) == 0: try: assert self._delegate is not None self._delegate.report(value, step) except Exception as e: err = e err = self._broadcast(err) else: err = self._broadcast(err) if err is not None: raise err @broadcast_properties def should_prune(self) -> bool: def func() -> bool: assert self._delegate is not None # Some pruners return numpy.bool_, which is incompatible with bool. return bool(self._delegate.should_prune()) # torch.bool seems to be the correct type, but the communication fails # due to the RuntimeError. return self._call_and_communicate(func, torch.uint8) @broadcast_properties def set_user_attr(self, key: str, value: Any) -> None: err = None if dist.get_rank(self._group) == 0: try: assert self._delegate is not None self._delegate.set_user_attr(key, value) except Exception as e: err = e err = self._broadcast(err) else: err = self._broadcast(err) if err is not None: raise err @broadcast_properties @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: err = None if dist.get_rank(self._group) == 0: try: assert self._delegate is not None self._delegate.storage.set_trial_system_attr( # type: ignore[attr-defined] self._delegate._trial_id, key, value # type: ignore[attr-defined] ) except Exception as e: err = e err = self._broadcast(err) else: err = self._broadcast(err) if err is not None: raise err @property def number(self) -> int: return self._number @property def params(self) -> Dict[str, Any]: return self._params @property def distributions(self) -> Dict[str, BaseDistribution]: return self._distributions @property def user_attrs(self) -> Dict[str, Any]: return self._user_attrs @property @deprecated_func("3.1.0", "5.0.0") def system_attrs(self) -> Dict[str, Any]: return self._system_attrs @property def datetime_start(self) -> Optional[datetime]: return self._datetime_start def _call_and_communicate(self, func: Callable, dtype: "torch.dtype") -> Any: buffer = torch.empty(1, dtype=dtype) rank = dist.get_rank(self._group) if rank == 0: result = func() buffer[0] = result dist.broadcast(buffer, src=0, group=self._group) return buffer.item() def _call_and_communicate_obj(self, func: Callable) -> Any: rank = dist.get_rank(self._group) result = func() if rank == 0 else None return self._broadcast(result) def _broadcast(self, value: Optional[Any]) -> Any: buffer = None size_buffer = torch.empty(1, dtype=torch.int) rank = dist.get_rank(self._group) if rank == 0: buffer = _to_tensor(value) size_buffer[0] = buffer.shape[0] dist.broadcast(size_buffer, src=0, group=self._group) buffer_size = int(size_buffer.item()) if rank != 0: buffer = torch.empty(buffer_size, dtype=torch.uint8) assert buffer is not None dist.broadcast(buffer, src=0, group=self._group) return _from_tensor(buffer) def _to_tensor(obj: Any) -> "torch.Tensor": b = bytearray(pickle.dumps(obj)) return torch.tensor(b, dtype=torch.uint8) def _from_tensor(tensor: "torch.Tensor") -> Any: b = bytearray(tensor.tolist()) return pickle.loads(b) optuna-3.5.0/optuna/integration/pytorch_ignite.py000066400000000000000000000026011453453102400222570ustar00rootroot00000000000000import optuna from optuna.trial import Trial with optuna._imports.try_import() as _imports: from ignite.engine import Engine class PyTorchIgnitePruningHandler: """PyTorch Ignite handler to prune unpromising trials. See `the example `__ if you want to add a pruning handler which observes validation accuracy. Args: trial: A :class:`~optuna.trial.Trial` corresponding to the current evaluation of the objective function. metric: A name of metric for pruning, e.g., ``accuracy`` and ``loss``. trainer: A trainer engine of PyTorch Ignite. Please refer to `ignite.engine.Engine reference `_ for further details. """ def __init__(self, trial: Trial, metric: str, trainer: "Engine") -> None: _imports.check() self._trial = trial self._metric = metric self._trainer = trainer def __call__(self, engine: "Engine") -> None: score = engine.state.metrics[self._metric] self._trial.report(score, self._trainer.state.epoch) if self._trial.should_prune(): message = "Trial was pruned at {} epoch.".format(self._trainer.state.epoch) raise optuna.TrialPruned(message) optuna-3.5.0/optuna/integration/pytorch_lightning.py000066400000000000000000000172271453453102400227750ustar00rootroot00000000000000import warnings from packaging import version import optuna from optuna.storages._cached_storage import _CachedStorage from optuna.storages._rdb.storage import RDBStorage # Define key names of `Trial.system_attrs`. _EPOCH_KEY = "ddp_pl:epoch" _INTERMEDIATE_VALUE = "ddp_pl:intermediate_value" _PRUNED_KEY = "ddp_pl:pruned" with optuna._imports.try_import() as _imports: import lightning.pytorch as pl from lightning.pytorch import LightningModule from lightning.pytorch import Trainer from lightning.pytorch.callbacks import Callback if not _imports.is_successful(): Callback = object # type: ignore[assignment, misc] # NOQA[F811] LightningModule = object # type: ignore[assignment, misc] # NOQA[F811] Trainer = object # type: ignore[assignment, misc] # NOQA[F811] class PyTorchLightningPruningCallback(Callback): """PyTorch Lightning callback to prune unpromising trials. See `the example `__ if you want to add a pruning callback which observes accuracy. Args: trial: A :class:`~optuna.trial.Trial` corresponding to the current evaluation of the objective function. monitor: An evaluation metric for pruning, e.g., ``val_loss`` or ``val_acc``. The metrics are obtained from the returned dictionaries from e.g. ``lightning.pytorch.LightningModule.training_step`` or ``lightning.pytorch.LightningModule.validation_epoch_end`` and the names thus depend on how this dictionary is formatted. .. note:: For the distributed data parallel training, the version of PyTorchLightning needs to be higher than or equal to v1.6.0. In addition, :class:`~optuna.study.Study` should be instantiated with RDB storage. .. note:: If you would like to use PyTorchLightningPruningCallback in a distributed training environment, you need to evoke `PyTorchLightningPruningCallback.check_pruned()` manually so that :class:`~optuna.exceptions.TrialPruned` is properly handled. """ def __init__(self, trial: optuna.trial.Trial, monitor: str) -> None: _imports.check() super().__init__() self._trial = trial self.monitor = monitor self.is_ddp_backend = False def on_fit_start(self, trainer: Trainer, pl_module: "pl.LightningModule") -> None: self.is_ddp_backend = trainer._accelerator_connector.is_distributed if self.is_ddp_backend: if version.parse(pl.__version__) < version.parse( # type: ignore[attr-defined] "1.6.0" ): raise ValueError("PyTorch Lightning>=1.6.0 is required in DDP.") # If it were not for this block, fitting is started even if unsupported storage # is used. Note that the ValueError is transformed into ProcessRaisedException inside # torch. if not ( isinstance(self._trial.study._storage, _CachedStorage) and isinstance(self._trial.study._storage._backend, RDBStorage) ): raise ValueError( "optuna.integration.PyTorchLightningPruningCallback" " supports only optuna.storages.RDBStorage in DDP." ) # It is necessary to store intermediate values directly in the backend storage because # they are not properly propagated to main process due to cached storage. # TODO(Shinichi) Remove intermediate_values from system_attr after PR #4431 is merged. if trainer.is_global_zero: self._trial.storage.set_trial_system_attr( self._trial._trial_id, _INTERMEDIATE_VALUE, dict(), ) def on_validation_end(self, trainer: Trainer, pl_module: LightningModule) -> None: # Trainer calls `on_validation_end` for sanity check. Therefore, it is necessary to avoid # calling `trial.report` multiple times at epoch 0. For more details, see # https://github.com/PyTorchLightning/pytorch-lightning/issues/1391. if trainer.sanity_checking: return current_score = trainer.callback_metrics.get(self.monitor) if current_score is None: message = ( f"The metric '{self.monitor}' is not in the evaluation logs for pruning. " "Please make sure you set the correct metric name." ) warnings.warn(message) return epoch = pl_module.current_epoch should_stop = False # Determine if the trial should be terminated in a single process. if not self.is_ddp_backend: self._trial.report(current_score.item(), step=epoch) if not self._trial.should_prune(): return raise optuna.TrialPruned(f"Trial was pruned at epoch {epoch}.") # Determine if the trial should be terminated in a DDP. if trainer.is_global_zero: self._trial.report(current_score.item(), step=epoch) should_stop = self._trial.should_prune() # Update intermediate value in the storage. _trial_id = self._trial._trial_id _study = self._trial.study _trial_system_attrs = _study._storage.get_trial_system_attrs(_trial_id) intermediate_values = _trial_system_attrs.get(_INTERMEDIATE_VALUE) intermediate_values[epoch] = current_score.item() # type: ignore[index] self._trial.storage.set_trial_system_attr( self._trial._trial_id, _INTERMEDIATE_VALUE, intermediate_values ) # Terminate every process if any world process decides to stop. should_stop = trainer.strategy.broadcast(should_stop) trainer.should_stop = trainer.should_stop or should_stop if not should_stop: return if trainer.is_global_zero: # Update system_attr from global zero process. self._trial.storage.set_trial_system_attr(self._trial._trial_id, _PRUNED_KEY, True) self._trial.storage.set_trial_system_attr(self._trial._trial_id, _EPOCH_KEY, epoch) def check_pruned(self) -> None: """Raise :class:`optuna.TrialPruned` manually if pruned. Currently, ``intermediate_values`` are not properly propagated between processes due to storage cache. Therefore, necessary information is kept in trial_system_attrs when the trial runs in a distributed situation. Please call this method right after calling ``lightning.pytorch.Trainer.fit()``. If a callback doesn't have any backend storage for DDP, this method does nothing. """ _trial_id = self._trial._trial_id _study = self._trial.study # Confirm if storage is not InMemory in case this method is called in a non-distributed # situation by mistake. if not isinstance(_study._storage, _CachedStorage): return _trial_system_attrs = _study._storage._backend.get_trial_system_attrs(_trial_id) is_pruned = _trial_system_attrs.get(_PRUNED_KEY) intermediate_values = _trial_system_attrs.get(_INTERMEDIATE_VALUE) # Confirm if DDP backend is used in case this method is called from a non-DDP situation by # mistake. if intermediate_values is None: return for epoch, score in intermediate_values.items(): self._trial.report(score, step=int(epoch)) if is_pruned: epoch = _trial_system_attrs.get(_EPOCH_KEY) raise optuna.TrialPruned(f"Trial was pruned at epoch {epoch}.") optuna-3.5.0/optuna/integration/shap.py000066400000000000000000000001521453453102400201620ustar00rootroot00000000000000from optuna_integration.shap import ShapleyImportanceEvaluator __all__ = ["ShapleyImportanceEvaluator"] optuna-3.5.0/optuna/integration/sklearn.py000066400000000000000000000777471453453102400207160ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Iterable from collections.abc import Mapping from logging import DEBUG from logging import INFO from logging import WARNING from numbers import Integral from numbers import Number from time import time from typing import Any from typing import List from typing import Union import numpy as np from optuna import distributions from optuna import logging from optuna import samplers from optuna import study as study_module from optuna import TrialPruned from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.study import StudyDirection from optuna.terminator import report_cross_validation_scores from optuna.trial import FrozenTrial from optuna.trial import Trial with try_import() as _imports: import pandas as pd import scipy as sp from scipy.sparse import spmatrix import sklearn from sklearn.base import BaseEstimator from sklearn.base import clone from sklearn.base import is_classifier from sklearn.metrics import check_scoring from sklearn.model_selection import BaseCrossValidator from sklearn.model_selection import check_cv from sklearn.model_selection import cross_validate from sklearn.utils import _safe_indexing as sklearn_safe_indexing from sklearn.utils import check_random_state from sklearn.utils.metaestimators import _safe_split from sklearn.utils.validation import check_is_fitted if not _imports.is_successful(): BaseEstimator = object # NOQA ArrayLikeType = Union[List, np.ndarray, "pd.Series", "spmatrix"] OneDimArrayLikeType = Union[List[float], np.ndarray, "pd.Series"] TwoDimArrayLikeType = Union[List[List[float]], np.ndarray, "pd.DataFrame", "spmatrix"] IterableType = Union[List, "pd.DataFrame", np.ndarray, "pd.Series", "spmatrix", None] IndexableType = Union[Iterable, None] _logger = logging.get_logger(__name__) def _check_fit_params( X: TwoDimArrayLikeType, fit_params: dict, indices: OneDimArrayLikeType ) -> dict: fit_params_validated = {} for key, value in fit_params.items(): # NOTE Original implementation: # https://github.com/scikit-learn/scikit-learn/blob/ \ # 2467e1b84aeb493a22533fa15ff92e0d7c05ed1c/sklearn/utils/validation.py#L1324-L1328 # Scikit-learn does not accept non-iterable inputs. # This line is for keeping backward compatibility. # (See: https://github.com/scikit-learn/scikit-learn/issues/15805) if not _is_arraylike(value) or _num_samples(value) != _num_samples(X): fit_params_validated[key] = value else: fit_params_validated[key] = _make_indexable(value) fit_params_validated[key] = _safe_indexing(fit_params_validated[key], indices) return fit_params_validated # NOTE Original implementation: # https://github.com/scikit-learn/scikit-learn/blob/ \ # 8caa93889f85254fc3ca84caa0a24a1640eebdd1/sklearn/utils/validation.py#L131-L135 def _is_arraylike(x: Any) -> bool: return hasattr(x, "__len__") or hasattr(x, "shape") or hasattr(x, "__array__") # NOTE Original implementation: # https://github.com/scikit-learn/scikit-learn/blob/ \ # 8caa93889f85254fc3ca84caa0a24a1640eebdd1/sklearn/utils/validation.py#L217-L234 def _make_indexable(iterable: IterableType) -> IndexableType: tocsr_func = getattr(iterable, "tocsr", None) if tocsr_func is not None and sp.sparse.issparse(iterable): return tocsr_func(iterable) elif hasattr(iterable, "__getitem__") or hasattr(iterable, "iloc"): return iterable elif iterable is None: return iterable return np.array(iterable) def _num_samples(x: ArrayLikeType) -> int: # NOTE For dask dataframes # https://github.com/scikit-learn/scikit-learn/blob/ \ # 8caa93889f85254fc3ca84caa0a24a1640eebdd1/sklearn/utils/validation.py#L155-L158 x_shape = getattr(x, "shape", None) if x_shape is not None: if isinstance(x_shape[0], Integral): return int(x_shape[0]) try: return len(x) except TypeError: raise TypeError("Expected sequence or array-like, got %s." % type(x)) from None def _safe_indexing( X: OneDimArrayLikeType | TwoDimArrayLikeType, indices: OneDimArrayLikeType ) -> OneDimArrayLikeType | TwoDimArrayLikeType: if X is None: return X return sklearn_safe_indexing(X, indices) class _Objective: """Callable that implements objective function. Args: estimator: Object to use to fit the data. This is assumed to implement the scikit-learn estimator interface. Either this needs to provide ``score``, or ``scoring`` must be passed. param_distributions: Dictionary where keys are parameters and values are distributions. Distributions are assumed to implement the optuna distribution interface. X: Training data. y: Target variable. cv: Cross-validation strategy. enable_pruning: If :obj:`True`, pruning is performed in the case where the underlying estimator supports ``partial_fit``. error_score: Value to assign to the score if an error occurs in fitting. If 'raise', the error is raised. If numeric, ``sklearn.exceptions.FitFailedWarning`` is raised. This does not affect the refit step, which will always raise the error. fit_params: Parameters passed to ``fit`` one the estimator. groups: Group labels for the samples used while splitting the dataset into train/validation set. max_iter: Maximum number of epochs. This is only used if the underlying estimator supports ``partial_fit``. return_train_score: If :obj:`True`, training scores will be included. Computing training scores is used to get insights on how different hyperparameter settings impact the overfitting/underfitting trade-off. However computing training scores can be computationally expensive and is not strictly required to select the hyperparameters that yield the best generalization performance. scoring: Scorer function. """ def __init__( self, estimator: "sklearn.base.BaseEstimator", param_distributions: Mapping[str, distributions.BaseDistribution], X: TwoDimArrayLikeType, y: OneDimArrayLikeType | TwoDimArrayLikeType | None, cv: "BaseCrossValidator", enable_pruning: bool, error_score: Number | float | str, fit_params: dict[str, Any], groups: OneDimArrayLikeType | None, max_iter: int, return_train_score: bool, scoring: Callable[..., Number], ) -> None: self.cv = cv self.enable_pruning = enable_pruning self.error_score = error_score self.estimator = estimator self.fit_params = fit_params self.groups = groups self.max_iter = max_iter self.param_distributions = param_distributions self.return_train_score = return_train_score self.scoring = scoring self.X = X self.y = y def __call__(self, trial: Trial) -> float: estimator = clone(self.estimator) params = self._get_params(trial) estimator.set_params(**params) if self.enable_pruning: scores = self._cross_validate_with_pruning(trial, estimator) else: try: scores = cross_validate( estimator, self.X, self.y, cv=self.cv, error_score=self.error_score, fit_params=self.fit_params, groups=self.groups, return_train_score=self.return_train_score, scoring=self.scoring, ) except ValueError: n_splits = self.cv.get_n_splits(self.X, self.y, self.groups) fit_time = np.array([np.nan] * n_splits) score_time = np.array([np.nan] * n_splits) test_score = np.array( [self.error_score if self.error_score is not None else np.nan] * n_splits ) scores = { "fit_time": fit_time, "score_time": score_time, "test_score": test_score, } self._store_scores(trial, scores) test_scores = scores["test_score"] scores_list = test_scores if isinstance(test_scores, list) else test_scores.tolist() report_cross_validation_scores(trial, scores_list) return trial.user_attrs["mean_test_score"] def _cross_validate_with_pruning( self, trial: Trial, estimator: "sklearn.base.BaseEstimator" ) -> Mapping[str, OneDimArrayLikeType]: if is_classifier(estimator): partial_fit_params = self.fit_params.copy() y = self.y.values if isinstance(self.y, pd.Series) else self.y classes = np.unique(y) partial_fit_params.setdefault("classes", classes) else: partial_fit_params = self.fit_params n_splits = self.cv.get_n_splits(self.X, self.y, groups=self.groups) estimators = [clone(estimator) for _ in range(n_splits)] scores = { "fit_time": np.zeros(n_splits), "score_time": np.zeros(n_splits), "test_score": np.empty(n_splits), } if self.return_train_score: scores["train_score"] = np.empty(n_splits) for step in range(self.max_iter): for i, (train, test) in enumerate(self.cv.split(self.X, self.y, groups=self.groups)): out = self._partial_fit_and_score(estimators[i], train, test, partial_fit_params) if self.return_train_score: scores["train_score"][i] = out.pop(0) scores["test_score"][i] = out[0] scores["fit_time"][i] += out[1] scores["score_time"][i] += out[2] intermediate_value = np.nanmean(scores["test_score"]) trial.report(intermediate_value, step=step) if trial.should_prune(): self._store_scores(trial, scores) raise TrialPruned("trial was pruned at iteration {}.".format(step)) return scores def _get_params(self, trial: Trial) -> dict[str, Any]: return { name: trial._suggest(name, distribution) for name, distribution in self.param_distributions.items() } def _partial_fit_and_score( self, estimator: "sklearn.base.BaseEstimator", train: list[int], test: list[int], partial_fit_params: dict[str, Any], ) -> list[Number]: X_train, y_train = _safe_split(estimator, self.X, self.y, train) X_test, y_test = _safe_split(estimator, self.X, self.y, test, train_indices=train) start_time = time() try: estimator.partial_fit(X_train, y_train, **partial_fit_params) except Exception as e: if self.error_score == "raise": raise e elif isinstance(self.error_score, Number): fit_time = time() - start_time test_score = self.error_score score_time = 0.0 if self.return_train_score: train_score = self.error_score else: raise ValueError("error_score must be 'raise' or numeric.") from e else: fit_time = time() - start_time test_score = self.scoring(estimator, X_test, y_test) score_time = time() - fit_time - start_time if self.return_train_score: train_score = self.scoring(estimator, X_train, y_train) # Required for type checking but is never expected to fail. assert isinstance(fit_time, Number) assert isinstance(score_time, Number) ret = [test_score, fit_time, score_time] if self.return_train_score: ret.insert(0, train_score) return ret def _store_scores(self, trial: Trial, scores: Mapping[str, OneDimArrayLikeType]) -> None: for name, array in scores.items(): if name in ["test_score", "train_score"]: for i, score in enumerate(array): trial.set_user_attr("split{}_{}".format(i, name), score) trial.set_user_attr("mean_{}".format(name), np.nanmean(array)) trial.set_user_attr("std_{}".format(name), np.nanstd(array)) @experimental_class("0.17.0") class OptunaSearchCV(BaseEstimator): """Hyperparameter search with cross-validation. Args: estimator: Object to use to fit the data. This is assumed to implement the scikit-learn estimator interface. Either this needs to provide ``score``, or ``scoring`` must be passed. param_distributions: Dictionary where keys are parameters and values are distributions. Distributions are assumed to implement the optuna distribution interface. cv: Cross-validation strategy. Possible inputs for cv are: - :obj:`None`, to use the default 5-fold cross validation, - integer to specify the number of folds in a CV splitter, - `CV splitter `_, - an iterable yielding (train, validation) splits as arrays of indices. For integer, if ``estimator`` is a classifier and ``y`` is either binary or multiclass, ``sklearn.model_selection.StratifiedKFold`` is used. otherwise, ``sklearn.model_selection.KFold`` is used. enable_pruning: If :obj:`True`, pruning is performed in the case where the underlying estimator supports ``partial_fit``. error_score: Value to assign to the score if an error occurs in fitting. If 'raise', the error is raised. If numeric, ``sklearn.exceptions.FitFailedWarning`` is raised. This does not affect the refit step, which will always raise the error. max_iter: Maximum number of epochs. This is only used if the underlying estimator supports ``partial_fit``. n_jobs: Number of :obj:`threading` based parallel jobs. :obj:`None` means ``1``. ``-1`` means using the number is set to CPU count. .. note:: ``n_jobs`` allows parallelization using :obj:`threading` and may suffer from `Python's GIL `_. It is recommended to use :ref:`process-based parallelization` if ``func`` is CPU bound. n_trials: Number of trials. If :obj:`None`, there is no limitation on the number of trials. If ``timeout`` is also set to :obj:`None`, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM. This trades off runtime vs quality of the solution. random_state: Seed of the pseudo random number generator. If int, this is the seed used by the random number generator. If ``numpy.random.RandomState`` object, this is the random number generator. If :obj:`None`, the global random state from ``numpy.random`` is used. refit: If :obj:`True`, refit the estimator with the best found hyperparameters. The refitted estimator is made available at the ``best_estimator_`` attribute and permits using ``predict`` directly. return_train_score: If :obj:`True`, training scores will be included. Computing training scores is used to get insights on how different hyperparameter settings impact the overfitting/underfitting trade-off. However computing training scores can be computationally expensive and is not strictly required to select the hyperparameters that yield the best generalization performance. scoring: String or callable to evaluate the predictions on the validation data. If :obj:`None`, ``score`` on the estimator is used. study: Study corresponds to the optimization task. If :obj:`None`, a new study is created. subsample: Proportion of samples that are used during hyperparameter search. - If int, then draw ``subsample`` samples. - If float, then draw ``subsample`` * ``X.shape[0]`` samples. timeout: Time limit in seconds for the search of appropriate models. If :obj:`None`, the study is executed without time limitation. If ``n_trials`` is also set to :obj:`None`, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM. This trades off runtime vs quality of the solution. verbose: Verbosity level. The higher, the more messages. callbacks: List of callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. .. seealso:: See the tutorial of :ref:`optuna_callback` for how to use and implement callback functions. Attributes: best_estimator_: Estimator that was chosen by the search. This is present only if ``refit`` is set to :obj:`True`. n_splits_: Number of cross-validation splits. refit_time_: Time for refitting the best estimator. This is present only if ``refit`` is set to :obj:`True`. sample_indices_: Indices of samples that are used during hyperparameter search. scorer_: Scorer function. study_: Actual study. Examples: .. testcode:: import optuna from sklearn.datasets import load_iris from sklearn.svm import SVC clf = SVC(gamma="auto") param_distributions = { "C": optuna.distributions.FloatDistribution(1e-10, 1e10, log=True) } optuna_search = optuna.integration.OptunaSearchCV(clf, param_distributions) X, y = load_iris(return_X_y=True) optuna_search.fit(X, y) y_pred = optuna_search.predict(X) .. note:: By following the scikit-learn convention for scorers, the direction of optimization is ``maximize``. See https://scikit-learn.org/stable/modules/model_evaluation.html. For the minimization problem, please multiply ``-1``. """ _required_parameters = ["estimator", "param_distributions"] @property def _estimator_type(self) -> str: return self.estimator._estimator_type @property def best_index_(self) -> int: """Trial number which corresponds to the best candidate parameter setting. Returned value is equivalent to ``optuna_search.best_trial_.number``. """ return self.best_trial_.number @property def best_params_(self) -> dict[str, Any]: """Parameters of the best trial in the :class:`~optuna.study.Study`.""" self._check_is_fitted() return self.study_.best_params @property def best_score_(self) -> float: """Mean cross-validated score of the best estimator.""" self._check_is_fitted() return self.study_.best_value @property def best_trial_(self) -> FrozenTrial: """Best trial in the :class:`~optuna.study.Study`.""" self._check_is_fitted() return self.study_.best_trial @property def classes_(self) -> OneDimArrayLikeType: """Class labels.""" self._check_is_fitted() return self.best_estimator_.classes_ @property def cv_results_(self) -> dict[str, Any]: """A dictionary mapping a metric name to a list of Cross-Validation results of all trials.""" cv_results_dict_in_list = [trial_.user_attrs for trial_ in self.trials_] if len(cv_results_dict_in_list) == 0: cv_results_list_in_dict = {} else: cv_results_list_in_dict = { key: [dict_[key] for dict_ in cv_results_dict_in_list] for key in cv_results_dict_in_list[0] } return cv_results_list_in_dict @property def n_trials_(self) -> int: """Actual number of trials.""" return len(self.trials_) @property def trials_(self) -> list[FrozenTrial]: """All trials in the :class:`~optuna.study.Study`.""" self._check_is_fitted() return self.study_.trials @property def user_attrs_(self) -> dict[str, Any]: """User attributes in the :class:`~optuna.study.Study`.""" self._check_is_fitted() return self.study_.user_attrs @property def decision_function(self) -> Callable[..., OneDimArrayLikeType | TwoDimArrayLikeType]: """Call ``decision_function`` on the best estimator. This is available only if the underlying estimator supports ``decision_function`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.decision_function @property def inverse_transform(self) -> Callable[..., TwoDimArrayLikeType]: """Call ``inverse_transform`` on the best estimator. This is available only if the underlying estimator supports ``inverse_transform`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.inverse_transform @property def predict(self) -> Callable[..., OneDimArrayLikeType | TwoDimArrayLikeType]: """Call ``predict`` on the best estimator. This is available only if the underlying estimator supports ``predict`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.predict @property def predict_log_proba(self) -> Callable[..., TwoDimArrayLikeType]: """Call ``predict_log_proba`` on the best estimator. This is available only if the underlying estimator supports ``predict_log_proba`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.predict_log_proba @property def predict_proba(self) -> Callable[..., TwoDimArrayLikeType]: """Call ``predict_proba`` on the best estimator. This is available only if the underlying estimator supports ``predict_proba`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.predict_proba @property def score_samples(self) -> Callable[..., OneDimArrayLikeType]: """Call ``score_samples`` on the best estimator. This is available only if the underlying estimator supports ``score_samples`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.score_samples @property def set_user_attr(self) -> Callable[..., None]: """Call ``set_user_attr`` on the :class:`~optuna.study.Study`.""" self._check_is_fitted() return self.study_.set_user_attr @property def transform(self) -> Callable[..., TwoDimArrayLikeType]: """Call ``transform`` on the best estimator. This is available only if the underlying estimator supports ``transform`` and ``refit`` is set to :obj:`True`. """ self._check_is_fitted() return self.best_estimator_.transform @property def trials_dataframe(self) -> Callable[..., "pd.DataFrame"]: """Call ``trials_dataframe`` on the :class:`~optuna.study.Study`.""" self._check_is_fitted() return self.study_.trials_dataframe def __init__( self, estimator: "sklearn.base.BaseEstimator", param_distributions: Mapping[str, distributions.BaseDistribution], *, cv: int | "BaseCrossValidator" | Iterable | None = None, enable_pruning: bool = False, error_score: Number | float | str = np.nan, max_iter: int = 1000, n_jobs: int | None = None, n_trials: int | None = 10, random_state: int | np.random.RandomState | None = None, refit: bool = True, return_train_score: bool = False, scoring: Callable[..., float] | str | None = None, study: study_module.Study | None = None, subsample: float | int = 1.0, timeout: float | None = None, verbose: int = 0, callbacks: list[Callable[[study_module.Study, FrozenTrial], None]] | None = None, ) -> None: _imports.check() if not isinstance(param_distributions, dict): raise TypeError("param_distributions must be a dictionary.") # Rejecting deprecated distributions as they may cause cryptic error # when cloning OptunaSearchCV instance. # https://github.com/optuna/optuna/issues/4084 for key, dist in param_distributions.items(): if dist != _convert_old_distribution_to_new_distribution(dist): raise ValueError( f"Deprecated distribution is specified in `{key}` of param_distributions. " "Rejecting this because it may cause unexpected behavior. " "Please use new distributions such as FloatDistribution etc." ) self.cv = cv self.enable_pruning = enable_pruning self.error_score = error_score self.estimator = estimator self.max_iter = max_iter self.n_trials = n_trials self.n_jobs = n_jobs if n_jobs else 1 self.param_distributions = param_distributions self.random_state = random_state self.refit = refit self.return_train_score = return_train_score self.scoring = scoring self.study = study self.subsample = subsample self.timeout = timeout self.verbose = verbose self.callbacks = callbacks def _check_is_fitted(self) -> None: attributes = ["n_splits_", "sample_indices_", "scorer_", "study_"] if self.refit: attributes += ["best_estimator_", "refit_time_"] check_is_fitted(self, attributes) def _check_params(self) -> None: if not hasattr(self.estimator, "fit"): raise ValueError("estimator must be a scikit-learn estimator.") for name, distribution in self.param_distributions.items(): if not isinstance(distribution, distributions.BaseDistribution): raise ValueError("Value of {} must be a optuna distribution.".format(name)) if self.enable_pruning and not hasattr(self.estimator, "partial_fit"): raise ValueError("estimator must support partial_fit.") if self.max_iter <= 0: raise ValueError("max_iter must be > 0, got {}.".format(self.max_iter)) if self.study is not None and self.study.direction != StudyDirection.MAXIMIZE: raise ValueError("direction of study must be 'maximize'.") def _more_tags(self) -> dict[str, bool]: return {"non_deterministic": True, "no_validation": True} def _refit( self, X: TwoDimArrayLikeType, y: OneDimArrayLikeType | TwoDimArrayLikeType | None = None, **fit_params: Any, ) -> "OptunaSearchCV": n_samples = _num_samples(X) self.best_estimator_ = clone(self.estimator) try: self.best_estimator_.set_params(**self.study_.best_params) except ValueError as e: _logger.exception(e) _logger.info("Refitting the estimator using {} samples...".format(n_samples)) start_time = time() self.best_estimator_.fit(X, y, **fit_params) self.refit_time_ = time() - start_time _logger.info("Finished refitting! (elapsed time: {:.3f} sec.)".format(self.refit_time_)) return self def fit( self, X: TwoDimArrayLikeType, y: OneDimArrayLikeType | TwoDimArrayLikeType | None = None, groups: OneDimArrayLikeType | None = None, **fit_params: Any, ) -> "OptunaSearchCV": """Run fit with all sets of parameters. Args: X: Training data. y: Target variable. groups: Group labels for the samples used while splitting the dataset into train/validation set. **fit_params: Parameters passed to ``fit`` on the estimator. Returns: self. """ self._check_params() random_state = check_random_state(self.random_state) max_samples = self.subsample n_samples = _num_samples(X) old_level = _logger.getEffectiveLevel() if self.verbose > 1: _logger.setLevel(DEBUG) elif self.verbose > 0: _logger.setLevel(INFO) else: _logger.setLevel(WARNING) self.sample_indices_ = np.arange(n_samples) if type(max_samples) is float: max_samples = int(max_samples * n_samples) if max_samples < n_samples: self.sample_indices_ = random_state.choice( self.sample_indices_, max_samples, replace=False ) self.sample_indices_.sort() X_res = _safe_indexing(X, self.sample_indices_) y_res = _safe_indexing(y, self.sample_indices_) groups_res = _safe_indexing(groups, self.sample_indices_) fit_params_res = fit_params if fit_params_res is not None: fit_params_res = _check_fit_params(X, fit_params, self.sample_indices_) classifier = is_classifier(self.estimator) cv = check_cv(self.cv, y_res, classifier=classifier) self.n_splits_ = cv.get_n_splits(X_res, y_res, groups=groups_res) self.scorer_ = check_scoring(self.estimator, scoring=self.scoring) if self.study is None: seed = random_state.randint(0, np.iinfo("int32").max) sampler = samplers.TPESampler(seed=seed) self.study_ = study_module.create_study(direction="maximize", sampler=sampler) else: self.study_ = self.study objective = _Objective( self.estimator, self.param_distributions, X_res, y_res, cv, self.enable_pruning, self.error_score, fit_params_res, groups_res, self.max_iter, self.return_train_score, self.scorer_, ) _logger.info( "Searching the best hyperparameters using {} " "samples...".format(_num_samples(self.sample_indices_)) ) self.study_.optimize( objective, n_jobs=self.n_jobs, n_trials=self.n_trials, timeout=self.timeout, callbacks=self.callbacks, ) _logger.info("Finished hyperparameter search!") if self.refit: self._refit(X, y, **fit_params) _logger.setLevel(old_level) return self def score( self, X: TwoDimArrayLikeType, y: OneDimArrayLikeType | TwoDimArrayLikeType | None = None, ) -> float: """Return the score on the given data. Args: X: Data. y: Target variable. Returns: Scaler score. """ return self.scorer_(self.best_estimator_, X, y) optuna-3.5.0/optuna/integration/skopt.py000066400000000000000000000332711453453102400203770ustar00rootroot00000000000000import copy import random from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import Tuple import warnings import numpy as np import optuna from optuna import distributions from optuna import samplers from optuna._deprecated import deprecated_class from optuna._imports import try_import from optuna.exceptions import ExperimentalWarning from optuna.samplers import BaseSampler from optuna.search_space import IntersectionSearchSpace from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState with try_import() as _imports: import skopt from skopt.space import space @deprecated_class("3.4.0", "4.0.0") class SkoptSampler(BaseSampler): """Sampler using Scikit-Optimize as the backend. The use of :class:`~optuna.integration.SkoptSampler` is highly not recommended, as the development of Scikit-Optimize has been inactive and we have identified compatibility issues with newer NumPy versions. Args: independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for :class:`~optuna.integration.SkoptSampler` is determined by :func:`~optuna.search_space.intersection_search_space()`. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. skopt_kwargs: Keyword arguments passed to the constructor of `skopt.Optimizer `_ class. Note that ``dimensions`` argument in ``skopt_kwargs`` will be ignored because it is added by :class:`~optuna.integration.SkoptSampler` automatically. n_startup_trials: The independent sampling is used until the given number of trials finish in the same study. consider_pruned_trials: If this is :obj:`True`, the PRUNED trials are considered for sampling. .. note:: Added in v2.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.0.0. .. note:: As the number of trials :math:`n` increases, each sampling takes longer and longer on a scale of :math:`O(n^3)`. And, if this is :obj:`True`, the number of trials will increase. So, it is suggested to set this flag :obj:`False` when each evaluation of the objective function is relatively faster than each sampling. On the other hand, it is suggested to set this flag :obj:`True` when each evaluation of the objective function is relatively slower than each sampling. seed: Seed for random number generator. """ def __init__( self, independent_sampler: Optional[BaseSampler] = None, warn_independent_sampling: bool = True, skopt_kwargs: Optional[Dict[str, Any]] = None, n_startup_trials: int = 1, *, consider_pruned_trials: bool = False, seed: Optional[int] = None, ) -> None: _imports.check() self._skopt_kwargs = skopt_kwargs or {} if "dimensions" in self._skopt_kwargs: del self._skopt_kwargs["dimensions"] self._independent_sampler = independent_sampler or samplers.RandomSampler(seed=seed) self._warn_independent_sampling = warn_independent_sampling self._n_startup_trials = n_startup_trials self._search_space = IntersectionSearchSpace() self._consider_pruned_trials = consider_pruned_trials if self._consider_pruned_trials: warnings.warn( "`consider_pruned_trials` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if seed is not None and "random_state" not in self._skopt_kwargs: self._skopt_kwargs["random_state"] = seed self._rng: Optional[np.random.RandomState] = None def reseed_rng(self) -> None: self._skopt_kwargs["random_state"] = random.randint(1, np.iinfo(np.int32).max) self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, distributions.BaseDistribution]: search_space = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): if not isinstance(distribution, distributions.CategoricalDistribution): # `skopt` cannot handle non-categorical distributions that contain just # a single value, so we skip this distribution. # # Note that `Trial` takes care of this distribution during suggestion. continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, distributions.BaseDistribution], ) -> Dict[str, Any]: self._raise_error_if_multi_objective(study) if len(search_space) == 0: return {} complete_trials = self._get_trials(study) if len(complete_trials) < self._n_startup_trials: return {} optimizer = _Optimizer(search_space, self._skopt_kwargs) if self._rng is not None: optimizer._optimizer.rng = self._rng optimizer.tell(study, complete_trials) params = optimizer.ask() self._rng = optimizer._optimizer.rng return params def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: distributions.BaseDistribution, ) -> Any: self._raise_error_if_multi_objective(study) if self._warn_independent_sampling: complete_trials = self._get_trials(study) if len(complete_trials) >= self._n_startup_trials: self._log_independent_sampling(trial, param_name) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def _log_independent_sampling(self, trial: FrozenTrial, param_name: str) -> None: logger = optuna.logging.get_logger(__name__) logger.warning( "The parameter '{}' in trial#{} is sampled independently " "by using `{}` instead of `SkoptSampler` " "(optimization performance may be degraded). " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `SkoptSampler`, " "if this independent sampling is intended behavior.".format( param_name, trial.number, self._independent_sampler.__class__.__name__ ) ) def _get_trials(self, study: Study) -> List[FrozenTrial]: complete_trials = [] for t in study._get_trials(deepcopy=False, use_cache=True): if t.state == TrialState.COMPLETE: complete_trials.append(t) elif ( t.state == TrialState.PRUNED and len(t.intermediate_values) > 0 and self._consider_pruned_trials ): _, value = max(t.intermediate_values.items()) if value is None: continue # We rewrite the value of the trial `t` for sampling, so we need a deepcopy. copied_t = copy.deepcopy(t) copied_t.value = value complete_trials.append(copied_t) return complete_trials def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: self._independent_sampler.after_trial(study, trial, state, values) class _Optimizer: def __init__( self, search_space: Dict[str, distributions.BaseDistribution], skopt_kwargs: Dict[str, Any] ) -> None: self._search_space = search_space dimensions = [] for name, distribution in sorted(self._search_space.items()): if isinstance(distribution, distributions.IntDistribution): if distribution.log: low = distribution.low - 0.5 high = distribution.high + 0.5 dimension = space.Real(low, high, prior="log-uniform") else: count = (distribution.high - distribution.low) // distribution.step dimension = space.Integer(0, count) elif isinstance(distribution, distributions.CategoricalDistribution): dimension = space.Categorical(distribution.choices) elif isinstance(distribution, distributions.FloatDistribution): # Convert the upper bound from exclusive (optuna) to inclusive (skopt). if distribution.log: high = np.nextafter(distribution.high, float("-inf")) dimension = space.Real(distribution.low, high, prior="log-uniform") elif distribution.step is not None: count = int((distribution.high - distribution.low) // distribution.step) dimension = space.Integer(0, count) else: high = np.nextafter(distribution.high, float("-inf")) dimension = space.Real(distribution.low, high) else: raise NotImplementedError( "The distribution {} is not implemented.".format(distribution) ) dimensions.append(dimension) self._optimizer = skopt.Optimizer(dimensions, **skopt_kwargs) def tell(self, study: Study, complete_trials: List[FrozenTrial]) -> None: xs = [] ys = [] for trial in complete_trials: if not self._is_compatible(trial): continue x, y = self._complete_trial_to_skopt_observation(study, trial) xs.append(x) ys.append(y) self._optimizer.tell(xs, ys) def ask(self) -> Dict[str, Any]: params = {} param_values = self._optimizer.ask() for (name, distribution), value in zip(sorted(self._search_space.items()), param_values): if isinstance(distribution, distributions.FloatDistribution): # Type of value is np.floating, so cast it to Python's built-in float. value = float(value) if distribution.step is not None: value = value * distribution.step + distribution.low elif isinstance(distribution, distributions.IntDistribution): if distribution.log: value = int(np.round(value)) value = min(max(value, distribution.low), distribution.high) else: value = int(value * distribution.step + distribution.low) params[name] = value return params def _is_compatible(self, trial: FrozenTrial) -> bool: # Thanks to `intersection_search_space()` function, in sequential optimization, # the parameters of complete trials are always compatible with the search space. # # However, in distributed optimization, incompatible trials may complete on a worker # just after an intersection search space is calculated on another worker. for name, distribution in self._search_space.items(): if name not in trial.params: return False distributions.check_distribution_compatibility(distribution, trial.distributions[name]) param_value = trial.params[name] param_internal_value = distribution.to_internal_repr(param_value) if not distribution._contains(param_internal_value): return False return True def _complete_trial_to_skopt_observation( self, study: Study, trial: FrozenTrial ) -> Tuple[List[Any], float]: param_values = [] for name, distribution in sorted(self._search_space.items()): param_value = trial.params[name] if isinstance(distribution, distributions.FloatDistribution): if distribution.step is not None: param_value = (param_value - distribution.low) // distribution.step elif isinstance(distribution, distributions.IntDistribution): if not distribution.log: param_value = (param_value - distribution.low) // distribution.step param_values.append(param_value) value = trial.value assert value is not None if study.direction == StudyDirection.MAXIMIZE: value = -value return param_values, value optuna-3.5.0/optuna/integration/skorch.py000066400000000000000000000001421453453102400205170ustar00rootroot00000000000000from optuna_integration.skorch import SkorchPruningCallback __all__ = ["SkorchPruningCallback"] optuna-3.5.0/optuna/integration/tensorboard.py000066400000000000000000000116341453453102400215600ustar00rootroot00000000000000import os from typing import Dict import optuna from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.logging import get_logger with try_import() as _imports: from tensorboard.plugins.hparams import api as hp import tensorflow as tf _logger = get_logger(__name__) @experimental_class("2.0.0") class TensorBoardCallback: """Callback to track Optuna trials with TensorBoard. This callback adds relevant information that is tracked by Optuna to TensorBoard. See `the example `_. Args: dirname: Directory to store TensorBoard logs. metric_name: Name of the metric. Since the metric itself is just a number, `metric_name` can be used to give it a name. So you know later if it was roc-auc or accuracy. """ def __init__(self, dirname: str, metric_name: str) -> None: _imports.check() self._dirname = dirname self._metric_name = metric_name self._hp_params: Dict[str, hp.HParam] = {} def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None: if len(self._hp_params) == 0: self._initialization(study) if trial.state != optuna.trial.TrialState.COMPLETE: return trial_value = trial.value if trial.value is not None else float("nan") hparams = {} for param_name, param_value in trial.params.items(): if param_name not in self._hp_params: self._add_distributions(trial.distributions) param = self._hp_params[param_name] if isinstance(param.domain, hp.Discrete): hparams[param] = param.domain.dtype(param_value) else: hparams[param] = param_value run_name = "trial-%d" % trial.number run_dir = os.path.join(self._dirname, run_name) with tf.summary.create_file_writer(run_dir).as_default(): hp.hparams(hparams, trial_id=run_name) # record the values used in this trial tf.summary.scalar(self._metric_name, trial_value, step=trial.number) def _add_distributions( self, distributions: Dict[str, optuna.distributions.BaseDistribution] ) -> None: supported_distributions = ( optuna.distributions.CategoricalDistribution, optuna.distributions.FloatDistribution, optuna.distributions.IntDistribution, ) for param_name, param_distribution in distributions.items(): if isinstance(param_distribution, optuna.distributions.FloatDistribution): self._hp_params[param_name] = hp.HParam( param_name, hp.RealInterval(float(param_distribution.low), float(param_distribution.high)), ) elif isinstance(param_distribution, optuna.distributions.IntDistribution): self._hp_params[param_name] = hp.HParam( param_name, hp.IntInterval(param_distribution.low, param_distribution.high), ) elif isinstance(param_distribution, optuna.distributions.CategoricalDistribution): choices = param_distribution.choices dtype = type(choices[0]) if any(not isinstance(choice, dtype) for choice in choices): _logger.warning( "Choices contains mixed types, which is not supported by TensorBoard. " "Converting all choices to strings." ) choices = tuple(map(str, choices)) elif dtype not in (int, float, bool, str): _logger.warning( f"Choices are of type {dtype}, which is not supported by TensorBoard. " "Converting all choices to strings." ) choices = tuple(map(str, choices)) self._hp_params[param_name] = hp.HParam( param_name, hp.Discrete(choices), ) else: distribution_list = [ distribution.__name__ for distribution in supported_distributions ] raise NotImplementedError( "The distribution {} is not implemented. " "The parameter distribution should be one of the {}".format( param_distribution, distribution_list ) ) def _initialization(self, study: optuna.Study) -> None: completed_trials = [ trial for trial in study.get_trials(deepcopy=False) if trial.state == optuna.trial.TrialState.COMPLETE ] for trial in completed_trials: self._add_distributions(trial.distributions) optuna-3.5.0/optuna/integration/tensorflow.py000066400000000000000000000001461453453102400214340ustar00rootroot00000000000000from optuna_integration.tensorflow import TensorFlowPruningHook __all__ = ["TensorFlowPruningHook"] optuna-3.5.0/optuna/integration/tfkeras.py000066400000000000000000000001451453453102400206700ustar00rootroot00000000000000from optuna_integration.tfkeras import TFKerasPruningCallback __all__ = ["TFKerasPruningCallback"] optuna-3.5.0/optuna/integration/wandb.py000066400000000000000000000202151453453102400203240ustar00rootroot00000000000000import functools from typing import Any from typing import Callable from typing import Dict from typing import Optional from typing import Sequence from typing import Union import optuna from optuna._experimental import experimental_class from optuna._experimental import experimental_func from optuna._imports import try_import from optuna.study.study import ObjectiveFuncType with try_import() as _imports: import wandb @experimental_class("2.9.0") class WeightsAndBiasesCallback: """Callback to track Optuna trials with Weights & Biases. This callback enables tracking of Optuna study in Weights & Biases. The study is tracked as a single experiment run, where all suggested hyperparameters and optimized metrics are logged and plotted as a function of optimizer steps. .. note:: User needs to be logged in to Weights & Biases before using this callback in online mode. For more information, please refer to `wandb setup `_. .. note:: Users who want to run multiple Optuna studies within the same process should call ``wandb.finish()`` between subsequent calls to ``study.optimize()``. Calling ``wandb.finish()`` is not necessary if you are running one Optuna study per process. .. note:: To ensure correct trial order in Weights & Biases, this callback should only be used with ``study.optimize(n_jobs=1)``. Example: Add Weights & Biases callback to Optuna optimization. .. code:: import optuna from optuna.integration.wandb import WeightsAndBiasesCallback def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study() wandb_kwargs = {"project": "my-project"} wandbc = WeightsAndBiasesCallback(wandb_kwargs=wandb_kwargs) study.optimize(objective, n_trials=10, callbacks=[wandbc]) Weights & Biases logging in multirun mode. .. code:: import optuna from optuna.integration.wandb import WeightsAndBiasesCallback wandb_kwargs = {"project": "my-project"} wandbc = WeightsAndBiasesCallback(wandb_kwargs=wandb_kwargs, as_multirun=True) @wandbc.track_in_wandb() def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study() study.optimize(objective, n_trials=10, callbacks=[wandbc]) Args: metric_name: Name assigned to optimized metric. In case of multi-objective optimization, list of names can be passed. Those names will be assigned to metrics in the order returned by objective function. If single name is provided, or this argument is left to default value, it will be broadcasted to each objective with a number suffix in order returned by objective function e.g. two objectives and default metric name will be logged as ``value_0`` and ``value_1``. The number of metrics must be the same as the number of values objective function returns. wandb_kwargs: Set of arguments passed when initializing Weights & Biases run. Please refer to `Weights & Biases API documentation `_ for more details. as_multirun: Creates new runs for each trial. Useful for generating W&B Sweeps like panels (for ex., parameter importance, parallel coordinates, etc). """ def __init__( self, metric_name: Union[str, Sequence[str]] = "value", wandb_kwargs: Optional[Dict[str, Any]] = None, as_multirun: bool = False, ) -> None: _imports.check() if not isinstance(metric_name, Sequence): raise TypeError( "Expected metric_name to be string or sequence of strings, got {}.".format( type(metric_name) ) ) self._metric_name = metric_name self._wandb_kwargs = wandb_kwargs or {} self._as_multirun = as_multirun if not self._as_multirun: self._initialize_run() def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None: if isinstance(self._metric_name, str): if len(trial.values) > 1: # Broadcast default name for multi-objective optimization. names = ["{}_{}".format(self._metric_name, i) for i in range(len(trial.values))] else: names = [self._metric_name] else: if len(self._metric_name) != len(trial.values): raise ValueError( "Running multi-objective optimization " "with {} objective values, but {} names specified. " "Match objective values and names, or use default broadcasting.".format( len(trial.values), len(self._metric_name) ) ) else: names = [*self._metric_name] metrics = {name: value for name, value in zip(names, trial.values)} if self._as_multirun: metrics["trial_number"] = trial.number attributes = {"direction": [d.name for d in study.directions]} step = trial.number if wandb.run else None run = wandb.run # Might create extra runs if a user logs in wandb but doesn't use the decorator. if not run: run = self._initialize_run() run.name = f"trial/{trial.number}/{run.name}" run.log({**trial.params, **metrics}, step=step) if self._as_multirun: run.config.update({**attributes, **trial.params}) run.tags = tuple(self._wandb_kwargs.get("tags", ())) + (study.study_name,) run.finish() else: run.config.update(attributes) @experimental_func("3.0.0") def track_in_wandb(self) -> Callable: """Decorator for using W&B for logging inside the objective function. The run is initialized with the same ``wandb_kwargs`` that are passed to the callback. All the metrics from inside the objective function will be logged into the same run which stores the parameters for a given trial. Example: Add additional logging to Weights & Biases. .. code:: import optuna from optuna.integration.wandb import WeightsAndBiasesCallback import wandb wandb_kwargs = {"project": "my-project"} wandbc = WeightsAndBiasesCallback(wandb_kwargs=wandb_kwargs, as_multirun=True) @wandbc.track_in_wandb() def objective(trial): x = trial.suggest_float("x", -10, 10) wandb.log({"power": 2, "base of metric": x - 2}) return (x - 2) ** 2 study = optuna.create_study() study.optimize(objective, n_trials=10, callbacks=[wandbc]) Returns: Objective function with W&B tracking enabled. """ def decorator(func: ObjectiveFuncType) -> ObjectiveFuncType: @functools.wraps(func) def wrapper(trial: optuna.trial.Trial) -> Union[float, Sequence[float]]: run = wandb.run # Uses global run when `as_multirun` is set to False. if not run: run = self._initialize_run() run.name = f"trial/{trial.number}/{run.name}" return func(trial) return wrapper return decorator def _initialize_run(self) -> "wandb.sdk.wandb_run.Run": """Initializes Weights & Biases run.""" run = wandb.init(**self._wandb_kwargs) if not isinstance(run, wandb.sdk.wandb_run.Run): raise RuntimeError( "Cannot create a Run. " "Expected wandb.sdk.wandb_run.Run as a return. " f"Got: {type(run)}." ) return run optuna-3.5.0/optuna/integration/xgboost.py000066400000000000000000000114031453453102400207150ustar00rootroot00000000000000from typing import Any import optuna use_callback_cls = True with optuna._imports.try_import() as _imports: import xgboost as xgb xgboost_version = xgb.__version__.split(".") xgboost_major_version = int(xgboost_version[0]) xgboost_minor_version = int(xgboost_version[1]) use_callback_cls = ( xgboost_major_version >= 1 and xgboost_minor_version >= 3 ) or xgboost_major_version >= 2 _doc = """Callback for XGBoost to prune unpromising trials. See `the example `__ if you want to add a pruning callback which observes validation accuracy of a XGBoost model. Args: trial: A :class:`~optuna.trial.Trial` corresponding to the current evaluation of the objective function. observation_key: An evaluation metric for pruning, e.g., ``validation-error`` and ``validation-merror``. When using the Scikit-Learn API, the index number of ``eval_set`` must be included in the ``observation_key``, e.g., ``validation_0-error`` and ``validation_0-merror``. Please refer to ``eval_metric`` in `XGBoost reference `_ for further details. """ if _imports.is_successful() and use_callback_cls: class XGBoostPruningCallback(xgb.callback.TrainingCallback): __doc__ = _doc def __init__(self, trial: optuna.trial.Trial, observation_key: str) -> None: self._trial = trial self._observation_key = observation_key self._is_cv = False def before_training(self, model: Any) -> Any: # The use of Any type is due to _PackedBooster is not yet being exposed # to public interface as of xgboost 1.3. if isinstance(model, xgb.Booster): self._is_cv = False else: self._is_cv = True return model def after_iteration(self, model: Any, epoch: int, evals_log: dict) -> bool: evaluation_results = {} # Flatten the evaluation history to `{dataset-metric: score}` layout. for dataset, metrics in evals_log.items(): for metric, scores in metrics.items(): assert isinstance(scores, list), scores key = dataset + "-" + metric if self._is_cv: # Remove stddev of the metric across the cross-validation # folds. evaluation_results[key] = scores[-1][0] else: evaluation_results[key] = scores[-1] current_score = evaluation_results[self._observation_key] self._trial.report(current_score, step=epoch) if self._trial.should_prune(): message = "Trial was pruned at iteration {}.".format(epoch) raise optuna.TrialPruned(message) # The training should not stop. return False elif _imports.is_successful(): def _get_callback_context(env: "xgb.core.CallbackEnv") -> str: # type: ignore """Return whether the current callback context is cv or train. .. note:: `Reference `_. """ if env.model is None and env.cvfolds is not None: context = "cv" else: context = "train" return context class XGBoostPruningCallback: # type: ignore __doc__ = _doc def __init__(self, trial: optuna.trial.Trial, observation_key: str) -> None: self._trial = trial self._observation_key = observation_key def __call__(self, env: "xgb.core.CallbackEnv") -> None: # type: ignore context = _get_callback_context(env) evaluation_result_list = env.evaluation_result_list if context == "cv": # Remove a third element: the stddev of the metric across the # cross-validation folds. evaluation_result_list = [ (key, metric) for key, metric, _ in evaluation_result_list ] current_score = dict(evaluation_result_list)[self._observation_key] self._trial.report(current_score, step=env.iteration) if self._trial.should_prune(): message = "Trial was pruned at iteration {}.".format(env.iteration) raise optuna.TrialPruned(message) else: class XGBoostPruningCallback: # type: ignore __doc__ = _doc def __init__(self, trial: optuna.trial.Trial, observation_key: str) -> None: _imports.check() optuna-3.5.0/optuna/logging.py000066400000000000000000000243241453453102400163410ustar00rootroot00000000000000import logging from logging import CRITICAL from logging import DEBUG from logging import ERROR from logging import FATAL from logging import INFO from logging import WARN from logging import WARNING import os import sys import threading from typing import Optional import colorlog __all__ = [ "CRITICAL", "DEBUG", "ERROR", "FATAL", "INFO", "WARN", "WARNING", ] _lock: threading.Lock = threading.Lock() _default_handler: Optional[logging.Handler] = None def create_default_formatter() -> logging.Formatter: """Create a default formatter of log messages. This function is not supposed to be directly accessed by library users. """ header = "[%(levelname)1.1s %(asctime)s]" message = "%(message)s" if _color_supported(): return colorlog.ColoredFormatter( f"%(log_color)s{header}%(reset)s {message}", ) return logging.Formatter(f"{header} {message}") def _color_supported() -> bool: """Detection of color support.""" # NO_COLOR environment variable: if os.environ.get("NO_COLOR", None): return False if not hasattr(sys.stderr, "isatty") or not sys.stderr.isatty(): return False else: return True def _get_library_name() -> str: return __name__.split(".")[0] def _get_library_root_logger() -> logging.Logger: return logging.getLogger(_get_library_name()) def _configure_library_root_logger() -> None: global _default_handler with _lock: if _default_handler: # This library has already configured the library root logger. return _default_handler = logging.StreamHandler() # Set sys.stderr as stream. _default_handler.setFormatter(create_default_formatter()) # Apply our default configuration to the library root logger. library_root_logger: logging.Logger = _get_library_root_logger() library_root_logger.addHandler(_default_handler) library_root_logger.setLevel(logging.INFO) library_root_logger.propagate = False def _reset_library_root_logger() -> None: global _default_handler with _lock: if not _default_handler: return library_root_logger: logging.Logger = _get_library_root_logger() library_root_logger.removeHandler(_default_handler) library_root_logger.setLevel(logging.NOTSET) _default_handler = None def get_logger(name: str) -> logging.Logger: """Return a logger with the specified name. This function is not supposed to be directly accessed by library users. """ _configure_library_root_logger() return logging.getLogger(name) def get_verbosity() -> int: """Return the current level for the Optuna's root logger. Example: Get the default verbosity level. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna # The default verbosity level of Optuna is `optuna.logging.INFO`. print(optuna.logging.get_verbosity()) # 20 print(optuna.logging.INFO) # 20 # There are logs of the INFO level. study = optuna.create_study() study.optimize(objective, n_trials=5) # [I 2021-10-31 05:35:17,232] A new study created ... # [I 2021-10-31 05:35:17,238] Trial 0 finished with value: ... # [I 2021-10-31 05:35:17,245] Trial 1 finished with value: ... # ... .. testoutput:: :hide: 20 20 Returns: Logging level, e.g., ``optuna.logging.DEBUG`` and ``optuna.logging.INFO``. .. note:: Optuna has following logging levels: - ``optuna.logging.CRITICAL``, ``optuna.logging.FATAL`` - ``optuna.logging.ERROR`` - ``optuna.logging.WARNING``, ``optuna.logging.WARN`` - ``optuna.logging.INFO`` - ``optuna.logging.DEBUG`` """ _configure_library_root_logger() return _get_library_root_logger().getEffectiveLevel() def set_verbosity(verbosity: int) -> None: """Set the level for the Optuna's root logger. Example: Set the logging level ``optuna.logging.WARNING``. .. testsetup:: def objective(trial): x = trial.suggest_int("x", -10, 10) return x**2 .. testcode:: import optuna # There are INFO level logs. study = optuna.create_study() study.optimize(objective, n_trials=10) # [I 2021-10-31 02:59:35,088] Trial 0 finished with value: 16.0 ... # [I 2021-10-31 02:59:35,091] Trial 1 finished with value: 1.0 ... # [I 2021-10-31 02:59:35,096] Trial 2 finished with value: 1.0 ... # Setting the logging level WARNING, the INFO logs are suppressed. optuna.logging.set_verbosity(optuna.logging.WARNING) study.optimize(objective, n_trials=10) .. testcleanup:: optuna.logging.set_verbosity(optuna.logging.INFO) Args: verbosity: Logging level, e.g., ``optuna.logging.DEBUG`` and ``optuna.logging.INFO``. .. note:: Optuna has following logging levels: - ``optuna.logging.CRITICAL``, ``optuna.logging.FATAL`` - ``optuna.logging.ERROR`` - ``optuna.logging.WARNING``, ``optuna.logging.WARN`` - ``optuna.logging.INFO`` - ``optuna.logging.DEBUG`` """ _configure_library_root_logger() _get_library_root_logger().setLevel(verbosity) def disable_default_handler() -> None: """Disable the default handler of the Optuna's root logger. Example: Stop and then resume logging to :obj:`sys.stderr`. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna study = optuna.create_study() # There are no logs in sys.stderr. optuna.logging.disable_default_handler() study.optimize(objective, n_trials=10) # There are logs in sys.stderr. optuna.logging.enable_default_handler() study.optimize(objective, n_trials=10) # [I 2020-02-23 17:00:54,314] Trial 10 finished with value: ... # [I 2020-02-23 17:00:54,356] Trial 11 finished with value: ... # ... """ _configure_library_root_logger() assert _default_handler is not None _get_library_root_logger().removeHandler(_default_handler) def enable_default_handler() -> None: """Enable the default handler of the Optuna's root logger. Please refer to the example shown in :func:`~optuna.logging.disable_default_handler()`. """ _configure_library_root_logger() assert _default_handler is not None _get_library_root_logger().addHandler(_default_handler) def disable_propagation() -> None: """Disable propagation of the library log outputs. Note that log propagation is disabled by default. You only need to use this function to stop log propagation when you use :func:`~optuna.logging.enable_propagation()`. Example: Stop propagating logs to the root logger on the second optimize call. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna import logging optuna.logging.disable_default_handler() # Disable the default handler. logger = logging.getLogger() logger.setLevel(logging.INFO) # Setup the root logger. logger.addHandler(logging.FileHandler("foo.log", mode="w")) optuna.logging.enable_propagation() # Propagate logs to the root logger. study = optuna.create_study() logger.info("Logs from first optimize call") # The logs are saved in the logs file. study.optimize(objective, n_trials=10) optuna.logging.disable_propagation() # Stop propogating logs to the root logger. logger.info("Logs from second optimize call") # The new logs for second optimize call are not saved. study.optimize(objective, n_trials=10) with open("foo.log") as f: assert f.readline().startswith("A new study created") assert f.readline() == "Logs from first optimize call\\n" # Check for logs after second optimize call. assert f.read().split("Logs from second optimize call\\n")[-1] == "" """ _configure_library_root_logger() _get_library_root_logger().propagate = False def enable_propagation() -> None: """Enable propagation of the library log outputs. Please disable the Optuna's default handler to prevent double logging if the root logger has been configured. Example: Propagate all log output to the root logger in order to save them to the file. .. testsetup:: def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y .. testcode:: import optuna import logging logger = logging.getLogger() logger.setLevel(logging.INFO) # Setup the root logger. logger.addHandler(logging.FileHandler("foo.log", mode="w")) optuna.logging.enable_propagation() # Propagate logs to the root logger. optuna.logging.disable_default_handler() # Stop showing logs in sys.stderr. study = optuna.create_study() logger.info("Start optimization.") study.optimize(objective, n_trials=10) with open("foo.log") as f: assert f.readline().startswith("A new study created") assert f.readline() == "Start optimization.\\n" """ _configure_library_root_logger() _get_library_root_logger().propagate = True optuna-3.5.0/optuna/multi_objective/000077500000000000000000000000001453453102400175205ustar00rootroot00000000000000optuna-3.5.0/optuna/multi_objective/__init__.py000066400000000000000000000006651453453102400216400ustar00rootroot00000000000000from optuna._imports import _LazyImport from optuna.multi_objective import samplers from optuna.multi_objective import study from optuna.multi_objective import trial from optuna.multi_objective.study import create_study from optuna.multi_objective.study import load_study visualization = _LazyImport("optuna.multi_objective.visualization") __all__ = [ "samplers", "study", "trial", "create_study", "load_study", ] optuna-3.5.0/optuna/multi_objective/samplers/000077500000000000000000000000001453453102400213465ustar00rootroot00000000000000optuna-3.5.0/optuna/multi_objective/samplers/__init__.py000066400000000000000000000011121453453102400234520ustar00rootroot00000000000000from optuna.multi_objective.samplers._adapter import _MultiObjectiveSamplerAdapter from optuna.multi_objective.samplers._base import BaseMultiObjectiveSampler from optuna.multi_objective.samplers._motpe import MOTPEMultiObjectiveSampler from optuna.multi_objective.samplers._nsga2 import NSGAIIMultiObjectiveSampler from optuna.multi_objective.samplers._random import RandomMultiObjectiveSampler __all__ = [ "_MultiObjectiveSamplerAdapter", "BaseMultiObjectiveSampler", "MOTPEMultiObjectiveSampler", "NSGAIIMultiObjectiveSampler", "RandomMultiObjectiveSampler", ] optuna-3.5.0/optuna/multi_objective/samplers/_adapter.py000066400000000000000000000037051453453102400235040ustar00rootroot00000000000000from typing import Any from typing import Dict from optuna import multi_objective from optuna.distributions import BaseDistribution from optuna.samplers import BaseSampler from optuna.study import Study from optuna.trial import FrozenTrial class _MultiObjectiveSamplerAdapter(BaseSampler): """Adapter for to :class:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler`. This class implements the :class:`~optuna.samplers.BaseSampler` interface. When a method is invoked, the handling will be delegated to the given :class:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler` instance. """ def __init__(self, mo_sampler: "multi_objective.samplers.BaseMultiObjectiveSampler") -> None: self._mo_sampler = mo_sampler def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: mo_study = multi_objective.study.MultiObjectiveStudy(study) mo_trial = multi_objective.trial.FrozenMultiObjectiveTrial(mo_study.n_objectives, trial) return self._mo_sampler.infer_relative_search_space(mo_study, mo_trial) def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: mo_study = multi_objective.study.MultiObjectiveStudy(study) mo_trial = multi_objective.trial.FrozenMultiObjectiveTrial(mo_study.n_objectives, trial) return self._mo_sampler.sample_relative(mo_study, mo_trial, search_space) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: mo_study = multi_objective.study.MultiObjectiveStudy(study) mo_trial = multi_objective.trial.FrozenMultiObjectiveTrial(mo_study.n_objectives, trial) return self._mo_sampler.sample_independent( mo_study, mo_trial, param_name, param_distribution ) optuna-3.5.0/optuna/multi_objective/samplers/_base.py000066400000000000000000000105621453453102400227750ustar00rootroot00000000000000import abc from typing import Any from typing import Dict from optuna import multi_objective from optuna._deprecated import deprecated_class from optuna.distributions import BaseDistribution @deprecated_class("2.4.0", "4.0.0") class BaseMultiObjectiveSampler(abc.ABC): """Base class for multi-objective samplers. The abstract methods of this class are the same as ones defined by :class:`~optuna.samplers.BaseSampler` except for taking multi-objective versions of study and trial instances as the arguments. """ @abc.abstractmethod def infer_relative_search_space( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", ) -> Dict[str, BaseDistribution]: """Infer the search space that will be used by relative sampling in the target trial. This method is called right before :func:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler.sample_relative` method, and the search space returned by this method is passed to it. The parameters not contained in the search space will be sampled by using :func:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler.sample_independent` method. Args: study: Target study object. trial: Target trial object. Returns: A dictionary containing the parameter names and parameter's distributions. .. seealso:: Please refer to :func:`~optuna.search_space.intersection_search_space` as an implementation of :func:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler.infer_relative_search_space`. """ raise NotImplementedError @abc.abstractmethod def sample_relative( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: """Sample parameters in a given search space. This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use the relationship between parameters. Args: study: Target study object. trial: Target trial object. search_space: The search space returned by :func:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler.infer_relative_search_space`. Returns: A dictionary containing the parameter names and the values. """ raise NotImplementedError @abc.abstractmethod def sample_independent( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: """Sample a parameter for a given distribution. This method is called only for the parameters not contained in the search space returned by :func:`~optuna.multi_objective.samplers.MultiObjectiveBaseSampler.sample_relative` method. This method is suitable for sampling algorithms that do not use the relationship between parameters such as random sampling. Args: study: Target study object. trial: Target trial object. param_name: Name of the sampled parameter. param_distribution: Distribution object that specifies a prior and/or scale of the sampling algorithm. Returns: A parameter value. """ raise NotImplementedError def reseed_rng(self) -> None: """Reseed sampler's random number generator. This method is called by the :class:`~optuna.multi_objective.study.MultiObjectiveStudy` instance if trials are executed in parallel with the option ``n_jobs>1``. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator. """ pass optuna-3.5.0/optuna/multi_objective/samplers/_motpe.py000066400000000000000000000202541453453102400232060ustar00rootroot00000000000000from typing import Any from typing import Callable from typing import Dict from typing import Optional import warnings import numpy as np import optuna from optuna import multi_objective from optuna._deprecated import deprecated_class from optuna.distributions import BaseDistribution from optuna.exceptions import ExperimentalWarning from optuna.multi_objective.samplers import _MultiObjectiveSamplerAdapter from optuna.multi_objective.samplers import BaseMultiObjectiveSampler from optuna.pruners import NopPruner from optuna.samplers import MOTPESampler from optuna.samplers._tpe.multi_objective_sampler import _default_weights_above from optuna.samplers._tpe.multi_objective_sampler import default_gamma from optuna.study import create_study from optuna.trial import create_trial from optuna.trial import FrozenTrial @deprecated_class("2.4.0", "4.0.0") class MOTPEMultiObjectiveSampler(BaseMultiObjectiveSampler): """Multi-objective sampler using the MOTPE algorithm. This sampler is a multi-objective version of :class:`~optuna.samplers.TPESampler`. .. note:: For `v2.4.0 `_ or later, :class:`~optuna.multi_objective.samplers.MOTPEMultiObjectiveSampler` is deprecated and :class:`~optuna.samplers.TPESampler` should be used instead. The following code shows how you apply :class:`~optuna.samplers.TPESampler` to a multi-objective task: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) f1 = x**2 + y f2 = -((x - 2) ** 2 + y) return f1, f2 # We minimize the first objective and maximize the second objective. sampler = optuna.samplers.TPESampler() study = optuna.create_study(directions=["minimize", "maximize"], sampler=sampler) study.optimize(objective, n_trials=100) For further information about MOTPE algorithm, please refer to the following papers: - `Multiobjective tree-structured parzen estimator for computationally expensive optimization problems `_ - `Multiobjective Tree-Structured Parzen Estimator `_ Args: consider_prior: Enhance the stability of Parzen estimator by imposing a Gaussian prior when :obj:`True`. The prior is only effective if the sampling distribution is either :class:`~optuna.distributions.FloatDistribution`, or :class:`~optuna.distributions.IntDistribution`. prior_weight: The weight of the prior. This argument is used in :class:`~optuna.distributions.FloatDistribution`, :class:`~optuna.distributions.IntDistribution`, and :class:`~optuna.distributions.CategoricalDistribution`. consider_magic_clip: Enable a heuristic to limit the smallest variances of Gaussians used in the Parzen estimator. consider_endpoints: Take endpoints of domains into account when calculating variances of Gaussians in Parzen estimator. See the original paper for details on the heuristics to calculate the variances. n_startup_trials: The random sampling is used instead of the MOTPE algorithm until the given number of trials finish in the same study. 11 * number of variables - 1 is recommended in the original paper. n_ehvi_candidates: Number of candidate samples used to calculate the expected hypervolume improvement. gamma: A function that takes the number of finished trials and returns the number of trials to form a density function for samples with low grains. See the original paper for more details. weights_above: A function that takes the number of finished trials and returns a weight for them. As default, weights are automatically calculated by the MOTPE's default strategy. seed: Seed for random number generator. .. note:: Initialization with Latin hypercube sampling may improve optimization performance. However, the current implementation only supports initialization with random sampling. Example: .. testcode:: import optuna seed = 128 num_variables = 9 n_startup_trials = 11 * num_variables - 1 def objective(trial): x = [] for i in range(1, num_variables + 1): x.append(trial.suggest_float(f"x{i}", 0.0, 2.0 * i)) return x sampler = optuna.multi_objective.samplers.MOTPEMultiObjectiveSampler( n_startup_trials=n_startup_trials, n_ehvi_candidates=24, seed=seed ) study = optuna.multi_objective.create_study( ["minimize"] * num_variables, sampler=sampler ) study.optimize(objective, n_trials=250) """ def __init__( self, consider_prior: bool = True, prior_weight: float = 1.0, consider_magic_clip: bool = True, consider_endpoints: bool = True, n_startup_trials: int = 10, n_ehvi_candidates: int = 24, gamma: Callable[[int], int] = default_gamma, weights_above: Callable[[int], np.ndarray] = _default_weights_above, seed: Optional[int] = None, ) -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", ExperimentalWarning) self._motpe_sampler = MOTPESampler( consider_prior=consider_prior, prior_weight=prior_weight, consider_magic_clip=consider_magic_clip, consider_endpoints=consider_endpoints, n_startup_trials=n_startup_trials, n_ehvi_candidates=n_ehvi_candidates, gamma=gamma, weights_above=weights_above, seed=seed, ) def reseed_rng(self) -> None: self._motpe_sampler.reseed_rng() def infer_relative_search_space( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", ) -> Dict[str, BaseDistribution]: return {} def sample_relative( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: return {} def sample_independent( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: return self._motpe_sampler.sample_independent( _create_study(study), _create_trial(trial), param_name, param_distribution ) def _create_study(mo_study: "multi_objective.study.MultiObjectiveStudy") -> "optuna.Study": study = create_study( storage=mo_study._storage, sampler=_MultiObjectiveSamplerAdapter(mo_study.sampler), pruner=NopPruner(), study_name="_motpe-" + mo_study._storage.get_study_name_from_id(mo_study._study_id), directions=mo_study.directions, load_if_exists=True, ) for mo_trial in mo_study.trials: with warnings.catch_warnings(): warnings.simplefilter("ignore", ExperimentalWarning) study.add_trial(_create_trial(mo_trial)) return study def _create_trial(mo_trial: "multi_objective.trial.FrozenMultiObjectiveTrial") -> FrozenTrial: with warnings.catch_warnings(): warnings.simplefilter("ignore", ExperimentalWarning) trial = create_trial( state=mo_trial.state, values=mo_trial.values, # type: ignore params=mo_trial.params, distributions=mo_trial.distributions, user_attrs=mo_trial.user_attrs, system_attrs=mo_trial.system_attrs, ) return trial optuna-3.5.0/optuna/multi_objective/samplers/_nsga2.py000066400000000000000000000330311453453102400230710ustar00rootroot00000000000000from collections import defaultdict import hashlib import itertools from typing import Any from typing import cast from typing import DefaultDict from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import Tuple import optuna from optuna import multi_objective from optuna._deprecated import deprecated_class from optuna.distributions import BaseDistribution from optuna.multi_objective.samplers import BaseMultiObjectiveSampler from optuna.samplers._lazy_random_state import LazyRandomState # Define key names of `Trial.system_attrs`. _GENERATION_KEY = "multi_objective:nsga2:generation" _PARENTS_KEY = "multi_objective:nsga2:parents" _POPULATION_CACHE_KEY_PREFIX = "multi_objective:nsga2:population" @deprecated_class("2.4.0", "4.0.0") class NSGAIIMultiObjectiveSampler(BaseMultiObjectiveSampler): """Multi-objective sampler using the NSGA-II algorithm. NSGA-II stands for "Nondominated Sorting Genetic Algorithm II", which is a well known, fast and elitist multi-objective genetic algorithm. For further information about NSGA-II, please refer to the following paper: - `A fast and elitist multiobjective genetic algorithm: NSGA-II `_ Args: population_size: Number of individuals (trials) in a generation. mutation_prob: Probability of mutating each parameter when creating a new individual. If :obj:`None` is specified, the value ``1.0 / len(parent_trial.params)`` is used where ``parent_trial`` is the parent trial of the target individual. crossover_prob: Probability that a crossover (parameters swapping between parents) will occur when creating a new individual. swapping_prob: Probability of swapping each parameter of the parents during crossover. seed: Seed for random number generator. """ def __init__( self, population_size: int = 50, mutation_prob: Optional[float] = None, crossover_prob: float = 0.9, swapping_prob: float = 0.5, seed: Optional[int] = None, ) -> None: # TODO(ohta): Reconsider the default value of each parameter. if not isinstance(population_size, int): raise TypeError("`population_size` must be an integer value.") if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") if not (mutation_prob is None or 0.0 <= mutation_prob <= 1.0): raise ValueError( "`mutation_prob` must be None or a float value within the range [0.0, 1.0]." ) if not (0.0 <= crossover_prob <= 1.0): raise ValueError("`crossover_prob` must be a float value within the range [0.0, 1.0].") if not (0.0 <= swapping_prob <= 1.0): raise ValueError("`swapping_prob` must be a float value within the range [0.0, 1.0].") self._population_size = population_size self._mutation_prob = mutation_prob self._crossover_prob = crossover_prob self._swapping_prob = swapping_prob self._random_sampler = multi_objective.samplers.RandomMultiObjectiveSampler(seed=seed) self._rng = LazyRandomState(seed) def reseed_rng(self) -> None: self._random_sampler.reseed_rng() self._rng.rng.seed() def infer_relative_search_space( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", ) -> Dict[str, BaseDistribution]: return {} def sample_relative( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: parent_generation, parent_population = self._collect_parent_population(study) trial_id = trial._trial_id generation = parent_generation + 1 study._storage.set_trial_system_attr(trial_id, _GENERATION_KEY, generation) if parent_generation >= 0: p0 = self._select_parent(study, parent_population) if self._rng.rng.rand() < self._crossover_prob: p1 = self._select_parent( study, [t for t in parent_population if t._trial_id != p0._trial_id] ) else: p1 = p0 study._storage.set_trial_system_attr( trial_id, _PARENTS_KEY, [p0._trial_id, p1._trial_id] ) return {} def sample_independent( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: if _PARENTS_KEY not in trial.system_attrs: return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) p0_id, p1_id = trial.system_attrs[_PARENTS_KEY] p0 = study._storage.get_trial(p0_id) p1 = study._storage.get_trial(p1_id) param = p0.params.get(param_name, None) parent_params_len = len(p0.params) if param is None or self._rng.rng.rand() < self._swapping_prob: param = p1.params.get(param_name, None) parent_params_len = len(p1.params) mutation_prob = self._mutation_prob if mutation_prob is None: mutation_prob = 1.0 / max(1.0, parent_params_len) if param is None or self._rng.rng.rand() < mutation_prob: return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) return param def _collect_parent_population( self, study: "multi_objective.study.MultiObjectiveStudy" ) -> Tuple[int, List["multi_objective.trial.FrozenMultiObjectiveTrial"]]: trials = [ multi_objective.trial.FrozenMultiObjectiveTrial(study.n_objectives, t) for t in study._storage.get_all_trials(study._study_id, deepcopy=False) ] generation_to_runnings = defaultdict(list) generation_to_population = defaultdict(list) for trial in trials: if _GENERATION_KEY not in trial.system_attrs: continue generation = trial.system_attrs[_GENERATION_KEY] if trial.state != optuna.trial.TrialState.COMPLETE: if trial.state == optuna.trial.TrialState.RUNNING: generation_to_runnings[generation].append(trial) continue generation_to_population[generation].append(trial) hasher = hashlib.sha256() parent_population: List[multi_objective.trial.FrozenMultiObjectiveTrial] = [] parent_generation = -1 while True: generation = parent_generation + 1 population = generation_to_population[generation] # Under multi-worker settings, the population size might become larger than # `self._population_size`. if len(population) < self._population_size: break # [NOTE] # It's generally safe to assume that once the above condition is satisfied, # there are no additional individuals added to the generation (i.e., the members of # the generation have been fixed). # If the number of parallel workers is huge, this assumption can be broken, but # this is a very rare case and doesn't significantly impact optimization performance. # So we can ignore the case. # The cache key is calculated based on the key of the previous generation and # the remaining running trials in the current population. # If there are no running trials, the new cache key becomes exactly the same as # the previous one, and the cached content will be overwritten. This allows us to # skip redundant cache key calculations when this method is called for the subsequent # trials. for trial in generation_to_runnings[generation]: hasher.update(bytes(str(trial.number), "utf-8")) cache_key = "{}:{}".format(_POPULATION_CACHE_KEY_PREFIX, hasher.hexdigest()) study_system_attrs = study._storage.get_study_system_attrs(study._study_id) cached_generation, cached_population_numbers = study_system_attrs.get( cache_key, (-1, []) ) if cached_generation >= generation: generation = cached_generation population = [trials[n] for n in cached_population_numbers] else: population.extend(parent_population) population = self._select_elite_population(study, population) # To reduce the number of system attribute entries, # we cache the population information only if there are no running trials # (i.e., the information of the population has been fixed). # Usually, if there are no too delayed running trials, the single entry # will be used. if len(generation_to_runnings[generation]) == 0: population_numbers = [t.number for t in population] study._storage.set_study_system_attr( study._study_id, cache_key, (generation, population_numbers) ) parent_generation = generation parent_population = population return parent_generation, parent_population def _select_elite_population( self, study: "multi_objective.study.MultiObjectiveStudy", population: List["multi_objective.trial.FrozenMultiObjectiveTrial"], ) -> List["multi_objective.trial.FrozenMultiObjectiveTrial"]: elite_population: List[multi_objective.trial.FrozenMultiObjectiveTrial] = [] population_per_rank = _fast_non_dominated_sort(population, study.directions) for population in population_per_rank: if len(elite_population) + len(population) < self._population_size: elite_population.extend(population) else: n = self._population_size - len(elite_population) _crowding_distance_sort(population) elite_population.extend(population[:n]) break return elite_population def _select_parent( self, study: "multi_objective.study.MultiObjectiveStudy", population: Sequence["multi_objective.trial.FrozenMultiObjectiveTrial"], ) -> "multi_objective.trial.FrozenMultiObjectiveTrial": # TODO(ohta): Consider to allow users to specify the number of parent candidates. population_size = len(population) candidate0 = population[self._rng.rng.choice(population_size)] candidate1 = population[self._rng.rng.choice(population_size)] # TODO(ohta): Consider crowding distance. if candidate0._dominates(candidate1, study.directions): return candidate0 else: return candidate1 def _fast_non_dominated_sort( population: List["multi_objective.trial.FrozenMultiObjectiveTrial"], directions: List[optuna.study.StudyDirection], ) -> List[List["multi_objective.trial.FrozenMultiObjectiveTrial"]]: dominated_count: DefaultDict[int, int] = defaultdict(int) dominates_list = defaultdict(list) for p, q in itertools.combinations(population, 2): if p._dominates(q, directions): dominates_list[p.number].append(q.number) dominated_count[q.number] += 1 elif q._dominates(p, directions): dominates_list[q.number].append(p.number) dominated_count[p.number] += 1 population_per_rank = [] while population: non_dominated_population = [] i = 0 while i < len(population): if dominated_count[population[i].number] == 0: individual = population[i] if i == len(population) - 1: population.pop() else: population[i] = population.pop() non_dominated_population.append(individual) else: i += 1 for x in non_dominated_population: for y in dominates_list[x.number]: dominated_count[y] -= 1 assert non_dominated_population population_per_rank.append(non_dominated_population) return population_per_rank def _crowding_distance_sort( population: List["multi_objective.trial.FrozenMultiObjectiveTrial"], ) -> None: manhattan_distances = defaultdict(float) for i in range(len(population[0].values)): population.sort(key=lambda x: cast(float, x.values[i])) v_min = population[0].values[i] v_max = population[-1].values[i] assert v_min is not None assert v_max is not None width = v_max - v_min if width == 0: continue manhattan_distances[population[0].number] = float("inf") manhattan_distances[population[-1].number] = float("inf") for j in range(1, len(population) - 1): v_high = population[j + 1].values[i] v_low = population[j - 1].values[i] assert v_high is not None assert v_low is not None manhattan_distances[population[j].number] += (v_high - v_low) / width population.sort(key=lambda x: manhattan_distances[x.number]) population.reverse() optuna-3.5.0/optuna/multi_objective/samplers/_random.py000066400000000000000000000052561453453102400233470ustar00rootroot00000000000000from typing import Any from typing import Dict from typing import Optional import optuna from optuna import multi_objective from optuna._deprecated import deprecated_class from optuna.distributions import BaseDistribution from optuna.multi_objective.samplers import BaseMultiObjectiveSampler @deprecated_class("2.4.0", "4.0.0") class RandomMultiObjectiveSampler(BaseMultiObjectiveSampler): """Multi-objective sampler using random sampling. This sampler is based on *independent sampling*. See also :class:`~optuna.multi_objective.samplers.BaseMultiObjectiveSampler` for more details of 'independent sampling'. Example: .. testcode:: import optuna from optuna.multi_objective.samplers import RandomMultiObjectiveSampler def objective(trial): x = trial.suggest_float("x", -5, 5) y = trial.suggest_float("y", -5, 5) return x**2, y + 10 study = optuna.multi_objective.create_study( ["minimize", "minimize"], sampler=RandomMultiObjectiveSampler() ) study.optimize(objective, n_trials=10) Args: seed: Seed for random number generator. """ def __init__(self, seed: Optional[int] = None) -> None: self._sampler = optuna.samplers.RandomSampler(seed=seed) def reseed_rng(self) -> None: self._sampler.reseed_rng() def infer_relative_search_space( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", ) -> Dict[str, BaseDistribution]: # TODO(ohta): Convert `study` and `trial` to single objective versions before passing. return self._sampler.infer_relative_search_space(study, trial) # type: ignore def sample_relative( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: # TODO(ohta): Convert `study` and `trial` to single objective versions before passing. return self._sampler.sample_relative(study, trial, search_space) # type: ignore def sample_independent( self, study: "multi_objective.study.MultiObjectiveStudy", trial: "multi_objective.trial.FrozenMultiObjectiveTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: # TODO(ohta): Convert `study` and `trial` to single objective versions before passing. return self._sampler.sample_independent( study, trial, param_name, param_distribution # type: ignore ) optuna-3.5.0/optuna/multi_objective/study.py000066400000000000000000000422221453453102400212440ustar00rootroot00000000000000import types from typing import Any from typing import Callable from typing import Container from typing import Dict from typing import Iterable from typing import List from typing import Optional from typing import Sequence from typing import Tuple from typing import Type from typing import Union from optuna import logging from optuna import multi_objective from optuna._deprecated import deprecated_class from optuna._deprecated import deprecated_func from optuna.pruners import NopPruner from optuna.storages import BaseStorage from optuna.study import create_study as _create_study from optuna.study import load_study as _load_study from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial import TrialState ObjectiveFuncType = Callable[["multi_objective.trial.MultiObjectiveTrial"], Sequence[float]] CallbackFuncType = Callable[ [ "multi_objective.study.MultiObjectiveStudy", "multi_objective.trial.FrozenMultiObjectiveTrial", ], None, ] _logger = logging.get_logger(__name__) # TODO(ohta): Reconsider the API design. # See https://github.com/optuna/optuna/pull/1054/files#r407255282 for the detail. # # TODO(ohta): Consider to add `objective_labels` argument. # See: https://github.com/optuna/optuna/pull/1054#issuecomment-616382152 @deprecated_func("2.4.0", "4.0.0") def create_study( directions: List[str], study_name: Optional[str] = None, storage: Optional[Union[str, BaseStorage]] = None, sampler: Optional["multi_objective.samplers.BaseMultiObjectiveSampler"] = None, load_if_exists: bool = False, ) -> "multi_objective.study.MultiObjectiveStudy": """Create a new :class:`~optuna.multi_objective.study.MultiObjectiveStudy`. Example: .. testcode:: import optuna def objective(trial): # Binh and Korn function. x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x**2 + 4 * y**2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.multi_objective.create_study(["minimize", "minimize"]) study.optimize(objective, n_trials=3) Args: directions: Optimization direction for each objective value. Set ``minimize`` for minimization and ``maximize`` for maximization. study_name: Study's name. If this argument is set to None, a unique name is generated automatically. storage: Database URL. If this argument is set to None, in-memory storage is used, and the :class:`~optuna.study.Study` will not be persistent. .. note:: When a database URL is passed, Optuna internally uses `SQLAlchemy`_ to handle the database. Please refer to `SQLAlchemy's document`_ for further details. If you want to specify non-default options to `SQLAlchemy Engine`_, you can instantiate :class:`~optuna.storages.RDBStorage` with your desired options and pass it to the ``storage`` argument instead of a URL. .. _SQLAlchemy: https://www.sqlalchemy.org/ .. _SQLAlchemy's document: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls .. _SQLAlchemy Engine: https://docs.sqlalchemy.org/en/latest/core/engines.html sampler: A sampler object that implements background algorithm for value suggestion. If :obj:`None` is specified, :class:`~optuna.multi_objective.samplers.NSGAIIMultiObjectiveSampler` is used as the default. See also :class:`~optuna.multi_objective.samplers`. load_if_exists: Flag to control the behavior to handle a conflict of study names. In the case where a study named ``study_name`` already exists in the ``storage``, a :class:`~optuna.exceptions.DuplicatedStudyError` is raised if ``load_if_exists`` is set to :obj:`False`. Otherwise, the creation of the study is skipped, and the existing one is returned. Returns: A :class:`~optuna.multi_objective.study.MultiObjectiveStudy` object. """ # TODO(ohta): Support pruner. mo_sampler = sampler or multi_objective.samplers.NSGAIIMultiObjectiveSampler() sampler_adapter = multi_objective.samplers._MultiObjectiveSamplerAdapter(mo_sampler) if not isinstance(directions, Iterable): raise TypeError("`directions` must be a list or other iterable types.") if not all(d in ["minimize", "maximize"] for d in directions): raise ValueError("`directions` includes unknown direction names.") study = _create_study( study_name=study_name, storage=storage, sampler=sampler_adapter, pruner=NopPruner(), load_if_exists=load_if_exists, ) study._storage.set_study_system_attr( study._study_id, "multi_objective:study:directions", list(directions) ) return MultiObjectiveStudy(study) @deprecated_func("2.4.0", "4.0.0") def load_study( study_name: str, storage: Union[str, BaseStorage], sampler: Optional["multi_objective.samplers.BaseMultiObjectiveSampler"] = None, ) -> "multi_objective.study.MultiObjectiveStudy": """Load the existing :class:`MultiObjectiveStudy` that has the specified name. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): # Binh and Korn function. x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x**2 + 4 * y**2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.multi_objective.create_study( directions=["minimize", "minimize"], study_name="my_study", storage="sqlite:///example.db", ) study.optimize(objective, n_trials=3) loaded_study = optuna.multi_objective.study.load_study( study_name="my_study", storage="sqlite:///example.db" ) assert len(loaded_study.trials) == len(study.trials) .. testcleanup:: os.remove("example.db") Args: study_name: Study's name. Each study has a unique name as an identifier. storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.multi_objective.study.create_study` for further details. sampler: A sampler object that implements background algorithm for value suggestion. If :obj:`None` is specified, :class:`~optuna.multi_objective.samplers.RandomMultiObjectiveSampler` is used as the default. See also :class:`~optuna.multi_objective.samplers`. Returns: A :class:`~optuna.multi_objective.study.MultiObjectiveStudy` object. """ mo_sampler = sampler or multi_objective.samplers.RandomMultiObjectiveSampler() sampler_adapter = multi_objective.samplers._MultiObjectiveSamplerAdapter(mo_sampler) study = _load_study(study_name=study_name, storage=storage, sampler=sampler_adapter) return MultiObjectiveStudy(study) @deprecated_class("2.4.0", "4.0.0") class MultiObjectiveStudy: """A study corresponds to a multi-objective optimization task, i.e., a set of trials. This object provides interfaces to run a new :class:`~optuna.multi_objective.trial.Trial`, access trials' history, set/get user-defined attributes of the study itself. Note that the direct use of this constructor is not recommended. To create and load a study, please refer to the documentation of :func:`~optuna.multi_objective.study.create_study` and :func:`~optuna.multi_objective.study.load_study` respectively. """ def __init__(self, study: Study): self._study = study self._directions = [] for d in study._storage.get_study_system_attrs(study._study_id)[ "multi_objective:study:directions" ]: if d == "minimize": self._directions.append(StudyDirection.MINIMIZE) elif d == "maximize": self._directions.append(StudyDirection.MAXIMIZE) else: raise ValueError("Unknown direction ({}) is specified.".format(d)) n_objectives = len(self._directions) if n_objectives < 1: raise ValueError("The number of objectives must be greater than 0.") self._study._log_completed_trial = types.MethodType( # type: ignore _log_completed_trial, self._study ) @property def n_objectives(self) -> int: """Return the number of objectives. Returns: Number of objectives. """ return len(self._directions) @property def directions(self) -> List[StudyDirection]: """Return the optimization direction list. Returns: A list that contains the optimization direction for each objective value. """ return self._directions @property def sampler(self) -> "multi_objective.samplers.BaseMultiObjectiveSampler": """Return the sampler. Returns: A :class:`~multi_objective.samplers.BaseMultiObjectiveSampler` object. """ adapter = self._study.sampler assert isinstance(adapter, multi_objective.samplers._MultiObjectiveSamplerAdapter) return adapter._mo_sampler def optimize( self, objective: ObjectiveFuncType, timeout: Optional[int] = None, n_trials: Optional[int] = None, n_jobs: int = 1, catch: Tuple[Type[Exception], ...] = (), callbacks: Optional[List[CallbackFuncType]] = None, gc_after_trial: bool = True, show_progress_bar: bool = False, ) -> None: """Optimize an objective function. This method is the same as :func:`optuna.study.Study.optimize` except for taking an objective function that returns multi-objective values as the argument. Please refer to the documentation of :func:`optuna.study.Study.optimize` for further details. Example: .. testcode:: import optuna def objective(trial): # Binh and Korn function. x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x**2 + 4 * y**2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.multi_objective.create_study(["minimize", "minimize"]) study.optimize(objective, n_trials=3) """ def mo_objective(trial: Trial) -> float: mo_trial = multi_objective.trial.MultiObjectiveTrial(trial) values = objective(mo_trial) mo_trial._report_complete_values(values) return 0.0 # Dummy value. # Wraps a multi-objective callback so that we can pass it to the `Study.optimize` method. def wrap_mo_callback(callback: CallbackFuncType) -> Callable[[Study, FrozenTrial], None]: return lambda study, trial: callback( MultiObjectiveStudy(study), multi_objective.trial.FrozenMultiObjectiveTrial(self.n_objectives, trial), ) if callbacks is None: wrapped_callbacks = None else: wrapped_callbacks = [wrap_mo_callback(callback) for callback in callbacks] self._study.optimize( mo_objective, timeout=timeout, n_trials=n_trials, n_jobs=n_jobs, catch=catch, callbacks=wrapped_callbacks, gc_after_trial=gc_after_trial, show_progress_bar=show_progress_bar, ) @property def user_attrs(self) -> Dict[str, Any]: """Return user attributes. Returns: A dictionary containing all user attributes. """ return self._study.user_attrs @property def system_attrs(self) -> Dict[str, Any]: """Return system attributes. Returns: A dictionary containing all system attributes. """ return self._study._storage.get_study_system_attrs(self._study._study_id) def set_user_attr(self, key: str, value: Any) -> None: """Set a user attribute to the study. Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self._study.set_user_attr(key, value) def set_system_attr(self, key: str, value: Any) -> None: """Set a system attribute to the study. Note that Optuna internally uses this method to save system messages. Please use :func:`~optuna.multi_objective.study.MultiObjectiveStudy.set_user_attr` to set users' attributes. Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self._study._storage.set_study_system_attr(self._study._study_id, key, value) def enqueue_trial(self, params: Dict[str, Any]) -> None: """Enqueue a trial with given parameter values. You can fix the next sampling parameters which will be evaluated in your objective function. Please refer to the documentation of :func:`optuna.study.Study.enqueue_trial` for further details. Args: params: Parameter values to pass your objective function. """ self._study.enqueue_trial(params, skip_if_exists=False) @property def trials(self) -> List["multi_objective.trial.FrozenMultiObjectiveTrial"]: """Return all trials in the study. The returned trials are ordered by trial number. This is a short form of ``self.get_trials(deepcopy=True, states=None)``. Returns: A list of :class:`~optuna.multi_objective.trial.FrozenMultiObjectiveTrial` objects. """ return self.get_trials(deepcopy=True, states=None) def get_trials( self, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List["multi_objective.trial.FrozenMultiObjectiveTrial"]: """Return all trials in the study. The returned trials are ordered by trial number. Args: deepcopy: Flag to control whether to apply ``copy.deepcopy()`` to the trials. Note that if you set the flag to :obj:`False`, you shouldn't mutate any fields of the returned trial. Otherwise the internal state of the study may corrupt and unexpected behavior may happen. states: Trial states to filter on. If :obj:`None`, include all states. Returns: A list of :class:`~optuna.multi_objective.trial.FrozenMultiObjectiveTrial` objects. """ return [ multi_objective.trial.FrozenMultiObjectiveTrial(self.n_objectives, t) for t in self._study.get_trials(deepcopy=deepcopy, states=states) ] def get_pareto_front_trials(self) -> List["multi_objective.trial.FrozenMultiObjectiveTrial"]: """Return trials located at the pareto front in the study. A trial is located at the pareto front if there are no trials that dominate the trial. It's called that a trial ``t0`` dominates another trial ``t1`` if ``all(v0 <= v1) for v0, v1 in zip(t0.values, t1.values)`` and ``any(v0 < v1) for v0, v1 in zip(t0.values, t1.values)`` are held. Returns: A list of :class:`~optuna.multi_objective.trial.FrozenMultiObjectiveTrial` objects. """ pareto_front = [] trials = [t for t in self.trials if t.state == TrialState.COMPLETE] # TODO(ohta): Optimize (use the fast non dominated sort defined in the NSGA-II paper). for trial in trials: dominated = False for other in trials: if other._dominates(trial, self.directions): dominated = True break if not dominated: pareto_front.append(trial) return pareto_front @property def _storage(self) -> BaseStorage: return self._study._storage @property def _study_id(self) -> int: return self._study._study_id def _log_completed_trial(self: Study, trial: FrozenTrial) -> None: if not _logger.isEnabledFor(logging.INFO): return n_objectives = len(self.directions) frozen_multi_objective_trial = multi_objective.trial.FrozenMultiObjectiveTrial( n_objectives, trial, ) actual_values = frozen_multi_objective_trial.values _logger.info( "Trial {} finished with values: {} with parameters: {}.".format( trial.number, actual_values, trial.params ) ) optuna-3.5.0/optuna/multi_objective/trial.py000066400000000000000000000331241453453102400212100ustar00rootroot00000000000000from datetime import datetime from typing import Any from typing import Dict from typing import List from typing import Optional from typing import overload from typing import Sequence from typing import Union from optuna import multi_objective from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_class from optuna.distributions import BaseDistribution from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial import TrialState from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS CategoricalChoiceType = Union[None, bool, int, float, str] @deprecated_class("2.4.0", "4.0.0") class MultiObjectiveTrial: """A trial is a process of evaluating an objective function. This object is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial's state, and set/get user-defined attributes of the trial. Note that the direct use of this constructor is not recommended. This object is seamlessly instantiated and passed to the objective function behind the :func:`optuna.multi_objective.study.MultiObjectiveStudy.optimize()` method; hence library users do not care about instantiation of this object. Args: trial: A :class:`~optuna.trial.Trial` object. """ def __init__(self, trial: Trial): self._trial = trial # TODO(ohta): Optimize the code below to eliminate the `MultiObjectiveStudy` construction. # See also: https://github.com/optuna/optuna/pull/1054/files#r407982636 self._n_objectives = multi_objective.study.MultiObjectiveStudy(trial.study).n_objectives def suggest_float( self, name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False, ) -> float: """Suggest a value for the floating point parameter. Please refer to the documentation of :func:`optuna.trial.Trial.suggest_float` for further details. """ return self._trial.suggest_float(name, low, high, step=step, log=log) def suggest_uniform(self, name: str, low: float, high: float) -> float: """Suggest a value for the continuous parameter. Please refer to the documentation of :func:`optuna.trial.Trial.suggest_uniform` for further details. """ return self._trial.suggest_uniform(name, low, high) def suggest_loguniform(self, name: str, low: float, high: float) -> float: """Suggest a value for the continuous parameter. Please refer to the documentation of :func:`optuna.trial.Trial.suggest_loguniform` for further details. """ return self._trial.suggest_loguniform(name, low, high) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: """Suggest a value for the discrete parameter. Please refer to the documentation of :func:`optuna.trial.Trial.suggest_discrete_uniform` for further details. """ return self._trial.suggest_discrete_uniform(name, low, high, q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: """Suggest a value for the integer parameter. Please refer to the documentation of :func:`optuna.trial.Trial.suggest_int` for further details. """ return self._trial.suggest_int(name, low, high, step=step, log=log) @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: """Suggest a value for the categorical parameter. Please refer to the documentation of :func:`optuna.trial.Trial.suggest_categorical` for further details. """ return self._trial.suggest_categorical(name, choices) def report(self, values: Sequence[float], step: int) -> None: """Report intermediate objective function values for a given step. The reported values are used by the pruners to determine whether this trial should be pruned. .. seealso:: Please refer to :class:`~optuna.pruners.BasePruner`. .. note:: The reported values are converted to ``float`` type by applying ``float()`` function internally. Thus, it accepts all float-like types (e.g., ``numpy.float32``). If the conversion fails, a ``TypeError`` is raised. Args: values: Intermediate objective function values for a given step. step: Step of the trial (e.g., Epoch of neural network training). """ # TODO(ohta): Allow users reporting a subset of target values. # See https://github.com/optuna/optuna/pull/1054/files#r401594785 for the detail. if len(values) != self._n_objectives: raise ValueError( "The number of the intermediate values {} at step {} is mismatched with" "the number of the objectives {}.", len(values), step, self._n_objectives, ) for i, value in enumerate(values): self._trial.report(value, self._n_objectives * (step + 1) + i) def _report_complete_values(self, values: Sequence[float]) -> None: if len(values) != self._n_objectives: raise ValueError( "The number of the values {} is mismatched with the number of the objectives {}.", len(values), self._n_objectives, ) for i, value in enumerate(values): self._trial.report(value, i) def set_user_attr(self, key: str, value: Any) -> None: """Set user attributes to the trial. Please refer to the documentation of :func:`optuna.trial.Trial.set_user_attr` for further details. """ self._trial.set_user_attr(key, value) def set_system_attr(self, key: str, value: Any) -> None: """Set system attributes to the trial. Please refer to the documentation of :func:`optuna.trial.Trial.set_system_attr` for further details. """ self._trial.storage.set_trial_system_attr(self._trial._trial_id, key, value) @property def number(self) -> int: """Return trial's number which is consecutive and unique in a study. Returns: A trial number. """ return self._trial.number @property def params(self) -> Dict[str, Any]: """Return parameters to be optimized. Returns: A dictionary containing all parameters. """ return self._trial.params @property def distributions(self) -> Dict[str, BaseDistribution]: """Return distributions of parameters to be optimized. Returns: A dictionary containing all distributions. """ return self._trial.distributions @property def user_attrs(self) -> Dict[str, Any]: """Return user attributes. Returns: A dictionary containing all user attributes. """ return self._trial.user_attrs @property def system_attrs(self) -> Dict[str, Any]: """Return system attributes. Returns: A dictionary containing all system attributes. """ return self._trial.system_attrs @property def datetime_start(self) -> Optional[datetime]: """Return start datetime. Returns: Datetime where the :class:`~optuna.trial.Trial` started. """ return self._trial.datetime_start # TODO(ohta): Add `to_single_objective` method. # This method would be helpful to use the existing pruning # integrations for multi-objective optimization. @deprecated_class("2.4.0", "4.0.0") class FrozenMultiObjectiveTrial: """Status and results of a :class:`~optuna.multi_objective.trial.MultiObjectiveTrial`. Attributes: number: Unique and consecutive number of :class:`~optuna.multi_objective.trial.MultiObjectiveTrial` for each :class:`~optuna.multi_objective.study.MultiObjectiveStudy`. Note that this field uses zero-based numbering. state: :class:`~optuna.trial.TrialState` of the :class:`~optuna.multi_objective.trial.MultiObjectiveTrial`. values: Objective values of the :class:`~optuna.multi_objective.trial.MultiObjectiveTrial`. datetime_start: Datetime where the :class:`~optuna.multi_objective.trial.MultiObjectiveTrial` started. datetime_complete: Datetime where the :class:`~optuna.multi_objective.trial.MultiObjectiveTrial` finished. params: Dictionary that contains suggested parameters. distributions: Dictionary that contains the distributions of :attr:`params`. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.multi_objective.trial.MultiObjectiveTrial` set with :func:`optuna.multi_objective.trial.MultiObjectiveTrial.set_user_attr`. intermediate_values: Intermediate objective values set with :func:`optuna.multi_objective.trial.MultiObjectiveTrial.report`. """ def __init__(self, n_objectives: int, trial: FrozenTrial): self.n_objectives = n_objectives self._trial = trial self.values = tuple(trial.intermediate_values.get(i) for i in range(n_objectives)) intermediate_values: Dict[int, List[Optional[float]]] = {} for key, value in trial.intermediate_values.items(): if key < n_objectives: continue step = key // n_objectives - 1 if step not in intermediate_values: intermediate_values[step] = [None for _ in range(n_objectives)] intermediate_values[step][key % n_objectives] = value self.intermediate_values = {k: tuple(v) for k, v in intermediate_values.items()} @property def number(self) -> int: return self._trial.number @property def _trial_id(self) -> int: return self._trial._trial_id @property def state(self) -> TrialState: return self._trial.state @property def datetime_start(self) -> Optional[datetime]: return self._trial.datetime_start @property def datetime_complete(self) -> Optional[datetime]: return self._trial.datetime_complete @property def params(self) -> Dict[str, Any]: return self._trial.params @property def user_attrs(self) -> Dict[str, Any]: return self._trial.user_attrs @property def system_attrs(self) -> Dict[str, Any]: return self._trial.system_attrs @property def last_step(self) -> Optional[int]: if len(self.intermediate_values) == 0: return None else: return max(self.intermediate_values.keys()) @property def distributions(self) -> Dict[str, BaseDistribution]: return self._trial.distributions def _dominates( self, other: "multi_objective.trial.FrozenMultiObjectiveTrial", directions: List[StudyDirection], ) -> bool: if len(self.values) != len(other.values): raise ValueError("Trials with different numbers of objectives cannot be compared.") if len(self.values) != len(directions): raise ValueError( "The number of the values and the number of the objectives are mismatched." ) if self.state != TrialState.COMPLETE: return False if other.state != TrialState.COMPLETE: return True values0 = [_normalize_value(v, d) for v, d in zip(self.values, directions)] values1 = [_normalize_value(v, d) for v, d in zip(other.values, directions)] if values0 == values1: return False return all(v0 <= v1 for v0, v1 in zip(values0, values1)) def __eq__(self, other: Any) -> bool: if not isinstance(other, FrozenMultiObjectiveTrial): return NotImplemented return self._trial == other._trial def __lt__(self, other: Any) -> bool: if not isinstance(other, FrozenMultiObjectiveTrial): return NotImplemented return self._trial < other._trial def __le__(self, other: Any) -> bool: if not isinstance(other, FrozenMultiObjectiveTrial): return NotImplemented return self._trial <= other._trial def __hash__(self) -> int: return hash(self._trial) # TODO(ohta): Implement `__repr__` method. def _normalize_value(value: Optional[float], direction: StudyDirection) -> float: if value is None: value = float("inf") if direction is StudyDirection.MAXIMIZE: value = -value return value optuna-3.5.0/optuna/multi_objective/visualization/000077500000000000000000000000001453453102400224215ustar00rootroot00000000000000optuna-3.5.0/optuna/multi_objective/visualization/__init__.py000066400000000000000000000002611453453102400245310ustar00rootroot00000000000000from optuna.multi_objective.visualization._pareto_front import plot_pareto_front from optuna.visualization import is_available __all__ = ["plot_pareto_front", "is_available"] optuna-3.5.0/optuna/multi_objective/visualization/_pareto_front.py000066400000000000000000000163001453453102400256340ustar00rootroot00000000000000import json from typing import List from typing import Optional import optuna from optuna import multi_objective from optuna._deprecated import deprecated_func from optuna.multi_objective.study import MultiObjectiveStudy from optuna.multi_objective.trial import FrozenMultiObjectiveTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = optuna.logging.get_logger(__name__) @deprecated_func("2.4.0", "4.0.0") def plot_pareto_front( study: MultiObjectiveStudy, names: Optional[List[str]] = None, include_dominated_trials: bool = False, axis_order: Optional[List[int]] = None, ) -> "go.Figure": """Plot the pareto front of a study. Example: The following code snippet shows how to plot the pareto front of a study. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.multi_objective.create_study(["minimize", "minimize"]) study.optimize(objective, n_trials=50) fig = optuna.multi_objective.visualization.plot_pareto_front(study) fig.show() Args: study: A :class:`~optuna.multi_objective.study.MultiObjectiveStudy` object whose trials are plotted for their objective values. ``study.n_objectives`` must be eigher 2 or 3. names: Objective name list used as the axis titles. If :obj:`None` is specified, "Objective {objective_index}" is used instead. include_dominated_trials: A flag to include all dominated trial's objective values. axis_order: A list of indices indicating the axis order. If :obj:`None` is specified, default order is used. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() if study.n_objectives == 2: return _get_pareto_front_2d(study, names, include_dominated_trials, axis_order) elif study.n_objectives == 3: return _get_pareto_front_3d(study, names, include_dominated_trials, axis_order) else: raise ValueError("`plot_pareto_front` function only supports 2 or 3 objective studies.") def _get_non_pareto_front_trials( study: MultiObjectiveStudy, pareto_trials: List["multi_objective.trial.FrozenMultiObjectiveTrial"], ) -> List["multi_objective.trial.FrozenMultiObjectiveTrial"]: non_pareto_trials = [] for trial in study.get_trials(): if trial.state == TrialState.COMPLETE and trial not in pareto_trials: non_pareto_trials.append(trial) return non_pareto_trials def _get_pareto_front_2d( study: MultiObjectiveStudy, names: Optional[List[str]], include_dominated_trials: bool = False, axis_order: Optional[List[int]] = None, ) -> "go.Figure": if names is None: names = ["Objective 0", "Objective 1"] elif len(names) != 2: raise ValueError("The length of `names` is supposed to be 2.") trials = study.get_pareto_front_trials() if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") point_colors = ["blue"] * len(trials) if include_dominated_trials: non_pareto_trials = _get_non_pareto_front_trials(study, trials) point_colors += ["red"] * len(non_pareto_trials) trials += non_pareto_trials if axis_order is None: axis_order = list(range(2)) else: if len(axis_order) != 2: raise ValueError( f"Size of `axis_order` {axis_order}. Expect: 2, Actual: {len(axis_order)}." ) if len(set(axis_order)) != 2: raise ValueError(f"Elements of given `axis_order` {axis_order} are not unique!") if max(axis_order) > 1: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {max(axis_order)} " "higher than 1." ) if min(axis_order) < 0: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {min(axis_order)} " "lower than 0." ) data = go.Scatter( x=[t.values[axis_order[0]] for t in trials], y=[t.values[axis_order[1]] for t in trials], text=[_make_hovertext(t) for t in trials], mode="markers", hovertemplate="%{text}", marker={"color": point_colors}, ) layout = go.Layout( title="Pareto-front Plot", xaxis_title=names[axis_order[0]], yaxis_title=names[axis_order[1]], ) return go.Figure(data=data, layout=layout) def _get_pareto_front_3d( study: MultiObjectiveStudy, names: Optional[List[str]], include_dominated_trials: bool = False, axis_order: Optional[List[int]] = None, ) -> "go.Figure": if names is None: names = ["Objective 0", "Objective 1", "Objective 2"] elif len(names) != 3: raise ValueError("The length of `names` is supposed to be 3.") trials = study.get_pareto_front_trials() if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") point_colors = ["blue"] * len(trials) if include_dominated_trials: non_pareto_trials = _get_non_pareto_front_trials(study, trials) point_colors += ["red"] * len(non_pareto_trials) trials += non_pareto_trials if axis_order is None: axis_order = list(range(3)) else: if len(axis_order) != 3: raise ValueError( f"Size of `axis_order` {axis_order}. Expect: 3, Actual: {len(axis_order)}." ) if len(set(axis_order)) != 3: raise ValueError(f"Elements of given `axis_order` {axis_order} are not unique!.") if max(axis_order) > 2: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {max(axis_order)} " "higher than 2." ) if min(axis_order) < 0: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {min(axis_order)} " "lower than 0." ) data = go.Scatter3d( x=[t.values[axis_order[0]] for t in trials], y=[t.values[axis_order[1]] for t in trials], z=[t.values[axis_order[2]] for t in trials], text=[_make_hovertext(t) for t in trials], mode="markers", hovertemplate="%{text}", marker={"color": point_colors}, ) layout = go.Layout( title="Pareto-front Plot", scene={ "xaxis_title": names[axis_order[0]], "yaxis_title": names[axis_order[1]], "zaxis_title": names[axis_order[2]], }, ) return go.Figure(data=data, layout=layout) def _make_hovertext(trial: FrozenMultiObjectiveTrial) -> str: text = json.dumps( {"number": trial.number, "values": trial.values, "params": trial.params}, indent=2 ) return text.replace("\n", "
") optuna-3.5.0/optuna/progress_bar.py000066400000000000000000000102141453453102400173740ustar00rootroot00000000000000from __future__ import annotations import logging from typing import Any from typing import TYPE_CHECKING import warnings from tqdm.auto import tqdm from optuna import logging as optuna_logging if TYPE_CHECKING: from optuna.study import Study _tqdm_handler: _TqdmLoggingHandler | None = None # Reference: https://gist.github.com/hvy/8b80c2cedf02b15c24f85d1fa17ebe02 class _TqdmLoggingHandler(logging.StreamHandler): def emit(self, record: Any) -> None: try: msg = self.format(record) tqdm.write(msg) self.flush() except (KeyboardInterrupt, SystemExit): raise except Exception: self.handleError(record) class _ProgressBar: """Progress Bar implementation for :func:`~optuna.study.Study.optimize` on the top of `tqdm`. Args: is_valid: Whether to show progress bars in :func:`~optuna.study.Study.optimize`. n_trials: The number of trials. timeout: Stop study after the given number of second(s). """ def __init__( self, is_valid: bool, n_trials: int | None = None, timeout: float | None = None, ) -> None: if is_valid and n_trials is None and timeout is None: warnings.warn("Progress bar won't be displayed because n_trials and timeout are None.") self._is_valid = is_valid and (n_trials or timeout) is not None self._n_trials = n_trials self._timeout = timeout self._last_elapsed_seconds = 0.0 if self._is_valid: if self._n_trials is not None: self._progress_bar = tqdm(total=self._n_trials) elif self._timeout is not None: total = tqdm.format_interval(self._timeout) fmt = "{desc} {percentage:3.0f}%|{bar}| {elapsed}/" + total self._progress_bar = tqdm(total=self._timeout, bar_format=fmt) else: assert False global _tqdm_handler _tqdm_handler = _TqdmLoggingHandler() _tqdm_handler.setLevel(logging.INFO) _tqdm_handler.setFormatter(optuna_logging.create_default_formatter()) optuna_logging.disable_default_handler() optuna_logging._get_library_root_logger().addHandler(_tqdm_handler) def update(self, elapsed_seconds: float, study: Study) -> None: """Update the progress bars if ``is_valid`` is :obj:`True`. Args: elapsed_seconds: The time past since :func:`~optuna.study.Study.optimize` started. study: The current study object. """ if self._is_valid: if not study._is_multi_objective(): # Not updating the progress bar when there are no complete trial. try: msg = ( f"Best trial: {study.best_trial.number}. " f"Best value: {study.best_value:.6g}" ) self._progress_bar.set_description(msg) except ValueError: pass if self._n_trials is not None: self._progress_bar.update(1) if self._timeout is not None: self._progress_bar.set_postfix_str( "{:.02f}/{} seconds".format(elapsed_seconds, self._timeout) ) elif self._timeout is not None: time_diff = elapsed_seconds - self._last_elapsed_seconds if elapsed_seconds > self._timeout: # Clip elapsed time to avoid tqdm warnings. time_diff -= elapsed_seconds - self._timeout self._progress_bar.update(time_diff) self._last_elapsed_seconds = elapsed_seconds else: assert False def close(self) -> None: """Close progress bars.""" if self._is_valid: self._progress_bar.close() assert _tqdm_handler is not None optuna_logging._get_library_root_logger().removeHandler(_tqdm_handler) optuna_logging.enable_default_handler() optuna-3.5.0/optuna/pruners/000077500000000000000000000000001453453102400160325ustar00rootroot00000000000000optuna-3.5.0/optuna/pruners/__init__.py000066400000000000000000000021231453453102400201410ustar00rootroot00000000000000from typing import TYPE_CHECKING from optuna.pruners._base import BasePruner from optuna.pruners._hyperband import HyperbandPruner from optuna.pruners._median import MedianPruner from optuna.pruners._nop import NopPruner from optuna.pruners._patient import PatientPruner from optuna.pruners._percentile import PercentilePruner from optuna.pruners._successive_halving import SuccessiveHalvingPruner from optuna.pruners._threshold import ThresholdPruner if TYPE_CHECKING: from optuna.study import Study from optuna.trial import FrozenTrial __all__ = [ "BasePruner", "HyperbandPruner", "MedianPruner", "NopPruner", "PatientPruner", "PercentilePruner", "SuccessiveHalvingPruner", "ThresholdPruner", ] def _filter_study(study: "Study", trial: "FrozenTrial") -> "Study": if isinstance(study.pruner, HyperbandPruner): # Create `_BracketStudy` to use trials that have the same bracket id. pruner: HyperbandPruner = study.pruner return pruner._create_bracket_study(study, pruner._get_bracket_id(study, trial)) else: return study optuna-3.5.0/optuna/pruners/_base.py000066400000000000000000000016161453453102400174610ustar00rootroot00000000000000import abc import optuna class BasePruner(abc.ABC): """Base class for pruners.""" @abc.abstractmethod def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: """Judge whether the trial should be pruned based on the reported values. Note that this method is not supposed to be called by library users. Instead, :func:`optuna.trial.Trial.report` and :func:`optuna.trial.Trial.should_prune` provide user interfaces to implement pruning mechanism in an objective function. Args: study: Study object of the target study. trial: FrozenTrial object of the target trial. Take a copy before modifying this object. Returns: A boolean value representing whether the trial should be pruned. """ raise NotImplementedError optuna-3.5.0/optuna/pruners/_hyperband.py000066400000000000000000000332661453453102400205310ustar00rootroot00000000000000import binascii import math from typing import Container from typing import List from typing import Optional from typing import Union import optuna from optuna import logging from optuna.pruners._base import BasePruner from optuna.pruners._successive_halving import SuccessiveHalvingPruner from optuna.trial._state import TrialState _logger = logging.get_logger(__name__) class HyperbandPruner(BasePruner): """Pruner using Hyperband. As SuccessiveHalving (SHA) requires the number of configurations :math:`n` as its hyperparameter. For a given finite budget :math:`B`, all the configurations have the resources of :math:`B \\over n` on average. As you can see, there will be a trade-off of :math:`B` and :math:`B \\over n`. `Hyperband `_ attacks this trade-off by trying different :math:`n` values for a fixed budget. .. note:: * In the Hyperband paper, the counterpart of :class:`~optuna.samplers.RandomSampler` is used. * Optuna uses :class:`~optuna.samplers.TPESampler` by default. * `The benchmark result `_ shows that :class:`optuna.pruners.HyperbandPruner` supports both samplers. .. note:: If you use ``HyperbandPruner`` with :class:`~optuna.samplers.TPESampler`, it's recommended to consider setting larger ``n_trials`` or ``timeout`` to make full use of the characteristics of :class:`~optuna.samplers.TPESampler` because :class:`~optuna.samplers.TPESampler` uses some (by default, :math:`10`) :class:`~optuna.trial.Trial`\\ s for its startup. As Hyperband runs multiple :class:`~optuna.pruners.SuccessiveHalvingPruner` and collects trials based on the current :class:`~optuna.trial.Trial`\\ 's bracket ID, each bracket needs to observe more than :math:`10` :class:`~optuna.trial.Trial`\\ s for :class:`~optuna.samplers.TPESampler` to adapt its search space. Thus, for example, if ``HyperbandPruner`` has :math:`4` pruners in it, at least :math:`4 \\times 10` trials are consumed for startup. .. note:: Hyperband has several :class:`~optuna.pruners.SuccessiveHalvingPruner`\\ s. Each :class:`~optuna.pruners.SuccessiveHalvingPruner` is referred to as "bracket" in the original paper. The number of brackets is an important factor to control the early stopping behavior of Hyperband and is automatically determined by ``min_resource``, ``max_resource`` and ``reduction_factor`` as :math:`\\mathrm{The\\ number\\ of\\ brackets} = \\mathrm{floor}(\\log_{\\texttt{reduction}\\_\\texttt{factor}} (\\frac{\\texttt{max}\\_\\texttt{resource}}{\\texttt{min}\\_\\texttt{resource}})) + 1`. Please set ``reduction_factor`` so that the number of brackets is not too large (about 4 – 6 in most use cases). Please see Section 3.6 of the `original paper `_ for the detail. Example: We minimize an objective function with Hyperband pruning algorithm. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) n_train_iter = 100 def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.HyperbandPruner( min_resource=1, max_resource=n_train_iter, reduction_factor=3 ), ) study.optimize(objective, n_trials=20) Args: min_resource: A parameter for specifying the minimum resource allocated to a trial noted as :math:`r` in the paper. A smaller :math:`r` will give a result faster, but a larger :math:`r` will give a better guarantee of successful judging between configurations. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. max_resource: A parameter for specifying the maximum resource allocated to a trial. :math:`R` in the paper corresponds to ``max_resource / min_resource``. This value represents and should match the maximum iteration steps (e.g., the number of epochs for neural networks). When this argument is "auto", the maximum resource is estimated according to the completed trials. The default value of this argument is "auto". .. note:: With "auto", the maximum resource will be the largest step reported by :meth:`~optuna.trial.Trial.report` in the first, or one of the first if trained in parallel, completed trial. No trials will be pruned until the maximum resource is determined. .. note:: If the step of the last intermediate value may change with each trial, please manually specify the maximum possible step to ``max_resource``. reduction_factor: A parameter for specifying reduction factor of promotable trials noted as :math:`\\eta` in the paper. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. bootstrap_count: Parameter specifying the number of trials required in a rung before any trial can be promoted. Incompatible with ``max_resource`` is ``"auto"``. See the details for :class:`~optuna.pruners.SuccessiveHalvingPruner`. """ def __init__( self, min_resource: int = 1, max_resource: Union[str, int] = "auto", reduction_factor: int = 3, bootstrap_count: int = 0, ) -> None: self._min_resource = min_resource self._max_resource = max_resource self._reduction_factor = reduction_factor self._pruners: List[SuccessiveHalvingPruner] = [] self._bootstrap_count = bootstrap_count self._total_trial_allocation_budget = 0 self._trial_allocation_budgets: List[int] = [] self._n_brackets: Optional[int] = None if not isinstance(self._max_resource, int) and self._max_resource != "auto": raise ValueError( "The 'max_resource' should be integer or 'auto'. " "But max_resource = {}".format(self._max_resource) ) if self._bootstrap_count > 0 and self._max_resource == "auto": raise ValueError( "bootstrap_count > 0 and max_resource == 'auto' " "are mutually incompatible, bootstrap_count is {}".format(self._bootstrap_count) ) def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: if len(self._pruners) == 0: self._try_initialization(study) if len(self._pruners) == 0: return False bracket_id = self._get_bracket_id(study, trial) _logger.debug("{}th bracket is selected".format(bracket_id)) bracket_study = self._create_bracket_study(study, bracket_id) return self._pruners[bracket_id].prune(bracket_study, trial) def _try_initialization(self, study: "optuna.study.Study") -> None: if self._max_resource == "auto": trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) n_steps = [t.last_step for t in trials if t.last_step is not None] if not n_steps: return self._max_resource = max(n_steps) + 1 assert isinstance(self._max_resource, int) if self._n_brackets is None: # In the original paper http://www.jmlr.org/papers/volume18/16-558/16-558.pdf, the # inputs of Hyperband are `R`: max resource and `\eta`: reduction factor. The # number of brackets (this is referred as `s_{max} + 1` in the paper) is calculated # by s_{max} + 1 = \floor{\log_{\eta} (R)} + 1 in Algorithm 1 of the original paper. # In this implementation, we combine this formula and that of ASHA paper # https://arxiv.org/abs/1502.07943 as # `n_brackets = floor(log_{reduction_factor}(max_resource / min_resource)) + 1` self._n_brackets = ( math.floor( math.log(self._max_resource / self._min_resource, self._reduction_factor) ) + 1 ) _logger.debug("Hyperband has {} brackets".format(self._n_brackets)) for bracket_id in range(self._n_brackets): trial_allocation_budget = self._calculate_trial_allocation_budget(bracket_id) self._total_trial_allocation_budget += trial_allocation_budget self._trial_allocation_budgets.append(trial_allocation_budget) pruner = SuccessiveHalvingPruner( min_resource=self._min_resource, reduction_factor=self._reduction_factor, min_early_stopping_rate=bracket_id, bootstrap_count=self._bootstrap_count, ) self._pruners.append(pruner) def _calculate_trial_allocation_budget(self, bracket_id: int) -> int: """Compute the trial allocated budget for a bracket of ``bracket_id``. In the `original paper `, the number of trials per one bracket is referred as ``n`` in Algorithm 1. Since we do not know the total number of trials in the leaning scheme of Optuna, we calculate the ratio of the number of trials here instead. """ assert self._n_brackets is not None s = self._n_brackets - 1 - bracket_id return math.ceil(self._n_brackets * (self._reduction_factor**s) / (s + 1)) def _get_bracket_id( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial" ) -> int: """Compute the index of bracket for a trial of ``trial_number``. The index of a bracket is noted as :math:`s` in `Hyperband paper `_. """ if len(self._pruners) == 0: return 0 assert self._n_brackets is not None n = ( binascii.crc32("{}_{}".format(study.study_name, trial.number).encode()) % self._total_trial_allocation_budget ) for bracket_id in range(self._n_brackets): n -= self._trial_allocation_budgets[bracket_id] if n < 0: return bracket_id assert False, "This line should be unreachable." def _create_bracket_study( self, study: "optuna.study.Study", bracket_id: int ) -> "optuna.study.Study": # This class is assumed to be passed to # `SuccessiveHalvingPruner.prune` in which `get_trials`, # `direction`, and `storage` are used. # But for safety, prohibit the other attributes explicitly. class _BracketStudy(optuna.study.Study): _VALID_ATTRS = ( "get_trials", "_get_trials", "directions", "direction", "_directions", "_storage", "_study_id", "pruner", "study_name", "_bracket_id", "sampler", "trials", "_is_multi_objective", "stop", "_study", "_thread_local", ) def __init__( self, study: "optuna.study.Study", pruner: HyperbandPruner, bracket_id: int ) -> None: super().__init__( study_name=study.study_name, storage=study._storage, sampler=study.sampler, pruner=pruner, ) self._study = study self._bracket_id = bracket_id def get_trials( self, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List["optuna.trial.FrozenTrial"]: trials = super()._get_trials(deepcopy=deepcopy, states=states) pruner = self.pruner assert isinstance(pruner, HyperbandPruner) return [t for t in trials if pruner._get_bracket_id(self, t) == self._bracket_id] def stop(self) -> None: # `stop` should stop the original study's optimization loop instead of # `_BracketStudy`. self._study.stop() def __getattribute__(self, attr_name): # type: ignore if attr_name not in _BracketStudy._VALID_ATTRS: raise AttributeError( "_BracketStudy does not have attribute of '{}'".format(attr_name) ) else: return object.__getattribute__(self, attr_name) return _BracketStudy(study, self, bracket_id) optuna-3.5.0/optuna/pruners/_median.py000066400000000000000000000056151453453102400200070ustar00rootroot00000000000000from optuna.pruners._percentile import PercentilePruner class MedianPruner(PercentilePruner): """Pruner using the median stopping rule. Prune if the trial's best intermediate result is worse than median of intermediate results of previous trials at the same step. Example: We minimize an objective function with the median stopping rule. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.MedianPruner( n_startup_trials=5, n_warmup_steps=30, interval_steps=10 ), ) study.optimize(objective, n_trials=20) Args: n_startup_trials: Pruning is disabled until the given number of trials finish in the same study. n_warmup_steps: Pruning is disabled until the trial exceeds the given number of step. Note that this feature assumes that ``step`` starts at zero. interval_steps: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. n_min_trials: Minimum number of reported trial results at a step to judge whether to prune. If the number of reported intermediate values from all trials at the current step is less than ``n_min_trials``, the trial will not be pruned. This can be used to ensure that a minimum number of trials are run to completion without being pruned. """ def __init__( self, n_startup_trials: int = 5, n_warmup_steps: int = 0, interval_steps: int = 1, *, n_min_trials: int = 1, ) -> None: super().__init__( 50.0, n_startup_trials, n_warmup_steps, interval_steps, n_min_trials=n_min_trials ) optuna-3.5.0/optuna/pruners/_nop.py000066400000000000000000000027401453453102400173420ustar00rootroot00000000000000import optuna from optuna.pruners import BasePruner class NopPruner(BasePruner): """Pruner which never prunes trials. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): assert False, "should_prune() should always return False with this pruner." raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize", pruner=optuna.pruners.NopPruner()) study.optimize(objective, n_trials=20) """ def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: return False optuna-3.5.0/optuna/pruners/_patient.py000066400000000000000000000101061453453102400202050ustar00rootroot00000000000000from typing import Optional import numpy as np import optuna from optuna._experimental import experimental_class from optuna.pruners import BasePruner from optuna.study._study_direction import StudyDirection @experimental_class("2.8.0") class PatientPruner(BasePruner): """Pruner which wraps another pruner with tolerance. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.PatientPruner(optuna.pruners.MedianPruner(), patience=1), ) study.optimize(objective, n_trials=20) Args: wrapped_pruner: Wrapped pruner to perform pruning when :class:`~optuna.pruners.PatientPruner` allows a trial to be pruned. If it is :obj:`None`, this pruner is equivalent to early-stopping taken the intermediate values in the individual trial. patience: Pruning is disabled until the objective doesn't improve for ``patience`` consecutive steps. min_delta: Tolerance value to check whether or not the objective improves. This value should be non-negative. """ def __init__( self, wrapped_pruner: Optional[BasePruner], patience: int, min_delta: float = 0.0 ) -> None: if patience < 0: raise ValueError(f"patience cannot be negative but got {patience}.") if min_delta < 0: raise ValueError(f"min_delta cannot be negative but got {min_delta}.") self._wrapped_pruner = wrapped_pruner self._patience = patience self._min_delta = min_delta def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: step = trial.last_step if step is None: return False intermediate_values = trial.intermediate_values steps = np.asarray(list(intermediate_values.keys())) # Do not prune if number of step to determine are insufficient. if steps.size <= self._patience + 1: return False steps.sort() # This is the score patience steps ago steps_before_patience = steps[: -self._patience - 1] scores_before_patience = np.asarray( list(intermediate_values[step] for step in steps_before_patience) ) # And these are the scores after that steps_after_patience = steps[-self._patience - 1 :] scores_after_patience = np.asarray( list(intermediate_values[step] for step in steps_after_patience) ) direction = study.direction if direction == StudyDirection.MINIMIZE: maybe_prune = np.nanmin(scores_before_patience) + self._min_delta < np.nanmin( scores_after_patience ) else: maybe_prune = np.nanmax(scores_before_patience) - self._min_delta > np.nanmax( scores_after_patience ) if maybe_prune: if self._wrapped_pruner is not None: return self._wrapped_pruner.prune(study, trial) else: return True else: return False optuna-3.5.0/optuna/pruners/_percentile.py000066400000000000000000000160161453453102400207010ustar00rootroot00000000000000import functools import math from typing import KeysView from typing import List import numpy as np import optuna from optuna.pruners import BasePruner from optuna.study._study_direction import StudyDirection from optuna.trial._state import TrialState def _get_best_intermediate_result_over_steps( trial: "optuna.trial.FrozenTrial", direction: StudyDirection ) -> float: values = np.asarray(list(trial.intermediate_values.values()), dtype=float) if direction == StudyDirection.MAXIMIZE: return np.nanmax(values) return np.nanmin(values) def _get_percentile_intermediate_result_over_trials( completed_trials: List["optuna.trial.FrozenTrial"], direction: StudyDirection, step: int, percentile: float, n_min_trials: int, ) -> float: if len(completed_trials) == 0: raise ValueError("No trials have been completed.") intermediate_values = [ t.intermediate_values[step] for t in completed_trials if step in t.intermediate_values ] if len(intermediate_values) < n_min_trials: return math.nan if direction == StudyDirection.MAXIMIZE: percentile = 100 - percentile return float( np.nanpercentile( np.array(intermediate_values, dtype=float), percentile, ) ) def _is_first_in_interval_step( step: int, intermediate_steps: KeysView[int], n_warmup_steps: int, interval_steps: int ) -> bool: nearest_lower_pruning_step = ( step - n_warmup_steps ) // interval_steps * interval_steps + n_warmup_steps assert nearest_lower_pruning_step >= 0 # `intermediate_steps` may not be sorted so we must go through all elements. second_last_step = functools.reduce( lambda second_last_step, s: s if s > second_last_step and s != step else second_last_step, intermediate_steps, -1, ) return second_last_step < nearest_lower_pruning_step class PercentilePruner(BasePruner): """Pruner to keep the specified percentile of the trials. Prune if the best intermediate value is in the bottom percentile among trials at the same step. Example: .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.PercentilePruner( 25.0, n_startup_trials=5, n_warmup_steps=30, interval_steps=10 ), ) study.optimize(objective, n_trials=20) Args: percentile: Percentile which must be between 0 and 100 inclusive (e.g., When given 25.0, top of 25th percentile trials are kept). n_startup_trials: Pruning is disabled until the given number of trials finish in the same study. n_warmup_steps: Pruning is disabled until the trial exceeds the given number of step. Note that this feature assumes that ``step`` starts at zero. interval_steps: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. Value must be at least 1. n_min_trials: Minimum number of reported trial results at a step to judge whether to prune. If the number of reported intermediate values from all trials at the current step is less than ``n_min_trials``, the trial will not be pruned. This can be used to ensure that a minimum number of trials are run to completion without being pruned. """ def __init__( self, percentile: float, n_startup_trials: int = 5, n_warmup_steps: int = 0, interval_steps: int = 1, *, n_min_trials: int = 1, ) -> None: if not 0.0 <= percentile <= 100: raise ValueError( "Percentile must be between 0 and 100 inclusive but got {}.".format(percentile) ) if n_startup_trials < 0: raise ValueError( "Number of startup trials cannot be negative but got {}.".format(n_startup_trials) ) if n_warmup_steps < 0: raise ValueError( "Number of warmup steps cannot be negative but got {}.".format(n_warmup_steps) ) if interval_steps < 1: raise ValueError( "Pruning interval steps must be at least 1 but got {}.".format(interval_steps) ) if n_min_trials < 1: raise ValueError( "Number of trials for pruning must be at least 1 but got {}.".format(n_min_trials) ) self._percentile = percentile self._n_startup_trials = n_startup_trials self._n_warmup_steps = n_warmup_steps self._interval_steps = interval_steps self._n_min_trials = n_min_trials def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) n_trials = len(completed_trials) if n_trials == 0: return False if n_trials < self._n_startup_trials: return False step = trial.last_step if step is None: return False n_warmup_steps = self._n_warmup_steps if step < n_warmup_steps: return False if not _is_first_in_interval_step( step, trial.intermediate_values.keys(), n_warmup_steps, self._interval_steps ): return False direction = study.direction best_intermediate_result = _get_best_intermediate_result_over_steps(trial, direction) if math.isnan(best_intermediate_result): return True p = _get_percentile_intermediate_result_over_trials( completed_trials, direction, step, self._percentile, self._n_min_trials ) if math.isnan(p): return False if direction == StudyDirection.MAXIMIZE: return best_intermediate_result < p return best_intermediate_result > p optuna-3.5.0/optuna/pruners/_successive_halving.py000066400000000000000000000241761453453102400224410ustar00rootroot00000000000000import math from typing import List from typing import Optional from typing import Union import optuna from optuna.pruners._base import BasePruner from optuna.study._study_direction import StudyDirection from optuna.trial._state import TrialState class SuccessiveHalvingPruner(BasePruner): """Pruner using Asynchronous Successive Halving Algorithm. `Successive Halving `_ is a bandit-based algorithm to identify the best one among multiple configurations. This class implements an asynchronous version of Successive Halving. Please refer to the paper of `Asynchronous Successive Halving `_ for detailed descriptions. Note that, this class does not take care of the parameter for the maximum resource, referred to as :math:`R` in the paper. The maximum resource allocated to a trial is typically limited inside the objective function (e.g., ``step`` number in `simple_pruning.py `_, ``EPOCH`` number in `chainer_integration.py `_). .. seealso:: Please refer to :meth:`~optuna.trial.Trial.report`. Example: We minimize an objective function with ``SuccessiveHalvingPruner``. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) classes = np.unique(y) def objective(trial): alpha = trial.suggest_float("alpha", 0.0, 1.0) clf = SGDClassifier(alpha=alpha) n_train_iter = 100 for step in range(n_train_iter): clf.partial_fit(X_train, y_train, classes=classes) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study( direction="maximize", pruner=optuna.pruners.SuccessiveHalvingPruner() ) study.optimize(objective, n_trials=20) Args: min_resource: A parameter for specifying the minimum resource allocated to a trial (in the `paper `_ this parameter is referred to as :math:`r`). This parameter defaults to 'auto' where the value is determined based on a heuristic that looks at the number of required steps for the first trial to complete. A trial is never pruned until it executes :math:`\\mathsf{min}\\_\\mathsf{resource} \\times \\mathsf{reduction}\\_\\mathsf{factor}^{ \\mathsf{min}\\_\\mathsf{early}\\_\\mathsf{stopping}\\_\\mathsf{rate}}` steps (i.e., the completion point of the first rung). When the trial completes the first rung, it will be promoted to the next rung only if the value of the trial is placed in the top :math:`{1 \\over \\mathsf{reduction}\\_\\mathsf{factor}}` fraction of the all trials that already have reached the point (otherwise it will be pruned there). If the trial won the competition, it runs until the next completion point (i.e., :math:`\\mathsf{min}\\_\\mathsf{resource} \\times \\mathsf{reduction}\\_\\mathsf{factor}^{ (\\mathsf{min}\\_\\mathsf{early}\\_\\mathsf{stopping}\\_\\mathsf{rate} + \\mathsf{rung})}` steps) and repeats the same procedure. .. note:: If the step of the last intermediate value may change with each trial, please manually specify the minimum possible step to ``min_resource``. reduction_factor: A parameter for specifying reduction factor of promotable trials (in the `paper `_ this parameter is referred to as :math:`\\eta`). At the completion point of each rung, about :math:`{1 \\over \\mathsf{reduction}\\_\\mathsf{factor}}` trials will be promoted. min_early_stopping_rate: A parameter for specifying the minimum early-stopping rate (in the `paper `_ this parameter is referred to as :math:`s`). bootstrap_count: Minimum number of trials that need to complete a rung before any trial is considered for promotion into the next rung. """ def __init__( self, min_resource: Union[str, int] = "auto", reduction_factor: int = 4, min_early_stopping_rate: int = 0, bootstrap_count: int = 0, ) -> None: if isinstance(min_resource, str) and min_resource != "auto": raise ValueError( "The value of `min_resource` is {}, " "but must be either `min_resource` >= 1 or 'auto'".format(min_resource) ) if isinstance(min_resource, int) and min_resource < 1: raise ValueError( "The value of `min_resource` is {}, " "but must be either `min_resource >= 1` or 'auto'".format(min_resource) ) if reduction_factor < 2: raise ValueError( "The value of `reduction_factor` is {}, " "but must be `reduction_factor >= 2`".format(reduction_factor) ) if min_early_stopping_rate < 0: raise ValueError( "The value of `min_early_stopping_rate` is {}, " "but must be `min_early_stopping_rate >= 0`".format(min_early_stopping_rate) ) if bootstrap_count < 0: raise ValueError( "The value of `bootstrap_count` is {}, " "but must be `bootstrap_count >= 0`".format(bootstrap_count) ) if bootstrap_count > 0 and min_resource == "auto": raise ValueError( "bootstrap_count > 0 and min_resource == 'auto' " "are mutually incompatible, bootstrap_count is {}".format(bootstrap_count) ) self._min_resource: Optional[int] = None if isinstance(min_resource, int): self._min_resource = min_resource self._reduction_factor = reduction_factor self._min_early_stopping_rate = min_early_stopping_rate self._bootstrap_count = bootstrap_count def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: step = trial.last_step if step is None: return False rung = _get_current_rung(trial) value = trial.intermediate_values[step] trials: Optional[List["optuna.trial.FrozenTrial"]] = None while True: if self._min_resource is None: if trials is None: trials = study.get_trials(deepcopy=False) self._min_resource = _estimate_min_resource(trials) if self._min_resource is None: return False assert self._min_resource is not None rung_promotion_step = self._min_resource * ( self._reduction_factor ** (self._min_early_stopping_rate + rung) ) if step < rung_promotion_step: return False if math.isnan(value): return True if trials is None: trials = study.get_trials(deepcopy=False) rung_key = _completed_rung_key(rung) study._storage.set_trial_system_attr(trial._trial_id, rung_key, value) competing = _get_competing_values(trials, value, rung_key) # 'competing' already includes the current trial # Therefore, we need to use the '<=' operator here if len(competing) <= self._bootstrap_count: return True if not _is_trial_promotable_to_next_rung( value, competing, self._reduction_factor, study.direction, ): return True rung += 1 def _estimate_min_resource(trials: List["optuna.trial.FrozenTrial"]) -> Optional[int]: n_steps = [ t.last_step for t in trials if t.state == TrialState.COMPLETE and t.last_step is not None ] if not n_steps: return None # Get the maximum number of steps and divide it by 100. last_step = max(n_steps) return max(last_step // 100, 1) def _get_current_rung(trial: "optuna.trial.FrozenTrial") -> int: # The following loop takes `O(log step)` iterations. rung = 0 while _completed_rung_key(rung) in trial.system_attrs: rung += 1 return rung def _completed_rung_key(rung: int) -> str: return "completed_rung_{}".format(rung) def _get_competing_values( trials: List["optuna.trial.FrozenTrial"], value: float, rung_key: str ) -> List[float]: competing_values = [t.system_attrs[rung_key] for t in trials if rung_key in t.system_attrs] competing_values.append(value) return competing_values def _is_trial_promotable_to_next_rung( value: float, competing_values: List[float], reduction_factor: int, study_direction: StudyDirection, ) -> bool: promotable_idx = (len(competing_values) // reduction_factor) - 1 if promotable_idx == -1: # Optuna does not support suspending or resuming ongoing trials. Therefore, for the first # `eta - 1` trials, this implementation instead promotes the trial if its value is the # smallest one among the competing values. promotable_idx = 0 competing_values.sort() if study_direction == StudyDirection.MAXIMIZE: return value >= competing_values[-(promotable_idx + 1)] return value <= competing_values[promotable_idx] optuna-3.5.0/optuna/pruners/_threshold.py000066400000000000000000000106261453453102400205440ustar00rootroot00000000000000import math from typing import Any from typing import Optional import optuna from optuna.pruners import BasePruner from optuna.pruners._percentile import _is_first_in_interval_step def _check_value(value: Any) -> float: try: # For convenience, we allow users to report a value that can be cast to `float`. value = float(value) except (TypeError, ValueError): message = "The `value` argument is of type '{}' but supposed to be a float.".format( type(value).__name__ ) raise TypeError(message) from None return value class ThresholdPruner(BasePruner): """Pruner to detect outlying metrics of the trials. Prune if a metric exceeds upper threshold, falls behind lower threshold or reaches ``nan``. Example: .. testcode:: from optuna import create_study from optuna.pruners import ThresholdPruner from optuna import TrialPruned def objective_for_upper(trial): for step, y in enumerate(ys_for_upper): trial.report(y, step) if trial.should_prune(): raise TrialPruned() return ys_for_upper[-1] def objective_for_lower(trial): for step, y in enumerate(ys_for_lower): trial.report(y, step) if trial.should_prune(): raise TrialPruned() return ys_for_lower[-1] ys_for_upper = [0.0, 0.1, 0.2, 0.5, 1.2] ys_for_lower = [100.0, 90.0, 0.1, 0.0, -1] study = create_study(pruner=ThresholdPruner(upper=1.0)) study.optimize(objective_for_upper, n_trials=10) study = create_study(pruner=ThresholdPruner(lower=0.0)) study.optimize(objective_for_lower, n_trials=10) Args: lower: A minimum value which determines whether pruner prunes or not. If an intermediate value is smaller than lower, it prunes. upper: A maximum value which determines whether pruner prunes or not. If an intermediate value is larger than upper, it prunes. n_warmup_steps: Pruning is disabled if the step is less than the given number of warmup steps. interval_steps: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. Value must be at least 1. """ def __init__( self, lower: Optional[float] = None, upper: Optional[float] = None, n_warmup_steps: int = 0, interval_steps: int = 1, ) -> None: if lower is None and upper is None: raise TypeError("Either lower or upper must be specified.") if lower is not None: lower = _check_value(lower) if upper is not None: upper = _check_value(upper) lower = lower if lower is not None else -float("inf") upper = upper if upper is not None else float("inf") if lower > upper: raise ValueError("lower should be smaller than upper.") if n_warmup_steps < 0: raise ValueError( "Number of warmup steps cannot be negative but got {}.".format(n_warmup_steps) ) if interval_steps < 1: raise ValueError( "Pruning interval steps must be at least 1 but got {}.".format(interval_steps) ) self._lower = lower self._upper = upper self._n_warmup_steps = n_warmup_steps self._interval_steps = interval_steps def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: step = trial.last_step if step is None: return False n_warmup_steps = self._n_warmup_steps if step < n_warmup_steps: return False if not _is_first_in_interval_step( step, trial.intermediate_values.keys(), n_warmup_steps, self._interval_steps ): return False latest_value = trial.intermediate_values[step] if math.isnan(latest_value): return True if latest_value < self._lower: return True if latest_value > self._upper: return True return False optuna-3.5.0/optuna/py.typed000066400000000000000000000000001453453102400160210ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/000077500000000000000000000000001453453102400161625ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/__init__.py000066400000000000000000000021021453453102400202660ustar00rootroot00000000000000from optuna.samplers import nsgaii from optuna.samplers._base import BaseSampler from optuna.samplers._brute_force import BruteForceSampler from optuna.samplers._cmaes import CmaEsSampler from optuna.samplers._grid import GridSampler from optuna.samplers._nsgaiii._sampler import NSGAIIISampler from optuna.samplers._partial_fixed import PartialFixedSampler from optuna.samplers._qmc import QMCSampler from optuna.samplers._random import RandomSampler from optuna.samplers._search_space import intersection_search_space from optuna.samplers._search_space import IntersectionSearchSpace from optuna.samplers._tpe.multi_objective_sampler import MOTPESampler from optuna.samplers._tpe.sampler import TPESampler from optuna.samplers.nsgaii._sampler import NSGAIISampler __all__ = [ "BaseSampler", "BruteForceSampler", "CmaEsSampler", "GridSampler", "IntersectionSearchSpace", "MOTPESampler", "NSGAIISampler", "NSGAIIISampler", "PartialFixedSampler", "QMCSampler", "RandomSampler", "TPESampler", "intersection_search_space", "nsgaii", ] optuna-3.5.0/optuna/samplers/_base.py000066400000000000000000000216331453453102400176120ustar00rootroot00000000000000import abc from typing import Any from typing import Callable from typing import Dict from typing import Optional from typing import Sequence import warnings import numpy as np from optuna.distributions import BaseDistribution from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState class BaseSampler(abc.ABC): """Base class for samplers. Optuna combines two types of sampling strategies, which are called *relative sampling* and *independent sampling*. *The relative sampling* determines values of multiple parameters simultaneously so that sampling algorithms can use relationship between parameters (e.g., correlation). Target parameters of the relative sampling are described in a relative search space, which is determined by :func:`~optuna.samplers.BaseSampler.infer_relative_search_space`. *The independent sampling* determines a value of a single parameter without considering any relationship between parameters. Target parameters of the independent sampling are the parameters not described in the relative search space. More specifically, parameters are sampled by the following procedure. At the beginning of a trial, :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space` is called to determine the relative search space for the trial. During the execution of the objective function, :meth:`~optuna.samplers.BaseSampler.sample_relative` is called only once when sampling the parameters belonging to the relative search space for the first time. :meth:`~optuna.samplers.BaseSampler.sample_independent` is used to sample parameters that don't belong to the relative search space. The following figure depicts the lifetime of a trial and how the above three methods are called in the trial. .. image:: ../../../../image/sampling-sequence.png | """ def __str__(self) -> str: return self.__class__.__name__ @abc.abstractmethod def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: """Infer the search space that will be used by relative sampling in the target trial. This method is called right before :func:`~optuna.samplers.BaseSampler.sample_relative` method, and the search space returned by this method is passed to it. The parameters not contained in the search space will be sampled by using :func:`~optuna.samplers.BaseSampler.sample_independent` method. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. Returns: A dictionary containing the parameter names and parameter's distributions. .. seealso:: Please refer to :func:`~optuna.search_space.intersection_search_space` as an implementation of :func:`~optuna.samplers.BaseSampler.infer_relative_search_space`. """ raise NotImplementedError @abc.abstractmethod def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: """Sample parameters in a given search space. This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use relationship between parameters such as Gaussian Process and CMA-ES. .. note:: The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers' perspective. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. search_space: The search space returned by :func:`~optuna.samplers.BaseSampler.infer_relative_search_space`. Returns: A dictionary containing the parameter names and the values. """ raise NotImplementedError @abc.abstractmethod def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: """Sample a parameter for a given distribution. This method is called only for the parameters not contained in the search space returned by :func:`~optuna.samplers.BaseSampler.sample_relative` method. This method is suitable for sampling algorithms that do not use relationship between parameters such as random sampling and TPE. .. note:: The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers' perspective. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. param_name: Name of the sampled parameter. param_distribution: Distribution object that specifies a prior and/or scale of the sampling algorithm. Returns: A parameter value. """ raise NotImplementedError def before_trial(self, study: Study, trial: FrozenTrial) -> None: """Trial pre-processing. This method is called before the objective function is called and right after the trial is instantiated. More precisely, this method is called during trial initialization, just before the :func:`~optuna.samplers.BaseSampler.infer_relative_search_space` call. In other words, it is responsible for pre-processing that should be done before inferring the search space. .. note:: Added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. Args: study: Target study object. trial: Target trial object. """ pass def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: """Trial post-processing. This method is called after the objective function returns and right before the trial is finished and its state is stored. .. note:: Added in v2.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.4.0. Args: study: Target study object. trial: Target trial object. Take a copy before modifying this object. state: Resulting trial state. values: Resulting trial values. Guaranteed to not be :obj:`None` if trial succeeded. """ pass def reseed_rng(self) -> None: """Reseed sampler's random number generator. This method is called by the :class:`~optuna.study.Study` instance if trials are executed in parallel with the option ``n_jobs>1``. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator. """ pass def _raise_error_if_multi_objective(self, study: Study) -> None: if study._is_multi_objective(): raise ValueError( "If the study is being used for multi-objective optimization, " f"{self.__class__.__name__} cannot be used." ) _CONSTRAINTS_KEY = "constraints" def _process_constraints_after_trial( constraints_func: Callable[[FrozenTrial], Sequence[float]], study: Study, trial: FrozenTrial, state: TrialState, ) -> None: if state not in [TrialState.COMPLETE, TrialState.PRUNED]: return constraints = None try: con = constraints_func(trial) if np.any(np.isnan(con)): raise ValueError("Constraint values cannot be NaN.") if not isinstance(con, (tuple, list)): warnings.warn( f"Constraints should be a sequence of floats but got {type(con).__name__}." ) constraints = tuple(con) finally: assert constraints is None or isinstance(constraints, tuple) study._storage.set_trial_system_attr( trial._trial_id, _CONSTRAINTS_KEY, constraints, ) optuna-3.5.0/optuna/samplers/_brute_force.py000066400000000000000000000232621453453102400211770ustar00rootroot00000000000000from dataclasses import dataclass import decimal from typing import Any from typing import Dict from typing import Iterable from typing import List from typing import Optional from typing import Sequence from typing import Tuple import numpy as np from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.study import Study from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState @dataclass class _TreeNode: # This is a class to represent the tree of search space. # A tree node has three states: # 1. Unexpanded. This is represented by children=None. # 2. Leaf. This is represented by children={} and param_name=None. # 3. Normal node. It has a param_name and non-empty children. param_name: Optional[str] = None children: Optional[Dict[float, "_TreeNode"]] = None def expand(self, param_name: Optional[str], search_space: Iterable[float]) -> None: # If the node is unexpanded, expand it. # Otherwise, check if the node is compatible with the given search space. if self.children is None: # Expand the node self.param_name = param_name self.children = {value: _TreeNode() for value in search_space} else: if self.param_name != param_name: raise ValueError(f"param_name mismatch: {self.param_name} != {param_name}") if self.children.keys() != set(search_space): raise ValueError( f"search_space mismatch: {set(self.children.keys())} != {set(search_space)}" ) def set_leaf(self) -> None: self.expand(None, []) def add_path( self, params_and_search_spaces: Iterable[Tuple[str, Iterable[float], float]] ) -> Optional["_TreeNode"]: # Add a path (i.e. a list of suggested parameters in one trial) to the tree. current_node = self for param_name, search_space, value in params_and_search_spaces: current_node.expand(param_name, search_space) assert current_node.children is not None if value not in current_node.children: return None current_node = current_node.children[value] return current_node def count_unexpanded(self) -> int: # Count the number of unexpanded nodes in the subtree. return ( 1 if self.children is None else sum(child.count_unexpanded() for child in self.children.values()) ) def sample_child(self, rng: np.random.RandomState) -> float: assert self.children is not None # Sample an unexpanded node in the subtree uniformly, and return the first # parameter value in the path to the node. # Equivalently, we sample the child node with weights proportional to the number # of unexpanded nodes in the subtree. weights = np.array( [child.count_unexpanded() for child in self.children.values()], dtype=np.float64 ) weights /= weights.sum() return rng.choice(list(self.children.keys()), p=weights) @experimental_class("3.1.0") class BruteForceSampler(BaseSampler): """Sampler using brute force. This sampler performs exhaustive search on the defined search space. Example: .. testcode:: import optuna def objective(trial): c = trial.suggest_categorical("c", ["float", "int"]) if c == "float": return trial.suggest_float("x", 1, 3, step=0.5) elif c == "int": a = trial.suggest_int("a", 1, 3) b = trial.suggest_int("b", a, 3) return a + b study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler()) study.optimize(objective) Note: The defined search space must be finite. Therefore, when using :class:`~optuna.distributions.FloatDistribution` or :func:`~optuna.trial.Trial.suggest_float`, ``step=None`` is not allowed. Note: The sampler may fail to try the entire search space in when the suggestion ranges or parameters are changed in the same :class:`~optuna.study.Study`. Args: seed: A seed to fix the order of trials as the search order randomly shuffled. Please note that it is not recommended using this option in distributed optimization settings since this option cannot ensure the order of trials and may increase the number of duplicate suggestions during distributed optimization. """ def __init__(self, seed: Optional[int] = None) -> None: self._rng = LazyRandomState(seed) def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: return {} def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: return {} @staticmethod def _populate_tree( tree: _TreeNode, trials: Iterable[FrozenTrial], params: Dict[str, Any] ) -> None: # Populate tree under given params from the given trials. incomplete_leaves: List[_TreeNode] = [] for trial in trials: if not all(p in trial.params and trial.params[p] == v for p, v in params.items()): continue leaf = tree.add_path( ( ( param_name, _enumerate_candidates(param_distribution), param_distribution.to_internal_repr(trial.params[param_name]), ) for param_name, param_distribution in trial.distributions.items() if param_name not in params ) ) if leaf is not None: # The parameters are on the defined grid. if trial.state.is_finished(): leaf.set_leaf() else: incomplete_leaves.append(leaf) # Add all incomplete leaf nodes at the end because they may not have complete search space. for leaf in incomplete_leaves: if leaf.children is None: leaf.set_leaf() def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: trials = study.get_trials( deepcopy=False, states=( TrialState.COMPLETE, TrialState.PRUNED, TrialState.RUNNING, TrialState.FAIL, ), ) tree = _TreeNode() candidates = _enumerate_candidates(param_distribution) tree.expand(param_name, candidates) # Populating must happen after the initialization above to prevent `tree` from # being initialized as an empty graph, which is created with n_jobs > 1 # where we get trials[i].params = {} for some i. self._populate_tree(tree, (t for t in trials if t.number != trial.number), trial.params) if tree.count_unexpanded() == 0: return param_distribution.to_external_repr(self._rng.rng.choice(candidates)) else: return param_distribution.to_external_repr(tree.sample_child(self._rng.rng)) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: trials = study.get_trials( deepcopy=False, states=( TrialState.COMPLETE, TrialState.PRUNED, TrialState.RUNNING, TrialState.FAIL, ), ) tree = _TreeNode() self._populate_tree( tree, ( t if t.number != trial.number else create_trial( state=state, # Set current trial as complete. values=values, params=trial.params, distributions=trial.distributions, ) for t in trials ), {}, ) if tree.count_unexpanded() == 0: study.stop() def _enumerate_candidates(param_distribution: BaseDistribution) -> Sequence[float]: if isinstance(param_distribution, FloatDistribution): if param_distribution.step is None: raise ValueError( "FloatDistribution.step must be given for BruteForceSampler" " (otherwise, the search space will be infinite)." ) low = decimal.Decimal(str(param_distribution.low)) high = decimal.Decimal(str(param_distribution.high)) step = decimal.Decimal(str(param_distribution.step)) ret = [] value = low while value <= high: ret.append(float(value)) value += step return ret elif isinstance(param_distribution, IntDistribution): return list( range(param_distribution.low, param_distribution.high + 1, param_distribution.step) ) elif isinstance(param_distribution, CategoricalDistribution): return list(range(len(param_distribution.choices))) # Internal representations. else: raise ValueError(f"Unknown distribution {param_distribution}.") optuna-3.5.0/optuna/samplers/_cmaes.py000066400000000000000000001006011453453102400177610ustar00rootroot00000000000000from __future__ import annotations import copy import math import pickle from typing import Any from typing import Callable from typing import cast from typing import Dict from typing import List from typing import NamedTuple from typing import Optional from typing import Sequence from typing import Tuple from typing import TYPE_CHECKING from typing import Union import warnings import numpy as np import optuna from optuna import logging from optuna._imports import _LazyImport from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.exceptions import ExperimentalWarning from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.search_space import IntersectionSearchSpace from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: import cmaes CmaClass = Union[cmaes.CMA, cmaes.SepCMA, cmaes.CMAwM] else: cmaes = _LazyImport("cmaes") _logger = logging.get_logger(__name__) _EPS = 1e-10 # The value of system_attrs must be less than 2046 characters on RDBStorage. _SYSTEM_ATTR_MAX_LENGTH = 2045 class _CmaEsAttrKeys(NamedTuple): optimizer: Callable[[int], str] generation: Callable[[int], str] popsize: Callable[[], str] n_restarts: Callable[[], str] n_restarts_with_large: str poptype: str small_n_eval: str large_n_eval: str class CmaEsSampler(BaseSampler): """A sampler using `cmaes `_ as the backend. Example: Optimize a simple quadratic function by using :class:`~optuna.samplers.CmaEsSampler`. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y sampler = optuna.samplers.CmaEsSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=20) Please note that this sampler does not support CategoricalDistribution. However, :class:`~optuna.distributions.FloatDistribution` with ``step``, (:func:`~optuna.trial.Trial.suggest_float`) and :class:`~optuna.distributions.IntDistribution` (:func:`~optuna.trial.Trial.suggest_int`) are supported. If your search space contains categorical parameters, I recommend you to use :class:`~optuna.samplers.TPESampler` instead. Furthermore, there is room for performance improvements in parallel optimization settings. This sampler cannot use some trials for updating the parameters of multivariate normal distribution. For further information about CMA-ES algorithm, please refer to the following papers: - `N. Hansen, The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772, 2016. `_ - `A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005), pages 1769–1776. IEEE Press, 2005. `_ - `N. Hansen. Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed. GECCO Workshop, 2009. `_ - `Raymond Ros, Nikolaus Hansen. A Simple Modification in CMA-ES Achieving Linear Time and Space Complexity. 10th International Conference on Parallel Problem Solving From Nature, Sep 2008, Dortmund, Germany. inria-00287367. `_ - `Masahiro Nomura, Shuhei Watanabe, Youhei Akimoto, Yoshihiko Ozaki, Masaki Onishi. Warm Starting CMA-ES for Hyperparameter Optimization, AAAI. 2021. `_ - `R. Hamano, S. Saito, M. Nomura, S. Shirakawa. CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization, GECCO. 2022. `_ - `M. Nomura, Y. Akimoto, I. Ono. CMA-ES with Learning Rate Adaptation: Can CMA-ES with Default Population Size Solve Multimodal and Noisy Problems?, GECCO. 2023. `_ .. seealso:: You can also use :class:`optuna.integration.PyCmaSampler` which is a sampler using cma library as the backend. Args: x0: A dictionary of an initial parameter values for CMA-ES. By default, the mean of ``low`` and ``high`` for each distribution is used. Note that ``x0`` is sampled uniformly within the search space domain for each restart if you specify ``restart_strategy`` argument. sigma0: Initial standard deviation of CMA-ES. By default, ``sigma0`` is set to ``min_range / 6``, where ``min_range`` denotes the minimum range of the distributions in the search space. seed: A random seed for CMA-ES. n_startup_trials: The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study. independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for :class:`~optuna.samplers.CmaEsSampler` is determined by :func:`~optuna.search_space.intersection_search_space()`. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case. restart_strategy: Strategy for restarting CMA-ES optimization when converges to a local minimum. If :obj:`None` is given, CMA-ES will not restart (default). If 'ipop' is given, CMA-ES will restart with increasing population size. if 'bipop' is given, CMA-ES will restart with the population size increased or decreased. Please see also ``inc_popsize`` parameter. .. note:: Added in v2.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.1.0. popsize: A population size of CMA-ES. When ``restart_strategy = 'ipop'`` or ``restart_strategy = 'bipop'`` is specified, this is used as the initial population size. inc_popsize: Multiplier for increasing population size before each restart. This argument will be used when ``restart_strategy = 'ipop'`` or ``restart_strategy = 'bipop'`` is specified. consider_pruned_trials: If this is :obj:`True`, the PRUNED trials are considered for sampling. .. note:: Added in v2.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.0.0. .. note:: It is suggested to set this flag :obj:`False` when the :class:`~optuna.pruners.MedianPruner` is used. On the other hand, it is suggested to set this flag :obj:`True` when the :class:`~optuna.pruners.HyperbandPruner` is used. Please see `the benchmark result `_ for the details. use_separable_cma: If this is :obj:`True`, the covariance matrix is constrained to be diagonal. Due to reduce the model complexity, the learning rate for the covariance matrix is increased. Consequently, this algorithm outperforms CMA-ES on separable functions. .. note:: Added in v2.6.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.6.0. with_margin: If this is :obj:`True`, CMA-ES with margin is used. This algorithm prevents samples in each discrete distribution (:class:`~optuna.distributions.FloatDistribution` with `step` and :class:`~optuna.distributions.IntDistribution`) from being fixed to a single point. Currently, this option cannot be used with ``use_separable_cma=True``. .. note:: Added in v3.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.1.0. lr_adapt: If this is :obj:`True`, CMA-ES with learning rate adaptation is used. This algorithm focuses on working well on multimodal and/or noisy problems with default settings. Currently, this option cannot be used with ``use_separable_cma=True`` or ``with_margin=True``. .. note:: Added in v3.3.0 or later, as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. source_trials: This option is for Warm Starting CMA-ES, a method to transfer prior knowledge on similar HPO tasks through the initialization of CMA-ES. This method estimates a promising distribution from ``source_trials`` and generates the parameter of multivariate gaussian distribution. Please note that it is prohibited to use ``x0``, ``sigma0``, or ``use_separable_cma`` argument together. .. note:: Added in v2.6.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.6.0. """ def __init__( self, x0: Optional[Dict[str, Any]] = None, sigma0: Optional[float] = None, n_startup_trials: int = 1, independent_sampler: Optional[BaseSampler] = None, warn_independent_sampling: bool = True, seed: Optional[int] = None, *, consider_pruned_trials: bool = False, restart_strategy: Optional[str] = None, popsize: Optional[int] = None, inc_popsize: int = 2, use_separable_cma: bool = False, with_margin: bool = False, lr_adapt: bool = False, source_trials: Optional[List[FrozenTrial]] = None, ) -> None: self._x0 = x0 self._sigma0 = sigma0 self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) self._n_startup_trials = n_startup_trials self._warn_independent_sampling = warn_independent_sampling self._cma_rng = LazyRandomState(seed) self._search_space = IntersectionSearchSpace() self._consider_pruned_trials = consider_pruned_trials self._restart_strategy = restart_strategy self._initial_popsize = popsize self._inc_popsize = inc_popsize self._use_separable_cma = use_separable_cma self._with_margin = with_margin self._lr_adapt = lr_adapt self._source_trials = source_trials if self._restart_strategy: warnings.warn( "`restart_strategy` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if self._consider_pruned_trials: warnings.warn( "`consider_pruned_trials` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if self._use_separable_cma: warnings.warn( "`use_separable_cma` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if self._source_trials is not None: warnings.warn( "`source_trials` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if self._with_margin: warnings.warn( "`with_margin` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if self._lr_adapt: warnings.warn( "`lr_adapt` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if source_trials is not None and (x0 is not None or sigma0 is not None): raise ValueError( "It is prohibited to pass `source_trials` argument when " "x0 or sigma0 is specified." ) # TODO(c-bata): Support WS-sep-CMA-ES. if source_trials is not None and use_separable_cma: raise ValueError( "It is prohibited to pass `source_trials` argument when using separable CMA-ES." ) if lr_adapt and (use_separable_cma or with_margin): raise ValueError( "It is prohibited to pass `use_separable_cma` or `with_margin` argument when " "using `lr_adapt`." ) if restart_strategy not in ( "ipop", "bipop", None, ): raise ValueError( "restart_strategy={} is unsupported. " "Please specify: 'ipop', 'bipop', or None.".format(restart_strategy) ) # TODO(knshnb): Support sep-CMA-ES with margin. if self._use_separable_cma and self._with_margin: raise ValueError( "Currently, we do not support `use_separable_cma=True` and `with_margin=True`." ) def reseed_rng(self) -> None: # _cma_rng doesn't require reseeding because the relative sampling reseeds in each trial. self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial" ) -> Dict[str, BaseDistribution]: search_space: Dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # `cma` cannot handle distributions that contain just a single value, so we skip # them. Note that the parameter values for such distributions are sampled in # `Trial`. continue if not isinstance(distribution, (FloatDistribution, IntDistribution)): # Categorical distribution is unsupported. continue search_space[name] = distribution return search_space def sample_relative( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: self._raise_error_if_multi_objective(study) if len(search_space) == 0: return {} completed_trials = self._get_trials(study) if len(completed_trials) < self._n_startup_trials: return {} if len(search_space) == 1: if self._warn_independent_sampling: _logger.warning( "`CmaEsSampler` only supports two or more dimensional continuous " "search space. `{}` is used instead of `CmaEsSampler`.".format( self._independent_sampler.__class__.__name__ ) ) self._warn_independent_sampling = False return {} # When `with_margin=True`, bounds in discrete dimensions are handled inside `CMAwM`. trans = _SearchSpaceTransform( search_space, transform_step=not self._with_margin, transform_0_1=True ) if self._initial_popsize is None: self._initial_popsize = 4 + math.floor(3 * math.log(len(trans.bounds))) popsize: int = self._initial_popsize n_restarts: int = 0 n_restarts_with_large: int = 0 poptype: str = "small" small_n_eval: int = 0 large_n_eval: int = 0 if len(completed_trials) != 0: latest_trial = completed_trials[-1] popsize_attr_key = self._attr_keys.popsize() if popsize_attr_key in latest_trial.system_attrs: popsize = latest_trial.system_attrs[popsize_attr_key] else: popsize = self._initial_popsize n_restarts_attr_key = self._attr_keys.n_restarts() n_restarts = latest_trial.system_attrs.get(n_restarts_attr_key, 0) n_restarts_with_large = latest_trial.system_attrs.get( self._attr_keys.n_restarts_with_large, 0 ) poptype = latest_trial.system_attrs.get(self._attr_keys.poptype, "small") small_n_eval = latest_trial.system_attrs.get(self._attr_keys.small_n_eval, 0) large_n_eval = latest_trial.system_attrs.get(self._attr_keys.large_n_eval, 0) optimizer = self._restore_optimizer(completed_trials, n_restarts) if optimizer is None: optimizer = self._init_optimizer( trans, study.direction, population_size=self._initial_popsize ) if optimizer.dim != len(trans.bounds): if self._warn_independent_sampling: _logger.warning( "`CmaEsSampler` does not support dynamic search space. " "`{}` is used instead of `CmaEsSampler`.".format( self._independent_sampler.__class__.__name__ ) ) self._warn_independent_sampling = False return {} # TODO(c-bata): Reduce the number of wasted trials during parallel optimization. # See https://github.com/optuna/optuna/pull/920#discussion_r385114002 for details. solution_trials = self._get_solution_trials( completed_trials, optimizer.generation, n_restarts ) if len(solution_trials) >= popsize: solutions: List[Tuple[np.ndarray, float]] = [] for t in solution_trials[:popsize]: assert t.value is not None, "completed trials must have a value" if isinstance(optimizer, cmaes.CMAwM): x = np.array(t.system_attrs["x_for_tell"]) else: x = trans.transform(t.params) y = t.value if study.direction == StudyDirection.MINIMIZE else -t.value solutions.append((x, y)) optimizer.tell(solutions) if self._restart_strategy == "ipop" and optimizer.should_stop(): n_restarts += 1 popsize = popsize * self._inc_popsize optimizer = self._init_optimizer( trans, study.direction, population_size=popsize, randomize_start_point=True ) if self._restart_strategy == "bipop" and optimizer.should_stop(): n_restarts += 1 n_eval = popsize * optimizer.generation if poptype == "small": small_n_eval += n_eval else: # poptype == "large" large_n_eval += n_eval if small_n_eval < large_n_eval: poptype = "small" popsize_multiplier = self._inc_popsize**n_restarts_with_large popsize = math.floor( self._initial_popsize * popsize_multiplier ** (self._cma_rng.rng.uniform() ** 2) ) else: poptype = "large" n_restarts_with_large += 1 popsize = self._initial_popsize * (self._inc_popsize**n_restarts_with_large) optimizer = self._init_optimizer( trans, study.direction, population_size=popsize, randomize_start_point=True ) # Store optimizer. optimizer_str = pickle.dumps(optimizer).hex() optimizer_attrs = self._split_optimizer_str(optimizer_str, n_restarts) for key in optimizer_attrs: study._storage.set_trial_system_attr(trial._trial_id, key, optimizer_attrs[key]) # Caution: optimizer should update its seed value. seed = self._cma_rng.rng.randint(1, 2**16) + trial.number optimizer._rng.seed(seed) if isinstance(optimizer, cmaes.CMAwM): params, x_for_tell = optimizer.ask() study._storage.set_trial_system_attr( trial._trial_id, "x_for_tell", x_for_tell.tolist() ) else: params = optimizer.ask() generation_attr_key = self._attr_keys.generation(n_restarts) study._storage.set_trial_system_attr( trial._trial_id, generation_attr_key, optimizer.generation ) popsize_attr_key = self._attr_keys.popsize() study._storage.set_trial_system_attr(trial._trial_id, popsize_attr_key, popsize) n_restarts_attr_key = self._attr_keys.n_restarts() study._storage.set_trial_system_attr(trial._trial_id, n_restarts_attr_key, n_restarts) study._storage.set_trial_system_attr( trial._trial_id, self._attr_keys.n_restarts_with_large, n_restarts_with_large ) study._storage.set_trial_system_attr(trial._trial_id, self._attr_keys.poptype, poptype) study._storage.set_trial_system_attr( trial._trial_id, self._attr_keys.small_n_eval, small_n_eval ) study._storage.set_trial_system_attr( trial._trial_id, self._attr_keys.large_n_eval, large_n_eval ) external_values = trans.untransform(params) return external_values @property def _attr_keys(self) -> _CmaEsAttrKeys: if self._use_separable_cma: attr_prefix = "sepcma:" elif self._with_margin: attr_prefix = "cmawm:" else: attr_prefix = "cma:" def optimizer_key_template(restart: int) -> str: if self._restart_strategy is None: return attr_prefix + "optimizer" else: return attr_prefix + "{}:restart_{}:optimizer".format( self._restart_strategy, restart ) def generation_attr_key_template(restart: int) -> str: if self._restart_strategy is None: return attr_prefix + "generation" else: return attr_prefix + "{}:restart_{}:generation".format( self._restart_strategy, restart ) def popsize_attr_key_template() -> str: if self._restart_strategy is None: return attr_prefix + "popsize" else: return attr_prefix + "{}:popsize".format(self._restart_strategy) def n_restarts_attr_key_template() -> str: if self._restart_strategy is None: return attr_prefix + "n_restarts" else: return attr_prefix + "{}:n_restarts".format(self._restart_strategy) return _CmaEsAttrKeys( optimizer_key_template, generation_attr_key_template, popsize_attr_key_template, n_restarts_attr_key_template, attr_prefix + "n_restarts_with_large", attr_prefix + "poptype", attr_prefix + "small_n_eval", attr_prefix + "large_n_eval", ) def _concat_optimizer_attrs(self, optimizer_attrs: Dict[str, str], n_restarts: int = 0) -> str: return "".join( optimizer_attrs["{}:{}".format(self._attr_keys.optimizer(n_restarts), i)] for i in range(len(optimizer_attrs)) ) def _split_optimizer_str(self, optimizer_str: str, n_restarts: int = 0) -> Dict[str, str]: optimizer_len = len(optimizer_str) attrs = {} for i in range(math.ceil(optimizer_len / _SYSTEM_ATTR_MAX_LENGTH)): start = i * _SYSTEM_ATTR_MAX_LENGTH end = min((i + 1) * _SYSTEM_ATTR_MAX_LENGTH, optimizer_len) attrs["{}:{}".format(self._attr_keys.optimizer(n_restarts), i)] = optimizer_str[ start:end ] return attrs def _restore_optimizer( self, completed_trials: "List[optuna.trial.FrozenTrial]", n_restarts: int = 0, ) -> Optional["CmaClass"]: # Restore a previous CMA object. for trial in reversed(completed_trials): optimizer_attrs = { key: value for key, value in trial.system_attrs.items() if key.startswith(self._attr_keys.optimizer(n_restarts)) } if len(optimizer_attrs) == 0: continue optimizer_str = self._concat_optimizer_attrs(optimizer_attrs, n_restarts) return pickle.loads(bytes.fromhex(optimizer_str)) return None def _init_optimizer( self, trans: _SearchSpaceTransform, direction: StudyDirection, population_size: Optional[int] = None, randomize_start_point: bool = False, ) -> "CmaClass": lower_bounds = trans.bounds[:, 0] upper_bounds = trans.bounds[:, 1] n_dimension = len(trans.bounds) if self._source_trials is None: if randomize_start_point: mean = lower_bounds + (upper_bounds - lower_bounds) * self._cma_rng.rng.rand( n_dimension ) elif self._x0 is None: mean = lower_bounds + (upper_bounds - lower_bounds) / 2 else: # `self._x0` is external representations. mean = trans.transform(self._x0) if self._sigma0 is None: sigma0 = np.min((upper_bounds - lower_bounds) / 6) else: sigma0 = self._sigma0 cov = None else: expected_states = [TrialState.COMPLETE] if self._consider_pruned_trials: expected_states.append(TrialState.PRUNED) # TODO(c-bata): Filter parameters by their values instead of checking search space. sign = 1 if direction == StudyDirection.MINIMIZE else -1 source_solutions = [ (trans.transform(t.params), sign * cast(float, t.value)) for t in self._source_trials if t.state in expected_states and _is_compatible_search_space(trans, t.distributions) ] if len(source_solutions) == 0: raise ValueError("No compatible source_trials") # TODO(c-bata): Add options to change prior parameters (alpha and gamma). mean, sigma0, cov = cmaes.get_warm_start_mgd(source_solutions) # Avoid ZeroDivisionError in cmaes. sigma0 = max(sigma0, _EPS) if self._use_separable_cma: return cmaes.SepCMA( mean=mean, sigma=sigma0, bounds=trans.bounds, seed=self._cma_rng.rng.randint(1, 2**31 - 2), n_max_resampling=10 * n_dimension, population_size=population_size, ) if self._with_margin: steps = np.empty(len(trans._search_space), dtype=float) for i, dist in enumerate(trans._search_space.values()): assert isinstance(dist, (IntDistribution, FloatDistribution)) # Set step 0.0 for continuous search space. if dist.step is None or dist.log: steps[i] = 0.0 elif dist.low == dist.high: steps[i] = 1.0 else: steps[i] = dist.step / (dist.high - dist.low) return cmaes.CMAwM( mean=mean, sigma=sigma0, bounds=trans.bounds, steps=steps, cov=cov, seed=self._cma_rng.rng.randint(1, 2**31 - 2), n_max_resampling=10 * n_dimension, population_size=population_size, ) return cmaes.CMA( mean=mean, sigma=sigma0, cov=cov, bounds=trans.bounds, seed=self._cma_rng.rng.randint(1, 2**31 - 2), n_max_resampling=10 * n_dimension, population_size=population_size, lr_adapt=self._lr_adapt, ) def sample_independent( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: self._raise_error_if_multi_objective(study) if self._warn_independent_sampling: complete_trials = self._get_trials(study) if len(complete_trials) >= self._n_startup_trials: self._log_independent_sampling(trial, param_name) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def _log_independent_sampling(self, trial: FrozenTrial, param_name: str) -> None: _logger.warning( "The parameter '{}' in trial#{} is sampled independently " "by using `{}` instead of `CmaEsSampler` " "(optimization performance may be degraded). " "`CmaEsSampler` does not support dynamic search space or `CategoricalDistribution`. " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `CmaEsSampler`, " "if this independent sampling is intended behavior.".format( param_name, trial.number, self._independent_sampler.__class__.__name__ ) ) def _get_trials(self, study: "optuna.Study") -> List[FrozenTrial]: complete_trials = [] for t in study._get_trials(deepcopy=False, use_cache=True): if t.state == TrialState.COMPLETE: complete_trials.append(t) elif ( t.state == TrialState.PRUNED and len(t.intermediate_values) > 0 and self._consider_pruned_trials ): _, value = max(t.intermediate_values.items()) if value is None: continue # We rewrite the value of the trial `t` for sampling, so we need a deepcopy. copied_t = copy.deepcopy(t) copied_t.value = value complete_trials.append(copied_t) return complete_trials def _get_solution_trials( self, trials: List[FrozenTrial], generation: int, n_restarts: int ) -> List[FrozenTrial]: generation_attr_key = self._attr_keys.generation(n_restarts) return [t for t in trials if generation == t.system_attrs.get(generation_attr_key, -1)] def before_trial(self, study: optuna.Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", state: TrialState, values: Optional[Sequence[float]], ) -> None: self._independent_sampler.after_trial(study, trial, state, values) def _is_compatible_search_space( trans: _SearchSpaceTransform, search_space: Dict[str, BaseDistribution] ) -> bool: intersection_size = len(set(trans._search_space.keys()).intersection(search_space.keys())) return intersection_size == len(trans._search_space) == len(search_space) optuna-3.5.0/optuna/samplers/_grid.py000066400000000000000000000260361453453102400176270ustar00rootroot00000000000000import itertools from numbers import Real from typing import Any from typing import Dict from typing import List from typing import Mapping from typing import Optional from typing import Sequence from typing import Union import warnings import numpy as np from optuna.distributions import BaseDistribution from optuna.logging import get_logger from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState GridValueType = Union[str, float, int, bool, None] _logger = get_logger(__name__) class GridSampler(BaseSampler): """Sampler using grid search. With :class:`~optuna.samplers.GridSampler`, the trials suggest all combinations of parameters in the given search space during the study. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2 search_space = {"x": [-50, 0, 50], "y": [-99, 0, 99]} study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space)) study.optimize(objective) Note: :class:`~optuna.samplers.GridSampler` automatically stops the optimization if all combinations in the passed ``search_space`` have already been evaluated, internally invoking the :func:`~optuna.study.Study.stop` method. Note: :class:`~optuna.samplers.GridSampler` does not take care of a parameter's quantization specified by discrete suggest methods but just samples one of values specified in the search space. E.g., in the following code snippet, either of ``-0.5`` or ``0.5`` is sampled as ``x`` instead of an integer point. .. testcode:: import optuna def objective(trial): # The following suggest method specifies integer points between -5 and 5. x = trial.suggest_float("x", -5, 5, step=1) return x**2 # Non-int points are specified in the grid. search_space = {"x": [-0.5, 0.5]} study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space)) study.optimize(objective, n_trials=2) Note: A parameter configuration in the grid is not considered finished until its trial is finished. Therefore, during distributed optimization where trials run concurrently, different workers will occasionally suggest the same parameter configuration. The total number of actual trials may therefore exceed the size of the grid. Note: All parameters must be specified when using :class:`~optuna.samplers.GridSampler` with :meth:`~optuna.study.Study.enqueue_trial`. Args: search_space: A dictionary whose key and value are a parameter name and the corresponding candidates of values, respectively. seed: A seed to fix the order of trials as the grid is randomly shuffled. Please note that it is not recommended using this option in distributed optimization settings since this option cannot ensure the order of trials and may increase the number of duplicate suggestions during distributed optimization. """ def __init__( self, search_space: Mapping[str, Sequence[GridValueType]], seed: Optional[int] = None ) -> None: for param_name, param_values in search_space.items(): for value in param_values: self._check_value(param_name, value) self._search_space = {} for param_name, param_values in sorted(search_space.items()): self._search_space[param_name] = list(param_values) self._all_grids = list(itertools.product(*self._search_space.values())) self._param_names = sorted(search_space.keys()) self._n_min_trials = len(self._all_grids) self._rng = LazyRandomState(seed) self._rng.rng.shuffle(self._all_grids) def reseed_rng(self) -> None: self._rng.rng.seed() def before_trial(self, study: Study, trial: FrozenTrial) -> None: # Instead of returning param values, GridSampler puts the target grid id as a system attr, # and the values are returned from `sample_independent`. This is because the distribution # object is hard to get at the beginning of trial, while we need the access to the object # to validate the sampled value. # When the trial is created by RetryFailedTrialCallback or enqueue_trial, we should not # assign a new grid_id. if "grid_id" in trial.system_attrs or "fixed_params" in trial.system_attrs: return if 0 <= trial.number and trial.number < self._n_min_trials: study._storage.set_trial_system_attr( trial._trial_id, "search_space", self._search_space ) study._storage.set_trial_system_attr(trial._trial_id, "grid_id", trial.number) return target_grids = self._get_unvisited_grid_ids(study) if len(target_grids) == 0: # This case may occur with distributed optimization or trial queue. If there is no # target grid, `GridSampler` evaluates a visited, duplicated point with the current # trial. After that, the optimization stops. _logger.warning( "`GridSampler` is re-evaluating a configuration because the grid has been " "exhausted. This may happen due to a timing issue during distributed optimization " "or when re-running optimizations on already finished studies." ) # One of all grids is randomly picked up in this case. target_grids = list(range(len(self._all_grids))) # In distributed optimization, multiple workers may simultaneously pick up the same grid. # To make the conflict less frequent, the grid is chosen randomly. grid_id = int(self._rng.rng.choice(target_grids)) study._storage.set_trial_system_attr(trial._trial_id, "search_space", self._search_space) study._storage.set_trial_system_attr(trial._trial_id, "grid_id", grid_id) def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: return {} def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: return {} def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: if "grid_id" not in trial.system_attrs: message = "All parameters must be specified when using GridSampler with enqueue_trial." raise ValueError(message) if param_name not in self._search_space: message = "The parameter name, {}, is not found in the given grid.".format(param_name) raise ValueError(message) # TODO(c-bata): Reduce the number of duplicated evaluations on multiple workers. # Current selection logic may evaluate the same parameters multiple times. # See https://gist.github.com/c-bata/f759f64becb24eea2040f4b2e3afce8f for details. grid_id = trial.system_attrs["grid_id"] param_value = self._all_grids[grid_id][self._param_names.index(param_name)] contains = param_distribution._contains(param_distribution.to_internal_repr(param_value)) if not contains: warnings.warn( f"The value `{param_value}` is out of range of the parameter `{param_name}`. " f"The value will be used but the actual distribution is: `{param_distribution}`." ) return param_value def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: target_grids = self._get_unvisited_grid_ids(study) if len(target_grids) == 0: study.stop() elif len(target_grids) == 1: grid_id = study._storage.get_trial_system_attrs(trial._trial_id)["grid_id"] if grid_id == target_grids[0]: study.stop() @staticmethod def _check_value(param_name: str, param_value: Any) -> None: if param_value is None or isinstance(param_value, (str, int, float, bool)): return message = ( "{} contains a value with the type of {}, which is not supported by " "`GridSampler`. Please make sure a value is `str`, `int`, `float`, `bool`" " or `None` for persistent storage.".format(param_name, type(param_value)) ) warnings.warn(message) def _get_unvisited_grid_ids(self, study: Study) -> List[int]: # List up unvisited grids based on already finished ones. visited_grids = [] running_grids = [] # We directly query the storage to get trials here instead of `study.get_trials`, # since some pruners such as `HyperbandPruner` use the study transformed # to filter trials. See https://github.com/optuna/optuna/issues/2327 for details. trials = study._storage.get_all_trials(study._study_id, deepcopy=False) for t in trials: if "grid_id" in t.system_attrs and self._same_search_space( t.system_attrs["search_space"] ): if t.state.is_finished(): visited_grids.append(t.system_attrs["grid_id"]) elif t.state == TrialState.RUNNING: running_grids.append(t.system_attrs["grid_id"]) unvisited_grids = set(range(self._n_min_trials)) - set(visited_grids) - set(running_grids) # If evaluations for all grids have been started, return grids that have not yet finished # because all grids should be evaluated before stopping the optimization. if len(unvisited_grids) == 0: unvisited_grids = set(range(self._n_min_trials)) - set(visited_grids) return list(unvisited_grids) @staticmethod def _grid_value_equal(value1: GridValueType, value2: GridValueType) -> bool: value1_is_nan = isinstance(value1, Real) and np.isnan(float(value1)) value2_is_nan = isinstance(value2, Real) and np.isnan(float(value2)) return (value1 == value2) or (value1_is_nan and value2_is_nan) def _same_search_space(self, search_space: Mapping[str, Sequence[GridValueType]]) -> bool: if set(search_space.keys()) != set(self._search_space.keys()): return False for param_name in search_space.keys(): if len(search_space[param_name]) != len(self._search_space[param_name]): return False for i, param_value in enumerate(search_space[param_name]): if not self._grid_value_equal(param_value, self._search_space[param_name][i]): return False return True optuna-3.5.0/optuna/samplers/_lazy_random_state.py000066400000000000000000000013351453453102400224140ustar00rootroot00000000000000from __future__ import annotations import numpy class LazyRandomState: """Lazy Random State class. This is a class to initialize the random state just before use to prevent duplication of the same random state when deepcopy is applied to the instance of sampler. """ def __init__(self, seed: int | None = None) -> None: self._rng: numpy.random.RandomState | None = None if seed is not None: self.rng.seed(seed=seed) def _set_rng(self) -> None: self._rng = numpy.random.RandomState() @property def rng(self) -> numpy.random.RandomState: if self._rng is None: self._set_rng() assert self._rng is not None return self._rng optuna-3.5.0/optuna/samplers/_nsgaiii/000077500000000000000000000000001453453102400177445ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/_nsgaiii/__init__.py000066400000000000000000000000001453453102400220430ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/_nsgaiii/_elite_population_selection_strategy.py000066400000000000000000000346701453453102400300320ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import itertools import math import numpy as np from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers.nsgaii._dominates import _constrained_dominates from optuna.samplers.nsgaii._dominates import _validate_constraints from optuna.samplers.nsgaii._elite_population_selection_strategy import _fast_non_dominated_sort from optuna.study import Study from optuna.study._multi_objective import _dominates from optuna.trial import FrozenTrial # Define a coefficient for scaling intervals, used in _filter_inf() to replace +-inf. _COEF = 3 class NSGAIIIElitePopulationSelectionStrategy: def __init__( self, *, population_size: int, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, reference_points: np.ndarray | None = None, dividing_parameter: int = 3, rng: LazyRandomState, ) -> None: if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") self._population_size = population_size self._constraints_func = constraints_func self._reference_points = reference_points self._dividing_parameter = dividing_parameter self._rng = rng def __call__(self, study: Study, population: list[FrozenTrial]) -> list[FrozenTrial]: """Select elite population from the given trials by NSGA-III algorithm. Args: study: Target study object. population: Trials in the study. Returns: A list of trials that are selected as elite population. """ _validate_constraints(population, self._constraints_func) dominates = _dominates if self._constraints_func is None else _constrained_dominates population_per_rank = _fast_non_dominated_sort(population, study.directions, dominates) elite_population: list[FrozenTrial] = [] for population in population_per_rank: if len(elite_population) + len(population) < self._population_size: elite_population.extend(population) else: n_objectives = len(study.directions) # Construct reference points in the first run. if self._reference_points is None: self._reference_points = _generate_default_reference_point( n_objectives, self._dividing_parameter ) elif np.shape(self._reference_points)[1] != n_objectives: raise ValueError( "The dimension of reference points vectors must be the same as the number " "of objectives of the study." ) # Normalize objective values after filtering +-inf. objective_matrix = _normalize_objective_values( _filter_inf(elite_population + population) ) ( closest_reference_points, distance_reference_points, ) = _associate_individuals_with_reference_points( objective_matrix, self._reference_points ) elite_population_num = len(elite_population) target_population_size = self._population_size - elite_population_num additional_elite_population = _preserve_niche_individuals( target_population_size, elite_population_num, population, closest_reference_points, distance_reference_points, self._rng.rng, ) elite_population.extend(additional_elite_population) break return elite_population # TODO(Shinichi) Replace with math.comb after support for python3.7 is deprecated. # This function calculates n multi-choose k, which is the total number of combinations with # repetition of size k from n items. This is equally re-written as math.comb(n+k-1, k) def _multi_choose(n: int, k: int) -> int: return math.factorial(n + k - 1) // math.factorial(k) // math.factorial(n - 1) def _generate_default_reference_point( n_objectives: int, dividing_parameter: int = 3 ) -> np.ndarray: """Generates default reference points which are `uniformly` spread on a hyperplane.""" reference_points = np.zeros( ( _multi_choose(n_objectives, dividing_parameter), n_objectives, ) ) for i, comb in enumerate( itertools.combinations_with_replacement(range(n_objectives), dividing_parameter) ): for j in comb: reference_points[i, j] += 1.0 return reference_points def _filter_inf(population: list[FrozenTrial]) -> np.ndarray: # Collect all objective values. n_objectives = len(population[0].values) objective_matrix = np.zeros((len(population), n_objectives)) for i, trial in enumerate(population): objective_matrix[i] = np.array(trial.values, dtype=float) mask_posinf = np.isposinf(objective_matrix) mask_neginf = np.isneginf(objective_matrix) # Replace +-inf with nan temporary to get max and min. objective_matrix[mask_posinf + mask_neginf] = np.nan nadir_point = np.nanmax(objective_matrix, axis=0) ideal_point = np.nanmin(objective_matrix, axis=0) interval = nadir_point - ideal_point # TODO(Shinichi) reconsider alternative value for inf. rows_posinf, cols_posinf = np.where(mask_posinf) objective_matrix[rows_posinf, cols_posinf] = ( nadir_point[cols_posinf] + _COEF * interval[cols_posinf] ) rows_neginf, cols_neginf = np.where(mask_neginf) objective_matrix[rows_neginf, cols_neginf] = ( ideal_point[cols_neginf] - _COEF * interval[cols_neginf] ) return objective_matrix def _normalize_objective_values(objective_matrix: np.ndarray) -> np.ndarray: """Normalizes objective values of population. An ideal point z* consists of minimums in each axis. Each objective value of population is then subtracted by the ideal point. An extreme point of each axis is (originally) defined as a minimum solution of achievement scalarizing function from the population. After that, intercepts are calculate as intercepts of hyperplane which has all the extreme points on it and used to rescale objective values. We adopt weights and achievement scalarizing function(ASF) used in pre-print of the NSGA-III paper (See https://www.egr.msu.edu/~kdeb/papers/k2012009.pdf). """ n_objectives = np.shape(objective_matrix)[1] # Subtract ideal point from objective values. objective_matrix -= np.min(objective_matrix, axis=0) # Initialize weights. weights = np.eye(n_objectives) weights[weights == 0] = 1e6 # Calculate extreme points to normalize objective values. # TODO(Shinichi) Reimplement to reduce time complexity. asf_value = np.max( np.einsum("nm,dm->dnm", objective_matrix, weights), axis=2, ) extreme_points = objective_matrix[np.argmin(asf_value, axis=1), :] # Normalize objective_matrix with extreme points. # Note that extreme_points can be degenerate, but no proper operation is remarked in the # paper. Therefore, the maximum value of population in each axis is used in such cases. if np.all(np.isfinite(extreme_points)) and np.linalg.matrix_rank(extreme_points) == len( extreme_points ): intercepts_inv = np.linalg.solve(extreme_points, np.ones(n_objectives)) else: intercepts_inv = 1 / np.max(objective_matrix, axis=0) objective_matrix *= np.where(np.isfinite(intercepts_inv), intercepts_inv, 1) return objective_matrix def _associate_individuals_with_reference_points( objective_matrix: np.ndarray, reference_points: np.ndarray ) -> tuple[np.ndarray, np.ndarray]: """Associates each objective value to the closest reference point. Associate each normalized objective value to the closest reference point. The distance is calculated by Euclidean norm. Args: objective_matrix: A 2 dimension ``numpy.ndarray`` with columns of objective dimension and rows of generation size. Each row is the normalized objective value of the corresponding individual. Returns: closest_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the index of the closest reference point to the corresponding individual. distance_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the distance from the corresponding individual to the closest reference point. """ # TODO(Shinichi) Implement faster assignment for the default reference points because it does # not seem necessary to calculate distance from all reference points. # TODO(Shinichi) Normalize reference_points in constructor to remove reference_point_norms. # In addition, the minimum distance from each reference point can be replaced with maximum # inner product between the given individual and each normalized reference points. # distance_from_reference_lines is a ndarray of shape (n, p), where n is the size of the # population and p is the number of reference points. Its (i,j) entry keeps distance between # the i-th individual values and the j-th reference line. reference_point_norm_squared = np.linalg.norm(reference_points, axis=1) ** 2 perpendicular_vectors_to_reference_lines = np.einsum( "ni,pi,p,pm->npm", objective_matrix, reference_points, 1 / reference_point_norm_squared, reference_points, ) distance_from_reference_lines = np.linalg.norm( objective_matrix[:, np.newaxis, :] - perpendicular_vectors_to_reference_lines, axis=2, ) closest_reference_points: np.ndarray = np.argmin(distance_from_reference_lines, axis=1) distance_reference_points: np.ndarray = np.min(distance_from_reference_lines, axis=1) return closest_reference_points, distance_reference_points def _preserve_niche_individuals( target_population_size: int, elite_population_num: int, population: list[FrozenTrial], closest_reference_points: np.ndarray, distance_reference_points: np.ndarray, rng: np.random.RandomState, ) -> list[FrozenTrial]: """Determine who survives form the borderline front. Who survive form the borderline front is determined according to the sparsity of each closest reference point. The algorithm picks a reference point from those who have the least neighbors in elite population and adds one of borderline front member who has the same closest reference point. Args: target_population_size: The number of individuals to select. elite_population_num: The number of individuals which are already selected as the elite population. population: List of all the trials in the current surviving generation. distance_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the distance from the corresponding individual to the closest reference point. closest_reference_points: A ``numpy.ndarray`` with rows of generation size. Each row is the index of the closest reference point to the corresponding individual. rng: Random number generator. Returns: A list of trials which are selected as the next generation. """ if len(population) < target_population_size: raise ValueError( "The population size must be greater than or equal to the target population size." ) # reference_point_to_borderline_population keeps pairs of a neighbor and the distance of # each reference point from borderline front population. reference_point_to_borderline_population = defaultdict(list) for i, reference_point_idx in enumerate(closest_reference_points[elite_population_num:]): population_idx = i + elite_population_num reference_point_to_borderline_population[reference_point_idx].append( (distance_reference_points[population_idx], i) ) # reference_points_to_elite_population_count keeps how many elite neighbors each reference # point has. reference_point_to_elite_population_count: dict[int, int] = defaultdict(int) for i, reference_point_idx in enumerate(closest_reference_points[:elite_population_num]): reference_point_to_elite_population_count[reference_point_idx] += 1 # nearest_points_count_to_reference_points classifies reference points which have at least one # closest borderline population member by the number of elite neighbors they have. Each key # corresponds to the number of elite neighbors and the value to the reference point indices. nearest_points_count_to_reference_points = defaultdict(list) for reference_point_idx in reference_point_to_borderline_population: elite_population_count = reference_point_to_elite_population_count[reference_point_idx] nearest_points_count_to_reference_points[elite_population_count].append( reference_point_idx ) count = -1 additional_elite_population: list[FrozenTrial] = [] is_shuffled: defaultdict[int, bool] = defaultdict(bool) while len(additional_elite_population) < target_population_size: if len(nearest_points_count_to_reference_points[count]) == 0: count += 1 rng.shuffle(nearest_points_count_to_reference_points[count]) continue reference_point_idx = nearest_points_count_to_reference_points[count].pop() if count > 0 and not is_shuffled[reference_point_idx]: rng.shuffle(reference_point_to_borderline_population[reference_point_idx]) is_shuffled[reference_point_idx] = True elif count == 0: reference_point_to_borderline_population[reference_point_idx].sort(reverse=True) _, selected_individual_id = reference_point_to_borderline_population[ reference_point_idx ].pop() additional_elite_population.append(population[selected_individual_id]) if reference_point_to_borderline_population[reference_point_idx]: nearest_points_count_to_reference_points[count + 1].append(reference_point_idx) return additional_elite_population optuna-3.5.0/optuna/samplers/_nsgaiii/_sampler.py000066400000000000000000000302651453453102400221260ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import hashlib from typing import Any import numpy as np import optuna from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( NSGAIIIElitePopulationSelectionStrategy, ) from optuna.samplers._random import RandomSampler from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.samplers.nsgaii._child_generation_strategy import NSGAIIChildGenerationStrategy from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._crossovers._uniform import UniformCrossover from optuna.search_space import IntersectionSearchSpace from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState # Define key names of `Trial.system_attrs`. _GENERATION_KEY = "nsga3:generation" _POPULATION_CACHE_KEY_PREFIX = "nsga3:population" @experimental_class("3.2.0") class NSGAIIISampler(BaseSampler): """Multi-objective sampler using the NSGA-III algorithm. NSGA-III stands for "Nondominated Sorting Genetic Algorithm III", which is a modified version of NSGA-II for many objective optimization problem. For further information about NSGA-III, please refer to the following papers: - `An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints `_ - `An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part II: Handling Constraints and Extending to an Adaptive Approach `_ Args: reference_points: A 2 dimension ``numpy.ndarray`` with objective dimension columns. Represents a list of reference points which is used to determine who to survive. After non-dominated sort, who out of borderline front are going to survived is determined according to how sparse the closest reference point of each individual is. In the default setting the algorithm uses `uniformly` spread points to diversify the result. It is also possible to reflect your `preferences` by giving an arbitrary set of `target` points since the algorithm prioritizes individuals around reference points. dividing_parameter: A parameter to determine the density of default reference points. This parameter determines how many divisions are made between reference points on each axis. The smaller this value is, the less reference points you have. The default value is 3. Note that this parameter is not used when ``reference_points`` is not :obj:`None`. .. note:: Other parameters than ``reference_points`` and ``dividing_parameter`` are the same as :class:`~optuna.samplers.NSGAIISampler`. """ def __init__( self, *, population_size: int = 50, mutation_prob: float | None = None, crossover: BaseCrossover | None = None, crossover_prob: float = 0.9, swapping_prob: float = 0.5, seed: int | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, reference_points: np.ndarray | None = None, dividing_parameter: int = 3, elite_population_selection_strategy: Callable[ [Study, list[FrozenTrial]], list[FrozenTrial] ] | None = None, child_generation_strategy: Callable[ [Study, dict[str, BaseDistribution], list[FrozenTrial]], dict[str, Any] ] | None = None, after_trial_strategy: Callable[ [Study, FrozenTrial, TrialState, Sequence[float] | None], None ] | None = None, ) -> None: # TODO(ohta): Reconsider the default value of each parameter. if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") if crossover is None: crossover = UniformCrossover(swapping_prob) if not isinstance(crossover, BaseCrossover): raise ValueError( f"'{crossover}' is not a valid crossover." " For valid crossovers see" " https://optuna.readthedocs.io/en/stable/reference/samplers.html." ) if population_size < crossover.n_parents: raise ValueError( f"Using {crossover}," f" the population size should be greater than or equal to {crossover.n_parents}." f" The specified `population_size` is {population_size}." ) self._population_size = population_size self._random_sampler = RandomSampler(seed=seed) self._rng = LazyRandomState(seed) self._constraints_func = constraints_func self._search_space = IntersectionSearchSpace() self._elite_population_selection_strategy = ( elite_population_selection_strategy or NSGAIIIElitePopulationSelectionStrategy( population_size=population_size, constraints_func=constraints_func, reference_points=reference_points, dividing_parameter=dividing_parameter, rng=self._rng, ) ) self._child_generation_strategy = ( child_generation_strategy or NSGAIIChildGenerationStrategy( crossover_prob=crossover_prob, mutation_prob=mutation_prob, swapping_prob=swapping_prob, crossover=crossover, constraints_func=constraints_func, rng=self._rng, ) ) self._after_trial_strategy = after_trial_strategy or NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) def reseed_rng(self) -> None: self._random_sampler.reseed_rng() self._rng.rng.seed() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: search_space: dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # The `untransform` method of `optuna._transform._SearchSpaceTransform` # does not assume a single value, # so single value objects are not sampled with the `sample_relative` method, # but with the `sample_independent` method. continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: parent_generation, parent_population = self._collect_parent_population(study) generation = parent_generation + 1 study._storage.set_trial_system_attr(trial._trial_id, _GENERATION_KEY, generation) if parent_generation < 0: return {} return self._child_generation_strategy(study, search_space, parent_population) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: # Following parameters are randomly sampled here. # 1. A parameter in the initial population/first generation. # 2. A parameter to mutate. # 3. A parameter excluded from the intersection search space. return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) def _collect_parent_population(self, study: Study) -> tuple[int, list[FrozenTrial]]: trials = study.get_trials(deepcopy=False) generation_to_runnings = defaultdict(list) generation_to_population = defaultdict(list) for trial in trials: if _GENERATION_KEY not in trial.system_attrs: continue generation = trial.system_attrs[_GENERATION_KEY] if trial.state != optuna.trial.TrialState.COMPLETE: if trial.state == optuna.trial.TrialState.RUNNING: generation_to_runnings[generation].append(trial) continue # Do not use trials whose states are not COMPLETE, or `constraint` will be unavailable. generation_to_population[generation].append(trial) hasher = hashlib.sha256() parent_population: list[FrozenTrial] = [] parent_generation = -1 while True: generation = parent_generation + 1 population = generation_to_population[generation] # Under multi-worker settings, the population size might become larger than # `self._population_size`. if len(population) < self._population_size: break # [NOTE] # It's generally safe to assume that once the above condition is satisfied, # there are no additional individuals added to the generation (i.e., the members of # the generation have been fixed). # If the number of parallel workers is huge, this assumption can be broken, but # this is a very rare case and doesn't significantly impact optimization performance. # So we can ignore the case. # The cache key is calculated based on the key of the previous generation and # the remaining running trials in the current population. # If there are no running trials, the new cache key becomes exactly the same as # the previous one, and the cached content will be overwritten. This allows us to # skip redundant cache key calculations when this method is called for the subsequent # trials. for trial in generation_to_runnings[generation]: hasher.update(bytes(str(trial.number), "utf-8")) cache_key = "{}:{}".format(_POPULATION_CACHE_KEY_PREFIX, hasher.hexdigest()) study_system_attrs = study._storage.get_study_system_attrs(study._study_id) cached_generation, cached_population_numbers = study_system_attrs.get( cache_key, (-1, []) ) if cached_generation >= generation: generation = cached_generation population = [trials[n] for n in cached_population_numbers] else: population.extend(parent_population) population = self._elite_population_selection_strategy(study, population) # To reduce the number of system attribute entries, # we cache the population information only if there are no running trials # (i.e., the information of the population has been fixed). # Usually, if there are no too delayed running trials, the single entry # will be used. if len(generation_to_runnings[generation]) == 0: population_numbers = [t.number for t in population] study._storage.set_study_system_attr( study._study_id, cache_key, (generation, population_numbers) ) parent_generation = generation parent_population = population return parent_generation, parent_population def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._random_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert state in [TrialState.COMPLETE, TrialState.FAIL, TrialState.PRUNED] self._after_trial_strategy(study, trial, state, values) self._random_sampler.after_trial(study, trial, state, values) optuna-3.5.0/optuna/samplers/_partial_fixed.py000066400000000000000000000073201453453102400215100ustar00rootroot00000000000000from typing import Any from typing import Dict from typing import Optional from typing import Sequence import warnings from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.samplers import BaseSampler from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState @experimental_class("2.4.0") class PartialFixedSampler(BaseSampler): """Sampler with partially fixed parameters. Example: After several steps of optimization, you can fix the value of ``y`` and re-optimize it. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y study = optuna.create_study() study.optimize(objective, n_trials=10) best_params = study.best_params fixed_params = {"y": best_params["y"]} partial_sampler = optuna.samplers.PartialFixedSampler(fixed_params, study.sampler) study.sampler = partial_sampler study.optimize(objective, n_trials=10) Args: fixed_params: A dictionary of parameters to be fixed. base_sampler: A sampler which samples unfixed parameters. """ def __init__(self, fixed_params: Dict[str, Any], base_sampler: BaseSampler) -> None: self._fixed_params = fixed_params self._base_sampler = base_sampler def reseed_rng(self) -> None: self._base_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: search_space = self._base_sampler.infer_relative_search_space(study, trial) # Remove fixed params from relative search space to return fixed values. for param_name in self._fixed_params.keys(): if param_name in search_space: del search_space[param_name] return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution], ) -> Dict[str, Any]: # Fixed params are never sampled here. return self._base_sampler.sample_relative(study, trial, search_space) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: if param_name not in self._fixed_params: # Unfixed params are sampled here. return self._base_sampler.sample_independent( study, trial, param_name, param_distribution ) else: # Fixed params are sampled here. # Check if a parameter value is contained in the range of this distribution. param_value = self._fixed_params[param_name] param_value_in_internal_repr = param_distribution.to_internal_repr(param_value) contained = param_distribution._contains(param_value_in_internal_repr) if not contained: warnings.warn( f"Fixed parameter '{param_name}' with value {param_value} is out of range " f"for distribution {param_distribution}." ) return param_value def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._base_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: self._base_sampler.after_trial(study, trial, state, values) optuna-3.5.0/optuna/samplers/_qmc.py000066400000000000000000000313651453453102400174630ustar00rootroot00000000000000from typing import Any from typing import Dict from typing import Optional from typing import Sequence import numpy as np import optuna from optuna import logging from optuna._experimental import experimental_class from optuna._imports import _LazyImport from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.samplers import BaseSampler from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = logging.get_logger(__name__) _SUGGESTED_STATES = (TrialState.COMPLETE, TrialState.PRUNED) @experimental_class("3.0.0") class QMCSampler(BaseSampler): """A Quasi Monte Carlo Sampler that generates low-discrepancy sequences. Quasi Monte Carlo (QMC) sequences are designed to have lower discrepancies than standard random sequences. They are known to perform better than the standard random sequences in hyperparameter optimization. For further information about the use of QMC sequences for hyperparameter optimization, please refer to the following paper: - `Bergstra, James, and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research 13.2, 2012. `_ We use the QMC implementations in Scipy. For the details of the QMC algorithm, see the Scipy API references on `scipy.stats.qmc `_. .. note: If your search space contains categorical parameters, it samples the categorical parameters by its `independent_sampler` without using QMC algorithm. .. note:: The search space of the sampler is determined by either previous trials in the study or the first trial that this sampler samples. If there are previous trials in the study, :class:`~optuna.samplers.QMCSampler` infers its search space using the trial which was created first in the study. Otherwise (if the study has no previous trials), :class:`~optuna.samplers.QMCSampler` samples the first trial using its `independent_sampler` and then infers the search space in the second trial. As mentioned above, the search space of the :class:`~optuna.samplers.QMCSampler` is determined by the first trial of the study. Once the search space is determined, it cannot be changed afterwards. Args: qmc_type: The type of QMC sequence to be sampled. This must be one of `"halton"` and `"sobol"`. Default is `"sobol"`. .. note:: Sobol' sequence is designed to have low-discrepancy property when the number of samples is :math:`n=2^m` for each positive integer :math:`m`. When it is possible to pre-specify the number of trials suggested by `QMCSampler`, it is recommended that the number of trials should be set as power of two. scramble: If this option is :obj:`True`, scrambling (randomization) is applied to the QMC sequences. seed: A seed for ``QMCSampler``. This argument is used only when ``scramble`` is :obj:`True`. If this is :obj:`None`, the seed is initialized randomly. Default is :obj:`None`. .. note:: When using multiple :class:`~optuna.samplers.QMCSampler`'s in parallel and/or distributed optimization, all the samplers must share the same seed when the `scrambling` is enabled. Otherwise, the low-discrepancy property of the samples will be degraded. independent_sampler: A :class:`~optuna.samplers.BaseSampler` instance that is used for independent sampling. The first trial of the study and the parameters not contained in the relative search space are sampled by this sampler. If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used as the default. .. seealso:: :class:`~optuna.samplers` module provides built-in independent samplers such as :class:`~optuna.samplers.RandomSampler` and :class:`~optuna.samplers.TPESampler`. warn_independent_sampling: If this is :obj:`True`, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. Note that the parameters of the first trial in a study are sampled via an independent sampler in most cases, so no warning messages are emitted in such cases. warn_asynchronous_seeding: If this is :obj:`True`, a warning message is emitted when the scrambling (randomization) is applied to the QMC sequence and the random seed of the sampler is not set manually. .. note:: When using parallel and/or distributed optimization without manually setting the seed, the seed is set randomly for each instances of :class:`~optuna.samplers.QMCSampler` for different workers, which ends up asynchronous seeding for multiple samplers used in the optimization. .. seealso:: See parameter ``seed`` in :class:`~optuna.samplers.QMCSampler`. Example: Optimize a simple quadratic function by using :class:`~optuna.samplers.QMCSampler`. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y sampler = optuna.samplers.QMCSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=8) """ def __init__( self, *, qmc_type: str = "sobol", scramble: bool = False, # default is False for simplicity in distributed environment. seed: Optional[int] = None, independent_sampler: Optional[BaseSampler] = None, warn_asynchronous_seeding: bool = True, warn_independent_sampling: bool = True, ) -> None: self._scramble = scramble self._seed = np.random.PCG64().random_raw() if seed is None else seed self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) self._initial_search_space: Optional[Dict[str, BaseDistribution]] = None self._warn_independent_sampling = warn_independent_sampling if qmc_type in ("halton", "sobol"): self._qmc_type = qmc_type else: message = ( f'The `qmc_type`, "{qmc_type}", is not a valid. ' 'It must be one of "halton" and "sobol".' ) raise ValueError(message) if seed is None and scramble and warn_asynchronous_seeding: # Sobol/Halton sequences without scrambling do not use seed. self._log_asynchronous_seeding() def reseed_rng(self) -> None: # We must not reseed the `self._seed` like below. Otherwise, workers will have different # seed under parallel execution because `self.reseed_rng()` is called when starting each # parallel executor. # >>> self._seed = np.random.MT19937().random_raw() self._independent_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: if self._initial_search_space is not None: return self._initial_search_space past_trials = study._get_trials(deepcopy=False, states=_SUGGESTED_STATES, use_cache=True) # The initial trial is sampled by the independent sampler. if len(past_trials) == 0: return {} # If an initial trial was already made, # construct search_space of this sampler from the initial trial. first_trial = min(past_trials, key=lambda t: t.number) self._initial_search_space = self._infer_initial_search_space(first_trial) return self._initial_search_space def _infer_initial_search_space(self, trial: FrozenTrial) -> Dict[str, BaseDistribution]: search_space: Dict[str, BaseDistribution] = {} for param_name, distribution in trial.distributions.items(): if isinstance(distribution, CategoricalDistribution): continue search_space[param_name] = distribution return search_space @staticmethod def _log_asynchronous_seeding() -> None: _logger.warning( "No seed is provided for `QMCSampler` and the seed is set randomly. " "If you are running multiple `QMCSampler`s in parallel and/or distributed " " environment, the same seed must be used in all samplers to ensure that resulting " "samples are taken from the same QMC sequence. " ) def _log_independent_sampling(self, trial: FrozenTrial, param_name: str) -> None: _logger.warning( f"The parameter '{param_name}' in trial#{trial.number} is sampled independently " f"by using `{self._independent_sampler.__class__.__name__}` instead of `QMCSampler` " "(optimization performance may be degraded). " "`QMCSampler` does not support dynamic search space or `CategoricalDistribution`. " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `QMCSampler`, " "if this independent sampling is intended behavior." ) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: if self._initial_search_space is not None: if self._warn_independent_sampling: self._log_independent_sampling(trial, param_name) return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution ) def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: if search_space == {}: return {} sample = self._sample_qmc(study, search_space) trans = _SearchSpaceTransform(search_space) sample = trans.bounds[:, 0] + sample * (trans.bounds[:, 1] - trans.bounds[:, 0]) return trans.untransform(sample[0, :]) def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._independent_sampler.before_trial(study, trial) def after_trial( self, study: "optuna.Study", trial: "optuna.trial.FrozenTrial", state: TrialState, values: Optional[Sequence[float]], ) -> None: self._independent_sampler.after_trial(study, trial, state, values) def _sample_qmc(self, study: Study, search_space: Dict[str, BaseDistribution]) -> np.ndarray: # Lazy import because the `scipy.stats.qmc` is slow to import. qmc_module = _LazyImport("scipy.stats.qmc") sample_id = self._find_sample_id(study) d = len(search_space) if self._qmc_type == "halton": qmc_engine = qmc_module.Halton(d, seed=self._seed, scramble=self._scramble) elif self._qmc_type == "sobol": qmc_engine = qmc_module.Sobol(d, seed=self._seed, scramble=self._scramble) else: raise ValueError("Invalid `qmc_type`") forward_size = sample_id # `sample_id` starts from 0. # Skip fast_forward with forward_size==0 because Sobol doesn't support the case, # and fast_forward(0) doesn't affect sampling. if forward_size > 0: qmc_engine.fast_forward(forward_size) sample = qmc_engine.random(1) return sample def _find_sample_id(self, study: Study) -> int: qmc_id = "" qmc_id += self._qmc_type # Sobol/Halton sequences without scrambling do not use seed. if self._scramble: qmc_id += f" (scramble=True, seed={self._seed})" else: qmc_id += " (scramble=False)" key_qmc_id = qmc_id + "'s last sample id" # TODO(kstoneriv3): Here, we ideally assume that the following block is # an atomic transaction. Without such an assumption, the current implementation # only ensures that each `sample_id` is sampled at least once. system_attrs = study._storage.get_study_system_attrs(study._study_id) if key_qmc_id in system_attrs.keys(): sample_id = system_attrs[key_qmc_id] sample_id += 1 else: sample_id = 0 study._storage.set_study_system_attr(study._study_id, key_qmc_id, sample_id) return sample_id optuna-3.5.0/optuna/samplers/_random.py000066400000000000000000000036201453453102400201540ustar00rootroot00000000000000from typing import Any from typing import Dict from typing import Optional from optuna import distributions from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.study import Study from optuna.trial import FrozenTrial class RandomSampler(BaseSampler): """Sampler using random sampling. This sampler is based on *independent sampling*. See also :class:`~optuna.samplers.BaseSampler` for more details of 'independent sampling'. Example: .. testcode:: import optuna from optuna.samplers import RandomSampler def objective(trial): x = trial.suggest_float("x", -5, 5) return x**2 study = optuna.create_study(sampler=RandomSampler()) study.optimize(objective, n_trials=10) Args: seed: Seed for random number generator. """ def __init__(self, seed: Optional[int] = None) -> None: self._rng = LazyRandomState(seed) def reseed_rng(self) -> None: self._rng.rng.seed() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: return {} def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: return {} def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: distributions.BaseDistribution, ) -> Any: search_space = {param_name: param_distribution} trans = _SearchSpaceTransform(search_space) trans_params = self._rng.rng.uniform(trans.bounds[:, 0], trans.bounds[:, 1]) return trans.untransform(trans_params)[param_name] optuna-3.5.0/optuna/samplers/_search_space/000077500000000000000000000000001453453102400207415ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/_search_space/__init__.py000066400000000000000000000003601453453102400230510ustar00rootroot00000000000000from optuna.samplers._search_space.intersection import intersection_search_space from optuna.samplers._search_space.intersection import IntersectionSearchSpace __all__ = [ "IntersectionSearchSpace", "intersection_search_space", ] optuna-3.5.0/optuna/samplers/_search_space/intersection.py000066400000000000000000000130031453453102400240160ustar00rootroot00000000000000import copy from typing import Dict from typing import Optional import optuna from optuna._deprecated import deprecated_class from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.study import Study @deprecated_class( "3.2.0", "4.0.0", name="optuna.samplers.IntersectionSearchSpace", text="Please use optuna.search_space.IntersectionSearchSpace instead.", ) class IntersectionSearchSpace: """A class to calculate the intersection search space of a :class:`~optuna.study.Study`. Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded). Note that an instance of this class is supposed to be used for only one study. If different studies are passed to :func:`~optuna.samplers.IntersectionSearchSpace.calculate`, a :obj:`ValueError` is raised. Args: include_pruned: Whether pruned trials should be included in the search space. """ def __init__(self, include_pruned: bool = False) -> None: self._cursor: int = -1 self._search_space: Optional[Dict[str, BaseDistribution]] = None self._study_id: Optional[int] = None self._include_pruned = include_pruned def calculate(self, study: Study, ordered_dict: bool = False) -> Dict[str, BaseDistribution]: """Returns the intersection search space of the :class:`~optuna.study.Study`. Args: study: A study with completed trials. The same study must be passed for one instance of this class through its lifetime. ordered_dict: A boolean flag determining the return type. If :obj:`False`, the returned object will be a :obj:`dict`. If :obj:`True`, the returned object will be a :obj:`dict` sorted by keys, i.e. parameter names. Returns: A dictionary containing the parameter names and parameter's distributions. """ if self._study_id is None: self._study_id = study._study_id else: # Note that the check below is meaningless when `InMemoryStorage` is used # because `InMemoryStorage.create_new_study` always returns the same study ID. if self._study_id != study._study_id: raise ValueError("`IntersectionSearchSpace` cannot handle multiple studies.") states_of_interest = [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.WAITING, optuna.trial.TrialState.RUNNING, ] if self._include_pruned: states_of_interest.append(optuna.trial.TrialState.PRUNED) trials = study._get_trials(deepcopy=False, states=states_of_interest, use_cache=False) next_cursor = trials[-1].number + 1 if len(trials) > 0 else -1 for trial in reversed(trials): if self._cursor > trial.number: break if not trial.state.is_finished(): next_cursor = trial.number continue if self._search_space is None: self._search_space = copy.copy(trial.distributions) continue self._search_space = { name: distribution for name, distribution in self._search_space.items() if trial.distributions.get(name) == distribution } self._cursor = next_cursor search_space = self._search_space or {} if ordered_dict: search_space = dict(sorted(search_space.items(), key=lambda x: x[0])) return copy.deepcopy(search_space) @deprecated_func( "3.2.0", "4.0.0", name="optuna.samplers.intersection_search_space", text="Please use optuna.search_space.intersection_search_space instead.", ) def intersection_search_space( study: Study, ordered_dict: bool = False, include_pruned: bool = False ) -> Dict[str, BaseDistribution]: """Return the intersection search space of the :class:`~optuna.study.Study`. Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded). .. note:: :class:`~optuna.samplers.IntersectionSearchSpace` provides the same functionality with a much faster way. Please consider using it if you want to reduce execution time as much as possible. Args: study: A study with completed trials. ordered_dict: A boolean flag determining the return type. If :obj:`False`, the returned object will be a :obj:`dict`. If :obj:`True`, the returned object will be a :obj:`dict` sorted by keys, i.e. parameter names. include_pruned: Whether pruned trials should be included in the search space. Returns: A dictionary containing the parameter names and parameter's distributions. """ return IntersectionSearchSpace(include_pruned=include_pruned).calculate( study, ordered_dict=ordered_dict ) optuna-3.5.0/optuna/samplers/_tpe/000077500000000000000000000000001453453102400171115ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/_tpe/__init__.py000066400000000000000000000000001453453102400212100ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/_tpe/_erf.py000066400000000000000000000134471453453102400204070ustar00rootroot00000000000000# This code is the modified version of erf function in FreeBSD's standard C library. # origin: FreeBSD /usr/src/lib/msun/src/s_erf.c # https://github.com/freebsd/freebsd-src/blob/main/lib/msun/src/s_erf.c # /* @(#)s_erf.c 5.1 93/09/24 */ # /* # * ==================================================== # * Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. # * # * Developed at SunPro, a Sun Microsystems, Inc. business. # * Permission to use, copy, modify, and distribute this # * software is freely granted, provided that this notice # * is preserved. # * ==================================================== # */ import numpy as np from numpy.polynomial import Polynomial half = 0.5 one = 1 two = 2 erx = 8.45062911510467529297e-01 # /* # * In the domain [0, 2**-28], only the first term in the power series # * expansion of erf(x) is used. The magnitude of the first neglected # * terms is less than 2**-84. # */ efx = 1.28379167095512586316e-01 efx8 = 1.02703333676410069053e00 # Coefficients for approximation to erf on [0,0.84375] pp0 = 1.28379167095512558561e-01 pp1 = -3.25042107247001499370e-01 pp2 = -2.84817495755985104766e-02 pp3 = -5.77027029648944159157e-03 pp4 = -2.37630166566501626084e-05 pp = Polynomial([pp0, pp1, pp2, pp3, pp4]) qq1 = 3.97917223959155352819e-01 qq2 = 6.50222499887672944485e-02 qq3 = 5.08130628187576562776e-03 qq4 = 1.32494738004321644526e-04 qq5 = -3.96022827877536812320e-06 qq = Polynomial([one, qq1, qq2, qq3, qq4, qq5]) # Coefficients for approximation to erf in [0.84375,1.25] pa0 = -2.36211856075265944077e-03 pa1 = 4.14856118683748331666e-01 pa2 = -3.72207876035701323847e-01 pa3 = 3.18346619901161753674e-01 pa4 = -1.10894694282396677476e-01 pa5 = 3.54783043256182359371e-02 pa6 = -2.16637559486879084300e-03 pa = Polynomial([pa0, pa1, pa2, pa3, pa4, pa5, pa6]) qa1 = 1.06420880400844228286e-01 qa2 = 5.40397917702171048937e-01 qa3 = 7.18286544141962662868e-02 qa4 = 1.26171219808761642112e-01 qa5 = 1.36370839120290507362e-02 qa6 = 1.19844998467991074170e-02 qa = Polynomial([one, qa1, qa2, qa3, qa4, qa5, qa6]) # Coefficients for approximation to erfc in [1.25,1/0.35] ra0 = -9.86494403484714822705e-03 ra1 = -6.93858572707181764372e-01 ra2 = -1.05586262253232909814e01 ra3 = -6.23753324503260060396e01 ra4 = -1.62396669462573470355e02 ra5 = -1.84605092906711035994e02 ra6 = -8.12874355063065934246e01 ra7 = -9.81432934416914548592e00 ra = Polynomial([ra0, ra1, ra2, ra3, ra4, ra5, ra6, ra7]) sa1 = 1.96512716674392571292e01 sa2 = 1.37657754143519042600e02 sa3 = 4.34565877475229228821e02 sa4 = 6.45387271733267880336e02 sa5 = 4.29008140027567833386e02 sa6 = 1.08635005541779435134e02 sa7 = 6.57024977031928170135e00 sa8 = -6.04244152148580987438e-02 sa = Polynomial([one, sa1, sa2, sa3, sa4, sa5, sa6, sa7, sa8]) # Coefficients for approximation to erfc in [1/.35,28] rb0 = -9.86494292470009928597e-03 rb1 = -7.99283237680523006574e-01 rb2 = -1.77579549177547519889e01 rb3 = -1.60636384855821916062e02 rb4 = -6.37566443368389627722e02 rb5 = -1.02509513161107724954e03 rb6 = -4.83519191608651397019e02 rb = Polynomial([rb0, rb1, rb2, rb3, rb4, rb5, rb6]) sb1 = 3.03380607434824582924e01 sb2 = 3.25792512996573918826e02 sb3 = 1.53672958608443695994e03 sb4 = 3.19985821950859553908e03 sb5 = 2.55305040643316442583e03 sb6 = 4.74528541206955367215e02 sb7 = -2.24409524465858183362e01 sb = Polynomial([one, sb1, sb2, sb3, sb4, sb5, sb6, sb7]) def erf(x: np.ndarray) -> np.ndarray: a = np.abs(x) case_nan = np.isnan(x) case_posinf = np.isposinf(x) case_neginf = np.isneginf(x) case_tiny = a < 2**-28 case_small1 = (2**-28 <= a) & (a < 0.84375) case_small2 = (0.84375 <= a) & (a < 1.25) case_med1 = (1.25 <= a) & (a < 1 / 0.35) case_med2 = (1 / 0.35 <= a) & (a < 6) case_big = a >= 6 def calc_case_tiny(x: np.ndarray) -> np.ndarray: return x + efx * x def calc_case_small1(x: np.ndarray) -> np.ndarray: z = x * x r = pp(z) s = qq(z) y = r / s return x + x * y def calc_case_small2(x: np.ndarray) -> np.ndarray: s = np.abs(x) - one P = pa(s) Q = qa(s) absout = erx + P / Q return absout * np.sign(x) def calc_case_med1(x: np.ndarray) -> np.ndarray: sign = np.sign(x) x = np.abs(x) s = one / (x * x) R = ra(s) S = sa(s) # the following 3 lines are omitted for the following reasons: # (1) there are no easy way to implement SET_LOW_WORD equivalent method in NumPy # (2) we don't need very high accuracy in our use case. # z = x # SET_LOW_WORD(z, 0) # r = np.exp(-z * z - 0.5625) * np.exp((z - x) * (z + x) + R / S) r = np.exp(-x * x - 0.5625) * np.exp(R / S) return (one - r / x) * sign def calc_case_med2(x: np.ndarray) -> np.ndarray: sign = np.sign(x) x = np.abs(x) s = one / (x * x) R = rb(s) S = sb(s) # z = x # SET_LOW_WORD(z, 0) # r = np.exp(-z * z - 0.5625) * np.exp((z - x) * (z + x) + R / S) r = np.exp(-x * x - 0.5625) * np.exp(R / S) return (one - r / x) * sign def calc_case_big(x: np.ndarray) -> np.ndarray: return np.sign(x) out = np.full_like(a, fill_value=np.nan, dtype=np.float64) out[case_nan] = np.nan out[case_posinf] = 1.0 out[case_neginf] = -1.0 if x[case_tiny].size: out[case_tiny] = calc_case_tiny(x[case_tiny]) if x[case_small1].size: out[case_small1] = calc_case_small1(x[case_small1]) if x[case_small2].size: out[case_small2] = calc_case_small2(x[case_small2]) if x[case_med1].size: out[case_med1] = calc_case_med1(x[case_med1]) if x[case_med2].size: out[case_med2] = calc_case_med2(x[case_med2]) if x[case_big].size: out[case_big] = calc_case_big(x[case_big]) return out optuna-3.5.0/optuna/samplers/_tpe/_truncnorm.py000066400000000000000000000166551453453102400216660ustar00rootroot00000000000000# This file contains the codes from SciPy project. # # Copyright (c) 2001-2002 Enthought, Inc. 2003-2022, SciPy Developers. # All rights reserved. # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above # copyright notice, this list of conditions and the following # disclaimer in the documentation and/or other materials provided # with the distribution. # 3. Neither the name of the copyright holder nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. import functools import math import sys from typing import Callable from typing import Optional from typing import Union import numpy as np from optuna.samplers._tpe._erf import erf _norm_pdf_C = math.sqrt(2 * math.pi) _norm_pdf_logC = math.log(_norm_pdf_C) def _log_sum(log_p: np.ndarray, log_q: np.ndarray) -> np.ndarray: return np.logaddexp(log_p, log_q) def _log_diff(log_p: np.ndarray, log_q: np.ndarray) -> np.ndarray: return log_p + np.log1p(-np.exp(log_q - log_p)) @functools.lru_cache(1000) def _ndtr_single(a: float) -> float: x = a / 2**0.5 if x < -1 / 2**0.5: y = 0.5 * math.erfc(-x) elif x < 1 / 2**0.5: y = 0.5 + 0.5 * math.erf(x) else: y = 1.0 - 0.5 * math.erfc(x) return y def _ndtr(a: np.ndarray) -> np.ndarray: # todo(amylase): implement erfc in _erf.py and use it for big |a| inputs. return 0.5 + 0.5 * erf(a / 2**0.5) @functools.lru_cache(1000) def _log_ndtr_single(a: float) -> float: if a > 6: return -_ndtr_single(-a) if a > -20: return math.log(_ndtr_single(a)) log_LHS = -0.5 * a**2 - math.log(-a) - 0.5 * math.log(2 * math.pi) last_total = 0.0 right_hand_side = 1.0 numerator = 1.0 denom_factor = 1.0 denom_cons = 1 / a**2 sign = 1 i = 0 while abs(last_total - right_hand_side) > sys.float_info.epsilon: i += 1 last_total = right_hand_side sign = -sign denom_factor *= denom_cons numerator *= 2 * i - 1 right_hand_side += sign * numerator * denom_factor return log_LHS + math.log(right_hand_side) def _log_ndtr(a: np.ndarray) -> np.ndarray: return np.frompyfunc(_log_ndtr_single, 1, 1)(a).astype(float) def _norm_logpdf(x: np.ndarray) -> np.ndarray: return -(x**2) / 2.0 - _norm_pdf_logC def _log_gauss_mass(a: np.ndarray, b: np.ndarray) -> np.ndarray: """Log of Gaussian probability mass within an interval""" # Calculations in right tail are inaccurate, so we'll exploit the # symmetry and work only in the left tail case_left = b <= 0 case_right = a > 0 case_central = ~(case_left | case_right) def mass_case_left(a: np.ndarray, b: np.ndarray) -> np.ndarray: return _log_diff(_log_ndtr(b), _log_ndtr(a)) def mass_case_right(a: np.ndarray, b: np.ndarray) -> np.ndarray: return mass_case_left(-b, -a) def mass_case_central(a: np.ndarray, b: np.ndarray) -> np.ndarray: # Previously, this was implemented as: # left_mass = mass_case_left(a, 0) # right_mass = mass_case_right(0, b) # return _log_sum(left_mass, right_mass) # Catastrophic cancellation occurs as np.exp(log_mass) approaches 1. # Correct for this with an alternative formulation. # We're not concerned with underflow here: if only one term # underflows, it was insignificant; if both terms underflow, # the result can't accurately be represented in logspace anyway # because sc.log1p(x) ~ x for small x. return np.log1p(-_ndtr(a) - _ndtr(-b)) # _lazyselect not working; don't care to debug it out = np.full_like(a, fill_value=np.nan, dtype=np.complex128) if a[case_left].size: out[case_left] = mass_case_left(a[case_left], b[case_left]) if a[case_right].size: out[case_right] = mass_case_right(a[case_right], b[case_right]) if a[case_central].size: out[case_central] = mass_case_central(a[case_central], b[case_central]) return np.real(out) # discard ~0j def _bisect(f: Callable[[float], float], a: float, b: float, c: float) -> float: if f(a) > c: a, b = b, a # TODO(amylase): Justify this constant for _ in range(100): m = (a + b) / 2 if f(m) < c: a = m else: b = m return m def _ndtri_exp_single(y: float) -> float: # TODO(amylase): Justify this constant return _bisect(_log_ndtr_single, -100, +100, y) def _ndtri_exp(y: np.ndarray) -> np.ndarray: return np.frompyfunc(_ndtri_exp_single, 1, 1)(y).astype(float) def ppf(q: np.ndarray, a: Union[np.ndarray, float], b: Union[np.ndarray, float]) -> np.ndarray: q, a, b = np.atleast_1d(q, a, b) q, a, b = np.broadcast_arrays(q, a, b) case_left = a < 0 case_right = ~case_left def ppf_left(q: np.ndarray, a: np.ndarray, b: np.ndarray) -> np.ndarray: log_Phi_x = _log_sum(_log_ndtr(a), np.log(q) + _log_gauss_mass(a, b)) return _ndtri_exp(log_Phi_x) def ppf_right(q: np.ndarray, a: np.ndarray, b: np.ndarray) -> np.ndarray: log_Phi_x = _log_sum(_log_ndtr(-b), np.log1p(-q) + _log_gauss_mass(a, b)) return -_ndtri_exp(log_Phi_x) out = np.empty_like(q) q_left = q[case_left] q_right = q[case_right] if q_left.size: out[case_left] = ppf_left(q_left, a[case_left], b[case_left]) if q_right.size: out[case_right] = ppf_right(q_right, a[case_right], b[case_right]) out[q == 0] = a[q == 0] out[q == 1] = b[q == 1] out[a == b] = math.nan return out def rvs( a: np.ndarray, b: np.ndarray, loc: Union[np.ndarray, float] = 0, scale: Union[np.ndarray, float] = 1, random_state: Optional[np.random.RandomState] = None, ) -> np.ndarray: random_state = random_state or np.random.RandomState() size = np.broadcast(a, b, loc, scale).shape percentiles = random_state.uniform(low=0, high=1, size=size) return ppf(percentiles, a, b) * scale + loc def logpdf( x: np.ndarray, a: Union[np.ndarray, float], b: Union[np.ndarray, float], loc: Union[np.ndarray, float] = 0, scale: Union[np.ndarray, float] = 1, ) -> np.ndarray: x = (x - loc) / scale x, a, b = np.atleast_1d(x, a, b) out = _norm_logpdf(x) - _log_gauss_mass(a, b) - np.log(scale) x, a, b = np.broadcast_arrays(x, a, b) out[(x < a) | (b < x)] = -np.inf out[a == b] = math.nan return out optuna-3.5.0/optuna/samplers/_tpe/multi_objective_sampler.py000066400000000000000000000125571453453102400244040ustar00rootroot00000000000000from typing import Callable from typing import Optional import numpy as np from optuna._deprecated import deprecated_class from optuna.samplers._tpe.sampler import TPESampler EPS = 1e-12 def default_gamma(x: int) -> int: return int(np.floor(0.1 * x)) def _default_weights_above(x: int) -> np.ndarray: return np.ones(x) @deprecated_class("2.9.0", "4.0.0") class MOTPESampler(TPESampler): """Multi-objective sampler using the MOTPE algorithm. This sampler is a multi-objective version of :class:`~optuna.samplers.TPESampler`. .. note:: For `v2.9.0 `_ or later, :class:`~optuna.samplers.MOTPESampler` is deprecated and :class:`~optuna.samplers.TPESampler` should be used instead. The following code shows how you apply :class:`~optuna.samplers.TPESampler` to a multi-objective task: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) f1 = x**2 + y f2 = -((x - 2) ** 2 + y) return f1, f2 # We minimize the first objective and maximize the second objective. sampler = optuna.samplers.TPESampler() study = optuna.create_study(directions=["minimize", "maximize"], sampler=sampler) study.optimize(objective, n_trials=100) For further information about MOTPE algorithm, please refer to the following papers: - `Multiobjective tree-structured parzen estimator for computationally expensive optimization problems `_ - `Multiobjective Tree-Structured Parzen Estimator `_ Args: consider_prior: Enhance the stability of Parzen estimator by imposing a Gaussian prior when :obj:`True`. The prior is only effective if the sampling distribution is either :class:`~optuna.distributions.FloatDistribution`, or :class:`~optuna.distributions.IntDistribution`. prior_weight: The weight of the prior. This argument is used in :class:`~optuna.distributions.FloatDistribution`, :class:`~optuna.distributions.IntDistribution`, and :class:`~optuna.distributions.CategoricalDistribution`. consider_magic_clip: Enable a heuristic to limit the smallest variances of Gaussians used in the Parzen estimator. consider_endpoints: Take endpoints of domains into account when calculating variances of Gaussians in Parzen estimator. See the original paper for details on the heuristics to calculate the variances. n_startup_trials: The random sampling is used instead of the MOTPE algorithm until the given number of trials finish in the same study. 11 * number of variables - 1 is recommended in the original paper. n_ehvi_candidates: Number of candidate samples used to calculate the expected hypervolume improvement. gamma: A function that takes the number of finished trials and returns the number of trials to form a density function for samples with low grains. See the original paper for more details. weights_above: A function that takes the number of finished trials and returns a weight for them. As default, weights are automatically calculated by the MOTPE's default strategy. seed: Seed for random number generator. .. note:: Initialization with Latin hypercube sampling may improve optimization performance. However, the current implementation only supports initialization with random sampling. Example: .. testcode:: import optuna seed = 128 num_variables = 2 n_startup_trials = 11 * num_variables - 1 def objective(trial): x = [] for i in range(1, num_variables + 1): x.append(trial.suggest_float(f"x{i}", 0.0, 2.0 * i)) return x sampler = optuna.samplers.MOTPESampler( n_startup_trials=n_startup_trials, n_ehvi_candidates=24, seed=seed ) study = optuna.create_study(directions=["minimize"] * num_variables, sampler=sampler) study.optimize(objective, n_trials=n_startup_trials + 10) """ def __init__( self, *, consider_prior: bool = True, prior_weight: float = 1.0, consider_magic_clip: bool = True, consider_endpoints: bool = True, n_startup_trials: int = 10, n_ehvi_candidates: int = 24, gamma: Callable[[int], int] = default_gamma, weights_above: Callable[[int], np.ndarray] = _default_weights_above, seed: Optional[int] = None, ) -> None: super().__init__( consider_prior=consider_prior, prior_weight=prior_weight, consider_magic_clip=consider_magic_clip, consider_endpoints=consider_endpoints, n_startup_trials=n_startup_trials, n_ei_candidates=n_ehvi_candidates, gamma=gamma, weights=weights_above, seed=seed, ) optuna-3.5.0/optuna/samplers/_tpe/parzen_estimator.py000066400000000000000000000265071453453102400230630ustar00rootroot00000000000000from typing import Callable from typing import Dict from typing import NamedTuple from typing import Optional import numpy as np from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDistributions from optuna.samplers._tpe.probability_distributions import _BatchedTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution EPS = 1e-12 class _ParzenEstimatorParameters( NamedTuple( "_ParzenEstimatorParameters", [ ("consider_prior", bool), ("prior_weight", Optional[float]), ("consider_magic_clip", bool), ("consider_endpoints", bool), ("weights", Callable[[int], np.ndarray]), ("multivariate", bool), ( "categorical_distance_func", Dict[str, Callable[[CategoricalChoiceType, CategoricalChoiceType], float]], ), ], ) ): pass class _ParzenEstimator: def __init__( self, observations: Dict[str, np.ndarray], search_space: Dict[str, BaseDistribution], parameters: _ParzenEstimatorParameters, predetermined_weights: Optional[np.ndarray] = None, ) -> None: if parameters.consider_prior: if parameters.prior_weight is None: raise ValueError("Prior weight must be specified when consider_prior==True.") elif parameters.prior_weight <= 0: raise ValueError("Prior weight must be positive.") self._search_space = search_space transformed_observations = self._transform(observations) assert predetermined_weights is None or len(transformed_observations) == len( predetermined_weights ) weights = ( predetermined_weights if predetermined_weights is not None else self._call_weights_func(parameters.weights, len(transformed_observations)) ) if len(transformed_observations) == 0: weights = np.array([1.0]) elif parameters.consider_prior: assert parameters.prior_weight is not None weights = np.append(weights, [parameters.prior_weight]) weights /= weights.sum() self._mixture_distribution = _MixtureOfProductDistribution( weights=weights, distributions=[ self._calculate_distributions( transformed_observations[:, i], param, search_space[param], parameters ) for i, param in enumerate(search_space) ], ) def sample(self, rng: np.random.RandomState, size: int) -> Dict[str, np.ndarray]: sampled = self._mixture_distribution.sample(rng, size) return self._untransform(sampled) def log_pdf(self, samples_dict: Dict[str, np.ndarray]) -> np.ndarray: transformed_samples = self._transform(samples_dict) return self._mixture_distribution.log_pdf(transformed_samples) @staticmethod def _call_weights_func(weights_func: Callable[[int], np.ndarray], n: int) -> np.ndarray: w = np.array(weights_func(n))[:n] if np.any(w < 0): raise ValueError( f"The `weights` function is not allowed to return negative values {w}. " + f"The argument of the `weights` function is {n}." ) if len(w) > 0 and np.sum(w) <= 0: raise ValueError( f"The `weight` function is not allowed to return all-zero values {w}." + f" The argument of the `weights` function is {n}." ) if not np.all(np.isfinite(w)): raise ValueError( "The `weights`function is not allowed to return infinite or NaN values " + f"{w}. The argument of the `weights` function is {n}." ) # TODO(HideakiImamura) Raise `ValueError` if the weight function returns an ndarray of # unexpected size. return w @staticmethod def _is_log(dist: BaseDistribution) -> bool: return isinstance(dist, (FloatDistribution, IntDistribution)) and dist.log def _transform(self, samples_dict: Dict[str, np.ndarray]) -> np.ndarray: return np.array( [ np.log(samples_dict[param]) if self._is_log(self._search_space[param]) else samples_dict[param] for param in self._search_space ] ).T def _untransform(self, samples_array: np.ndarray) -> Dict[str, np.ndarray]: res = { param: np.exp(samples_array[:, i]) if self._is_log(self._search_space[param]) else samples_array[:, i] for i, param in enumerate(self._search_space) } # TODO(contramundum53): Remove this line after fixing log-Int hack. return { param: np.clip( dist.low + np.round((res[param] - dist.low) / dist.step) * dist.step, dist.low, dist.high, ) if isinstance(dist, IntDistribution) else res[param] for (param, dist) in self._search_space.items() } def _calculate_distributions( self, transformed_observations: np.ndarray, param_name: str, search_space: BaseDistribution, parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: if isinstance(search_space, CategoricalDistribution): return self._calculate_categorical_distributions( transformed_observations, param_name, search_space, parameters ) else: assert isinstance(search_space, (FloatDistribution, IntDistribution)) if search_space.log: low = np.log(search_space.low) high = np.log(search_space.high) else: low = search_space.low high = search_space.high step = search_space.step # TODO(contramundum53): This is a hack and should be fixed. if step is not None and search_space.log: low = np.log(search_space.low - step / 2) high = np.log(search_space.high + step / 2) step = None return self._calculate_numerical_distributions( transformed_observations, low, high, step, parameters ) def _calculate_categorical_distributions( self, observations: np.ndarray, param_name: str, search_space: CategoricalDistribution, parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: consider_prior = parameters.consider_prior or len(observations) == 0 assert parameters.prior_weight is not None weights = np.full( shape=(len(observations) + consider_prior, len(search_space.choices)), fill_value=parameters.prior_weight / (len(observations) + consider_prior), ) if param_name in parameters.categorical_distance_func: dist_func = parameters.categorical_distance_func[param_name] for i, observation in enumerate(observations.astype(int)): dists = [ dist_func(search_space.choices[observation], search_space.choices[j]) for j in range(len(search_space.choices)) ] exponent = -( (np.array(dists) / max(dists)) ** 2 * np.log((len(observations) + consider_prior) / parameters.prior_weight) * (np.log(len(search_space.choices)) / np.log(6)) ) weights[i] = np.exp(exponent) else: weights[np.arange(len(observations)), observations.astype(int)] += 1 weights /= weights.sum(axis=1, keepdims=True) return _BatchedCategoricalDistributions(weights) def _calculate_numerical_distributions( self, observations: np.ndarray, low: float, high: float, step: Optional[float], parameters: _ParzenEstimatorParameters, ) -> _BatchedDistributions: step_or_0 = step or 0 mus = observations consider_prior = parameters.consider_prior or len(observations) == 0 def compute_sigmas() -> np.ndarray: if parameters.multivariate: SIGMA0_MAGNITUDE = 0.2 sigma = ( SIGMA0_MAGNITUDE * max(len(observations), 1) ** (-1.0 / (len(self._search_space) + 4)) * (high - low + step_or_0) ) sigmas = np.full(shape=(len(observations),), fill_value=sigma) else: # TODO(contramundum53): Remove dependency on prior_mu prior_mu = 0.5 * (low + high) mus_with_prior = np.append(mus, prior_mu) if consider_prior else mus sorted_indices = np.argsort(mus_with_prior) sorted_mus = mus_with_prior[sorted_indices] sorted_mus_with_endpoints = np.empty(len(mus_with_prior) + 2, dtype=float) sorted_mus_with_endpoints[0] = low - step_or_0 / 2 sorted_mus_with_endpoints[1:-1] = sorted_mus sorted_mus_with_endpoints[-1] = high + step_or_0 / 2 sorted_sigmas = np.maximum( sorted_mus_with_endpoints[1:-1] - sorted_mus_with_endpoints[0:-2], sorted_mus_with_endpoints[2:] - sorted_mus_with_endpoints[1:-1], ) if not parameters.consider_endpoints and sorted_mus_with_endpoints.shape[0] >= 4: sorted_sigmas[0] = sorted_mus_with_endpoints[2] - sorted_mus_with_endpoints[1] sorted_sigmas[-1] = ( sorted_mus_with_endpoints[-2] - sorted_mus_with_endpoints[-3] ) sigmas = sorted_sigmas[np.argsort(sorted_indices)][: len(observations)] # We adjust the range of the 'sigmas' according to the 'consider_magic_clip' flag. maxsigma = 1.0 * (high - low + step_or_0) if parameters.consider_magic_clip: # TODO(contramundum53): Remove dependency of minsigma on consider_prior. minsigma = ( 1.0 * (high - low + step_or_0) / min(100.0, (1.0 + len(observations) + consider_prior)) ) else: minsigma = EPS return np.asarray(np.clip(sigmas, minsigma, maxsigma)) sigmas = compute_sigmas() if consider_prior: prior_mu = 0.5 * (low + high) prior_sigma = 1.0 * (high - low + step_or_0) mus = np.append(mus, [prior_mu]) sigmas = np.append(sigmas, [prior_sigma]) if step is None: return _BatchedTruncNormDistributions(mus, sigmas, low, high) else: return _BatchedDiscreteTruncNormDistributions(mus, sigmas, low, high, step) optuna-3.5.0/optuna/samplers/_tpe/probability_distributions.py000066400000000000000000000116251453453102400247720ustar00rootroot00000000000000from typing import List from typing import NamedTuple from typing import Union import numpy as np from optuna.samplers._tpe import _truncnorm class _BatchedCategoricalDistributions(NamedTuple): weights: np.ndarray class _BatchedTruncNormDistributions(NamedTuple): mu: np.ndarray sigma: np.ndarray low: float # Currently, low and high do not change per trial. high: float class _BatchedDiscreteTruncNormDistributions(NamedTuple): mu: np.ndarray sigma: np.ndarray low: float # Currently, low, high and step do not change per trial. high: float step: float _BatchedDistributions = Union[ _BatchedCategoricalDistributions, _BatchedTruncNormDistributions, _BatchedDiscreteTruncNormDistributions, ] class _MixtureOfProductDistribution(NamedTuple): weights: np.ndarray distributions: List[_BatchedDistributions] def sample(self, rng: np.random.RandomState, batch_size: int) -> np.ndarray: active_indices = rng.choice(len(self.weights), p=self.weights, size=batch_size) ret = np.empty((batch_size, len(self.distributions)), dtype=np.float64) for i, d in enumerate(self.distributions): if isinstance(d, _BatchedCategoricalDistributions): active_weights = d.weights[active_indices, :] rnd_quantile = rng.rand(batch_size) cum_probs = np.cumsum(active_weights, axis=-1) assert np.isclose(cum_probs[:, -1], 1).all() cum_probs[:, -1] = 1 # Avoid numerical errors. ret[:, i] = np.sum(cum_probs < rnd_quantile[:, None], axis=-1) elif isinstance(d, _BatchedTruncNormDistributions): active_mus = d.mu[active_indices] active_sigmas = d.sigma[active_indices] ret[:, i] = _truncnorm.rvs( a=(d.low - active_mus) / active_sigmas, b=(d.high - active_mus) / active_sigmas, loc=active_mus, scale=active_sigmas, random_state=rng, ) elif isinstance(d, _BatchedDiscreteTruncNormDistributions): active_mus = d.mu[active_indices] active_sigmas = d.sigma[active_indices] samples = _truncnorm.rvs( a=(d.low - d.step / 2 - active_mus) / active_sigmas, b=(d.high + d.step / 2 - active_mus) / active_sigmas, loc=active_mus, scale=active_sigmas, random_state=rng, ) ret[:, i] = np.clip( d.low + np.round((samples - d.low) / d.step) * d.step, d.low, d.high ) else: assert False return ret def log_pdf(self, x: np.ndarray) -> np.ndarray: batch_size, n_vars = x.shape log_pdfs = np.empty((batch_size, len(self.weights), n_vars), dtype=np.float64) for i, d in enumerate(self.distributions): xi = x[:, i] if isinstance(d, _BatchedCategoricalDistributions): log_pdfs[:, :, i] = np.log( np.take_along_axis( d.weights[None, :, :], xi[:, None, None].astype(np.int64), axis=-1 ) )[:, :, 0] elif isinstance(d, _BatchedTruncNormDistributions): log_pdfs[:, :, i] = _truncnorm.logpdf( x=xi[:, None], a=(d.low - d.mu[None, :]) / d.sigma[None, :], b=(d.high - d.mu[None, :]) / d.sigma[None, :], loc=d.mu[None, :], scale=d.sigma[None, :], ) elif isinstance(d, _BatchedDiscreteTruncNormDistributions): lower_limit = d.low - d.step / 2 upper_limit = d.high + d.step / 2 x_lower = np.maximum(xi - d.step / 2, lower_limit) x_upper = np.minimum(xi + d.step / 2, upper_limit) log_gauss_mass = _truncnorm._log_gauss_mass( (x_lower[:, None] - d.mu[None, :]) / d.sigma[None, :], (x_upper[:, None] - d.mu[None, :]) / d.sigma[None, :], ) log_p_accept = _truncnorm._log_gauss_mass( (d.low - d.step / 2 - d.mu[None, :]) / d.sigma[None, :], (d.high + d.step / 2 - d.mu[None, :]) / d.sigma[None, :], ) log_pdfs[:, :, i] = log_gauss_mass - log_p_accept else: assert False weighted_log_pdf = np.sum(log_pdfs, axis=-1) + np.log(self.weights[None, :]) max_ = weighted_log_pdf.max(axis=1) # We need to avoid (-inf) - (-inf) when the probability is zero. max_[np.isneginf(max_)] = 0 with np.errstate(divide="ignore"): # Suppress warning in log(0). return np.log(np.exp(weighted_log_pdf - max_[:, None]).sum(axis=1)) + max_ optuna-3.5.0/optuna/samplers/_tpe/sampler.py000066400000000000000000001023311453453102400211260ustar00rootroot00000000000000from __future__ import annotations import math from typing import Any from typing import Callable from typing import cast from typing import Dict from typing import Optional from typing import Sequence from typing import Union import warnings import numpy as np from optuna._hypervolume import WFG from optuna._hypervolume.hssp import _solve_hssp from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.exceptions import ExperimentalWarning from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._base import _process_constraints_after_trial from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers._random import RandomSampler from optuna.samplers._tpe.parzen_estimator import _ParzenEstimator from optuna.samplers._tpe.parzen_estimator import _ParzenEstimatorParameters from optuna.search_space import IntersectionSearchSpace from optuna.search_space.group_decomposed import _GroupDecomposedSearchSpace from optuna.search_space.group_decomposed import _SearchSpaceGroup from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState EPS = 1e-12 _logger = get_logger(__name__) def default_gamma(x: int) -> int: return min(int(np.ceil(0.1 * x)), 25) def hyperopt_default_gamma(x: int) -> int: return min(int(np.ceil(0.25 * np.sqrt(x))), 25) def default_weights(x: int) -> np.ndarray: if x == 0: return np.asarray([]) elif x < 25: return np.ones(x) else: ramp = np.linspace(1.0 / x, 1.0, num=x - 25) flat = np.ones(25) return np.concatenate([ramp, flat], axis=0) class TPESampler(BaseSampler): """Sampler using TPE (Tree-structured Parzen Estimator) algorithm. This sampler is based on *independent sampling*. See also :class:`~optuna.samplers.BaseSampler` for more details of 'independent sampling'. On each trial, for each parameter, TPE fits one Gaussian Mixture Model (GMM) ``l(x)`` to the set of parameter values associated with the best objective values, and another GMM ``g(x)`` to the remaining parameter values. It chooses the parameter value ``x`` that maximizes the ratio ``l(x)/g(x)``. For further information about TPE algorithm, please refer to the following papers: - `Algorithms for Hyper-Parameter Optimization `_ - `Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures `_ - `Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance `_ For multi-objective TPE (MOTPE), please refer to the following papers: - `Multiobjective Tree-Structured Parzen Estimator for Computationally Expensive Optimization Problems `_ - `Multiobjective Tree-Structured Parzen Estimator `_ Example: An example of a single-objective optimization is as follows: .. testcode:: import optuna from optuna.samplers import TPESampler def objective(trial): x = trial.suggest_float("x", -10, 10) return x**2 study = optuna.create_study(sampler=TPESampler()) study.optimize(objective, n_trials=10) .. note:: :class:`~optuna.samplers.TPESampler` can handle a multi-objective task as well and the following shows an example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) f1 = x**2 + y f2 = -((x - 2) ** 2 + y) return f1, f2 # We minimize the first objective and maximize the second objective. sampler = optuna.samplers.TPESampler() study = optuna.create_study(directions=["minimize", "maximize"], sampler=sampler) study.optimize(objective, n_trials=100) Args: consider_prior: Enhance the stability of Parzen estimator by imposing a Gaussian prior when :obj:`True`. The prior is only effective if the sampling distribution is either :class:`~optuna.distributions.FloatDistribution`, or :class:`~optuna.distributions.IntDistribution`. prior_weight: The weight of the prior. This argument is used in :class:`~optuna.distributions.FloatDistribution`, :class:`~optuna.distributions.IntDistribution`, and :class:`~optuna.distributions.CategoricalDistribution`. consider_magic_clip: Enable a heuristic to limit the smallest variances of Gaussians used in the Parzen estimator. consider_endpoints: Take endpoints of domains into account when calculating variances of Gaussians in Parzen estimator. See the original paper for details on the heuristics to calculate the variances. n_startup_trials: The random sampling is used instead of the TPE algorithm until the given number of trials finish in the same study. n_ei_candidates: Number of candidate samples used to calculate the expected improvement. gamma: A function that takes the number of finished trials and returns the number of trials to form a density function for samples with low grains. See the original paper for more details. weights: A function that takes the number of finished trials and returns a weight for them. See `Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures `_ for more details. .. note:: In the multi-objective case, this argument is only used to compute the weights of bad trials, i.e., trials to construct `g(x)` in the `paper `_ ). The weights of good trials, i.e., trials to construct `l(x)`, are computed by a rule based on the hypervolume contribution proposed in the `paper of MOTPE `_. seed: Seed for random number generator. multivariate: If this is :obj:`True`, the multivariate TPE is used when suggesting parameters. The multivariate TPE is reported to outperform the independent TPE. See `BOHB: Robust and Efficient Hyperparameter Optimization at Scale `_ for more details. .. note:: Added in v2.2.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.2.0. group: If this and ``multivariate`` are :obj:`True`, the multivariate TPE with the group decomposed search space is used when suggesting parameters. The sampling algorithm decomposes the search space based on past trials and samples from the joint distribution in each decomposed subspace. The decomposed subspaces are a partition of the whole search space. Each subspace is a maximal subset of the whole search space, which satisfies the following: for a trial in completed trials, the intersection of the subspace and the search space of the trial becomes subspace itself or an empty set. Sampling from the joint distribution on the subspace is realized by multivariate TPE. If ``group`` is :obj:`True`, ``multivariate`` must be :obj:`True` as well. .. note:: Added in v2.8.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.8.0. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_categorical("x", ["A", "B"]) if x == "A": return trial.suggest_float("y", -10, 10) else: return trial.suggest_int("z", -10, 10) sampler = optuna.samplers.TPESampler(multivariate=True, group=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) warn_independent_sampling: If this is :obj:`True` and ``multivariate=True``, a warning message is emitted when the value of a parameter is sampled by using an independent sampler. If ``multivariate=False``, this flag has no effect. constant_liar: If :obj:`True`, penalize running trials to avoid suggesting parameter configurations nearby. .. note:: Abnormally terminated trials often leave behind a record with a state of ``RUNNING`` in the storage. Such "zombie" trial parameters will be avoided by the constant liar algorithm during subsequent sampling. When using an :class:`~optuna.storages.RDBStorage`, it is possible to enable the ``heartbeat_interval`` to change the records for abnormally terminated trials to ``FAIL``. .. note:: It is recommended to set this value to :obj:`True` during distributed optimization to avoid having multiple workers evaluating similar parameter configurations. In particular, if each objective function evaluation is costly and the durations of the running states are significant, and/or the number of workers is high. .. note:: Added in v2.8.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.8.0. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible. If ``constraints_func`` returns more than one value for a trial, that trial is considered feasible if and only if all values are equal to 0 or smaller. The ``constraints_func`` will be evaluated after each successful trial. The function won't be called when trials fail or they are pruned, but this behavior is subject to change in the future releases. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. categorical_distance_func: A dictionary of distance functions for categorical parameters. The key is the name of the categorical parameter and the value is a distance function that takes two :class:`~optuna.distributions.CategoricalChoiceType` s and returns a :obj:`float` value. The distance function must return a non-negative value. While categorical choices are handled equally by default, this option allows users to specify prior knowledge on the structure of categorical parameters. When specified, categorical choices closer to current best choices are more likely to be sampled. .. note:: Added in v3.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.4.0. """ def __init__( self, consider_prior: bool = True, prior_weight: float = 1.0, consider_magic_clip: bool = True, consider_endpoints: bool = False, n_startup_trials: int = 10, n_ei_candidates: int = 24, gamma: Callable[[int], int] = default_gamma, weights: Callable[[int], np.ndarray] = default_weights, seed: Optional[int] = None, *, multivariate: bool = False, group: bool = False, warn_independent_sampling: bool = True, constant_liar: bool = False, constraints_func: Optional[Callable[[FrozenTrial], Sequence[float]]] = None, categorical_distance_func: Optional[ dict[str, Callable[[CategoricalChoiceType, CategoricalChoiceType], float]] ] = None, ) -> None: self._parzen_estimator_parameters = _ParzenEstimatorParameters( consider_prior, prior_weight, consider_magic_clip, consider_endpoints, weights, multivariate, categorical_distance_func or {}, ) self._n_startup_trials = n_startup_trials self._n_ei_candidates = n_ei_candidates self._gamma = gamma self._warn_independent_sampling = warn_independent_sampling self._rng = LazyRandomState(seed) self._random_sampler = RandomSampler(seed=seed) self._multivariate = multivariate self._group = group self._group_decomposed_search_space: Optional[_GroupDecomposedSearchSpace] = None self._search_space_group: Optional[_SearchSpaceGroup] = None self._search_space = IntersectionSearchSpace(include_pruned=True) self._constant_liar = constant_liar self._constraints_func = constraints_func if multivariate: warnings.warn( "``multivariate`` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if group: if not multivariate: raise ValueError( "``group`` option can only be enabled when ``multivariate`` is enabled." ) warnings.warn( "``group`` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) self._group_decomposed_search_space = _GroupDecomposedSearchSpace(True) if constant_liar: warnings.warn( "``constant_liar`` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if constraints_func is not None: warnings.warn( "The ``constraints_func`` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if categorical_distance_func is not None: warnings.warn( "The ``categorical_distance_func`` option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) def reseed_rng(self) -> None: self._rng.rng.seed() self._random_sampler.reseed_rng() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> Dict[str, BaseDistribution]: if not self._multivariate: return {} search_space: Dict[str, BaseDistribution] = {} if self._group: assert self._group_decomposed_search_space is not None self._search_space_group = self._group_decomposed_search_space.calculate(study) for sub_space in self._search_space_group.search_spaces: # Sort keys because Python's string hashing is nondeterministic. for name, distribution in sorted(sub_space.items()): if distribution.single(): continue search_space[name] = distribution return search_space for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: if self._group: assert self._search_space_group is not None params = {} for sub_space in self._search_space_group.search_spaces: search_space = {} # Sort keys because Python's string hashing is nondeterministic. for name, distribution in sorted(sub_space.items()): if not distribution.single(): search_space[name] = distribution params.update(self._sample_relative(study, trial, search_space)) return params else: return self._sample_relative(study, trial, search_space) def _sample_relative( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: if search_space == {}: return {} states = (TrialState.COMPLETE, TrialState.PRUNED) trials = study._get_trials(deepcopy=False, states=states, use_cache=True) # If the number of samples is insufficient, we run random trial. if len(trials) < self._n_startup_trials: return {} return self._sample(study, trial, search_space) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: states = (TrialState.COMPLETE, TrialState.PRUNED) trials = study._get_trials(deepcopy=False, states=states, use_cache=True) # If the number of samples is insufficient, we run random trial. if len(trials) < self._n_startup_trials: return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) if self._warn_independent_sampling and self._multivariate: # Avoid independent warning at the first sampling of `param_name`. if any(param_name in trial.params for trial in trials): _logger.warning( f"The parameter '{param_name}' in trial#{trial.number} is sampled " "independently instead of being sampled by multivariate TPE sampler. " "(optimization performance may be degraded). " "You can suppress this warning by setting `warn_independent_sampling` " "to `False` in the constructor of `TPESampler`, " "if this independent sampling is intended behavior." ) return self._sample(study, trial, {param_name: param_distribution})[param_name] def _get_internal_repr( self, trials: list[FrozenTrial], search_space: dict[str, BaseDistribution] ) -> dict[str, np.ndarray]: values: dict[str, list[float]] = {param_name: [] for param_name in search_space} for trial in trials: if all((param_name in trial.params) for param_name in search_space): for param_name in search_space: param = trial.params[param_name] distribution = trial.distributions[param_name] values[param_name].append(distribution.to_internal_repr(param)) return {k: np.asarray(v) for k, v in values.items()} def _sample( self, study: Study, trial: FrozenTrial, search_space: Dict[str, BaseDistribution] ) -> Dict[str, Any]: if self._constant_liar: states = [TrialState.COMPLETE, TrialState.PRUNED, TrialState.RUNNING] else: states = [TrialState.COMPLETE, TrialState.PRUNED] use_cache = not self._constant_liar trials = study._get_trials(deepcopy=False, states=states, use_cache=use_cache) # We divide data into below and above. n = sum(trial.state != TrialState.RUNNING for trial in trials) # Ignore running trials. below_trials, above_trials = _split_trials( study, trials, self._gamma(n), self._constraints_func is not None, ) below = self._get_internal_repr(below_trials, search_space) above = self._get_internal_repr(above_trials, search_space) # We then sample by maximizing log likelihood ratio. if study._is_multi_objective(): param_mask_below = [] for trial in below_trials: param_mask_below.append( all((param_name in trial.params) for param_name in search_space) ) weights_below = _calculate_weights_below_for_multi_objective( study, below_trials, self._constraints_func )[param_mask_below] mpe_below = _ParzenEstimator( below, search_space, self._parzen_estimator_parameters, weights_below ) else: mpe_below = _ParzenEstimator(below, search_space, self._parzen_estimator_parameters) mpe_above = _ParzenEstimator(above, search_space, self._parzen_estimator_parameters) samples_below = mpe_below.sample(self._rng.rng, self._n_ei_candidates) log_likelihoods_below = mpe_below.log_pdf(samples_below) log_likelihoods_above = mpe_above.log_pdf(samples_below) ret = TPESampler._compare(samples_below, log_likelihoods_below, log_likelihoods_above) for param_name, dist in search_space.items(): ret[param_name] = dist.to_external_repr(ret[param_name]) return ret @classmethod def _compare( cls, samples: Dict[str, np.ndarray], log_l: np.ndarray, log_g: np.ndarray, ) -> Dict[str, Union[float, int]]: sample_size = next(iter(samples.values())).size if sample_size: score = log_l - log_g if sample_size != score.size: raise ValueError( "The size of the 'samples' and that of the 'score' " "should be same. " "But (samples.size, score.size) = ({}, {})".format(sample_size, score.size) ) best = np.argmax(score) return {k: v[best].item() for k, v in samples.items()} else: raise ValueError( "The size of 'samples' should be more than 0." "But samples.size = {}".format(sample_size) ) @staticmethod def hyperopt_parameters() -> Dict[str, Any]: """Return the the default parameters of hyperopt (v0.1.2). :class:`~optuna.samplers.TPESampler` can be instantiated with the parameters returned by this method. Example: Create a :class:`~optuna.samplers.TPESampler` instance with the default parameters of `hyperopt `_. .. testcode:: import optuna from optuna.samplers import TPESampler def objective(trial): x = trial.suggest_float("x", -10, 10) return x**2 sampler = TPESampler(**TPESampler.hyperopt_parameters()) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) Returns: A dictionary containing the default parameters of hyperopt. """ return { "consider_prior": True, "prior_weight": 1.0, "consider_magic_clip": True, "consider_endpoints": False, "n_startup_trials": 20, "n_ei_candidates": 24, "gamma": hyperopt_default_gamma, "weights": default_weights, } def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._random_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Optional[Sequence[float]], ) -> None: assert state in [TrialState.COMPLETE, TrialState.FAIL, TrialState.PRUNED] if self._constraints_func is not None: _process_constraints_after_trial(self._constraints_func, study, trial, state) self._random_sampler.after_trial(study, trial, state, values) def _calculate_nondomination_rank(loss_vals: np.ndarray, n_below: int) -> np.ndarray: ranks = np.full(len(loss_vals), -1) num_ranked = 0 rank = 0 domination_mat = np.all(loss_vals[:, None, :] >= loss_vals[None, :, :], axis=2) & np.any( loss_vals[:, None, :] > loss_vals[None, :, :], axis=2 ) while num_ranked < n_below: counts = np.sum((ranks == -1)[None, :] & domination_mat, axis=1) num_ranked += np.sum((counts == 0) & (ranks == -1)) ranks[(counts == 0) & (ranks == -1)] = rank rank += 1 return ranks def _split_trials( study: Study, trials: list[FrozenTrial], n_below: int, constraints_enabled: bool, ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: complete_trials = [] pruned_trials = [] running_trials = [] infeasible_trials = [] for trial in trials: if constraints_enabled and _get_infeasible_trial_score(trial) > 0: infeasible_trials.append(trial) elif trial.state == TrialState.COMPLETE: complete_trials.append(trial) elif trial.state == TrialState.PRUNED: pruned_trials.append(trial) elif trial.state == TrialState.RUNNING: running_trials.append(trial) else: assert False # We divide data into below and above. below_complete, above_complete = _split_complete_trials(complete_trials, study, n_below) # This ensures `n_below` is non-negative to prevent unexpected trial splits. n_below = max(0, n_below - len(below_complete)) below_pruned, above_pruned = _split_pruned_trials(pruned_trials, study, n_below) # This ensures `n_below` is non-negative to prevent unexpected trial splits. n_below = max(0, n_below - len(below_pruned)) below_infeasible, above_infeasible = _split_infeasible_trials(infeasible_trials, n_below) below_trials = below_complete + below_pruned + below_infeasible above_trials = above_complete + above_pruned + above_infeasible + running_trials below_trials.sort(key=lambda trial: trial.number) above_trials.sort(key=lambda trial: trial.number) return below_trials, above_trials def _split_complete_trials( trials: Sequence[FrozenTrial], study: Study, n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: n_below = min(n_below, len(trials)) if len(study.directions) <= 1: return _split_complete_trials_single_objective(trials, study, n_below) else: return _split_complete_trials_multi_objective(trials, study, n_below) def _split_complete_trials_single_objective( trials: Sequence[FrozenTrial], study: Study, n_below: int, ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: if study.direction == StudyDirection.MINIMIZE: sorted_trials = sorted(trials, key=lambda trial: cast(float, trial.value)) else: sorted_trials = sorted(trials, key=lambda trial: cast(float, trial.value), reverse=True) return sorted_trials[:n_below], sorted_trials[n_below:] def _split_complete_trials_multi_objective( trials: Sequence[FrozenTrial], study: Study, n_below: int, ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: if n_below == 0: # The type of trials must be `list`, but not `Sequence`. return [], list(trials) lvals = np.asarray([trial.values for trial in trials]) for i, direction in enumerate(study.directions): if direction == StudyDirection.MAXIMIZE: lvals[:, i] *= -1 # Solving HSSP for variables number of times is a waste of time. nondomination_ranks = _calculate_nondomination_rank(lvals, n_below) assert 0 <= n_below <= len(lvals) indices = np.array(range(len(lvals))) indices_below = np.empty(n_below, dtype=int) # Nondomination rank-based selection i = 0 last_idx = 0 while last_idx < n_below and last_idx + sum(nondomination_ranks == i) <= n_below: length = indices[nondomination_ranks == i].shape[0] indices_below[last_idx : last_idx + length] = indices[nondomination_ranks == i] last_idx += length i += 1 # Hypervolume subset selection problem (HSSP)-based selection subset_size = n_below - last_idx if subset_size > 0: rank_i_lvals = lvals[nondomination_ranks == i] rank_i_indices = indices[nondomination_ranks == i] worst_point = np.max(rank_i_lvals, axis=0) reference_point = np.maximum(1.1 * worst_point, 0.9 * worst_point) reference_point[reference_point == 0] = EPS selected_indices = _solve_hssp(rank_i_lvals, rank_i_indices, subset_size, reference_point) indices_below[last_idx:] = selected_indices below_trials = [] above_trials = [] for index in range(len(trials)): if index in indices_below: below_trials.append(trials[index]) else: above_trials.append(trials[index]) return below_trials, above_trials def _get_pruned_trial_score(trial: FrozenTrial, study: Study) -> tuple[float, float]: if len(trial.intermediate_values) > 0: step, intermediate_value = max(trial.intermediate_values.items()) if math.isnan(intermediate_value): return -step, float("inf") elif study.direction == StudyDirection.MINIMIZE: return -step, intermediate_value else: return -step, -intermediate_value else: return 1, 0.0 def _split_pruned_trials( trials: Sequence[FrozenTrial], study: Study, n_below: int, ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: n_below = min(n_below, len(trials)) sorted_trials = sorted(trials, key=lambda trial: _get_pruned_trial_score(trial, study)) return sorted_trials[:n_below], sorted_trials[n_below:] def _get_infeasible_trial_score(trial: FrozenTrial) -> float: constraint = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraint is None: warnings.warn( f"Trial {trial.number} does not have constraint values." " It will be treated as a lower priority than other trials." ) return float("inf") else: # Violation values of infeasible dimensions are summed up. return sum(v for v in constraint if v > 0) def _split_infeasible_trials( trials: Sequence[FrozenTrial], n_below: int ) -> tuple[list[FrozenTrial], list[FrozenTrial]]: n_below = min(n_below, len(trials)) sorted_trials = sorted(trials, key=_get_infeasible_trial_score) return sorted_trials[:n_below], sorted_trials[n_below:] def _calculate_weights_below_for_multi_objective( study: Study, below_trials: list[FrozenTrial], constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> np.ndarray: loss_vals = [] feasible_mask = np.ones(len(below_trials), dtype=bool) for i, trial in enumerate(below_trials): # Hypervolume contributions are calculated only using feasible trials. if constraints_func is not None: if any(constraint > 0 for constraint in constraints_func(trial)): feasible_mask[i] = False continue values = [] for value, direction in zip(trial.values, study.directions): if direction == StudyDirection.MINIMIZE: values.append(value) else: values.append(-value) loss_vals.append(values) lvals = np.asarray(loss_vals, dtype=float) # Calculate weights based on hypervolume contributions. n_below = len(lvals) weights_below: np.ndarray if n_below == 0: weights_below = np.asarray([]) elif n_below == 1: weights_below = np.asarray([1.0]) else: worst_point = np.max(lvals, axis=0) reference_point = np.maximum(1.1 * worst_point, 0.9 * worst_point) reference_point[reference_point == 0] = EPS hv = WFG().compute(lvals, reference_point) indices_mat = ~np.eye(n_below).astype(bool) contributions = np.asarray( [hv - WFG().compute(lvals[indices_mat[i]], reference_point) for i in range(n_below)] ) contributions += EPS weights_below = np.clip(contributions / np.max(contributions), 0, 1) # For now, EPS weight is assigned to infeasible trials. weights_below_all = np.full(len(below_trials), EPS) weights_below_all[feasible_mask] = weights_below return weights_below_all optuna-3.5.0/optuna/samplers/nsgaii/000077500000000000000000000000001453453102400174345ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/nsgaii/__init__.py000066400000000000000000000012071453453102400215450ustar00rootroot00000000000000from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._crossovers._blxalpha import BLXAlphaCrossover from optuna.samplers.nsgaii._crossovers._sbx import SBXCrossover from optuna.samplers.nsgaii._crossovers._spx import SPXCrossover from optuna.samplers.nsgaii._crossovers._undx import UNDXCrossover from optuna.samplers.nsgaii._crossovers._uniform import UniformCrossover from optuna.samplers.nsgaii._crossovers._vsbx import VSBXCrossover __all__ = [ "BaseCrossover", "BLXAlphaCrossover", "SBXCrossover", "SPXCrossover", "UNDXCrossover", "UniformCrossover", "VSBXCrossover", ] optuna-3.5.0/optuna/samplers/nsgaii/_after_trial_strategy.py000066400000000000000000000020511453453102400243610ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from optuna.samplers._base import _process_constraints_after_trial from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState class NSGAIIAfterTrialStrategy: def __init__( self, *, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None ) -> None: self._constraints_func = constraints_func def __call__( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None = None, ) -> None: """Carry out the after trial process of default NSGA-II. This method is called after each trial of the study, examines whether the trial result is valid in terms of constraints, and store the results in system_attrs of the study. """ if self._constraints_func is not None: _process_constraints_after_trial(self._constraints_func, study, trial, state) optuna-3.5.0/optuna/samplers/nsgaii/_child_generation_strategy.py000066400000000000000000000072731453453102400253760ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import Any from optuna.distributions import BaseDistribution from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers.nsgaii._crossover import perform_crossover from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._dominates import _constrained_dominates from optuna.study import Study from optuna.study._multi_objective import _dominates from optuna.trial import FrozenTrial class NSGAIIChildGenerationStrategy: def __init__( self, *, mutation_prob: float | None = None, crossover: BaseCrossover, crossover_prob: float, swapping_prob: float, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, rng: LazyRandomState, ) -> None: if not (mutation_prob is None or 0.0 <= mutation_prob <= 1.0): raise ValueError( "`mutation_prob` must be None or a float value within the range [0.0, 1.0]." ) if not (0.0 <= crossover_prob <= 1.0): raise ValueError("`crossover_prob` must be a float value within the range [0.0, 1.0].") if not (0.0 <= swapping_prob <= 1.0): raise ValueError("`swapping_prob` must be a float value within the range [0.0, 1.0].") if not isinstance(crossover, BaseCrossover): raise ValueError( f"'{crossover}' is not a valid crossover." " For valid crossovers see" " https://optuna.readthedocs.io/en/stable/reference/samplers.html." ) self._crossover_prob = crossover_prob self._mutation_prob = mutation_prob self._swapping_prob = swapping_prob self._crossover = crossover self._constraints_func = constraints_func self._rng = rng def __call__( self, study: Study, search_space: dict[str, BaseDistribution], parent_population: list[FrozenTrial], ) -> dict[str, Any]: """Generate a child parameter from the given parent population by NSGA-II algorithm. Args: study: Target study object. search_space: A dictionary containing the parameter names and parameter's distributions. parent_population: A list of trials that are selected as parent population. Returns: A dictionary containing the parameter names and parameter's values. """ dominates = _dominates if self._constraints_func is None else _constrained_dominates # We choose a child based on the specified crossover method. if self._rng.rng.rand() < self._crossover_prob: child_params = perform_crossover( self._crossover, study, parent_population, search_space, self._rng.rng, self._swapping_prob, dominates, ) else: parent_population_size = len(parent_population) parent_params = parent_population[self._rng.rng.choice(parent_population_size)].params child_params = {name: parent_params[name] for name in search_space.keys()} n_params = len(child_params) if self._mutation_prob is None: mutation_prob = 1.0 / max(1.0, n_params) else: mutation_prob = self._mutation_prob params = {} for param_name in child_params.keys(): if self._rng.rng.rand() >= mutation_prob: params[param_name] = child_params[param_name] return params optuna-3.5.0/optuna/samplers/nsgaii/_crossover.py000066400000000000000000000134171453453102400222000ustar00rootroot00000000000000from typing import Any from typing import Callable from typing import Dict from typing import List from typing import Optional from typing import Sequence import numpy as np from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import FrozenTrial _NUMERICAL_DISTRIBUTIONS = ( FloatDistribution, IntDistribution, ) def _try_crossover( parents: List[FrozenTrial], crossover: BaseCrossover, study: Study, rng: np.random.RandomState, swapping_prob: float, categorical_search_space: Dict[str, BaseDistribution], numerical_search_space: Dict[str, BaseDistribution], numerical_transform: Optional[_SearchSpaceTransform], ) -> Dict[str, Any]: child_params: Dict[str, Any] = {} if len(categorical_search_space) > 0: parents_categorical_params = np.array( [ [parent.params[p] for p in categorical_search_space] for parent in [parents[0], parents[-1]] ], dtype=object, ) child_categorical_array = _inlined_categorical_uniform_crossover( parents_categorical_params, rng, swapping_prob, categorical_search_space ) child_categorical_params = { param: value for param, value in zip(categorical_search_space, child_categorical_array) } child_params.update(child_categorical_params) if numerical_transform is None: return child_params # The following is applied only for numerical parameters. parents_numerical_params = np.stack( [ numerical_transform.transform( { param_key: parent.params[param_key] for param_key in numerical_search_space.keys() } ) for parent in parents ] ) # Parent individual with NUMERICAL_DISTRIBUTIONS parameter. child_numerical_array = crossover.crossover( parents_numerical_params, rng, study, numerical_transform.bounds ) child_numerical_params = numerical_transform.untransform(child_numerical_array) child_params.update(child_numerical_params) return child_params def perform_crossover( crossover: BaseCrossover, study: Study, parent_population: Sequence[FrozenTrial], search_space: Dict[str, BaseDistribution], rng: np.random.RandomState, swapping_prob: float, dominates: Callable[[FrozenTrial, FrozenTrial, Sequence[StudyDirection]], bool], ) -> Dict[str, Any]: numerical_search_space: Dict[str, BaseDistribution] = {} categorical_search_space: Dict[str, BaseDistribution] = {} for key, value in search_space.items(): if isinstance(value, _NUMERICAL_DISTRIBUTIONS): numerical_search_space[key] = value else: categorical_search_space[key] = value numerical_transform: Optional[_SearchSpaceTransform] = None if len(numerical_search_space) != 0: numerical_transform = _SearchSpaceTransform(numerical_search_space) while True: # Repeat while parameters lie outside search space boundaries. parents = _select_parents(crossover, study, parent_population, rng, dominates) child_params = _try_crossover( parents, crossover, study, rng, swapping_prob, categorical_search_space, numerical_search_space, numerical_transform, ) if _is_contained(child_params, search_space): break return child_params def _select_parents( crossover: BaseCrossover, study: Study, parent_population: Sequence[FrozenTrial], rng: np.random.RandomState, dominates: Callable[[FrozenTrial, FrozenTrial, Sequence[StudyDirection]], bool], ) -> List[FrozenTrial]: parents: List[FrozenTrial] = [] for _ in range(crossover.n_parents): parent = _select_parent( study, [t for t in parent_population if t not in parents], rng, dominates ) parents.append(parent) return parents def _select_parent( study: Study, parent_population: Sequence[FrozenTrial], rng: np.random.RandomState, dominates: Callable[[FrozenTrial, FrozenTrial, Sequence[StudyDirection]], bool], ) -> FrozenTrial: population_size = len(parent_population) candidate0 = parent_population[rng.choice(population_size)] candidate1 = parent_population[rng.choice(population_size)] # TODO(ohta): Consider crowding distance. if dominates(candidate0, candidate1, study.directions): return candidate0 else: return candidate1 def _is_contained(params: Dict[str, Any], search_space: Dict[str, BaseDistribution]) -> bool: for param_name in params.keys(): param, param_distribution = params[param_name], search_space[param_name] if not param_distribution._contains(param_distribution.to_internal_repr(param)): return False return True def _inlined_categorical_uniform_crossover( parent_params: np.ndarray, rng: np.random.RandomState, swapping_prob: float, search_space: Dict[str, BaseDistribution], ) -> np.ndarray: # We can't use uniform crossover implementation of `BaseCrossover` for # parameters from `CategoricalDistribution`, since categorical params are # passed to crossover untransformed, which is not what `BaseCrossover` # implementations expect. n_categorical_params = len(search_space) masks = (rng.rand(n_categorical_params) >= swapping_prob).astype(int) return parent_params[masks, range(n_categorical_params)] optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/000077500000000000000000000000001453453102400220035ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/__init__.py000066400000000000000000000000001453453102400241020ustar00rootroot00000000000000optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_base.py000066400000000000000000000036631453453102400234360ustar00rootroot00000000000000import abc import numpy as np from optuna.study import Study class BaseCrossover(abc.ABC): """Base class for crossovers. A crossover operation is used by :class:`~optuna.samplers.NSGAIISampler` to create new parameter combination from parameters of ``n`` parent individuals. .. note:: Concrete implementations of this class are expected to only accept parameters from numerical distributions. At the moment, only crossover operation for categorical parameters (uniform crossover) is built-in into :class:`~optuna.samplers.NSGAIISampler`. """ def __str__(self) -> str: return self.__class__.__name__ @property @abc.abstractmethod def n_parents(self) -> int: """Number of parent individuals required to perform crossover.""" raise NotImplementedError @abc.abstractmethod def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: """Perform crossover of selected parent individuals. This method is called in :func:`~optuna.samplers.NSGAIISampler.sample_relative`. Args: parents_params: A ``numpy.ndarray`` with dimensions ``num_parents x num_parameters``. Represents a parameter space for each parent individual. This space is continuous for numerical parameters. rng: An instance of ``numpy.random.RandomState``. study: Target study object. search_space_bounds: A ``numpy.ndarray`` with dimensions ``len_search_space x 2`` representing numerical distribution bounds constructed from transformed search space. Returns: A 1-dimensional ``numpy.ndarray`` containing new parameter combination. """ raise NotImplementedError optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_blxalpha.py000066400000000000000000000031601453453102400243070ustar00rootroot00000000000000import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study @experimental_class("3.0.0") class BLXAlphaCrossover(BaseCrossover): """Blend Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Uniformly samples child individuals from the hyper-rectangles created by the two parent individuals. For further information about BLX-alpha crossover, please refer to the following paper: - `Eshelman, L. and J. D. Schaffer. Real-Coded Genetic Algorithms and Interval-Schemata. FOGA (1992). `_ Args: alpha: Parametrizes blend operation. """ n_parents = 2 def __init__(self, alpha: float = 0.5) -> None: self._alpha = alpha def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.465.6900&rep=rep1&type=pdf # Section 2 Crossover Operators for RCGA 2.1 Blend Crossover parents_min = parents_params.min(axis=0) parents_max = parents_params.max(axis=0) diff = self._alpha * (parents_max - parents_min) # Equation (1). low = parents_min - diff # Equation (1). high = parents_max + diff # Equation (1). r = rng.rand(len(search_space_bounds)) child_params = (high - low) * r + low return child_params optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_sbx.py000066400000000000000000000074731453453102400233230ustar00rootroot00000000000000from typing import Optional import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study @experimental_class("3.0.0") class SBXCrossover(BaseCrossover): """Simulated Binary Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Generates a child from two parent individuals according to the polynomial probability distribution. - `Deb, K. and R. Agrawal. “Simulated Binary Crossover for Continuous Search Space.†Complex Syst. 9 (1995): n. pag. `_ Args: eta: Distribution index. A small value of ``eta`` allows distant solutions to be selected as children solutions. If not specified, takes default value of ``2`` for single objective functions and ``20`` for multi objective. """ n_parents = 2 def __init__(self, eta: Optional[float] = None) -> None: self._eta = eta def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://www.researchgate.net/profile/M-M-Raghuwanshi/publication/267198495_Simulated_Binary_Crossover_with_Lognormal_Distribution/links/5576c78408ae7536375205d7/Simulated-Binary-Crossover-with-Lognormal-Distribution.pdf # Section 2 Simulated Binary Crossover (SBX) # To avoid generating solutions that violate the box constraints, # alpha1, alpha2, xls and xus are introduced, unlike the reference. xls = search_space_bounds[..., 0] xus = search_space_bounds[..., 1] xs_min = np.min(parents_params, axis=0) xs_max = np.max(parents_params, axis=0) if self._eta is None: eta = 20.0 if study._is_multi_objective() else 2.0 else: eta = self._eta xs_diff = np.clip(xs_max - xs_min, 1e-10, None) beta1 = 1 + 2 * (xs_min - xls) / xs_diff beta2 = 1 + 2 * (xus - xs_max) / xs_diff alpha1 = 2 - np.power(beta1, -(eta + 1)) alpha2 = 2 - np.power(beta2, -(eta + 1)) us = rng.rand(len(search_space_bounds)) mask1 = us > 1 / alpha1 # Equation (3). betaq1 = np.power(us * alpha1, 1 / (eta + 1)) # Equation (3). betaq1[mask1] = np.power((1 / (2 - us * alpha1)), 1 / (eta + 1))[mask1] # Equation (3). mask2 = us > 1 / alpha2 # Equation (3). betaq2 = np.power(us * alpha2, 1 / (eta + 1)) # Equation (3) betaq2[mask2] = np.power((1 / (2 - us * alpha2)), 1 / (eta + 1))[mask2] # Equation (3). c1 = 0.5 * ((xs_min + xs_max) - betaq1 * xs_diff) # Equation (4). c2 = 0.5 * ((xs_min + xs_max) + betaq2 * xs_diff) # Equation (5). # SBX applies crossover with establishment 0.5, and with probability 0.5, # the gene of the parent individual is the gene of the child individual. # The original SBX creates two child individuals, # but optuna's implementation creates only one child individual. # Therefore, when there is no crossover, # the gene is selected with equal probability from the parent individuals x1 and x2. child_params_list = [] for c1_i, c2_i, x1_i, x2_i in zip(c1, c2, parents_params[0], parents_params[1]): if rng.rand() < 0.5: if rng.rand() < 0.5: child_params_list.append(c1_i) else: child_params_list.append(c2_i) else: if rng.rand() < 0.5: child_params_list.append(x1_i) else: child_params_list.append(x2_i) child_params = np.array(child_params_list) return child_params optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_spx.py000066400000000000000000000041201453453102400233230ustar00rootroot00000000000000from typing import Optional import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study @experimental_class("3.0.0") class SPXCrossover(BaseCrossover): """Simplex Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Uniformly samples child individuals from within a single simplex that is similar to the simplex produced by the parent individual. For further information about SPX crossover, please refer to the following paper: - `Shigeyoshi Tsutsui and Shigeyoshi Tsutsui and David E. Goldberg and David E. Goldberg and Kumara Sastry and Kumara Sastry Progress Toward Linkage Learning in Real-Coded GAs with Simplex Crossover. IlliGAL Report. 2000. `_ Args: epsilon: Expansion rate. If not specified, defaults to ``sqrt(len(search_space) + 2)``. """ n_parents = 3 def __init__(self, epsilon: Optional[float] = None) -> None: self._epsilon = epsilon def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://www.researchgate.net/publication/2388486_Progress_Toward_Linkage_Learning_in_Real-Coded_GAs_with_Simplex_Crossover # Section 2 A Brief Review of SPX n = self.n_parents - 1 G = np.mean(parents_params, axis=0) # Equation (1). rs = np.power(rng.rand(n), 1 / (np.arange(n) + 1)) # Equation (2). epsilon = np.sqrt(len(search_space_bounds) + 2) if self._epsilon is None else self._epsilon xks = [G + epsilon * (pk - G) for pk in parents_params] # Equation (3). ck = 0 # Equation (4). for k in range(1, self.n_parents): ck = rs[k - 1] * (xks[k - 1] - xks[k] + ck) child_params = xks[-1] + ck # Equation (5). return child_params optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_undx.py000066400000000000000000000077631453453102400235070ustar00rootroot00000000000000from typing import Optional import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study @experimental_class("3.0.0") class UNDXCrossover(BaseCrossover): """Unimodal Normal Distribution Crossover used by :class:`~optuna.samplers.NSGAIISampler`. Generates child individuals from the three parents using a multivariate normal distribution. - `H. Kita, I. Ono and S. Kobayashi, Multi-parental extension of the unimodal normal distribution crossover for real-coded genetic algorithms, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406), 1999, pp. 1581-1588 Vol. 2 `_ Args: sigma_xi: Parametrizes normal distribution from which ``xi`` is drawn. sigma_eta: Parametrizes normal distribution from which ``etas`` are drawn. If not specified, defaults to ``0.35 / sqrt(len(search_space))``. """ n_parents = 3 def __init__(self, sigma_xi: float = 0.5, sigma_eta: Optional[float] = None) -> None: self._sigma_xi = sigma_xi self._sigma_eta = sigma_eta def _distance_from_x_to_psl(self, parents_params: np.ndarray) -> np.floating: # The line connecting x1 to x2 is called psl (primary search line). # Compute the 2-norm of the vector orthogonal to psl from x3. e_12 = UNDXCrossover._normalized_x1_to_x2( parents_params ) # Normalized vector from x1 to x2. v_13 = parents_params[2] - parents_params[0] # Vector from x1 to x3. v_12_3 = v_13 - np.dot(v_13, e_12) * e_12 # Vector orthogonal to v_12 through x3. m_12_3 = np.linalg.norm(v_12_3, ord=2) # 2-norm of v_12_3. return m_12_3 def _orthonormal_basis_vector_to_psl(self, parents_params: np.ndarray, n: int) -> np.ndarray: # Compute orthogonal basis vectors for the subspace orthogonal to psl. e_12 = UNDXCrossover._normalized_x1_to_x2( parents_params ) # Normalized vector from x1 to x2. basis_matrix = np.identity(n) if np.count_nonzero(e_12) != 0: basis_matrix[0] = e_12 basis_matrix_t = basis_matrix.T Q, _ = np.linalg.qr(basis_matrix_t) return Q.T[1:] def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://ieeexplore.ieee.org/document/782672 # Section 2 Unimodal Normal Distribution Crossover n = len(search_space_bounds) xp = (parents_params[0] + parents_params[1]) / 2 # Section 2 (2). d = parents_params[0] - parents_params[1] # Section 2 (3). if self._sigma_eta is None: sigma_eta = 0.35 / np.sqrt(n) else: sigma_eta = self._sigma_eta etas = rng.normal(0, sigma_eta**2, size=n) xi = rng.normal(0, self._sigma_xi**2) es = self._orthonormal_basis_vector_to_psl( parents_params, n ) # Orthonormal basis vectors of the subspace orthogonal to the psl. one = xp # Section 2 (5). two = xi * d # Section 2 (5). if n > 1: # When n=1, there is no subsearch component. three = np.zeros(n) # Section 2 (5). D = self._distance_from_x_to_psl(parents_params) # Section 2 (4). for i in range(n - 1): three += etas[i] * es[i] three *= D child_params = one + two + three else: child_params = one + two return child_params @staticmethod def _normalized_x1_to_x2(parents_params: np.ndarray) -> np.ndarray: # Compute the normalized vector from x1 to x2. v_12 = parents_params[1] - parents_params[0] m_12 = np.linalg.norm(v_12, ord=2) e_12 = v_12 / np.clip(m_12, 1e-10, None) return e_12 optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_uniform.py000066400000000000000000000032011453453102400241670ustar00rootroot00000000000000import numpy as np from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study class UniformCrossover(BaseCrossover): """Uniform Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. Select each parameter with equal probability from the two parent individuals. For further information about uniform crossover, please refer to the following paper: - `Gilbert Syswerda. 1989. Uniform Crossover in Genetic Algorithms. In Proceedings of the 3rd International Conference on Genetic Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2-9. `_ Args: swapping_prob: Probability of swapping each parameter of the parents during crossover. """ n_parents = 2 def __init__(self, swapping_prob: float = 0.5) -> None: if not (0.0 <= swapping_prob <= 1.0): raise ValueError("`swapping_prob` must be a float value within the range [0.0, 1.0].") self._swapping_prob = swapping_prob def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://www.researchgate.net/publication/201976488_Uniform_Crossover_in_Genetic_Algorithms # Section 1 Introduction n_params = len(search_space_bounds) masks = (rng.rand(n_params) >= self._swapping_prob).astype(int) child_params = parents_params[masks, range(n_params)] return child_params optuna-3.5.0/optuna/samplers/nsgaii/_crossovers/_vsbx.py000066400000000000000000000062641453453102400235060ustar00rootroot00000000000000from typing import Optional import numpy as np from optuna._experimental import experimental_class from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.study import Study @experimental_class("3.0.0") class VSBXCrossover(BaseCrossover): """Modified Simulated Binary Crossover operation used by :class:`~optuna.samplers.NSGAIISampler`. vSBX generates child individuals without excluding any region of the parameter space, while maintaining the excellent properties of SBX. - `Pedro J. Ballester, Jonathan N. Carter. Real-Parameter Genetic Algorithms for Finding Multiple Optimal Solutions in Multi-modal Optimization. GECCO 2003: 706-717 `_ Args: eta: Distribution index. A small value of ``eta`` allows distant solutions to be selected as children solutions. If not specified, takes default value of ``2`` for single objective functions and ``20`` for multi objective. """ n_parents = 2 def __init__(self, eta: Optional[float] = None) -> None: self._eta = eta def crossover( self, parents_params: np.ndarray, rng: np.random.RandomState, study: Study, search_space_bounds: np.ndarray, ) -> np.ndarray: # https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.422.952&rep=rep1&type=pdf # Section 3.2 Crossover Schemes (vSBX) if self._eta is None: eta = 20.0 if study._is_multi_objective() else 2.0 else: eta = self._eta us = rng.rand(len(search_space_bounds)) beta_1 = np.power(1 / 2 * us, 1 / (eta + 1)) beta_2 = np.power(1 / 2 * (1 - us), 1 / (eta + 1)) mask = us > 0.5 c1 = 0.5 * ((1 + beta_1) * parents_params[0] + (1 - beta_1) * parents_params[1]) c1[mask] = ( 0.5 * ((1 - beta_1) * parents_params[0] + (1 + beta_1) * parents_params[1])[mask] ) c2 = 0.5 * ((3 - beta_2) * parents_params[0] - (1 - beta_2) * parents_params[1]) c2[mask] = ( 0.5 * (-(1 - beta_2) * parents_params[0] + (3 - beta_2) * parents_params[1])[mask] ) # vSBX applies crossover with establishment 0.5, and with probability 0.5, # the gene of the parent individual is the gene of the child individual. # The original SBX creates two child individuals, # but optuna's implementation creates only one child individual. # Therefore, when there is no crossover, # the gene is selected with equal probability from the parent individuals x1 and x2. child_params_list = [] for c1_i, c2_i, x1_i, x2_i in zip(c1, c2, parents_params[0], parents_params[1]): if rng.rand() < 0.5: if rng.rand() < 0.5: child_params_list.append(c1_i) else: child_params_list.append(c2_i) else: if rng.rand() < 0.5: child_params_list.append(x1_i) else: child_params_list.append(x2_i) child_params = np.array(child_params_list) return child_params optuna-3.5.0/optuna/samplers/nsgaii/_dominates.py000066400000000000000000000064401453453102400221340ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import warnings import numpy as np from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import StudyDirection from optuna.study._multi_objective import _dominates from optuna.trial import FrozenTrial from optuna.trial import TrialState def _constrained_dominates( trial0: FrozenTrial, trial1: FrozenTrial, directions: Sequence[StudyDirection] ) -> bool: """Checks constrained-domination. A trial x is said to constrained-dominate a trial y, if any of the following conditions is true: 1) Trial x is feasible and trial y is not. 2) Trial x and y are both infeasible, but solution x has a smaller overall constraint violation. 3) Trial x and y are feasible and trial x dominates trial y. """ constraints0 = trial0.system_attrs.get(_CONSTRAINTS_KEY) constraints1 = trial1.system_attrs.get(_CONSTRAINTS_KEY) if constraints0 is None: warnings.warn( f"Trial {trial0.number} does not have constraint values." " It will be dominated by the other trials." ) if constraints1 is None: warnings.warn( f"Trial {trial1.number} does not have constraint values." " It will be dominated by the other trials." ) if constraints0 is None and constraints1 is None: # Neither Trial x nor y has constraints values return _dominates(trial0, trial1, directions) if constraints0 is not None and constraints1 is None: # Trial x has constraint values, but y doesn't. return True if constraints0 is None and constraints1 is not None: # If Trial y has constraint values, but x doesn't. return False assert isinstance(constraints0, (list, tuple)) assert isinstance(constraints1, (list, tuple)) if len(constraints0) != len(constraints1): raise ValueError("Trials with different numbers of constraints cannot be compared.") if trial0.state != TrialState.COMPLETE: return False if trial1.state != TrialState.COMPLETE: return True satisfy_constraints0 = all(v <= 0 for v in constraints0) satisfy_constraints1 = all(v <= 0 for v in constraints1) if satisfy_constraints0 and satisfy_constraints1: # Both trials satisfy the constraints. return _dominates(trial0, trial1, directions) if satisfy_constraints0: # trial0 satisfies the constraints, but trial1 violates them. return True if satisfy_constraints1: # trial1 satisfies the constraints, but trial0 violates them. return False # Both trials violate the constraints. violation0 = sum(v for v in constraints0 if v > 0) violation1 = sum(v for v in constraints1 if v > 0) return violation0 < violation1 def _validate_constraints( population: list[FrozenTrial], constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> None: if constraints_func is None: return for _trial in population: _constraints = _trial.system_attrs.get(_CONSTRAINTS_KEY) if _constraints is None: continue if np.any(np.isnan(np.array(_constraints))): raise ValueError("NaN is not acceptable as constraint value.") optuna-3.5.0/optuna/samplers/nsgaii/_elite_population_selection_strategy.py000066400000000000000000000131671453453102400275200ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import itertools import optuna from optuna.samplers.nsgaii._dominates import _constrained_dominates from optuna.samplers.nsgaii._dominates import _validate_constraints from optuna.study import Study from optuna.study._multi_objective import _dominates from optuna.trial import FrozenTrial class NSGAIIElitePopulationSelectionStrategy: def __init__( self, *, population_size: int, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> None: if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") self._population_size = population_size self._constraints_func = constraints_func def __call__(self, study: Study, population: list[FrozenTrial]) -> list[FrozenTrial]: """Select elite population from the given trials by NSGA-II algorithm. Args: study: Target study object. population: Trials in the study. Returns: A list of trials that are selected as elite population. """ _validate_constraints(population, self._constraints_func) dominates = _dominates if self._constraints_func is None else _constrained_dominates population_per_rank = _fast_non_dominated_sort(population, study.directions, dominates) elite_population: list[FrozenTrial] = [] for individuals in population_per_rank: if len(elite_population) + len(individuals) < self._population_size: elite_population.extend(individuals) else: n = self._population_size - len(elite_population) _crowding_distance_sort(individuals) elite_population.extend(individuals[:n]) break return elite_population def _calc_crowding_distance(population: list[FrozenTrial]) -> defaultdict[int, float]: """Calculates the crowding distance of population. We define the crowding distance as the summation of the crowding distance of each dimension of value calculated as follows: * If all values in that dimension are the same, i.e., [1, 1, 1] or [inf, inf], the crowding distances of all trials in that dimension are zero. * Otherwise, the crowding distances of that dimension is the difference between two nearest values besides that value, one above and one below, divided by the difference between the maximal and minimal finite value of that dimension. Please note that: * the nearest value below the minimum is considered to be -inf and the nearest value above the maximum is considered to be inf, and * inf - inf and (-inf) - (-inf) is considered to be zero. """ manhattan_distances: defaultdict[int, float] = defaultdict(float) if len(population) == 0: return manhattan_distances for i in range(len(population[0].values)): population.sort(key=lambda x: x.values[i]) # If all trials in population have the same value in the i-th dimension, ignore the # objective dimension since it does not make difference. if population[0].values[i] == population[-1].values[i]: continue vs = [-float("inf")] + [trial.values[i] for trial in population] + [float("inf")] # Smallest finite value. v_min = next(x for x in vs if x != -float("inf")) # Largest finite value. v_max = next(x for x in reversed(vs) if x != float("inf")) width = v_max - v_min if width <= 0: # width == 0 or width == -inf width = 1.0 for j in range(len(population)): # inf - inf and (-inf) - (-inf) is considered to be zero. gap = 0.0 if vs[j] == vs[j + 2] else vs[j + 2] - vs[j] manhattan_distances[population[j].number] += gap / width return manhattan_distances def _crowding_distance_sort(population: list[FrozenTrial]) -> None: manhattan_distances = _calc_crowding_distance(population) population.sort(key=lambda x: manhattan_distances[x.number]) population.reverse() def _fast_non_dominated_sort( population: list[FrozenTrial], directions: list[optuna.study.StudyDirection], dominates: Callable[[FrozenTrial, FrozenTrial, list[optuna.study.StudyDirection]], bool], ) -> list[list[FrozenTrial]]: dominated_count: defaultdict[int, int] = defaultdict(int) dominates_list = defaultdict(list) for p, q in itertools.combinations(population, 2): if dominates(p, q, directions): dominates_list[p.number].append(q.number) dominated_count[q.number] += 1 elif dominates(q, p, directions): dominates_list[q.number].append(p.number) dominated_count[p.number] += 1 population_per_rank = [] while population: non_dominated_population = [] i = 0 while i < len(population): if dominated_count[population[i].number] == 0: individual = population[i] if i == len(population) - 1: population.pop() else: population[i] = population.pop() non_dominated_population.append(individual) else: i += 1 for x in non_dominated_population: for y in dominates_list[x.number]: dominated_count[y] -= 1 assert non_dominated_population population_per_rank.append(non_dominated_population) return population_per_rank optuna-3.5.0/optuna/samplers/nsgaii/_sampler.py000066400000000000000000000373611453453102400216220ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from collections.abc import Callable from collections.abc import Sequence import hashlib from typing import Any import warnings import optuna from optuna.distributions import BaseDistribution from optuna.exceptions import ExperimentalWarning from optuna.samplers._base import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers._random import RandomSampler from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.samplers.nsgaii._child_generation_strategy import NSGAIIChildGenerationStrategy from optuna.samplers.nsgaii._crossovers._base import BaseCrossover from optuna.samplers.nsgaii._crossovers._uniform import UniformCrossover from optuna.samplers.nsgaii._elite_population_selection_strategy import ( NSGAIIElitePopulationSelectionStrategy, ) from optuna.search_space import IntersectionSearchSpace from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState # Define key names of `Trial.system_attrs`. _GENERATION_KEY = "nsga2:generation" _POPULATION_CACHE_KEY_PREFIX = "nsga2:population" class NSGAIISampler(BaseSampler): """Multi-objective sampler using the NSGA-II algorithm. NSGA-II stands for "Nondominated Sorting Genetic Algorithm II", which is a well known, fast and elitist multi-objective genetic algorithm. For further information about NSGA-II, please refer to the following paper: - `A fast and elitist multiobjective genetic algorithm: NSGA-II `_ Args: population_size: Number of individuals (trials) in a generation. ``population_size`` must be greater than or equal to ``crossover.n_parents``. For :class:`~optuna.samplers.nsgaii.UNDXCrossover` and :class:`~optuna.samplers.nsgaii.SPXCrossover`, ``n_parents=3``, and for the other algorithms, ``n_parents=2``. mutation_prob: Probability of mutating each parameter when creating a new individual. If :obj:`None` is specified, the value ``1.0 / len(parent_trial.params)`` is used where ``parent_trial`` is the parent trial of the target individual. crossover: Crossover to be applied when creating child individuals. The available crossovers are listed here: https://optuna.readthedocs.io/en/stable/reference/samplers/nsgaii.html. :class:`~optuna.samplers.nsgaii.UniformCrossover` is always applied to parameters sampled from :class:`~optuna.distributions.CategoricalDistribution`, and by default for parameters sampled from other distributions unless this argument is specified. For more information on each of the crossover method, please refer to specific crossover documentation. crossover_prob: Probability that a crossover (parameters swapping between parents) will occur when creating a new individual. swapping_prob: Probability of swapping each parameter of the parents during crossover. seed: Seed for random number generator. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible. If ``constraints_func`` returns more than one value for a trial, that trial is considered feasible if and only if all values are equal to 0 or smaller. The ``constraints_func`` will be evaluated after each successful trial. The function won't be called when trials fail or they are pruned, but this behavior is subject to change in the future releases. The constraints are handled by the constrained domination. A trial x is said to constrained-dominate a trial y, if any of the following conditions is true: 1. Trial x is feasible and trial y is not. 2. Trial x and y are both infeasible, but trial x has a smaller overall violation. 3. Trial x and y are feasible and trial x dominates trial y. .. note:: Added in v2.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.5.0. elite_population_selection_strategy: The selection strategy for determining the individuals to survive from the current population pool. Default to :obj:`None`. .. note:: The arguments ``elite_population_selection_strategy`` was added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. child_generation_strategy: The strategy for generating child parameters from parent trials. Defaults to :obj:`None`. .. note:: The arguments ``child_generation_strategy`` was added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. after_trial_strategy: A set of procedure to be conducted after each trial. Defaults to :obj:`None`. .. note:: The arguments ``after_trial_strategy`` was added in v3.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.3.0. """ def __init__( self, *, population_size: int = 50, mutation_prob: float | None = None, crossover: BaseCrossover | None = None, crossover_prob: float = 0.9, swapping_prob: float = 0.5, seed: int | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, elite_population_selection_strategy: Callable[ [Study, list[FrozenTrial]], list[FrozenTrial] ] | None = None, child_generation_strategy: Callable[ [Study, dict[str, BaseDistribution], list[FrozenTrial]], dict[str, Any] ] | None = None, after_trial_strategy: Callable[ [Study, FrozenTrial, TrialState, Sequence[float] | None], None ] | None = None, ) -> None: # TODO(ohta): Reconsider the default value of each parameter. if population_size < 2: raise ValueError("`population_size` must be greater than or equal to 2.") if constraints_func is not None: warnings.warn( "The constraints_func option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if after_trial_strategy is not None: warnings.warn( "The after_trial_strategy option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if child_generation_strategy is not None: warnings.warn( "The child_generation_strategy option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if elite_population_selection_strategy is not None: warnings.warn( "The elite_population_selection_strategy option is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) if crossover is None: crossover = UniformCrossover(swapping_prob) if not isinstance(crossover, BaseCrossover): raise ValueError( f"'{crossover}' is not a valid crossover." " For valid crossovers see" " https://optuna.readthedocs.io/en/stable/reference/samplers.html." ) if population_size < crossover.n_parents: raise ValueError( f"Using {crossover}," f" the population size should be greater than or equal to {crossover.n_parents}." f" The specified `population_size` is {population_size}." ) self._population_size = population_size self._random_sampler = RandomSampler(seed=seed) self._rng = LazyRandomState(seed) self._constraints_func = constraints_func self._search_space = IntersectionSearchSpace() self._elite_population_selection_strategy = ( elite_population_selection_strategy or NSGAIIElitePopulationSelectionStrategy( population_size=population_size, constraints_func=constraints_func ) ) self._child_generation_strategy = ( child_generation_strategy or NSGAIIChildGenerationStrategy( crossover_prob=crossover_prob, mutation_prob=mutation_prob, swapping_prob=swapping_prob, crossover=crossover, constraints_func=constraints_func, rng=self._rng, ) ) self._after_trial_strategy = after_trial_strategy or NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) def reseed_rng(self) -> None: self._random_sampler.reseed_rng() self._rng.rng.seed() def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: search_space: dict[str, BaseDistribution] = {} for name, distribution in self._search_space.calculate(study).items(): if distribution.single(): # The `untransform` method of `optuna._transform._SearchSpaceTransform` # does not assume a single value, # so single value objects are not sampled with the `sample_relative` method, # but with the `sample_independent` method. continue search_space[name] = distribution return search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: parent_generation, parent_population = self._collect_parent_population(study) generation = parent_generation + 1 study._storage.set_trial_system_attr(trial._trial_id, _GENERATION_KEY, generation) if parent_generation < 0: return {} return self._child_generation_strategy(study, search_space, parent_population) def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: # Following parameters are randomly sampled here. # 1. A parameter in the initial population/first generation. # 2. A parameter to mutate. # 3. A parameter excluded from the intersection search space. return self._random_sampler.sample_independent( study, trial, param_name, param_distribution ) def _collect_parent_population(self, study: Study) -> tuple[int, list[FrozenTrial]]: trials = study._get_trials(deepcopy=False, use_cache=True) generation_to_runnings = defaultdict(list) generation_to_population = defaultdict(list) for trial in trials: if _GENERATION_KEY not in trial.system_attrs: continue generation = trial.system_attrs[_GENERATION_KEY] if trial.state != optuna.trial.TrialState.COMPLETE: if trial.state == optuna.trial.TrialState.RUNNING: generation_to_runnings[generation].append(trial) continue # Do not use trials whose states are not COMPLETE, or `constraint` will be unavailable. generation_to_population[generation].append(trial) hasher = hashlib.sha256() parent_population: list[FrozenTrial] = [] parent_generation = -1 while True: generation = parent_generation + 1 population = generation_to_population[generation] # Under multi-worker settings, the population size might become larger than # `self._population_size`. if len(population) < self._population_size: break # [NOTE] # It's generally safe to assume that once the above condition is satisfied, # there are no additional individuals added to the generation (i.e., the members of # the generation have been fixed). # If the number of parallel workers is huge, this assumption can be broken, but # this is a very rare case and doesn't significantly impact optimization performance. # So we can ignore the case. # The cache key is calculated based on the key of the previous generation and # the remaining running trials in the current population. # If there are no running trials, the new cache key becomes exactly the same as # the previous one, and the cached content will be overwritten. This allows us to # skip redundant cache key calculations when this method is called for the subsequent # trials. for trial in generation_to_runnings[generation]: hasher.update(bytes(str(trial.number), "utf-8")) cache_key = "{}:{}".format(_POPULATION_CACHE_KEY_PREFIX, hasher.hexdigest()) study_system_attrs = study._storage.get_study_system_attrs(study._study_id) cached_generation, cached_population_numbers = study_system_attrs.get( cache_key, (-1, []) ) if cached_generation >= generation: generation = cached_generation population = [trials[n] for n in cached_population_numbers] else: population.extend(parent_population) population = self._elite_population_selection_strategy(study, population) # To reduce the number of system attribute entries, # we cache the population information only if there are no running trials # (i.e., the information of the population has been fixed). # Usually, if there are no too delayed running trials, the single entry # will be used. if len(generation_to_runnings[generation]) == 0: population_numbers = [t.number for t in population] study._storage.set_study_system_attr( study._study_id, cache_key, (generation, population_numbers) ) parent_generation = generation parent_population = population return parent_generation, parent_population def before_trial(self, study: Study, trial: FrozenTrial) -> None: self._random_sampler.before_trial(study, trial) def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert state in [TrialState.COMPLETE, TrialState.FAIL, TrialState.PRUNED] self._after_trial_strategy(study, trial, state, values) self._random_sampler.after_trial(study, trial, state, values) optuna-3.5.0/optuna/search_space/000077500000000000000000000000001453453102400167545ustar00rootroot00000000000000optuna-3.5.0/optuna/search_space/__init__.py000066400000000000000000000006501453453102400210660ustar00rootroot00000000000000from optuna.search_space.group_decomposed import _GroupDecomposedSearchSpace from optuna.search_space.group_decomposed import _SearchSpaceGroup from optuna.search_space.intersection import intersection_search_space from optuna.search_space.intersection import IntersectionSearchSpace __all__ = [ "_GroupDecomposedSearchSpace", "_SearchSpaceGroup", "IntersectionSearchSpace", "intersection_search_space", ] optuna-3.5.0/optuna/search_space/group_decomposed.py000066400000000000000000000044421453453102400226700ustar00rootroot00000000000000import copy from typing import Dict from typing import List from typing import Optional from typing import Tuple from optuna.distributions import BaseDistribution from optuna.study import Study from optuna.trial import TrialState class _SearchSpaceGroup: def __init__(self) -> None: self._search_spaces: List[Dict[str, BaseDistribution]] = [] @property def search_spaces(self) -> List[Dict[str, BaseDistribution]]: return self._search_spaces def add_distributions(self, distributions: Dict[str, BaseDistribution]) -> None: dist_keys = set(distributions.keys()) next_search_spaces = [] for search_space in self._search_spaces: keys = set(search_space.keys()) next_search_spaces.append({name: search_space[name] for name in keys & dist_keys}) next_search_spaces.append({name: search_space[name] for name in keys - dist_keys}) dist_keys -= keys next_search_spaces.append({name: distributions[name] for name in dist_keys}) self._search_spaces = list( filter(lambda search_space: len(search_space) > 0, next_search_spaces) ) class _GroupDecomposedSearchSpace: def __init__(self, include_pruned: bool = False) -> None: self._search_space = _SearchSpaceGroup() self._study_id: Optional[int] = None self._include_pruned = include_pruned def calculate(self, study: Study) -> _SearchSpaceGroup: if self._study_id is None: self._study_id = study._study_id else: # Note that the check below is meaningless when `InMemoryStorage` is used # because `InMemoryStorage.create_new_study` always returns the same study ID. if self._study_id != study._study_id: raise ValueError("`_GroupDecomposedSearchSpace` cannot handle multiple studies.") states_of_interest: Tuple[TrialState, ...] if self._include_pruned: states_of_interest = (TrialState.COMPLETE, TrialState.PRUNED) else: states_of_interest = (TrialState.COMPLETE,) for trial in study._get_trials(deepcopy=False, states=states_of_interest, use_cache=False): self._search_space.add_distributions(trial.distributions) return copy.deepcopy(self._search_space) optuna-3.5.0/optuna/search_space/intersection.py000066400000000000000000000121021453453102400220300ustar00rootroot00000000000000from __future__ import annotations import copy from typing import Dict from typing import Tuple import optuna from optuna.distributions import BaseDistribution from optuna.study import Study def _calculate( trials: list[optuna.trial.FrozenTrial], include_pruned: bool = False, search_space: Dict[str, BaseDistribution] | None = None, cached_trial_number: int = -1, ) -> Tuple[Dict[str, BaseDistribution] | None, int]: states_of_interest = [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.WAITING, optuna.trial.TrialState.RUNNING, ] if include_pruned: states_of_interest.append(optuna.trial.TrialState.PRUNED) trials_of_interest = [trial for trial in trials if trial.state in states_of_interest] next_cached_trial_number = ( trials_of_interest[-1].number + 1 if len(trials_of_interest) > 0 else -1 ) for trial in reversed(trials_of_interest): if cached_trial_number > trial.number: break if not trial.state.is_finished(): next_cached_trial_number = trial.number continue if search_space is None: search_space = copy.copy(trial.distributions) continue search_space = { name: distribution for name, distribution in search_space.items() if trial.distributions.get(name) == distribution } return search_space, next_cached_trial_number class IntersectionSearchSpace: """A class to calculate the intersection search space of a :class:`~optuna.study.Study`. Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded). Note that an instance of this class is supposed to be used for only one study. If different studies are passed to :func:`~optuna.search_space.IntersectionSearchSpace.calculate`, a :obj:`ValueError` is raised. Args: include_pruned: Whether pruned trials should be included in the search space. """ def __init__(self, include_pruned: bool = False) -> None: self._cached_trial_number: int = -1 self._search_space: Dict[str, BaseDistribution] | None = None self._study_id: int | None = None self._include_pruned = include_pruned def calculate(self, study: Study) -> Dict[str, BaseDistribution]: """Returns the intersection search space of the :class:`~optuna.study.Study`. Args: study: A study with completed trials. The same study must be passed for one instance of this class through its lifetime. Returns: A dictionary containing the parameter names and parameter's distributions sorted by parameter names. """ if self._study_id is None: self._study_id = study._study_id else: # Note that the check below is meaningless when `InMemoryStorage` is used # because `InMemoryStorage.create_new_study` always returns the same study ID. if self._study_id != study._study_id: raise ValueError("`IntersectionSearchSpace` cannot handle multiple studies.") self._search_space, self._cached_trial_number = _calculate( study.get_trials(deepcopy=False), self._include_pruned, self._search_space, self._cached_trial_number, ) search_space = self._search_space or {} search_space = dict(sorted(search_space.items(), key=lambda x: x[0])) return copy.deepcopy(search_space) def intersection_search_space( trials: list[optuna.trial.FrozenTrial], include_pruned: bool = False, ) -> Dict[str, BaseDistribution]: """Return the intersection search space of the given trials. Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded). .. note:: :class:`~optuna.search_space.IntersectionSearchSpace` provides the same functionality with a much faster way. Please consider using it if you want to reduce execution time as much as possible. Args: trials: A list of trials. include_pruned: Whether pruned trials should be included in the search space. Returns: A dictionary containing the parameter names and parameter's distributions sorted by parameter names. """ search_space, _ = _calculate(trials, include_pruned) search_space = search_space or {} search_space = dict(sorted(search_space.items(), key=lambda x: x[0])) return search_space optuna-3.5.0/optuna/storages/000077500000000000000000000000001453453102400161635ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/__init__.py000066400000000000000000000031341453453102400202750ustar00rootroot00000000000000from typing import Union from optuna._callbacks import RetryFailedTrialCallback from optuna.storages._base import BaseStorage from optuna.storages._cached_storage import _CachedStorage from optuna.storages._heartbeat import fail_stale_trials from optuna.storages._in_memory import InMemoryStorage from optuna.storages._journal.base import BaseJournalLogStorage from optuna.storages._journal.file import JournalFileOpenLock from optuna.storages._journal.file import JournalFileStorage from optuna.storages._journal.file import JournalFileSymlinkLock from optuna.storages._journal.redis import JournalRedisStorage from optuna.storages._journal.storage import JournalStorage from optuna.storages._rdb.storage import RDBStorage __all__ = [ "BaseStorage", "BaseJournalLogStorage", "InMemoryStorage", "RDBStorage", "JournalStorage", "JournalFileSymlinkLock", "JournalFileOpenLock", "JournalFileStorage", "JournalRedisStorage", "RetryFailedTrialCallback", "_CachedStorage", "fail_stale_trials", ] def get_storage(storage: Union[None, str, BaseStorage]) -> BaseStorage: """Only for internal usage. It might be deprecated in the future.""" if storage is None: return InMemoryStorage() if isinstance(storage, str): if storage.startswith("redis"): raise ValueError( "RedisStorage is removed at Optuna v3.1.0. Please use JournalRedisStorage instead." ) return _CachedStorage(RDBStorage(storage)) elif isinstance(storage, RDBStorage): return _CachedStorage(storage) else: return storage optuna-3.5.0/optuna/storages/_base.py000066400000000000000000000455231453453102400176170ustar00rootroot00000000000000import abc from typing import Any from typing import cast from typing import Container from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import Tuple from typing import Union from optuna._typing import JSONSerializable from optuna.distributions import BaseDistribution from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState DEFAULT_STUDY_NAME_PREFIX = "no-name-" class BaseStorage(abc.ABC): """Base class for storages. This class is not supposed to be directly accessed by library users. This class abstracts a backend database and provides internal interfaces to read/write histories of studies and trials. A storage class implementing this class must meet the following requirements. **Thread safety** A storage class instance can be shared among multiple threads, and must therefore be thread-safe. It must guarantee that a data instance read from the storage must not be modified by subsequent writes. For example, `FrozenTrial` instance returned by `get_trial` should not be updated by the subsequent `set_trial_xxx`. This is usually achieved by replacing the old data with a copy on `set_trial_xxx`. A storage class can also assume that a data instance returned are never modified by its user. When a user modifies a return value from a storage class, the internal state of the storage may become inconsistent. Consequences are undefined. **Ownership of RUNNING trials** Trials in finished states are not allowed to be modified. Trials in the WAITING state are not allowed to be modified except for the `state` field. """ # Basic study manipulation @abc.abstractmethod def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: """Create a new study from a name. If no name is specified, the storage class generates a name. The returned study ID is unique among all current and deleted studies. Args: directions: A sequence of direction whose element is either :obj:`~optuna.study.StudyDirection.MAXIMIZE` or :obj:`~optuna.study.StudyDirection.MINIMIZE`. study_name: Name of the new study to create. Returns: ID of the created study. Raises: :exc:`optuna.exceptions.DuplicatedStudyError`: If a study with the same ``study_name`` already exists. """ # TODO(ytsmiling) Fix RDB storage implementation to ensure unique `study_id`. raise NotImplementedError @abc.abstractmethod def delete_study(self, study_id: int) -> None: """Delete a study. Args: study_id: ID of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: """Register a user-defined attribute to a study. This method overwrites any existing attribute. Args: study_id: ID of the study. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: """Register an optuna-internal attribute to a study. This method overwrites any existing attribute. Args: study_id: ID of the study. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError # Basic study access @abc.abstractmethod def get_study_id_from_name(self, study_name: str) -> int: """Read the ID of a study. Args: study_name: Name of the study. Returns: ID of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_name`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_name_from_id(self, study_id: int) -> str: """Read the study name of a study. Args: study_id: ID of the study. Returns: Name of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_directions(self, study_id: int) -> List[StudyDirection]: """Read whether a study maximizes or minimizes an objective. Args: study_id: ID of a study. Returns: Optimization directions list of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: """Read the user-defined attributes of a study. Args: study_id: ID of the study. Returns: Dictionary with the user attributes of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: """Read the optuna-internal attributes of a study. Args: study_id: ID of the study. Returns: Dictionary with the optuna-internal attributes of the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_all_studies(self) -> List[FrozenStudy]: """Read a list of :class:`~optuna.study.FrozenStudy` objects. Returns: A list of :class:`~optuna.study.FrozenStudy` objects, sorted by ``study_id``. """ raise NotImplementedError # Basic trial manipulation @abc.abstractmethod def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: """Create and add a new trial to a study. The returned trial ID is unique among all current and deleted trials. Args: study_id: ID of the study. template_trial: Template :class:`~optuna.trial.FrozenTrial` with default user-attributes, system-attributes, intermediate-values, and a state. Returns: ID of the created trial. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError @abc.abstractmethod def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: BaseDistribution, ) -> None: """Set a parameter to a trial. Args: trial_id: ID of the trial. param_name: Name of the parameter. param_value_internal: Internal representation of the parameter value. distribution: Sampled distribution of the parameter. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: """Read the trial ID of a trial. Args: study_id: ID of the study. trial_number: Number of the trial. Returns: ID of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``study_id`` and ``trial_number`` exists. """ trials = self.get_all_trials(study_id, deepcopy=False) if len(trials) <= trial_number: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) return trials[trial_number]._trial_id def get_trial_number_from_id(self, trial_id: int) -> int: """Read the trial number of a trial. .. note:: The trial number is only unique within a study, and is sequential. Args: trial_id: ID of the trial. Returns: Number of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).number def get_trial_param(self, trial_id: int, param_name: str) -> float: """Read the parameter of a trial. Args: trial_id: ID of the trial. param_name: Name of the parameter. Returns: Internal representation of the parameter. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. If no such parameter exists. """ trial = self.get_trial(trial_id) return trial.distributions[param_name].to_internal_repr(trial.params[param_name]) @abc.abstractmethod def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: """Update the state and values of a trial. Set return values of an objective function to values argument. If values argument is not :obj:`None`, this method overwrites any existing trial values. Args: trial_id: ID of the trial. state: New state of the trial. values: Values of the objective function. Returns: :obj:`True` if the state is successfully updated. :obj:`False` if the state is kept the same. The latter happens when this method tries to update the state of :obj:`~optuna.trial.TrialState.RUNNING` trial to :obj:`~optuna.trial.TrialState.RUNNING`. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError @abc.abstractmethod def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: """Report an intermediate value of an objective function. This method overwrites any existing intermediate value associated with the given step. Args: trial_id: ID of the trial. step: Step of the trial (e.g., the epoch when training a neural network). intermediate_value: Intermediate value corresponding to the step. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError @abc.abstractmethod def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: """Set a user-defined attribute to a trial. This method overwrites any existing attribute. Args: trial_id: ID of the trial. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError @abc.abstractmethod def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: """Set an optuna-internal attribute to a trial. This method overwrites any existing attribute. Args: trial_id: ID of the trial. key: Attribute key. value: Attribute value. It should be JSON serializable. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. :exc:`RuntimeError`: If the trial is already finished. """ raise NotImplementedError # Basic trial access @abc.abstractmethod def get_trial(self, trial_id: int) -> FrozenTrial: """Read a trial. Args: trial_id: ID of the trial. Returns: Trial with a matching trial ID. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ raise NotImplementedError @abc.abstractmethod def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: """Read all trials in a study. Args: study_id: ID of the study. deepcopy: Whether to copy the list of trials before returning. Set to :obj:`True` if you intend to update the list or elements of the list. states: Trial states to filter on. If :obj:`None`, include all states. Returns: List of trials in the study, sorted by ``trial_id``. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ raise NotImplementedError def get_n_trials( self, study_id: int, state: Optional[Union[Tuple[TrialState, ...], TrialState]] = None ) -> int: """Count the number of trials in a study. Args: study_id: ID of the study. state: Trial states to filter on. If :obj:`None`, include all states. Returns: Number of trials in the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. """ # TODO(hvy): Align the name and the behavior or the `state` parameter with # `get_all_trials`'s `states`. if isinstance(state, TrialState): state = (state,) return len(self.get_all_trials(study_id, deepcopy=False, states=state)) def get_best_trial(self, study_id: int) -> FrozenTrial: """Return the trial with the best value in a study. This method is valid only during single-objective optimization. Args: study_id: ID of the study. Returns: The trial with the best objective value among all finished trials in the study. Raises: :exc:`KeyError`: If no study with the matching ``study_id`` exists. :exc:`RuntimeError`: If the study has more than one direction. :exc:`ValueError`: If no trials have been completed. """ all_trials = self.get_all_trials(study_id, deepcopy=False, states=[TrialState.COMPLETE]) if len(all_trials) == 0: raise ValueError("No trials are completed yet.") directions = self.get_study_directions(study_id) if len(directions) > 1: raise RuntimeError( "Best trial can be obtained only for single-objective optimization." ) direction = directions[0] if direction == StudyDirection.MAXIMIZE: best_trial = max(all_trials, key=lambda t: cast(float, t.value)) else: best_trial = min(all_trials, key=lambda t: cast(float, t.value)) return best_trial def get_trial_params(self, trial_id: int) -> Dict[str, Any]: """Read the parameter dictionary of a trial. Args: trial_id: ID of the trial. Returns: Dictionary of a parameters. Keys are parameter names and values are internal representations of the parameter values. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).params def get_trial_user_attrs(self, trial_id: int) -> Dict[str, Any]: """Read the user-defined attributes of a trial. Args: trial_id: ID of the trial. Returns: Dictionary with the user-defined attributes of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).user_attrs def get_trial_system_attrs(self, trial_id: int) -> Dict[str, Any]: """Read the optuna-internal attributes of a trial. Args: trial_id: ID of the trial. Returns: Dictionary with the optuna-internal attributes of the trial. Raises: :exc:`KeyError`: If no trial with the matching ``trial_id`` exists. """ return self.get_trial(trial_id).system_attrs def remove_session(self) -> None: """Clean up all connections to a database.""" pass def check_trial_is_updatable(self, trial_id: int, trial_state: TrialState) -> None: """Check whether a trial state is updatable. Args: trial_id: ID of the trial. Only used for an error message. trial_state: Trial state to check. Raises: :exc:`RuntimeError`: If the trial is already finished. """ if trial_state.is_finished(): trial = self.get_trial(trial_id) raise RuntimeError( "Trial#{} has already finished and can not be updated.".format(trial.number) ) optuna-3.5.0/optuna/storages/_cached_storage.py000066400000000000000000000260741453453102400216400ustar00rootroot00000000000000import copy import threading from typing import Any from typing import Callable from typing import Container from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import Set from typing import Tuple from typing import Union import optuna from optuna import distributions from optuna._typing import JSONSerializable from optuna.storages import BaseStorage from optuna.storages._heartbeat import BaseHeartbeat from optuna.storages._rdb.storage import RDBStorage from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState class _StudyInfo: def __init__(self) -> None: # Trial number to corresponding FrozenTrial. self.trials: Dict[int, FrozenTrial] = {} # A list of trials which do not require storage access to read latest attributes. self.finished_trial_ids: Set[int] = set() # Cache distributions to avoid storage access on distribution consistency check. self.param_distribution: Dict[str, distributions.BaseDistribution] = {} self.directions: Optional[List[StudyDirection]] = None self.name: Optional[str] = None class _CachedStorage(BaseStorage, BaseHeartbeat): """A wrapper class of storage backends. This class is used in :func:`~optuna.get_storage` function and automatically wraps :class:`~optuna.storages.RDBStorage` class. :class:`~optuna.storages._CachedStorage` meets the following **Data persistence** requirements. **Data persistence** :class:`~optuna.storages._CachedStorage` does not guarantee that write operations are logged into a persistent storage, even when write methods succeed. Thus, when process failure occurs, some writes might be lost. As exceptions, when a persistent storage is available, any writes on any attributes of `Study` and writes on `state` of `Trial` are guaranteed to be persistent. Additionally, any preceding writes on any attributes of `Trial` are guaranteed to be written into a persistent storage before writes on `state` of `Trial` succeed. The same applies for `param`, `user_attrs', 'system_attrs' and 'intermediate_values` attributes. Args: backend: :class:`~optuna.storages.RDBStorage` class instance to wrap. """ def __init__(self, backend: RDBStorage) -> None: self._backend = backend self._studies: Dict[int, _StudyInfo] = {} self._trial_id_to_study_id_and_number: Dict[int, Tuple[int, int]] = {} self._study_id_and_number_to_trial_id: Dict[Tuple[int, int], int] = {} self._lock = threading.Lock() def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["_lock"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) self._lock = threading.Lock() def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: study_id = self._backend.create_new_study(directions=directions, study_name=study_name) with self._lock: study = _StudyInfo() study.name = study_name study.directions = list(directions) self._studies[study_id] = study return study_id def delete_study(self, study_id: int) -> None: with self._lock: if study_id in self._studies: for trial_number in self._studies[study_id].trials: trial_id = self._study_id_and_number_to_trial_id.get((study_id, trial_number)) if trial_id in self._trial_id_to_study_id_and_number: del self._trial_id_to_study_id_and_number[trial_id] if (study_id, trial_number) in self._study_id_and_number_to_trial_id: del self._study_id_and_number_to_trial_id[(study_id, trial_number)] del self._studies[study_id] self._backend.delete_study(study_id) def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: self._backend.set_study_user_attr(study_id, key, value) def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: self._backend.set_study_system_attr(study_id, key, value) def get_study_id_from_name(self, study_name: str) -> int: return self._backend.get_study_id_from_name(study_name) def get_study_name_from_id(self, study_id: int) -> str: with self._lock: if study_id in self._studies: name = self._studies[study_id].name if name is not None: return name name = self._backend.get_study_name_from_id(study_id) with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() self._studies[study_id].name = name return name def get_study_directions(self, study_id: int) -> List[StudyDirection]: with self._lock: if study_id in self._studies: directions = self._studies[study_id].directions if directions is not None: return directions directions = self._backend.get_study_directions(study_id) with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() self._studies[study_id].directions = directions return directions def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: return self._backend.get_study_user_attrs(study_id) def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: return self._backend.get_study_system_attrs(study_id) def get_all_studies(self) -> List[FrozenStudy]: return self._backend.get_all_studies() def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: frozen_trial = self._backend._create_new_trial(study_id, template_trial) trial_id = frozen_trial._trial_id with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() study = self._studies[study_id] self._add_trials_to_cache(study_id, [frozen_trial]) # Since finished trials will not be modified by any worker, we do not # need storage access for them. if frozen_trial.state.is_finished(): study.finished_trial_ids.add(frozen_trial._trial_id) return trial_id def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: self._backend.set_trial_param(trial_id, param_name, param_value_internal, distribution) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: key = (study_id, trial_number) with self._lock: if key in self._study_id_and_number_to_trial_id: return self._study_id_and_number_to_trial_id[key] return self._backend.get_trial_id_from_study_id_trial_number(study_id, trial_number) def get_best_trial(self, study_id: int) -> FrozenTrial: return self._backend.get_best_trial(study_id) def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: return self._backend.set_trial_state_values(trial_id, state=state, values=values) def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: self._backend.set_trial_intermediate_value(trial_id, step, intermediate_value) def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: self._backend.set_trial_user_attr(trial_id, key=key, value=value) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: self._backend.set_trial_system_attr(trial_id, key=key, value=value) def _get_cached_trial(self, trial_id: int) -> Optional[FrozenTrial]: if trial_id not in self._trial_id_to_study_id_and_number: return None study_id, number = self._trial_id_to_study_id_and_number[trial_id] study = self._studies[study_id] return study.trials[number] if trial_id in study.finished_trial_ids else None def get_trial(self, trial_id: int) -> FrozenTrial: with self._lock: trial = self._get_cached_trial(trial_id) if trial is not None: return trial return self._backend.get_trial(trial_id) def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: self._read_trials_from_remote_storage(study_id) with self._lock: study = self._studies[study_id] # We need to sort trials by their number because some samplers assume this behavior. # The following two lines are latency-sensitive. trials: Union[Dict[int, FrozenTrial], List[FrozenTrial]] if states is not None: trials = {number: t for number, t in study.trials.items() if t.state in states} else: trials = study.trials trials = list(sorted(trials.values(), key=lambda t: t.number)) return copy.deepcopy(trials) if deepcopy else trials def _read_trials_from_remote_storage(self, study_id: int) -> None: with self._lock: if study_id not in self._studies: self._studies[study_id] = _StudyInfo() study = self._studies[study_id] trials = self._backend._get_trials( study_id, states=None, excluded_trial_ids=study.finished_trial_ids ) if trials: self._add_trials_to_cache(study_id, trials) for trial in trials: if trial.state.is_finished(): study.finished_trial_ids.add(trial._trial_id) def _add_trials_to_cache(self, study_id: int, trials: List[FrozenTrial]) -> None: study = self._studies[study_id] for trial in trials: self._trial_id_to_study_id_and_number[trial._trial_id] = ( study_id, trial.number, ) self._study_id_and_number_to_trial_id[(study_id, trial.number)] = trial._trial_id study.trials[trial.number] = trial def record_heartbeat(self, trial_id: int) -> None: self._backend.record_heartbeat(trial_id) def _get_stale_trial_ids(self, study_id: int) -> List[int]: return self._backend._get_stale_trial_ids(study_id) def get_heartbeat_interval(self) -> Optional[int]: return self._backend.get_heartbeat_interval() def get_failed_trial_callback(self) -> Optional[Callable[["optuna.Study", FrozenTrial], None]]: return self._backend.get_failed_trial_callback() optuna-3.5.0/optuna/storages/_heartbeat.py000066400000000000000000000135631453453102400206430ustar00rootroot00000000000000import abc import copy from threading import Event from threading import Thread from types import TracebackType from typing import Callable from typing import List from typing import Optional from typing import Type import optuna from optuna._experimental import experimental_func from optuna.storages import BaseStorage from optuna.trial import FrozenTrial from optuna.trial import TrialState class BaseHeartbeat(metaclass=abc.ABCMeta): """Base class for heartbeat. This class is not supposed to be directly accessed by library users. The heartbeat mechanism periodically checks whether each trial process is alive during an optimization loop. To support this mechanism, the methods of :class:`~optuna.storages._heartbeat.BaseHeartbeat` is implemented for the target database backend, typically with multiple inheritance of :class:`~optuna.storages._base.BaseStorage` and :class:`~optuna.storages._heartbeat.BaseHeartbeat`. .. seealso:: See :class:`~optuna.storages.RDBStorage`, where the backend supports heartbeat. """ @abc.abstractmethod def record_heartbeat(self, trial_id: int) -> None: """Record the heartbeat of the trial. Args: trial_id: ID of the trial. """ raise NotImplementedError() @abc.abstractmethod def _get_stale_trial_ids(self, study_id: int) -> List[int]: """Get the stale trial ids of the study. Args: study_id: ID of the study. Returns: List of IDs of trials whose heartbeat has not been updated for a long time. """ raise NotImplementedError() @abc.abstractmethod def get_heartbeat_interval(self) -> Optional[int]: """Get the heartbeat interval if it is set. Returns: The heartbeat interval if it is set, otherwise :obj:`None`. """ raise NotImplementedError() @abc.abstractmethod def get_failed_trial_callback(self) -> Optional[Callable[["optuna.Study", FrozenTrial], None]]: """Get the failed trial callback function. Returns: The failed trial callback function if it is set, otherwise :obj:`None`. """ raise NotImplementedError() class BaseHeartbeatThread(metaclass=abc.ABCMeta): def __enter__(self) -> None: self.start() def __exit__( self, exc_type: Optional[Type[Exception]], exc_value: Optional[Exception], traceback: Optional[TracebackType], ) -> None: self.join() @abc.abstractmethod def start(self) -> None: raise NotImplementedError() @abc.abstractmethod def join(self) -> None: raise NotImplementedError() class NullHeartbeatThread(BaseHeartbeatThread): def __init__(self) -> None: pass def start(self) -> None: pass def join(self) -> None: pass class HeartbeatThread(BaseHeartbeatThread): def __init__(self, trial_id: int, heartbeat: BaseHeartbeat) -> None: self._trial_id = trial_id self._heartbeat = heartbeat self._thread: Optional[Thread] = None self._stop_event: Optional[Event] = None def start(self) -> None: self._stop_event = Event() self._thread = Thread( target=self._record_heartbeat, args=(self._trial_id, self._heartbeat, self._stop_event) ) self._thread.start() def join(self) -> None: assert self._stop_event is not None assert self._thread is not None self._stop_event.set() self._thread.join() @staticmethod def _record_heartbeat(trial_id: int, heartbeat: BaseHeartbeat, stop_event: Event) -> None: heartbeat_interval = heartbeat.get_heartbeat_interval() assert heartbeat_interval is not None while True: heartbeat.record_heartbeat(trial_id) if stop_event.wait(timeout=heartbeat_interval): return def get_heartbeat_thread(trial_id: int, storage: BaseStorage) -> BaseHeartbeatThread: if is_heartbeat_enabled(storage): assert isinstance(storage, BaseHeartbeat) return HeartbeatThread(trial_id, storage) else: return NullHeartbeatThread() @experimental_func("2.9.0") def fail_stale_trials(study: "optuna.Study") -> None: """Fail stale trials and run their failure callbacks. The running trials whose heartbeat has not been updated for a long time will be failed, that is, those states will be changed to :obj:`~optuna.trial.TrialState.FAIL`. .. seealso:: See :class:`~optuna.storages.RDBStorage`. Args: study: Study holding the trials to check. """ storage = study._storage if not isinstance(storage, BaseHeartbeat): return if not is_heartbeat_enabled(storage): return failed_trial_ids = [] for trial_id in storage._get_stale_trial_ids(study._study_id): try: if storage.set_trial_state_values(trial_id, state=TrialState.FAIL): failed_trial_ids.append(trial_id) except RuntimeError: # If another process fails the trial, the storage raises RuntimeError. pass failed_trial_callback = storage.get_failed_trial_callback() if failed_trial_callback is not None: for trial_id in failed_trial_ids: failed_trial = copy.deepcopy(storage.get_trial(trial_id)) failed_trial_callback(study, failed_trial) def is_heartbeat_enabled(storage: BaseStorage) -> bool: """Check whether the storage enables the heartbeat. Returns: :obj:`True` if the storage also inherits :class:`~optuna.storages._heartbeat.BaseHeartbeat` and the return value of :meth:`~optuna.storages.BaseStorage.get_heartbeat_interval` is an integer, otherwise :obj:`False`. """ return isinstance(storage, BaseHeartbeat) and storage.get_heartbeat_interval() is not None optuna-3.5.0/optuna/storages/_in_memory.py000066400000000000000000000340511453453102400206750ustar00rootroot00000000000000import copy from datetime import datetime import threading from typing import Any from typing import Container from typing import Dict from typing import List from typing import Optional from typing import Sequence from typing import Tuple import uuid import optuna from optuna import distributions # NOQA from optuna._typing import JSONSerializable from optuna.exceptions import DuplicatedStudyError from optuna.storages import BaseStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = optuna.logging.get_logger(__name__) class InMemoryStorage(BaseStorage): """Storage class that stores data in memory of the Python process. This class is not supposed to be directly accessed by library users. """ def __init__(self) -> None: self._trial_id_to_study_id_and_number: Dict[int, Tuple[int, int]] = {} self._study_name_to_id: Dict[str, int] = {} self._studies: Dict[int, _StudyInfo] = {} self._max_study_id = -1 self._max_trial_id = -1 self._lock = threading.RLock() def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["_lock"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) self._lock = threading.RLock() def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: with self._lock: study_id = self._max_study_id + 1 self._max_study_id += 1 if study_name is not None: if study_name in self._study_name_to_id: raise DuplicatedStudyError else: study_uuid = str(uuid.uuid4()) study_name = DEFAULT_STUDY_NAME_PREFIX + study_uuid self._studies[study_id] = _StudyInfo(study_name, list(directions)) self._study_name_to_id[study_name] = study_id _logger.info("A new study created in memory with name: {}".format(study_name)) return study_id def delete_study(self, study_id: int) -> None: with self._lock: self._check_study_id(study_id) for trial in self._studies[study_id].trials: del self._trial_id_to_study_id_and_number[trial._trial_id] study_name = self._studies[study_id].name del self._study_name_to_id[study_name] del self._studies[study_id] def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: with self._lock: self._check_study_id(study_id) self._studies[study_id].user_attrs[key] = value def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: with self._lock: self._check_study_id(study_id) self._studies[study_id].system_attrs[key] = value def get_study_id_from_name(self, study_name: str) -> int: with self._lock: if study_name not in self._study_name_to_id: raise KeyError("No such study {}.".format(study_name)) return self._study_name_to_id[study_name] def get_study_name_from_id(self, study_id: int) -> str: with self._lock: self._check_study_id(study_id) return self._studies[study_id].name def get_study_directions(self, study_id: int) -> List[StudyDirection]: with self._lock: self._check_study_id(study_id) return self._studies[study_id].directions def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: with self._lock: self._check_study_id(study_id) return self._studies[study_id].user_attrs def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: with self._lock: self._check_study_id(study_id) return self._studies[study_id].system_attrs def get_all_studies(self) -> List[FrozenStudy]: with self._lock: return [self._build_frozen_study(study_id) for study_id in self._studies] def _build_frozen_study(self, study_id: int) -> FrozenStudy: study = self._studies[study_id] return FrozenStudy( study_name=study.name, direction=None, directions=study.directions, user_attrs=copy.deepcopy(study.user_attrs), system_attrs=copy.deepcopy(study.system_attrs), study_id=study_id, ) def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: with self._lock: self._check_study_id(study_id) if template_trial is None: trial = self._create_running_trial() else: trial = copy.deepcopy(template_trial) trial_id = self._max_trial_id + 1 self._max_trial_id += 1 trial.number = len(self._studies[study_id].trials) trial._trial_id = trial_id self._trial_id_to_study_id_and_number[trial_id] = (study_id, trial.number) self._studies[study_id].trials.append(trial) self._update_cache(trial_id, study_id) return trial_id @staticmethod def _create_running_trial() -> FrozenTrial: return FrozenTrial( trial_id=-1, # dummy value. number=-1, # dummy value. state=TrialState.RUNNING, params={}, distributions={}, user_attrs={}, system_attrs={}, value=None, intermediate_values={}, datetime_start=datetime.now(), datetime_complete=None, ) def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: with self._lock: trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) study_id = self._trial_id_to_study_id_and_number[trial_id][0] # Check param distribution compatibility with previous trial(s). if param_name in self._studies[study_id].param_distribution: distributions.check_distribution_compatibility( self._studies[study_id].param_distribution[param_name], distribution ) # Set param distribution. self._studies[study_id].param_distribution[param_name] = distribution # Set param. trial = copy.copy(trial) trial.params = copy.copy(trial.params) trial.params[param_name] = distribution.to_external_repr(param_value_internal) trial.distributions = copy.copy(trial.distributions) trial.distributions[param_name] = distribution self._set_trial(trial_id, trial) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: with self._lock: study = self._studies.get(study_id) if study is None: raise KeyError("No study with study_id {} exists.".format(study_id)) trials = study.trials if len(trials) <= trial_number: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) trial = trials[trial_number] assert trial.number == trial_number return trial._trial_id def get_trial_number_from_id(self, trial_id: int) -> int: with self._lock: self._check_trial_id(trial_id) return self._trial_id_to_study_id_and_number[trial_id][1] def get_best_trial(self, study_id: int) -> FrozenTrial: with self._lock: self._check_study_id(study_id) best_trial_id = self._studies[study_id].best_trial_id if best_trial_id is None: raise ValueError("No trials are completed yet.") elif len(self._studies[study_id].directions) > 1: raise RuntimeError( "Best trial can be obtained only for single-objective optimization." ) return self.get_trial(best_trial_id) def get_trial_param(self, trial_id: int, param_name: str) -> float: with self._lock: trial = self._get_trial(trial_id) distribution = trial.distributions[param_name] return distribution.to_internal_repr(trial.params[param_name]) def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: with self._lock: trial = copy.copy(self._get_trial(trial_id)) self.check_trial_is_updatable(trial_id, trial.state) if state == TrialState.RUNNING and trial.state != TrialState.WAITING: return False trial.state = state if values is not None: trial.values = values if state == TrialState.RUNNING: trial.datetime_start = datetime.now() if state.is_finished(): trial.datetime_complete = datetime.now() self._set_trial(trial_id, trial) study_id = self._trial_id_to_study_id_and_number[trial_id][0] self._update_cache(trial_id, study_id) else: self._set_trial(trial_id, trial) return True def _update_cache(self, trial_id: int, study_id: int) -> None: trial = self._get_trial(trial_id) if trial.state != TrialState.COMPLETE: return best_trial_id = self._studies[study_id].best_trial_id if best_trial_id is None: self._studies[study_id].best_trial_id = trial_id return _directions = self.get_study_directions(study_id) if len(_directions) > 1: return direction = _directions[0] best_trial = self._get_trial(best_trial_id) assert best_trial is not None if best_trial.value is None: self._studies[study_id].best_trial_id = trial_id return # Complete trials do not have `None` values. assert trial.value is not None best_value = best_trial.value new_value = trial.value if direction == StudyDirection.MAXIMIZE: if best_value < new_value: self._studies[study_id].best_trial_id = trial_id else: if best_value > new_value: self._studies[study_id].best_trial_id = trial_id def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: with self._lock: trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) trial = copy.copy(trial) trial.intermediate_values = copy.copy(trial.intermediate_values) trial.intermediate_values[step] = intermediate_value self._set_trial(trial_id, trial) def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: with self._lock: self._check_trial_id(trial_id) trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) trial = copy.copy(trial) trial.user_attrs = copy.copy(trial.user_attrs) trial.user_attrs[key] = value self._set_trial(trial_id, trial) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: with self._lock: trial = self._get_trial(trial_id) self.check_trial_is_updatable(trial_id, trial.state) trial = copy.copy(trial) trial.system_attrs = copy.copy(trial.system_attrs) trial.system_attrs[key] = value self._set_trial(trial_id, trial) def get_trial(self, trial_id: int) -> FrozenTrial: with self._lock: return self._get_trial(trial_id) def _get_trial(self, trial_id: int) -> FrozenTrial: self._check_trial_id(trial_id) study_id, trial_number = self._trial_id_to_study_id_and_number[trial_id] return self._studies[study_id].trials[trial_number] def _set_trial(self, trial_id: int, trial: FrozenTrial) -> None: study_id, trial_number = self._trial_id_to_study_id_and_number[trial_id] self._studies[study_id].trials[trial_number] = trial def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: with self._lock: self._check_study_id(study_id) trials = self._studies[study_id].trials if states is not None: trials = [t for t in trials if t.state in states] if deepcopy: trials = copy.deepcopy(trials) else: # This copy is required for the replacing trick in `set_trial_xxx`. trials = copy.copy(trials) return trials def _check_study_id(self, study_id: int) -> None: if study_id not in self._studies: raise KeyError("No study with study_id {} exists.".format(study_id)) def _check_trial_id(self, trial_id: int) -> None: if trial_id not in self._trial_id_to_study_id_and_number: raise KeyError("No trial with trial_id {} exists.".format(trial_id)) class _StudyInfo: def __init__(self, name: str, directions: List[StudyDirection]) -> None: self.trials: List[FrozenTrial] = [] self.param_distribution: Dict[str, distributions.BaseDistribution] = {} self.user_attrs: Dict[str, Any] = {} self.system_attrs: Dict[str, Any] = {} self.name: str = name self.directions: List[StudyDirection] = directions self.best_trial_id: Optional[int] = None optuna-3.5.0/optuna/storages/_journal/000077500000000000000000000000001453453102400177745ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/_journal/__init__.py000066400000000000000000000000001453453102400220730ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/_journal/base.py000066400000000000000000000041201453453102400212550ustar00rootroot00000000000000import abc from typing import Any from typing import Dict from typing import List from typing import Optional class BaseJournalLogStorage(abc.ABC): """Base class for Journal storages. Storage classes implementing this base class must guarantee process safety. This means, multiple processes might concurrently call ``read_logs`` and ``append_logs``. If the backend storage does not internally support mutual exclusion mechanisms, such as locks, you might want to use :class:`~optuna.storages.JournalFileSymlinkLock` or :class:`~optuna.storages.JournalFileOpenLock` for creating a critical section. """ @abc.abstractmethod def read_logs(self, log_number_from: int) -> List[Dict[str, Any]]: """Read logs with a log number greater than or equal to ``log_number_from``. If ``log_number_from`` is 0, read all the logs. Args: log_number_from: A non-negative integer value indicating which logs to read. Returns: Logs with log number greater than or equal to ``log_number_from``. """ raise NotImplementedError @abc.abstractmethod def append_logs(self, logs: List[Dict[str, Any]]) -> None: """Append logs to the backend. Args: logs: A list that contains json-serializable logs. """ raise NotImplementedError class BaseJournalLogSnapshot(abc.ABC): """Optional base class for Journal storages. Storage classes implementing this base class may work faster when constructing the internal state from the large amount of logs. """ @abc.abstractmethod def save_snapshot(self, snapshot: bytes) -> None: """Save snapshot to the backend. Args: snapshot: A serialized snapshot (bytes) """ raise NotImplementedError @abc.abstractmethod def load_snapshot(self) -> Optional[bytes]: """Load snapshot from the backend. Returns: A serialized snapshot (bytes) if found, otherwise :obj:`None`. """ raise NotImplementedError optuna-3.5.0/optuna/storages/_journal/file.py000066400000000000000000000147331453453102400212750ustar00rootroot00000000000000import abc from contextlib import contextmanager import errno import json import os import time from typing import Any from typing import Dict from typing import Iterator from typing import List from typing import Optional import uuid from optuna.storages._journal.base import BaseJournalLogStorage LOCK_FILE_SUFFIX = ".lock" RENAME_FILE_SUFFIX = ".rename" class JournalFileBaseLock(abc.ABC): @abc.abstractmethod def acquire(self) -> bool: raise NotImplementedError @abc.abstractmethod def release(self) -> None: raise NotImplementedError class JournalFileSymlinkLock(JournalFileBaseLock): """Lock class for synchronizing processes for NFSv2 or later. On acquiring the lock, link system call is called to create an exclusive file. The file is deleted when the lock is released. In NFS environments prior to NFSv3, use this instead of :class:`~optuna.storages.JournalFileOpenLock` Args: filepath: The path of the file whose race condition must be protected. """ def __init__(self, filepath: str) -> None: self._lock_target_file = filepath self._lock_file = filepath + LOCK_FILE_SUFFIX self._lock_rename_file = self._lock_file + str(uuid.uuid4()) + RENAME_FILE_SUFFIX def acquire(self) -> bool: """Acquire a lock in a blocking way by creating a symbolic link of a file. Returns: :obj:`True` if it succeeded in creating a symbolic link of `self._lock_target_file`. """ sleep_secs = 0.001 while True: try: os.symlink(self._lock_target_file, self._lock_file) return True except OSError as err: if err.errno == errno.EEXIST: time.sleep(sleep_secs) sleep_secs = min(sleep_secs * 2, 1) continue raise err except BaseException: self.release() raise def release(self) -> None: """Release a lock by removing the symbolic link.""" try: os.rename(self._lock_file, self._lock_rename_file) os.unlink(self._lock_rename_file) except OSError: raise RuntimeError("Error: did not possess lock") except BaseException: os.unlink(self._lock_rename_file) raise class JournalFileOpenLock(JournalFileBaseLock): """Lock class for synchronizing processes for NFSv3 or later. On acquiring the lock, open system call is called with the O_EXCL option to create an exclusive file. The file is deleted when the lock is released. This class is only supported when using NFSv3 or later on kernel 2.6 or later. In prior NFS environments, use :class:`~optuna.storages.JournalFileSymlinkLock`. Args: filepath: The path of the file whose race condition must be protected. """ def __init__(self, filepath: str) -> None: self._lock_file = filepath + LOCK_FILE_SUFFIX self._lock_rename_file = self._lock_file + str(uuid.uuid4()) + RENAME_FILE_SUFFIX def acquire(self) -> bool: """Acquire a lock in a blocking way by creating a lock file. Returns: :obj:`True` if it succeeded in creating a `self._lock_file` """ sleep_secs = 0.001 while True: try: open_flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY os.close(os.open(self._lock_file, open_flags)) return True except OSError as err: if err.errno == errno.EEXIST: time.sleep(sleep_secs) sleep_secs = min(sleep_secs * 2, 1) continue raise err except BaseException: self.release() raise def release(self) -> None: """Release a lock by removing the created file.""" try: os.rename(self._lock_file, self._lock_rename_file) os.unlink(self._lock_rename_file) except OSError: raise RuntimeError("Error: did not possess lock") except BaseException: os.unlink(self._lock_rename_file) raise @contextmanager def get_lock_file(lock_obj: JournalFileBaseLock) -> Iterator[None]: lock_obj.acquire() try: yield finally: lock_obj.release() class JournalFileStorage(BaseJournalLogStorage): """File storage class for Journal log backend. Args: file_path: Path of file to persist the log to. lock_obj: Lock object for process exclusivity. """ def __init__(self, file_path: str, lock_obj: Optional[JournalFileBaseLock] = None) -> None: self._file_path: str = file_path self._lock = lock_obj or JournalFileSymlinkLock(self._file_path) if not os.path.exists(self._file_path): open(self._file_path, "ab").close() # Create a file if it does not exist self._log_number_offset: Dict[int, int] = {0: 0} def read_logs(self, log_number_from: int) -> List[Dict[str, Any]]: logs = [] with open(self._file_path, "rb") as f: log_number_start = 0 if log_number_from in self._log_number_offset: f.seek(self._log_number_offset[log_number_from]) log_number_start = log_number_from last_decode_error = None for log_number, line in enumerate(f, start=log_number_start): if last_decode_error is not None: raise last_decode_error if log_number + 1 not in self._log_number_offset: byte_len = len(line) self._log_number_offset[log_number + 1] = ( self._log_number_offset[log_number] + byte_len ) if log_number < log_number_from: continue try: logs.append(json.loads(line)) except json.JSONDecodeError as err: last_decode_error = err del self._log_number_offset[log_number + 1] return logs def append_logs(self, logs: List[Dict[str, Any]]) -> None: with get_lock_file(self._lock): what_to_write = "\n".join([json.dumps(log) for log in logs]) + "\n" with open(self._file_path, "ab") as f: f.write(what_to_write.encode("utf-8")) f.flush() os.fsync(f.fileno()) optuna-3.5.0/optuna/storages/_journal/redis.py000066400000000000000000000073421453453102400214620ustar00rootroot00000000000000import json import time from typing import Any from typing import Dict from typing import List from typing import Optional from optuna._experimental import experimental_class from optuna._imports import try_import from optuna.storages._journal.base import BaseJournalLogSnapshot from optuna.storages._journal.base import BaseJournalLogStorage with try_import() as _imports: import redis @experimental_class("3.1.0") class JournalRedisStorage(BaseJournalLogStorage, BaseJournalLogSnapshot): """Redis storage class for Journal log backend. Args: url: URL of the redis storage, password and db are optional. (ie: ``redis://localhost:6379``) use_cluster: Flag whether you use the Redis cluster. If this is :obj:`False`, it is assumed that you use the standalone Redis server and ensured that a write operation is atomic. This provides the consistency of the preserved logs. If this is :obj:`True`, it is assumed that you use the Redis cluster and not ensured that a write operation is atomic. This means the preserved logs can be inconsistent due to network errors, and may cause errors. prefix: Prefix of the preserved key of logs. This is useful when multiple users work on one Redis server. """ def __init__(self, url: str, use_cluster: bool = False, prefix: str = "") -> None: _imports.check() self._url = url self._redis = redis.Redis.from_url(url) self._use_cluster = use_cluster self._prefix = prefix def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["_redis"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) self._redis = redis.Redis.from_url(self._url) def read_logs(self, log_number_from: int) -> List[Dict[str, Any]]: max_log_number_bytes = self._redis.get(f"{self._prefix}:log_number") if max_log_number_bytes is None: return [] max_log_number = int(max_log_number_bytes) logs = [] for log_number in range(log_number_from, max_log_number + 1): sleep_secs = 0.1 while True: log = self._redis.get(self._key_log_id(log_number)) if log is not None: break time.sleep(sleep_secs) sleep_secs = min(sleep_secs * 2, 10) try: logs.append(json.loads(log)) except json.JSONDecodeError as err: if log_number != max_log_number: raise err return logs def append_logs(self, logs: List[Dict[str, Any]]) -> None: self._redis.setnx(f"{self._prefix}:log_number", -1) for log in logs: if not self._use_cluster: self._redis.eval( # type: ignore "local i = redis.call('incr', string.format('%s:log_number', ARGV[1])) " "redis.call('set', string.format('%s:log:%d', ARGV[1], i), ARGV[2])", 0, self._prefix, json.dumps(log), ) else: log_number = self._redis.incr(f"{self._prefix}:log_number", 1) self._redis.set(self._key_log_id(log_number), json.dumps(log)) def save_snapshot(self, snapshot: bytes) -> None: self._redis.set(f"{self._prefix}:snapshot", snapshot) def load_snapshot(self) -> Optional[bytes]: snapshot_bytes = self._redis.get(f"{self._prefix}:snapshot") return snapshot_bytes def _key_log_id(self, log_number: int) -> str: return f"{self._prefix}:log:{log_number}" optuna-3.5.0/optuna/storages/_journal/storage.py000066400000000000000000000624361453453102400220250ustar00rootroot00000000000000import copy import datetime import enum import pickle import threading from typing import Any from typing import Container from typing import Dict from typing import List from typing import Optional from typing import Sequence import uuid import optuna from optuna._experimental import experimental_class from optuna._typing import JSONSerializable from optuna.distributions import BaseDistribution from optuna.distributions import check_distribution_compatibility from optuna.distributions import distribution_to_json from optuna.distributions import json_to_distribution from optuna.exceptions import DuplicatedStudyError from optuna.storages import BaseStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.storages._journal.base import BaseJournalLogSnapshot from optuna.storages._journal.base import BaseJournalLogStorage from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = optuna.logging.get_logger(__name__) NOT_FOUND_MSG = "Record does not exist." # A heuristic interval number to dump snapshots SNAPSHOT_INTERVAL = 100 class JournalOperation(enum.IntEnum): CREATE_STUDY = 0 DELETE_STUDY = 1 SET_STUDY_USER_ATTR = 2 SET_STUDY_SYSTEM_ATTR = 3 CREATE_TRIAL = 4 SET_TRIAL_PARAM = 5 SET_TRIAL_STATE_VALUES = 6 SET_TRIAL_INTERMEDIATE_VALUE = 7 SET_TRIAL_USER_ATTR = 8 SET_TRIAL_SYSTEM_ATTR = 9 @experimental_class("3.1.0") class JournalStorage(BaseStorage): """Storage class for Journal storage backend. Note that library users can instantiate this class, but the attributes provided by this class are not supposed to be directly accessed by them. Journal storage writes a record of every operation to the database as it is executed and at the same time, keeps a latest snapshot of the database in-memory. If the database crashes for any reason, the storage can re-establish the contents in memory by replaying the operations stored from the beginning. Journal storage has several benefits over the conventional value logging storages. 1. The number of IOs can be reduced because of larger granularity of logs. 2. Journal storage has simpler backend API than value logging storage. 3. Journal storage keeps a snapshot in-memory so no need to add more cache. Example: .. code:: import optuna def objective(trial): ... storage = optuna.storages.JournalStorage( optuna.storages.JournalFileStorage("./journal.log"), ) study = optuna.create_study(storage=storage) study.optimize(objective) In a Windows environment, an error message "A required privilege is not held by the client" may appear. In this case, you can solve the problem with creating storage by specifying :class:`~optuna.storages.JournalFileOpenLock` as follows. .. code:: file_path = "./journal.log" lock_obj = optuna.storages.JournalFileOpenLock(file_path) storage = optuna.storages.JournalStorage( optuna.storages.JournalFileStorage(file_path, lock_obj=lock_obj), ) """ def __init__(self, log_storage: BaseJournalLogStorage) -> None: self._worker_id_prefix = str(uuid.uuid4()) + "-" self._backend = log_storage self._thread_lock = threading.Lock() self._replay_result = JournalStorageReplayResult(self._worker_id_prefix) with self._thread_lock: if isinstance(self._backend, BaseJournalLogSnapshot): snapshot = self._backend.load_snapshot() if snapshot is not None: self.restore_replay_result(snapshot) self._sync_with_backend() def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["_worker_id_prefix"] del state["_replay_result"] del state["_thread_lock"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) self._worker_id_prefix = str(uuid.uuid4()) + "-" self._replay_result = JournalStorageReplayResult(self._worker_id_prefix) self._thread_lock = threading.Lock() def restore_replay_result(self, snapshot: bytes) -> None: try: r: Optional[JournalStorageReplayResult] = pickle.loads(snapshot) except (pickle.UnpicklingError, KeyError): _logger.warning("Failed to restore `JournalStorageReplayResult`.") return if r is None: return if not isinstance(r, JournalStorageReplayResult): _logger.warning("The restored object is not `JournalStorageReplayResult`.") return r._worker_id_prefix = self._worker_id_prefix r._worker_id_to_owned_trial_id = {} r._last_created_trial_id_by_this_process = -1 self._replay_result = r def _write_log(self, op_code: int, extra_fields: Dict[str, Any]) -> None: worker_id = self._replay_result.worker_id self._backend.append_logs([{"op_code": op_code, "worker_id": worker_id, **extra_fields}]) def _sync_with_backend(self) -> None: logs = self._backend.read_logs(self._replay_result.log_number_read) self._replay_result.apply_logs(logs) def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: study_name = study_name or DEFAULT_STUDY_NAME_PREFIX + str(uuid.uuid4()) with self._thread_lock: self._write_log( JournalOperation.CREATE_STUDY, {"study_name": study_name, "directions": directions} ) self._sync_with_backend() for frozen_study in self._replay_result.get_all_studies(): if frozen_study.study_name != study_name: continue _logger.info("A new study created in Journal with name: {}".format(study_name)) study_id = frozen_study._study_id # Dump snapshot here. if ( isinstance(self._backend, BaseJournalLogSnapshot) and study_id != 0 and study_id % SNAPSHOT_INTERVAL == 0 ): self._backend.save_snapshot(pickle.dumps(self._replay_result)) return study_id assert False, "Should not reach." def delete_study(self, study_id: int) -> None: with self._thread_lock: self._write_log(JournalOperation.DELETE_STUDY, {"study_id": study_id}) self._sync_with_backend() def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: log: Dict[str, Any] = {"study_id": study_id, "user_attr": {key: value}} with self._thread_lock: self._write_log(JournalOperation.SET_STUDY_USER_ATTR, log) self._sync_with_backend() def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: log: Dict[str, Any] = {"study_id": study_id, "system_attr": {key: value}} with self._thread_lock: self._write_log(JournalOperation.SET_STUDY_SYSTEM_ATTR, log) self._sync_with_backend() def get_study_id_from_name(self, study_name: str) -> int: with self._thread_lock: self._sync_with_backend() for study in self._replay_result.get_all_studies(): if study.study_name == study_name: return study._study_id raise KeyError(NOT_FOUND_MSG) def get_study_name_from_id(self, study_id: int) -> str: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).study_name def get_study_directions(self, study_id: int) -> List[StudyDirection]: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).directions def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).user_attrs def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_study(study_id).system_attrs def get_all_studies(self) -> List[FrozenStudy]: with self._thread_lock: self._sync_with_backend() return copy.deepcopy(self._replay_result.get_all_studies()) # Basic trial manipulation def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: log: Dict[str, Any] = { "study_id": study_id, "datetime_start": datetime.datetime.now().isoformat(timespec="microseconds"), } if template_trial: log["state"] = template_trial.state if template_trial.values is not None and len(template_trial.values) > 1: log["value"] = None log["values"] = template_trial.values else: log["value"] = template_trial.value log["values"] = None if template_trial.datetime_start: log["datetime_start"] = template_trial.datetime_start.isoformat( timespec="microseconds" ) else: log["datetime_start"] = None if template_trial.datetime_complete: log["datetime_complete"] = template_trial.datetime_complete.isoformat( timespec="microseconds" ) log["distributions"] = { k: distribution_to_json(dist) for k, dist in template_trial.distributions.items() } log["params"] = { k: template_trial.distributions[k].to_internal_repr(param) for k, param in template_trial.params.items() } log["user_attrs"] = template_trial.user_attrs log["system_attrs"] = template_trial.system_attrs log["intermediate_values"] = template_trial.intermediate_values with self._thread_lock: self._write_log(JournalOperation.CREATE_TRIAL, log) self._sync_with_backend() trial_id = self._replay_result._last_created_trial_id_by_this_process # Dump snapshot here. if ( isinstance(self._backend, BaseJournalLogSnapshot) and trial_id != 0 and trial_id % SNAPSHOT_INTERVAL == 0 ): self._backend.save_snapshot(pickle.dumps(self._replay_result)) return trial_id def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: BaseDistribution, ) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "param_name": param_name, "param_value_internal": param_value_internal, "distribution": distribution_to_json(distribution), } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_PARAM, log) self._sync_with_backend() def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: with self._thread_lock: self._sync_with_backend() if len(self._replay_result._study_id_to_trial_ids[study_id]) <= trial_number: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) return self._replay_result._study_id_to_trial_ids[study_id][trial_number] def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: log: Dict[str, Any] = { "trial_id": trial_id, "state": state, "values": values, } if state == TrialState.RUNNING: log["datetime_start"] = datetime.datetime.now().isoformat(timespec="microseconds") elif state.is_finished(): log["datetime_complete"] = datetime.datetime.now().isoformat(timespec="microseconds") with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_STATE_VALUES, log) self._sync_with_backend() if state == TrialState.RUNNING and trial_id != self._replay_result.owned_trial_id: return False else: return True def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "step": step, "intermediate_value": intermediate_value, } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_INTERMEDIATE_VALUE, log) self._sync_with_backend() def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "user_attr": {key: value}, } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_USER_ATTR, log) self._sync_with_backend() def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: log: Dict[str, Any] = { "trial_id": trial_id, "system_attr": {key: value}, } with self._thread_lock: self._write_log(JournalOperation.SET_TRIAL_SYSTEM_ATTR, log) self._sync_with_backend() def get_trial(self, trial_id: int) -> FrozenTrial: with self._thread_lock: self._sync_with_backend() return self._replay_result.get_trial(trial_id) def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: with self._thread_lock: self._sync_with_backend() frozen_trials = self._replay_result.get_all_trials(study_id, states) if deepcopy: return copy.deepcopy(frozen_trials) return frozen_trials class JournalStorageReplayResult: def __init__(self, worker_id_prefix: str) -> None: self.log_number_read = 0 self._worker_id_prefix = worker_id_prefix self._studies: Dict[int, FrozenStudy] = {} self._trials: Dict[int, FrozenTrial] = {} self._study_id_to_trial_ids: Dict[int, List[int]] = {} self._trial_id_to_study_id: Dict[int, int] = {} self._next_study_id: int = 0 self._worker_id_to_owned_trial_id: Dict[str, int] = {} def apply_logs(self, logs: List[Dict[str, Any]]) -> None: for log in logs: self.log_number_read += 1 op = log["op_code"] if op == JournalOperation.CREATE_STUDY: self._apply_create_study(log) elif op == JournalOperation.DELETE_STUDY: self._apply_delete_study(log) elif op == JournalOperation.SET_STUDY_USER_ATTR: self._apply_set_study_user_attr(log) elif op == JournalOperation.SET_STUDY_SYSTEM_ATTR: self._apply_set_study_system_attr(log) elif op == JournalOperation.CREATE_TRIAL: self._apply_create_trial(log) elif op == JournalOperation.SET_TRIAL_PARAM: self._apply_set_trial_param(log) elif op == JournalOperation.SET_TRIAL_STATE_VALUES: self._apply_set_trial_state_values(log) elif op == JournalOperation.SET_TRIAL_INTERMEDIATE_VALUE: self._apply_set_trial_intermediate_value(log) elif op == JournalOperation.SET_TRIAL_USER_ATTR: self._apply_set_trial_user_attr(log) elif op == JournalOperation.SET_TRIAL_SYSTEM_ATTR: self._apply_set_trial_system_attr(log) else: assert False, "Should not reach." def get_study(self, study_id: int) -> FrozenStudy: if study_id not in self._studies: raise KeyError(NOT_FOUND_MSG) return self._studies[study_id] def get_all_studies(self) -> List[FrozenStudy]: return list(self._studies.values()) def get_trial(self, trial_id: int) -> FrozenTrial: if trial_id not in self._trials: raise KeyError(NOT_FOUND_MSG) return self._trials[trial_id] def get_all_trials( self, study_id: int, states: Optional[Container[TrialState]] ) -> List[FrozenTrial]: if study_id not in self._studies: raise KeyError(NOT_FOUND_MSG) frozen_trials: List[FrozenTrial] = [] for trial_id in self._study_id_to_trial_ids[study_id]: trial = self._trials[trial_id] if states is None or trial.state in states: frozen_trials.append(trial) return frozen_trials @property def worker_id(self) -> str: return self._worker_id_prefix + str(threading.get_ident()) @property def owned_trial_id(self) -> Optional[int]: return self._worker_id_to_owned_trial_id.get(self.worker_id) def _is_issued_by_this_worker(self, log: Dict[str, Any]) -> bool: return log["worker_id"] == self.worker_id def _study_exists(self, study_id: int, log: Dict[str, Any]) -> bool: if study_id in self._studies: return True if self._is_issued_by_this_worker(log): raise KeyError(NOT_FOUND_MSG) return False def _apply_create_study(self, log: Dict[str, Any]) -> None: study_name = log["study_name"] directions = [StudyDirection(d) for d in log["directions"]] if study_name in [s.study_name for s in self._studies.values()]: if self._is_issued_by_this_worker(log): raise DuplicatedStudyError( "Another study with name '{}' already exists. " "Please specify a different name, or reuse the existing one " "by setting `load_if_exists` (for Python API) or " "`--skip-if-exists` flag (for CLI).".format(study_name) ) return study_id = self._next_study_id self._next_study_id += 1 self._studies[study_id] = FrozenStudy( study_name=study_name, direction=None, user_attrs={}, system_attrs={}, study_id=study_id, directions=directions, ) self._study_id_to_trial_ids[study_id] = [] def _apply_delete_study(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if self._study_exists(study_id, log): fs = self._studies.pop(study_id) assert fs._study_id == study_id def _apply_set_study_user_attr(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if self._study_exists(study_id, log): assert len(log["user_attr"]) == 1 self._studies[study_id].user_attrs.update(log["user_attr"]) def _apply_set_study_system_attr(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if self._study_exists(study_id, log): assert len(log["system_attr"]) == 1 self._studies[study_id].system_attrs.update(log["system_attr"]) def _apply_create_trial(self, log: Dict[str, Any]) -> None: study_id = log["study_id"] if not self._study_exists(study_id, log): return trial_id = len(self._trials) distributions = {} if "distributions" in log: distributions = {k: json_to_distribution(v) for k, v in log["distributions"].items()} params = {} if "params" in log: params = {k: distributions[k].to_external_repr(p) for k, p in log["params"].items()} if log["datetime_start"] is not None: datetime_start = datetime.datetime.fromisoformat(log["datetime_start"]) else: datetime_start = None if "datetime_complete" in log: datetime_complete = datetime.datetime.fromisoformat(log["datetime_complete"]) else: datetime_complete = None self._trials[trial_id] = FrozenTrial( trial_id=trial_id, number=len(self._study_id_to_trial_ids[study_id]), state=TrialState(log.get("state", TrialState.RUNNING.value)), params=params, distributions=distributions, user_attrs=log.get("user_attrs", {}), system_attrs=log.get("system_attrs", {}), value=log.get("value", None), intermediate_values={int(k): v for k, v in log.get("intermediate_values", {}).items()}, datetime_start=datetime_start, datetime_complete=datetime_complete, values=log.get("values", None), ) self._study_id_to_trial_ids[study_id].append(trial_id) self._trial_id_to_study_id[trial_id] = study_id if self._is_issued_by_this_worker(log): self._last_created_trial_id_by_this_process = trial_id if self._trials[trial_id].state == TrialState.RUNNING: self._worker_id_to_owned_trial_id[self.worker_id] = trial_id def _apply_set_trial_param(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if not self._trial_exists_and_updatable(trial_id, log): return param_name = log["param_name"] param_value_internal = log["param_value_internal"] distribution = json_to_distribution(log["distribution"]) study_id = self._trial_id_to_study_id[trial_id] for prev_trial_id in self._study_id_to_trial_ids[study_id]: prev_trial = self._trials[prev_trial_id] if param_name in prev_trial.params.keys(): try: check_distribution_compatibility( prev_trial.distributions[param_name], distribution ) except Exception: if self._is_issued_by_this_worker(log): raise return break trial = copy.copy(self._trials[trial_id]) trial.params = { **copy.copy(trial.params), param_name: distribution.to_external_repr(param_value_internal), } trial.distributions = {**copy.copy(trial.distributions), param_name: distribution} self._trials[trial_id] = trial def _apply_set_trial_state_values(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if not self._trial_exists_and_updatable(trial_id, log): return state = TrialState(log["state"]) if state == self._trials[trial_id].state and state == TrialState.RUNNING: return trial = copy.copy(self._trials[trial_id]) if state == TrialState.RUNNING: trial.datetime_start = datetime.datetime.fromisoformat(log["datetime_start"]) if self._is_issued_by_this_worker(log): self._worker_id_to_owned_trial_id[self.worker_id] = trial_id if state.is_finished(): trial.datetime_complete = datetime.datetime.fromisoformat(log["datetime_complete"]) trial.state = state if log["values"] is not None: trial.values = log["values"] self._trials[trial_id] = trial def _apply_set_trial_intermediate_value(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if self._trial_exists_and_updatable(trial_id, log): trial = copy.copy(self._trials[trial_id]) trial.intermediate_values = { **copy.copy(trial.intermediate_values), log["step"]: log["intermediate_value"], } self._trials[trial_id] = trial def _apply_set_trial_user_attr(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if self._trial_exists_and_updatable(trial_id, log): assert len(log["user_attr"]) == 1 trial = copy.copy(self._trials[trial_id]) trial.user_attrs = {**copy.copy(trial.user_attrs), **log["user_attr"]} self._trials[trial_id] = trial def _apply_set_trial_system_attr(self, log: Dict[str, Any]) -> None: trial_id = log["trial_id"] if self._trial_exists_and_updatable(trial_id, log): assert len(log["system_attr"]) == 1 trial = copy.copy(self._trials[trial_id]) trial.system_attrs = { **copy.copy(trial.system_attrs), **log["system_attr"], } self._trials[trial_id] = trial def _trial_exists_and_updatable(self, trial_id: int, log: Dict[str, Any]) -> bool: if trial_id not in self._trials: if self._is_issued_by_this_worker(log): raise KeyError(NOT_FOUND_MSG) return False elif self._trials[trial_id].state.is_finished(): if self._is_issued_by_this_worker(log): raise RuntimeError( "Trial#{} has already finished and can not be updated.".format( self._trials[trial_id].number ) ) return False else: return True optuna-3.5.0/optuna/storages/_rdb/000077500000000000000000000000001453453102400170715ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/_rdb/__init__.py000066400000000000000000000000001453453102400211700ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/_rdb/alembic.ini000066400000000000000000000032411453453102400211660ustar00rootroot00000000000000# A generic, single database configuration. [alembic] # path to migration scripts script_location = alembic # template used to generate migration files # file_template = %%(rev)s_%%(slug)s # timezone to use when rendering the date # within the migration file as well as the filename. # string value is passed to dateutil.tz.gettz() # leave blank for localtime # timezone = # max length of characters to apply to the # "slug" field #truncate_slug_length = 40 # set to 'true' to run the environment during # the 'revision' command, regardless of autogenerate # revision_environment = false # set to 'true' to allow .pyc and .pyo files without # a source .py file to be detected as revisions in the # versions/ directory # sourceless = false # version location specification; this defaults # to alembic/versions. When using multiple version # directories, initial revisions must be specified with --version-path # version_locations = %(here)s/bar %(here)s/bat alembic/versions # the output encoding used when revision files # are written from script.py.mako # output_encoding = utf-8 # NOTE: This URL is only used when generating migration scripts. sqlalchemy.url = sqlite:///alembic.db # Logging configuration [loggers] keys = root,sqlalchemy,alembic [handlers] keys = console [formatters] keys = generic [logger_root] level = WARN handlers = console qualname = [logger_sqlalchemy] level = WARN handlers = qualname = sqlalchemy.engine [logger_alembic] level = INFO handlers = qualname = alembic [handler_console] class = StreamHandler args = (sys.stderr,) level = NOTSET formatter = generic [formatter_generic] format = %(levelname)-5.5s [%(name)s] %(message)s datefmt = %H:%M:%S optuna-3.5.0/optuna/storages/_rdb/alembic/000077500000000000000000000000001453453102400204655ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/_rdb/alembic/env.py000066400000000000000000000041611453453102400216310ustar00rootroot00000000000000import logging from logging.config import fileConfig from alembic import context from sqlalchemy import engine_from_config from sqlalchemy import pool import optuna.storages._rdb.models # this is the Alembic Config object, which provides # access to the values within the .ini file in use. config = context.config # Interpret the config file for Python logging. # This line sets up loggers basically. if len(logging.getLogger().handlers) == 0: fileConfig(config.config_file_name) # add your model's MetaData object here # for 'autogenerate' support # from myapp import mymodel # target_metadata = mymodel.Base.metadata target_metadata = optuna.storages._rdb.models.BaseModel.metadata # other values from the config, defined by the needs of env.py, # can be acquired: # my_important_option = config.get_main_option("my_important_option") # ... etc. def run_migrations_offline(): """Run migrations in 'offline' mode. This configures the context with just a URL and not an Engine, though an Engine is acceptable here as well. By skipping the Engine creation we don't even need a DBAPI to be available. Calls to context.execute() here emit the given string to the script output. """ url = config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, literal_binds=True, render_as_batch=True ) with context.begin_transaction(): context.run_migrations() def run_migrations_online(): """Run migrations in 'online' mode. In this scenario we need to create an Engine and associate a connection with the context. """ connectable = engine_from_config( config.get_section(config.config_ini_section), prefix="sqlalchemy.", poolclass=pool.NullPool, ) with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, render_as_batch=True ) with context.begin_transaction(): context.run_migrations() if context.is_offline_mode(): run_migrations_offline() else: run_migrations_online() optuna-3.5.0/optuna/storages/_rdb/alembic/script.py.mako000066400000000000000000000007561453453102400233010ustar00rootroot00000000000000"""${message} Revision ID: ${up_revision} Revises: ${down_revision | comma,n} Create Date: ${create_date} """ from alembic import op import sqlalchemy as sa ${imports if imports else ""} # revision identifiers, used by Alembic. revision = ${repr(up_revision)} down_revision = ${repr(down_revision)} branch_labels = ${repr(branch_labels)} depends_on = ${repr(depends_on)} def upgrade(): ${upgrades if upgrades else "pass"} def downgrade(): ${downgrades if downgrades else "pass"} optuna-3.5.0/optuna/storages/_rdb/alembic/versions/000077500000000000000000000000001453453102400223355ustar00rootroot00000000000000optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v0.9.0.a.py000066400000000000000000000125171453453102400237660ustar00rootroot00000000000000"""empty message Revision ID: v0.9.0.a Revises: Create Date: 2019-03-12 12:30:31.178819 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "v0.9.0.a" down_revision = None branch_labels = None depends_on = None def upgrade(): # ### commands auto generated by Alembic - please adjust! ### op.create_table( "studies", sa.Column("study_id", sa.Integer(), nullable=False), sa.Column("study_name", sa.String(length=512), nullable=False), sa.Column( "direction", sa.Enum("NOT_SET", "MINIMIZE", "MAXIMIZE", name="studydirection"), nullable=False, ), sa.PrimaryKeyConstraint("study_id"), ) op.create_index(op.f("ix_studies_study_name"), "studies", ["study_name"], unique=True) op.create_table( "version_info", sa.Column("version_info_id", sa.Integer(), autoincrement=False, nullable=False), sa.Column("schema_version", sa.Integer(), nullable=True), sa.Column("library_version", sa.String(length=256), nullable=True), sa.CheckConstraint("version_info_id=1"), sa.PrimaryKeyConstraint("version_info_id"), ) op.create_table( "study_system_attributes", sa.Column("study_system_attribute_id", sa.Integer(), nullable=False), sa.Column("study_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["study_id"], ["studies.study_id"]), sa.PrimaryKeyConstraint("study_system_attribute_id"), sa.UniqueConstraint("study_id", "key"), ) op.create_table( "study_user_attributes", sa.Column("study_user_attribute_id", sa.Integer(), nullable=False), sa.Column("study_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["study_id"], ["studies.study_id"]), sa.PrimaryKeyConstraint("study_user_attribute_id"), sa.UniqueConstraint("study_id", "key"), ) op.create_table( "trials", sa.Column("trial_id", sa.Integer(), nullable=False), sa.Column("study_id", sa.Integer(), nullable=True), sa.Column( "state", sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", name="trialstate"), nullable=False, ), sa.Column("value", sa.Float(), nullable=True), sa.Column("datetime_start", sa.DateTime(), nullable=True), sa.Column("datetime_complete", sa.DateTime(), nullable=True), sa.ForeignKeyConstraint(["study_id"], ["studies.study_id"]), sa.PrimaryKeyConstraint("trial_id"), ) op.create_table( "trial_params", sa.Column("param_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("param_name", sa.String(length=512), nullable=True), sa.Column("param_value", sa.Float(), nullable=True), sa.Column("distribution_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("param_id"), sa.UniqueConstraint("trial_id", "param_name"), ) op.create_table( "trial_system_attributes", sa.Column("trial_system_attribute_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("trial_system_attribute_id"), sa.UniqueConstraint("trial_id", "key"), ) op.create_table( "trial_user_attributes", sa.Column("trial_user_attribute_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("key", sa.String(length=512), nullable=True), sa.Column("value_json", sa.String(length=2048), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("trial_user_attribute_id"), sa.UniqueConstraint("trial_id", "key"), ) op.create_table( "trial_values", sa.Column("trial_value_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=True), sa.Column("step", sa.Integer(), nullable=True), sa.Column("value", sa.Float(), nullable=True), sa.ForeignKeyConstraint(["trial_id"], ["trials.trial_id"]), sa.PrimaryKeyConstraint("trial_value_id"), sa.UniqueConstraint("trial_id", "step"), ) # ### end Alembic commands ### def downgrade(): # ### commands auto generated by Alembic - please adjust! ### op.drop_table("trial_values") op.drop_table("trial_user_attributes") op.drop_table("trial_system_attributes") op.drop_table("trial_params") op.drop_table("trials") op.drop_table("study_user_attributes") op.drop_table("study_system_attributes") op.drop_table("version_info") op.drop_index(op.f("ix_studies_study_name"), table_name="studies") op.drop_table("studies") # ### end Alembic commands ### optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v1.2.0.a.py000066400000000000000000000017031453453102400237530ustar00rootroot00000000000000"""empty message Revision ID: v1.2.0.a Revises: v0.9.0.a Create Date: 2020-02-05 15:17:41.458947 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "v1.2.0.a" down_revision = "v0.9.0.a" branch_labels = None depends_on = None def upgrade(): with op.batch_alter_table("trials") as batch_op: batch_op.alter_column( "state", type_=sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", "WAITING", name="trialstate"), existing_type=sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", name="trialstate"), ) def downgrade(): with op.batch_alter_table("trials") as batch_op: batch_op.alter_column( "state", type_=sa.Enum("RUNNING", "COMPLETE", "PRUNED", "FAIL", name="trialstate"), existing_type=sa.Enum( "RUNNING", "COMPLETE", "PRUNED", "FAIL", "WAITING", name="trialstate" ), ) optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v1.3.0.a.py000066400000000000000000000054141453453102400237570ustar00rootroot00000000000000"""empty message Revision ID: v1.3.0.a Revises: v1.2.0.a Create Date: 2020-02-14 16:23:04.800808 """ import json from alembic import op import sqlalchemy as sa from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v1.3.0.a" down_revision = "v1.2.0.a" branch_labels = None depends_on = None # Model definition MAX_INDEXED_STRING_LENGTH = 512 MAX_STRING_LENGTH = 2048 BaseModel = declarative_base() class TrialModel(BaseModel): __tablename__ = "trials" trial_id = sa.Column(sa.Integer, primary_key=True) number = sa.Column(sa.Integer) class TrialSystemAttributeModel(BaseModel): __tablename__ = "trial_system_attributes" trial_system_attribute_id = sa.Column(sa.Integer, primary_key=True) trial_id = sa.Column(sa.Integer, sa.ForeignKey("trials.trial_id")) key = sa.Column(sa.String(MAX_INDEXED_STRING_LENGTH)) value_json = sa.Column(sa.String(MAX_STRING_LENGTH)) def upgrade(): bind = op.get_bind() session = orm.Session(bind=bind) with op.batch_alter_table("trials") as batch_op: batch_op.add_column(sa.Column("number", sa.Integer(), nullable=True, default=None)) try: number_records = ( session.query(TrialSystemAttributeModel) .filter(TrialSystemAttributeModel.key == "_number") .all() ) mapping = [ {"trial_id": r.trial_id, "number": json.loads(r.value_json)} for r in number_records ] session.bulk_update_mappings(TrialModel, mapping) stmt = ( sa.delete(TrialSystemAttributeModel) .where(TrialSystemAttributeModel.key == "_number") .execution_options(synchronize_session=False) ) session.execute(stmt) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade(): bind = op.get_bind() session = orm.Session(bind=bind) try: number_attrs = [] trials = session.query(TrialModel).all() for trial in trials: number_attrs.append( TrialSystemAttributeModel( trial_id=trial.trial_id, key="_number", value_json=json.dumps(trial.number) ) ) session.bulk_save_objects(number_attrs) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("trials") as batch_op: batch_op.drop_column("number") optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v2.4.0.a.py000066400000000000000000000144061453453102400237620ustar00rootroot00000000000000"""empty message Revision ID: v2.4.0.a Revises: v1.3.0.a Create Date: 2020-11-17 02:16:16.536171 """ from alembic import op import sqlalchemy as sa from typing import Any from sqlalchemy import Column from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import Integer from sqlalchemy import UniqueConstraint from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm from optuna.study import StudyDirection try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v2.4.0.a" down_revision = "v1.3.0.a" branch_labels = None depends_on = None # Model definition BaseModel = declarative_base() class StudyModel(BaseModel): __tablename__ = "studies" study_id = Column(Integer, primary_key=True) direction = sa.Column(sa.Enum(StudyDirection)) class StudyDirectionModel(BaseModel): __tablename__ = "study_directions" __table_args__: Any = (UniqueConstraint("study_id", "objective"),) study_direction_id = Column(Integer, primary_key=True) direction = Column(Enum(StudyDirection), nullable=False) study_id = Column(Integer, ForeignKey("studies.study_id"), nullable=False) objective = Column(Integer, nullable=False) class TrialModel(BaseModel): __tablename__ = "trials" trial_id = Column(Integer, primary_key=True) number = Column(Integer) study_id = Column(Integer, ForeignKey("studies.study_id")) value = sa.Column(sa.Float) class TrialValueModel(BaseModel): __tablename__ = "trial_values" __table_args__: Any = (UniqueConstraint("trial_id", "objective"),) trial_value_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id"), nullable=False) objective = Column(Integer, nullable=False) value = Column(Float, nullable=False) step = sa.Column(sa.Integer) class TrialIntermediateValueModel(BaseModel): __tablename__ = "trial_intermediate_values" __table_args__: Any = (UniqueConstraint("trial_id", "step"),) trial_intermediate_value_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id"), nullable=False) step = Column(Integer, nullable=False) intermediate_value = Column(Float, nullable=False) def upgrade(): bind = op.get_bind() inspector = sa.inspect(bind) tables = inspector.get_table_names() if "study_directions" not in tables: op.create_table( "study_directions", sa.Column("study_direction_id", sa.Integer(), nullable=False), sa.Column( "direction", sa.Enum("NOT_SET", "MINIMIZE", "MAXIMIZE", name="studydirection"), nullable=False, ), sa.Column("study_id", sa.Integer(), nullable=False), sa.Column("objective", sa.Integer(), nullable=False), sa.ForeignKeyConstraint( ["study_id"], ["studies.study_id"], ), sa.PrimaryKeyConstraint("study_direction_id"), sa.UniqueConstraint("study_id", "objective"), ) if "trial_intermediate_values" not in tables: op.create_table( "trial_intermediate_values", sa.Column("trial_intermediate_value_id", sa.Integer(), nullable=False), sa.Column("trial_id", sa.Integer(), nullable=False), sa.Column("step", sa.Integer(), nullable=False), sa.Column("intermediate_value", sa.Float(), nullable=False), sa.ForeignKeyConstraint( ["trial_id"], ["trials.trial_id"], ), sa.PrimaryKeyConstraint("trial_intermediate_value_id"), sa.UniqueConstraint("trial_id", "step"), ) session = orm.Session(bind=bind) try: studies_records = session.query(StudyModel).all() objects = [ StudyDirectionModel(study_id=r.study_id, direction=r.direction, objective=0) for r in studies_records ] session.bulk_save_objects(objects) intermediate_values_records = session.query( TrialValueModel.trial_id, TrialValueModel.value, TrialValueModel.step ).all() objects = [ TrialIntermediateValueModel( trial_id=r.trial_id, intermediate_value=r.value, step=r.step ) for r in intermediate_values_records ] session.bulk_save_objects(objects) session.query(TrialValueModel).delete() session.commit() with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.add_column(sa.Column("objective", sa.Integer(), nullable=False)) # The name of this constraint is manually determined. # In the future, the naming convention may be determined based on # https://alembic.sqlalchemy.org/en/latest/naming.html batch_op.create_unique_constraint( "uq_trial_values_trial_id_objective", ["trial_id", "objective"] ) trials_records = session.query(TrialModel).all() objects = [ TrialValueModel(trial_id=r.trial_id, value=r.value, objective=0) for r in trials_records ] session.bulk_save_objects(objects) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("studies", schema=None) as batch_op: batch_op.drop_column("direction") with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.drop_column("step") with op.batch_alter_table("trials", schema=None) as batch_op: batch_op.drop_column("value") for c in inspector.get_unique_constraints("trial_values"): # MySQL changes the uniq constraint of (trial_id, step) to that of trial_id. if c["column_names"] == ["trial_id"]: with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.drop_constraint(c["name"], type_="unique") break # TODO(imamura): Implement downgrade def downgrade(): pass optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v2.6.0.a_.py000066400000000000000000000032721453453102400241220ustar00rootroot00000000000000"""empty message Revision ID: v2.6.0.a Revises: v2.4.0.a Create Date: 2021-03-01 11:30:32.214196 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "v2.6.0.a" down_revision = "v2.4.0.a" branch_labels = None depends_on = None MAX_STRING_LENGTH = 2048 def upgrade(): with op.batch_alter_table("study_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("study_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("trial_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("trial_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.TEXT) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column("distribution_json", type_=sa.TEXT) def downgrade(): with op.batch_alter_table("study_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("study_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("trial_user_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("trial_system_attributes") as batch_op: batch_op.alter_column("value_json", type_=sa.String(MAX_STRING_LENGTH)) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column("distribution_json", type_=sa.String(MAX_STRING_LENGTH)) optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v3.0.0.a.py000066400000000000000000000140761453453102400237620ustar00rootroot00000000000000"""unify existing distributions to {int,float} distribution Revision ID: v3.0.0.a Revises: v2.6.0.a Create Date: 2021-11-21 23:48:42.424430 """ from typing import Any from typing import List import sqlalchemy as sa from alembic import op from sqlalchemy import Column from sqlalchemy import DateTime from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import Integer from sqlalchemy import orm from sqlalchemy import String from sqlalchemy import Text from sqlalchemy import UniqueConstraint from sqlalchemy.exc import SQLAlchemyError from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.distributions import BaseDistribution from optuna.distributions import DiscreteUniformDistribution from optuna.distributions import distribution_to_json from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.distributions import IntLogUniformDistribution from optuna.distributions import IntUniformDistribution from optuna.distributions import json_to_distribution from optuna.distributions import LogUniformDistribution from optuna.distributions import UniformDistribution from optuna.trial import TrialState try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.a" down_revision = "v2.6.0.a" branch_labels = None depends_on = None MAX_INDEXED_STRING_LENGTH = 512 BATCH_SIZE = 5000 BaseModel = declarative_base() class StudyModel(BaseModel): __tablename__ = "studies" study_id = Column(Integer, primary_key=True) study_name = Column(String(MAX_INDEXED_STRING_LENGTH), index=True, unique=True, nullable=False) class TrialModel(BaseModel): __tablename__ = "trials" trial_id = Column(Integer, primary_key=True) number = Column(Integer) study_id = Column(Integer, ForeignKey("studies.study_id")) state = Column(Enum(TrialState), nullable=False) datetime_start = Column(DateTime) datetime_complete = Column(DateTime) class TrialParamModel(BaseModel): __tablename__ = "trial_params" __table_args__: Any = (UniqueConstraint("trial_id", "param_name"),) param_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id")) param_name = Column(String(MAX_INDEXED_STRING_LENGTH)) param_value = Column(Float) distribution_json = Column(Text()) def migrate_new_distribution(distribution_json: str) -> str: distribution = json_to_distribution(distribution_json) new_distribution = _convert_old_distribution_to_new_distribution( distribution, suppress_warning=True, ) return distribution_to_json(new_distribution) def restore_old_distribution(distribution_json: str) -> str: distribution = json_to_distribution(distribution_json) old_distribution: BaseDistribution # Float distributions. if isinstance(distribution, FloatDistribution): if distribution.log: old_distribution = LogUniformDistribution( low=distribution.low, high=distribution.high, ) else: if distribution.step is not None: old_distribution = DiscreteUniformDistribution( low=distribution.low, high=distribution.high, q=distribution.step, ) else: old_distribution = UniformDistribution( low=distribution.low, high=distribution.high, ) # Integer distributions. elif isinstance(distribution, IntDistribution): if distribution.log: old_distribution = IntLogUniformDistribution( low=distribution.low, high=distribution.high, step=distribution.step, ) else: old_distribution = IntUniformDistribution( low=distribution.low, high=distribution.high, step=distribution.step, ) # Categorical distribution. else: old_distribution = distribution return distribution_to_json(old_distribution) def persist(session: orm.Session, distributions: List[BaseDistribution]) -> None: if len(distributions) == 0: return session.bulk_save_objects(distributions) session.commit() def upgrade() -> None: bind = op.get_bind() inspector = sa.inspect(bind) tables = inspector.get_table_names() assert "trial_params" in tables session = orm.Session(bind=bind) try: distributions: List[BaseDistribution] = [] for distribution in session.query(TrialParamModel).yield_per(BATCH_SIZE): distribution.distribution_json = migrate_new_distribution( distribution.distribution_json, ) distributions.append(distribution) if len(distributions) == BATCH_SIZE: persist(session, distributions) distributions = [] persist(session, distributions) except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade() -> None: bind = op.get_bind() inspector = sa.inspect(bind) tables = inspector.get_table_names() assert "trial_params" in tables session = orm.Session(bind=bind) try: distributions = [] for distribution in session.query(TrialParamModel).yield_per(BATCH_SIZE): distribution.distribution_json = restore_old_distribution( distribution.distribution_json, ) distributions.append(distribution) if len(distributions) == BATCH_SIZE: persist(session, distributions) distributions = [] persist(session, distributions) except SQLAlchemyError as e: session.rollback() raise e finally: session.close() optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v3.0.0.b.py000066400000000000000000000056131453453102400237600ustar00rootroot00000000000000"""Change floating point precision and make intermediate_value nullable. Revision ID: v3.0.0.b Revises: v3.0.0.a Create Date: 2022-04-27 16:31:42.012666 """ import enum from alembic import op from sqlalchemy import and_ from sqlalchemy import Column from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import Integer from sqlalchemy.orm import Session try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.b" down_revision = "v3.0.0.a" branch_labels = None depends_on = None BaseModel = declarative_base() FLOAT_PRECISION = 53 class TrialState(enum.Enum): RUNNING = 0 COMPLETE = 1 PRUNED = 2 FAIL = 3 WAITING = 4 class TrialModel(BaseModel): __tablename__ = "trials" trial_id = Column(Integer, primary_key=True) number = Column(Integer) state = Column(Enum(TrialState), nullable=False) class TrialValueModel(BaseModel): __tablename__ = "trial_values" trial_value_id = Column(Integer, primary_key=True) trial_id = Column(Integer, ForeignKey("trials.trial_id"), nullable=False) value = Column(Float, nullable=False) def upgrade(): bind = op.get_bind() session = Session(bind=bind) if ( session.query(TrialValueModel) .join(TrialModel, TrialValueModel.trial_id == TrialModel.trial_id) .filter(and_(TrialModel.state == TrialState.COMPLETE, TrialValueModel.value.is_(None))) .count() ) != 0: raise ValueError("Found invalid trial_values records (value=None and state='COMPLETE')") session.query(TrialValueModel).filter(TrialValueModel.value.is_(None)).delete() with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.alter_column( "intermediate_value", type_=Float(precision=FLOAT_PRECISION), nullable=True, ) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column( "param_value", type_=Float(precision=FLOAT_PRECISION), existing_nullable=True, ) with op.batch_alter_table("trial_values") as batch_op: batch_op.alter_column( "value", type_=Float(precision=FLOAT_PRECISION), nullable=False, ) def downgrade(): with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.alter_column("intermediate_value", type_=Float, nullable=False) with op.batch_alter_table("trial_params") as batch_op: batch_op.alter_column("param_value", type_=Float, existing_nullable=True) with op.batch_alter_table("trial_values") as batch_op: batch_op.alter_column("value", type_=Float, existing_nullable=False) optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v3.0.0.c.py000066400000000000000000000144271453453102400237640ustar00rootroot00000000000000"""Add intermediate_value_type column to represent +inf and -inf Revision ID: v3.0.0.c Revises: v3.0.0.b Create Date: 2022-05-16 17:17:28.810792 """ import enum import numpy as np from alembic import op import sqlalchemy as sa from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm from typing import Optional from typing import Tuple try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.c" down_revision = "v3.0.0.b" branch_labels = None depends_on = None BaseModel = declarative_base() RDB_MAX_FLOAT = np.finfo(np.float32).max RDB_MIN_FLOAT = np.finfo(np.float32).min FLOAT_PRECISION = 53 class IntermediateValueModel(BaseModel): class TrialIntermediateValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 NAN = 4 __tablename__ = "trial_intermediate_values" trial_intermediate_value_id = sa.Column(sa.Integer, primary_key=True) intermediate_value = sa.Column(sa.Float(precision=FLOAT_PRECISION), nullable=True) intermediate_value_type = sa.Column(sa.Enum(TrialIntermediateValueType), nullable=False) @classmethod def intermediate_value_to_stored_repr( cls, value: float, ) -> Tuple[Optional[float], TrialIntermediateValueType]: if np.isnan(value): return (None, cls.TrialIntermediateValueType.NAN) elif value == float("inf"): return (None, cls.TrialIntermediateValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialIntermediateValueType.INF_NEG) else: return (value, cls.TrialIntermediateValueType.FINITE) def upgrade(): bind = op.get_bind() inspector = sa.inspect(bind) column_names = [c["name"] for c in inspector.get_columns("trial_intermediate_values")] sa.Enum(IntermediateValueModel.TrialIntermediateValueType).create(bind, checkfirst=True) # MySQL and PostgreSQL supports DEFAULT clause like 'ALTER TABLE # ADD COLUMN ... DEFAULT "FINITE_OR_NAN"', but seemingly Alembic # does not support such a SQL statement. So first add a column with schema-level # default value setting, then remove it by `batch_op.alter_column()`. if "intermediate_value_type" not in column_names: with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.add_column( sa.Column( "intermediate_value_type", sa.Enum( "FINITE", "INF_POS", "INF_NEG", "NAN", name="trialintermediatevaluetype" ), nullable=False, server_default="FINITE", ), ) with op.batch_alter_table("trial_intermediate_values") as batch_op: batch_op.alter_column( "intermediate_value_type", existing_type=sa.Enum( "FINITE", "INF_POS", "INF_NEG", "NAN", name="trialintermediatevaluetype" ), existing_nullable=False, server_default=None, ) session = orm.Session(bind=bind) try: records = ( session.query(IntermediateValueModel) .filter( sa.or_( IntermediateValueModel.intermediate_value > 1e16, IntermediateValueModel.intermediate_value < -1e16, IntermediateValueModel.intermediate_value.is_(None), ) ) .all() ) mapping = [] for r in records: value: float if r.intermediate_value is None or np.isnan(r.intermediate_value): value = float("nan") elif np.isclose(r.intermediate_value, RDB_MAX_FLOAT) or np.isposinf( r.intermediate_value ): value = float("inf") elif np.isclose(r.intermediate_value, RDB_MIN_FLOAT) or np.isneginf( r.intermediate_value ): value = float("-inf") else: value = r.intermediate_value ( stored_value, float_type, ) = IntermediateValueModel.intermediate_value_to_stored_repr(value) mapping.append( { "trial_intermediate_value_id": r.trial_intermediate_value_id, "intermediate_value_type": float_type, "intermediate_value": stored_value, } ) session.bulk_update_mappings(IntermediateValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade(): bind = op.get_bind() session = orm.Session(bind=bind) try: records = session.query(IntermediateValueModel).all() mapping = [] for r in records: if ( r.intermediate_value_type == IntermediateValueModel.TrialIntermediateValueType.FINITE or r.intermediate_value_type == IntermediateValueModel.TrialIntermediateValueType.NAN ): continue _intermediate_value = r.intermediate_value if ( r.intermediate_value_type == IntermediateValueModel.TrialIntermediateValueType.INF_POS ): _intermediate_value = RDB_MAX_FLOAT else: _intermediate_value = RDB_MIN_FLOAT mapping.append( { "trial_intermediate_value_id": r.trial_intermediate_value_id, "intermediate_value": _intermediate_value, } ) session.bulk_update_mappings(IntermediateValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("trial_intermediate_values", schema=None) as batch_op: batch_op.drop_column("intermediate_value_type") sa.Enum(IntermediateValueModel.FloatTypeEnum).drop(bind, checkfirst=True) optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v3.0.0.d.py000066400000000000000000000133371453453102400237640ustar00rootroot00000000000000"""Handle inf/-inf for trial_values table. Revision ID: v3.0.0.d Revises: v3.0.0.c Create Date: 2022-06-02 09:57:22.818798 """ import enum import numpy as np from alembic import op import sqlalchemy as sa from sqlalchemy.exc import SQLAlchemyError from sqlalchemy import orm from typing import Optional from typing import Tuple try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base # revision identifiers, used by Alembic. revision = "v3.0.0.d" down_revision = "v3.0.0.c" branch_labels = None depends_on = None BaseModel = declarative_base() RDB_MAX_FLOAT = np.finfo(np.float32).max RDB_MIN_FLOAT = np.finfo(np.float32).min FLOAT_PRECISION = 53 class TrialValueModel(BaseModel): class TrialValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 __tablename__ = "trial_values" trial_value_id = sa.Column(sa.Integer, primary_key=True) value = sa.Column(sa.Float(precision=FLOAT_PRECISION), nullable=True) value_type = sa.Column(sa.Enum(TrialValueType), nullable=False) @classmethod def value_to_stored_repr( cls, value: float, ) -> Tuple[Optional[float], TrialValueType]: if value == float("inf"): return (None, cls.TrialValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialValueType.INF_NEG) else: return (value, cls.TrialValueType.FINITE) @classmethod def stored_repr_to_value(cls, value: Optional[float], float_type: TrialValueType) -> float: if float_type == cls.TrialValueType.INF_POS: assert value is None return float("inf") elif float_type == cls.TrialValueType.INF_NEG: assert value is None return float("-inf") else: assert float_type == cls.TrialValueType.FINITE assert value is not None return value def upgrade(): bind = op.get_bind() inspector = sa.inspect(bind) column_names = [c["name"] for c in inspector.get_columns("trial_values")] sa.Enum(TrialValueModel.TrialValueType).create(bind, checkfirst=True) # MySQL and PostgreSQL supports DEFAULT clause like 'ALTER TABLE # ADD COLUMN ... DEFAULT "FINITE"', but seemingly Alembic # does not support such a SQL statement. So first add a column with schema-level # default value setting, then remove it by `batch_op.alter_column()`. if "value_type" not in column_names: with op.batch_alter_table("trial_values") as batch_op: batch_op.add_column( sa.Column( "value_type", sa.Enum("FINITE", "INF_POS", "INF_NEG", name="trialvaluetype"), nullable=False, server_default="FINITE", ), ) with op.batch_alter_table("trial_values") as batch_op: batch_op.alter_column( "value_type", existing_type=sa.Enum("FINITE", "INF_POS", "INF_NEG", name="trialvaluetype"), existing_nullable=False, server_default=None, ) batch_op.alter_column( "value", existing_type=sa.Float(precision=FLOAT_PRECISION), nullable=True, ) session = orm.Session(bind=bind) try: records = ( session.query(TrialValueModel) .filter( sa.or_( TrialValueModel.value > 1e16, TrialValueModel.value < -1e16, ) ) .all() ) mapping = [] for r in records: value: float if np.isclose(r.value, RDB_MAX_FLOAT) or np.isposinf(r.value): value = float("inf") elif np.isclose(r.value, RDB_MIN_FLOAT) or np.isneginf(r.value): value = float("-inf") else: value = r.value ( stored_value, float_type, ) = TrialValueModel.value_to_stored_repr(value) mapping.append( { "trial_value_id": r.trial_value_id, "value_type": float_type, "value": stored_value, } ) session.bulk_update_mappings(TrialValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() def downgrade(): bind = op.get_bind() session = orm.Session(bind=bind) try: records = session.query(TrialValueModel).all() mapping = [] for r in records: if r.value_type == TrialValueModel.TrialValueType.FINITE: continue _value = r.value if r.value_type == TrialValueModel.TrialValueType.INF_POS: _value = RDB_MAX_FLOAT else: _value = RDB_MIN_FLOAT mapping.append( { "trial_value_id": r.trial_value_id, "value": _value, } ) session.bulk_update_mappings(TrialValueModel, mapping) session.commit() except SQLAlchemyError as e: session.rollback() raise e finally: session.close() with op.batch_alter_table("trial_values", schema=None) as batch_op: batch_op.drop_column("value_type") batch_op.alter_column( "value", existing_type=sa.Float(precision=FLOAT_PRECISION), nullable=False, ) sa.Enum(TrialValueModel.TrialValueType).drop(bind, checkfirst=True) optuna-3.5.0/optuna/storages/_rdb/alembic/versions/v3.2.0.a_.py000066400000000000000000000013111453453102400241070ustar00rootroot00000000000000"""Add index to study_id column in trials table Revision ID: v3.2.0.a Revises: v3.0.0.d Create Date: 2023-02-25 13:21:00.730272 """ from alembic import op # revision identifiers, used by Alembic. revision = "v3.2.0.a" down_revision = "v3.0.0.d" branch_labels = None depends_on = None def upgrade(): op.create_index(op.f("trials_study_id_key"), "trials", ["study_id"], unique=False) def downgrade(): # The following operation doesn't work on MySQL due to a foreign key constraint. # # mysql> DROP INDEX ix_trials_study_id ON trials; # ERROR: Cannot drop index 'ix_trials_study_id': needed in a foreign key constraint. op.drop_index(op.f("trials_study_id_key"), table_name="trials") optuna-3.5.0/optuna/storages/_rdb/models.py000066400000000000000000000464701453453102400207410ustar00rootroot00000000000000import enum import math from typing import Any from typing import List from typing import Optional from typing import Tuple from sqlalchemy import asc from sqlalchemy import case from sqlalchemy import CheckConstraint from sqlalchemy import DateTime from sqlalchemy import desc from sqlalchemy import Enum from sqlalchemy import Float from sqlalchemy import ForeignKey from sqlalchemy import func from sqlalchemy import Integer from sqlalchemy import orm from sqlalchemy import String from sqlalchemy import Text from sqlalchemy import UniqueConstraint from optuna import distributions from optuna.study._study_direction import StudyDirection from optuna.trial import TrialState try: from sqlalchemy.orm import declarative_base except ImportError: # TODO(c-bata): Remove this after dropping support for SQLAlchemy v1.3 or prior. from sqlalchemy.ext.declarative import declarative_base try: from sqlalchemy.orm import mapped_column _Column = mapped_column except ImportError: # TODO(Shinichi): Remove this after dropping support for SQLAlchemy<2.0. from sqlalchemy import Column as _Column # type: ignore[assignment] # Don't modify this version number anymore. # The schema management functionality has been moved to alembic. SCHEMA_VERSION = 12 MAX_INDEXED_STRING_LENGTH = 512 MAX_VERSION_LENGTH = 256 NOT_FOUND_MSG = "Record does not exist." FLOAT_PRECISION = 53 BaseModel: Any = declarative_base() class StudyModel(BaseModel): __tablename__ = "studies" study_id = _Column(Integer, primary_key=True) study_name = _Column( String(MAX_INDEXED_STRING_LENGTH), index=True, unique=True, nullable=False ) @classmethod def find_or_raise_by_id( cls, study_id: int, session: orm.Session, for_update: bool = False ) -> "StudyModel": query = session.query(cls).filter(cls.study_id == study_id) if for_update: query = query.with_for_update() study = query.one_or_none() if study is None: raise KeyError(NOT_FOUND_MSG) return study @classmethod def find_by_name(cls, study_name: str, session: orm.Session) -> Optional["StudyModel"]: study = session.query(cls).filter(cls.study_name == study_name).one_or_none() return study @classmethod def find_or_raise_by_name(cls, study_name: str, session: orm.Session) -> "StudyModel": study = cls.find_by_name(study_name, session) if study is None: raise KeyError(NOT_FOUND_MSG) return study class StudyDirectionModel(BaseModel): __tablename__ = "study_directions" __table_args__: Any = (UniqueConstraint("study_id", "objective"),) study_direction_id = _Column(Integer, primary_key=True) direction = _Column(Enum(StudyDirection), nullable=False) study_id = _Column(Integer, ForeignKey("studies.study_id"), nullable=False) objective = _Column(Integer, nullable=False) study = orm.relationship( StudyModel, backref=orm.backref("directions", cascade="all, delete-orphan") ) @classmethod def where_study_id(cls, study_id: int, session: orm.Session) -> List["StudyDirectionModel"]: return session.query(cls).filter(cls.study_id == study_id).all() class StudyUserAttributeModel(BaseModel): __tablename__ = "study_user_attributes" __table_args__: Any = (UniqueConstraint("study_id", "key"),) study_user_attribute_id = _Column(Integer, primary_key=True) study_id = _Column(Integer, ForeignKey("studies.study_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) study = orm.relationship( StudyModel, backref=orm.backref("user_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_study_and_key( cls, study: StudyModel, key: str, session: orm.Session ) -> Optional["StudyUserAttributeModel"]: attribute = ( session.query(cls) .filter(cls.study_id == study.study_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_study_id( cls, study_id: int, session: orm.Session ) -> List["StudyUserAttributeModel"]: return session.query(cls).filter(cls.study_id == study_id).all() class StudySystemAttributeModel(BaseModel): __tablename__ = "study_system_attributes" __table_args__: Any = (UniqueConstraint("study_id", "key"),) study_system_attribute_id = _Column(Integer, primary_key=True) study_id = _Column(Integer, ForeignKey("studies.study_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) study = orm.relationship( StudyModel, backref=orm.backref("system_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_study_and_key( cls, study: StudyModel, key: str, session: orm.Session ) -> Optional["StudySystemAttributeModel"]: attribute = ( session.query(cls) .filter(cls.study_id == study.study_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_study_id( cls, study_id: int, session: orm.Session ) -> List["StudySystemAttributeModel"]: return session.query(cls).filter(cls.study_id == study_id).all() class TrialModel(BaseModel): __tablename__ = "trials" trial_id = _Column(Integer, primary_key=True) # No `UniqueConstraint` is put on the `number` columns although it in practice is constrained # to be unique. This is to reduce code complexity as table-level locking would be required # otherwise. See https://github.com/optuna/optuna/pull/939#discussion_r387447632. number = _Column(Integer) study_id = _Column(Integer, ForeignKey("studies.study_id"), index=True) state = _Column(Enum(TrialState), nullable=False) datetime_start = _Column(DateTime) datetime_complete = _Column(DateTime) study = orm.relationship( StudyModel, backref=orm.backref("trials", cascade="all, delete-orphan") ) @classmethod def find_max_value_trial( cls, study_id: int, objective: int, session: orm.Session ) -> "TrialModel": trial = ( session.query(cls) .filter(cls.study_id == study_id) .filter(cls.state == TrialState.COMPLETE) .join(TrialValueModel) .filter(TrialValueModel.objective == objective) .order_by( desc( case( {"INF_NEG": -1, "FINITE": 0, "INF_POS": 1}, value=TrialValueModel.value_type, ) ), desc(TrialValueModel.value), ) .limit(1) .one_or_none() ) if trial is None: raise ValueError(NOT_FOUND_MSG) return trial @classmethod def find_min_value_trial( cls, study_id: int, objective: int, session: orm.Session ) -> "TrialModel": trial = ( session.query(cls) .filter(cls.study_id == study_id) .filter(cls.state == TrialState.COMPLETE) .join(TrialValueModel) .filter(TrialValueModel.objective == objective) .order_by( asc( case( {"INF_NEG": -1, "FINITE": 0, "INF_POS": 1}, value=TrialValueModel.value_type, ) ), asc(TrialValueModel.value), ) .limit(1) .one_or_none() ) if trial is None: raise ValueError(NOT_FOUND_MSG) return trial @classmethod def find_or_raise_by_id( cls, trial_id: int, session: orm.Session, for_update: bool = False ) -> "TrialModel": query = session.query(cls).filter(cls.trial_id == trial_id) # "FOR UPDATE" clause is used for row-level locking. # Please note that SQLite3 doesn't support this clause. if for_update: query = query.with_for_update() trial = query.one_or_none() if trial is None: raise KeyError(NOT_FOUND_MSG) return trial @classmethod def count( cls, session: orm.Session, study: Optional[StudyModel] = None, state: Optional[TrialState] = None, ) -> int: trial_count = session.query(func.count(cls.trial_id)) if study is not None: trial_count = trial_count.filter(cls.study_id == study.study_id) if state is not None: trial_count = trial_count.filter(cls.state == state) return trial_count.scalar() def count_past_trials(self, session: orm.Session) -> int: trial_count = session.query(func.count(TrialModel.trial_id)).filter( TrialModel.study_id == self.study_id, TrialModel.trial_id < self.trial_id ) return trial_count.scalar() class TrialUserAttributeModel(BaseModel): __tablename__ = "trial_user_attributes" __table_args__: Any = (UniqueConstraint("trial_id", "key"),) trial_user_attribute_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) trial = orm.relationship( TrialModel, backref=orm.backref("user_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_trial_and_key( cls, trial: TrialModel, key: str, session: orm.Session ) -> Optional["TrialUserAttributeModel"]: attribute = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> List["TrialUserAttributeModel"]: return session.query(cls).filter(cls.trial_id == trial_id).all() class TrialSystemAttributeModel(BaseModel): __tablename__ = "trial_system_attributes" __table_args__: Any = (UniqueConstraint("trial_id", "key"),) trial_system_attribute_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id")) key = _Column(String(MAX_INDEXED_STRING_LENGTH)) value_json = _Column(Text()) trial = orm.relationship( TrialModel, backref=orm.backref("system_attributes", cascade="all, delete-orphan") ) @classmethod def find_by_trial_and_key( cls, trial: TrialModel, key: str, session: orm.Session ) -> Optional["TrialSystemAttributeModel"]: attribute = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.key == key) .one_or_none() ) return attribute @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> List["TrialSystemAttributeModel"]: return session.query(cls).filter(cls.trial_id == trial_id).all() class TrialParamModel(BaseModel): __tablename__ = "trial_params" __table_args__: Any = (UniqueConstraint("trial_id", "param_name"),) param_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id")) param_name = _Column(String(MAX_INDEXED_STRING_LENGTH)) param_value = _Column(Float(precision=FLOAT_PRECISION)) distribution_json = _Column(Text()) trial = orm.relationship( TrialModel, backref=orm.backref("params", cascade="all, delete-orphan") ) def check_and_add(self, session: orm.Session) -> None: self._check_compatibility_with_previous_trial_param_distributions(session) session.add(self) def _check_compatibility_with_previous_trial_param_distributions( self, session: orm.Session ) -> None: trial = TrialModel.find_or_raise_by_id(self.trial_id, session) previous_record = ( session.query(TrialParamModel) .join(TrialModel) .filter(TrialModel.study_id == trial.study_id) .filter(TrialParamModel.param_name == self.param_name) .first() ) if previous_record is not None: distributions.check_distribution_compatibility( distributions.json_to_distribution(previous_record.distribution_json), distributions.json_to_distribution(self.distribution_json), ) @classmethod def find_by_trial_and_param_name( cls, trial: TrialModel, param_name: str, session: orm.Session ) -> Optional["TrialParamModel"]: param_distribution = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.param_name == param_name) .one_or_none() ) return param_distribution @classmethod def find_or_raise_by_trial_and_param_name( cls, trial: TrialModel, param_name: str, session: orm.Session ) -> "TrialParamModel": param_distribution = cls.find_by_trial_and_param_name(trial, param_name, session) if param_distribution is None: raise KeyError(NOT_FOUND_MSG) return param_distribution @classmethod def where_trial_id(cls, trial_id: int, session: orm.Session) -> List["TrialParamModel"]: trial_params = session.query(cls).filter(cls.trial_id == trial_id).all() return trial_params class TrialValueModel(BaseModel): class TrialValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 __tablename__ = "trial_values" __table_args__: Any = (UniqueConstraint("trial_id", "objective"),) trial_value_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id"), nullable=False) objective = _Column(Integer, nullable=False) value = _Column(Float(precision=FLOAT_PRECISION), nullable=True) value_type = _Column(Enum(TrialValueType), nullable=False) trial = orm.relationship( TrialModel, backref=orm.backref("values", cascade="all, delete-orphan") ) @classmethod def value_to_stored_repr( cls, value: float, ) -> Tuple[Optional[float], TrialValueType]: if value == float("inf"): return (None, cls.TrialValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialValueType.INF_NEG) else: return (value, cls.TrialValueType.FINITE) @classmethod def stored_repr_to_value(cls, value: Optional[float], float_type: TrialValueType) -> float: if float_type == cls.TrialValueType.INF_POS: assert value is None return float("inf") elif float_type == cls.TrialValueType.INF_NEG: assert value is None return float("-inf") else: assert float_type == cls.TrialValueType.FINITE assert value is not None return value @classmethod def find_by_trial_and_objective( cls, trial: TrialModel, objective: int, session: orm.Session ) -> Optional["TrialValueModel"]: trial_value = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.objective == objective) .one_or_none() ) return trial_value @classmethod def where_trial_id(cls, trial_id: int, session: orm.Session) -> List["TrialValueModel"]: trial_values = ( session.query(cls).filter(cls.trial_id == trial_id).order_by(asc(cls.objective)).all() ) return trial_values class TrialIntermediateValueModel(BaseModel): class TrialIntermediateValueType(enum.Enum): FINITE = 1 INF_POS = 2 INF_NEG = 3 NAN = 4 __tablename__ = "trial_intermediate_values" __table_args__: Any = (UniqueConstraint("trial_id", "step"),) trial_intermediate_value_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id"), nullable=False) step = _Column(Integer, nullable=False) intermediate_value = _Column(Float(precision=FLOAT_PRECISION), nullable=True) intermediate_value_type = _Column(Enum(TrialIntermediateValueType), nullable=False) trial = orm.relationship( TrialModel, backref=orm.backref("intermediate_values", cascade="all, delete-orphan") ) @classmethod def intermediate_value_to_stored_repr( cls, value: float, ) -> Tuple[Optional[float], TrialIntermediateValueType]: if math.isnan(value): return (None, cls.TrialIntermediateValueType.NAN) elif value == float("inf"): return (None, cls.TrialIntermediateValueType.INF_POS) elif value == float("-inf"): return (None, cls.TrialIntermediateValueType.INF_NEG) else: return (value, cls.TrialIntermediateValueType.FINITE) @classmethod def stored_repr_to_intermediate_value( cls, value: Optional[float], float_type: TrialIntermediateValueType ) -> float: if float_type == cls.TrialIntermediateValueType.NAN: assert value is None return float("nan") elif float_type == cls.TrialIntermediateValueType.INF_POS: assert value is None return float("inf") elif float_type == cls.TrialIntermediateValueType.INF_NEG: assert value is None return float("-inf") else: assert float_type == cls.TrialIntermediateValueType.FINITE assert value is not None return value @classmethod def find_by_trial_and_step( cls, trial: TrialModel, step: int, session: orm.Session ) -> Optional["TrialIntermediateValueModel"]: trial_intermediate_value = ( session.query(cls) .filter(cls.trial_id == trial.trial_id) .filter(cls.step == step) .one_or_none() ) return trial_intermediate_value @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> List["TrialIntermediateValueModel"]: trial_intermediate_values = session.query(cls).filter(cls.trial_id == trial_id).all() return trial_intermediate_values class TrialHeartbeatModel(BaseModel): __tablename__ = "trial_heartbeats" __table_args__: Any = (UniqueConstraint("trial_id"),) trial_heartbeat_id = _Column(Integer, primary_key=True) trial_id = _Column(Integer, ForeignKey("trials.trial_id"), nullable=False) heartbeat = _Column(DateTime, nullable=False, default=func.current_timestamp()) trial = orm.relationship( TrialModel, backref=orm.backref("heartbeats", cascade="all, delete-orphan") ) @classmethod def where_trial_id( cls, trial_id: int, session: orm.Session ) -> Optional["TrialHeartbeatModel"]: return session.query(cls).filter(cls.trial_id == trial_id).one_or_none() class VersionInfoModel(BaseModel): __tablename__ = "version_info" # setting check constraint to ensure the number of rows is at most 1 __table_args__: Any = (CheckConstraint("version_info_id=1"),) version_info_id = _Column(Integer, primary_key=True, autoincrement=False, default=1) schema_version = _Column(Integer) library_version = _Column(String(MAX_VERSION_LENGTH)) @classmethod def find(cls, session: orm.Session) -> Optional["VersionInfoModel"]: version_info = session.query(cls).one_or_none() return version_info optuna-3.5.0/optuna/storages/_rdb/storage.py000066400000000000000000001365731453453102400211260ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict from contextlib import contextmanager import copy from datetime import datetime import json import logging import os from typing import Any from typing import Callable from typing import Container from typing import Dict from typing import Generator from typing import Iterable from typing import List from typing import Optional from typing import Sequence from typing import Set from typing import TYPE_CHECKING import uuid import optuna from optuna import distributions from optuna import version from optuna._imports import _LazyImport from optuna._typing import JSONSerializable from optuna.storages._base import BaseStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.storages._heartbeat import BaseHeartbeat from optuna.storages._rdb.models import TrialValueModel from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState if TYPE_CHECKING: import alembic.command as alembic_command import alembic.config as alembic_config import alembic.migration as alembic_migration import alembic.script as alembic_script import sqlalchemy import sqlalchemy.exc as sqlalchemy_exc import sqlalchemy.orm as sqlalchemy_orm import sqlalchemy.sql.functions as sqlalchemy_sql_functions from optuna.storages._rdb import models else: alembic_command = _LazyImport("alembic.command") alembic_config = _LazyImport("alembic.config") alembic_migration = _LazyImport("alembic.migration") alembic_script = _LazyImport("alembic.script") sqlalchemy = _LazyImport("sqlalchemy") sqlalchemy_exc = _LazyImport("sqlalchemy.exc") sqlalchemy_orm = _LazyImport("sqlalchemy.orm") sqlalchemy_sql_functions = _LazyImport("sqlalchemy.sql.functions") models = _LazyImport("optuna.storages._rdb.models") _logger = optuna.logging.get_logger(__name__) @contextmanager def _create_scoped_session( scoped_session: "sqlalchemy_orm.scoped_session", ignore_integrity_error: bool = False, ) -> Generator["sqlalchemy_orm.Session", None, None]: session = scoped_session() try: yield session session.commit() except sqlalchemy_exc.IntegrityError as e: session.rollback() if ignore_integrity_error: _logger.debug( "Ignoring {}. This happens due to a timing issue among threads/processes/nodes. " "Another one might have committed a record with the same key(s).".format(repr(e)) ) else: raise except sqlalchemy_exc.SQLAlchemyError as e: session.rollback() message = ( "An exception is raised during the commit. " "This typically happens due to invalid data in the commit, " "e.g. exceeding max length. " ) raise optuna.exceptions.StorageInternalError(message) from e except Exception: session.rollback() raise finally: session.close() class RDBStorage(BaseStorage, BaseHeartbeat): """Storage class for RDB backend. Note that library users can instantiate this class, but the attributes provided by this class are not supposed to be directly accessed by them. Example: Create an :class:`~optuna.storages.RDBStorage` instance with customized ``pool_size`` and ``timeout`` settings. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) return x**2 storage = optuna.storages.RDBStorage( url="sqlite:///:memory:", engine_kwargs={"pool_size": 20, "connect_args": {"timeout": 10}}, ) study = optuna.create_study(storage=storage) study.optimize(objective, n_trials=10) Args: url: URL of the storage. engine_kwargs: A dictionary of keyword arguments that is passed to `sqlalchemy.engine.create_engine`_ function. skip_compatibility_check: Flag to skip schema compatibility check if set to :obj:`True`. heartbeat_interval: Interval to record the heartbeat. It is recorded every ``interval`` seconds. ``heartbeat_interval`` must be :obj:`None` or a positive integer. .. note:: The heartbeat is supposed to be used with :meth:`~optuna.study.Study.optimize`. If you use :meth:`~optuna.study.Study.ask` and :meth:`~optuna.study.Study.tell` instead, it will not work. grace_period: Grace period before a running trial is failed from the last heartbeat. ``grace_period`` must be :obj:`None` or a positive integer. If it is :obj:`None`, the grace period will be `2 * heartbeat_interval`. failed_trial_callback: A callback function that is invoked after failing each stale trial. The function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. .. note:: The procedure to fail existing stale trials is called just before asking the study for a new trial. skip_table_creation: Flag to skip table creation if set to :obj:`True`. .. _sqlalchemy.engine.create_engine: https://docs.sqlalchemy.org/en/latest/core/engines.html#sqlalchemy.create_engine .. note:: If you use MySQL, `pool_pre_ping`_ will be set to :obj:`True` by default to prevent connection timeout. You can turn it off with ``engine_kwargs['pool_pre_ping']=False``, but it is recommended to keep the setting if execution time of your objective function is longer than the `wait_timeout` of your MySQL configuration. .. _pool_pre_ping: https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params. pool_pre_ping .. note:: We would never recommend SQLite3 for parallel optimization. Please see the FAQ :ref:`sqlite_concurrency` for details. .. note:: Mainly in a cluster environment, running trials are often killed unexpectedly. If you want to detect a failure of trials, please use the heartbeat mechanism. Set ``heartbeat_interval``, ``grace_period``, and ``failed_trial_callback`` appropriately according to your use case. For more details, please refer to the :ref:`tutorial ` and `Example page `_. .. seealso:: You can use :class:`~optuna.storages.RetryFailedTrialCallback` to automatically retry failed trials detected by heartbeat. """ def __init__( self, url: str, engine_kwargs: Optional[Dict[str, Any]] = None, skip_compatibility_check: bool = False, *, heartbeat_interval: Optional[int] = None, grace_period: Optional[int] = None, failed_trial_callback: Optional[ Callable[["optuna.study.Study", FrozenTrial], None] ] = None, skip_table_creation: bool = False, ) -> None: self.engine_kwargs = engine_kwargs or {} self.url = self._fill_storage_url_template(url) self.skip_compatibility_check = skip_compatibility_check if heartbeat_interval is not None and heartbeat_interval <= 0: raise ValueError("The value of `heartbeat_interval` should be a positive integer.") if grace_period is not None and grace_period <= 0: raise ValueError("The value of `grace_period` should be a positive integer.") self.heartbeat_interval = heartbeat_interval self.grace_period = grace_period self.failed_trial_callback = failed_trial_callback self._set_default_engine_kwargs_for_mysql(url, self.engine_kwargs) try: self.engine = sqlalchemy.engine.create_engine(self.url, **self.engine_kwargs) except ImportError as e: raise ImportError( "Failed to import DB access module for the specified storage URL. " "Please install appropriate one." ) from e self.scoped_session = sqlalchemy_orm.scoped_session( sqlalchemy_orm.sessionmaker(bind=self.engine) ) if not skip_table_creation: models.BaseModel.metadata.create_all(self.engine) self._version_manager = _VersionManager(self.url, self.engine, self.scoped_session) if not skip_compatibility_check: self._version_manager.check_table_schema_compatibility() def __getstate__(self) -> Dict[Any, Any]: state = self.__dict__.copy() del state["scoped_session"] del state["engine"] del state["_version_manager"] return state def __setstate__(self, state: Dict[Any, Any]) -> None: self.__dict__.update(state) try: self.engine = sqlalchemy.engine.create_engine(self.url, **self.engine_kwargs) except ImportError as e: raise ImportError( "Failed to import DB access module for the specified storage URL. " "Please install appropriate one." ) from e self.scoped_session = sqlalchemy_orm.scoped_session( sqlalchemy_orm.sessionmaker(bind=self.engine) ) models.BaseModel.metadata.create_all(self.engine) self._version_manager = _VersionManager(self.url, self.engine, self.scoped_session) if not self.skip_compatibility_check: self._version_manager.check_table_schema_compatibility() def create_new_study( self, directions: Sequence[StudyDirection], study_name: Optional[str] = None ) -> int: try: with _create_scoped_session(self.scoped_session) as session: if study_name is None: study_name = self._create_unique_study_name(session) direction_models = [ models.StudyDirectionModel(objective=objective, direction=d) for objective, d in enumerate(list(directions)) ] session.add(models.StudyModel(study_name=study_name, directions=direction_models)) except sqlalchemy_exc.IntegrityError: raise optuna.exceptions.DuplicatedStudyError( "Another study with name '{}' already exists. " "Please specify a different name, or reuse the existing one " "by setting `load_if_exists` (for Python API) or " "`--skip-if-exists` flag (for CLI).".format(study_name) ) _logger.info("A new study created in RDB with name: {}".format(study_name)) return self.get_study_id_from_name(study_name) def delete_study(self, study_id: int) -> None: with _create_scoped_session(self.scoped_session, True) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) session.delete(study) @staticmethod def _create_unique_study_name(session: "sqlalchemy_orm.Session") -> str: while True: study_uuid = str(uuid.uuid4()) study_name = DEFAULT_STUDY_NAME_PREFIX + study_uuid study = models.StudyModel.find_by_name(study_name, session) if study is None: break return study_name def set_study_user_attr(self, study_id: int, key: str, value: Any) -> None: with _create_scoped_session(self.scoped_session, True) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) attribute = models.StudyUserAttributeModel.find_by_study_and_key(study, key, session) if attribute is None: attribute = models.StudyUserAttributeModel( study_id=study_id, key=key, value_json=json.dumps(value) ) session.add(attribute) else: attribute.value_json = json.dumps(value) def set_study_system_attr(self, study_id: int, key: str, value: JSONSerializable) -> None: with _create_scoped_session(self.scoped_session, True) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) attribute = models.StudySystemAttributeModel.find_by_study_and_key(study, key, session) if attribute is None: attribute = models.StudySystemAttributeModel( study_id=study_id, key=key, value_json=json.dumps(value) ) session.add(attribute) else: attribute.value_json = json.dumps(value) def get_study_id_from_name(self, study_name: str) -> int: with _create_scoped_session(self.scoped_session) as session: study = models.StudyModel.find_or_raise_by_name(study_name, session) study_id = study.study_id return study_id def get_study_name_from_id(self, study_id: int) -> str: with _create_scoped_session(self.scoped_session) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) study_name = study.study_name return study_name def get_study_directions(self, study_id: int) -> List[StudyDirection]: with _create_scoped_session(self.scoped_session) as session: study = models.StudyModel.find_or_raise_by_id(study_id, session) directions = [d.direction for d in study.directions] return directions def get_study_user_attrs(self, study_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure that that study exists. models.StudyModel.find_or_raise_by_id(study_id, session) attributes = models.StudyUserAttributeModel.where_study_id(study_id, session) user_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return user_attrs def get_study_system_attrs(self, study_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure that that study exists. models.StudyModel.find_or_raise_by_id(study_id, session) attributes = models.StudySystemAttributeModel.where_study_id(study_id, session) system_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return system_attrs def get_trial_user_attrs(self, trial_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure trial exists. models.TrialModel.find_or_raise_by_id(trial_id, session) attributes = models.TrialUserAttributeModel.where_trial_id(trial_id, session) user_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return user_attrs def get_trial_system_attrs(self, trial_id: int) -> Dict[str, Any]: with _create_scoped_session(self.scoped_session) as session: # Ensure trial exists. models.TrialModel.find_or_raise_by_id(trial_id, session) attributes = models.TrialSystemAttributeModel.where_trial_id(trial_id, session) system_attrs = {attr.key: json.loads(attr.value_json) for attr in attributes} return system_attrs def get_all_studies(self) -> List[FrozenStudy]: with _create_scoped_session(self.scoped_session) as session: studies = ( session.query( models.StudyModel.study_id, models.StudyModel.study_name, ) .order_by(models.StudyModel.study_id) .all() ) _directions = defaultdict(list) for direction_model in session.query(models.StudyDirectionModel).all(): _directions[direction_model.study_id].append(direction_model.direction) _user_attrs = defaultdict(list) for attribute_model in session.query(models.StudyUserAttributeModel).all(): _user_attrs[attribute_model.study_id].append(attribute_model) _system_attrs = defaultdict(list) for attribute_model in session.query(models.StudySystemAttributeModel).all(): _system_attrs[attribute_model.study_id].append(attribute_model) frozen_studies = [] for study in studies: directions = _directions[study.study_id] user_attrs = _user_attrs.get(study.study_id, []) system_attrs = _system_attrs.get(study.study_id, []) frozen_studies.append( FrozenStudy( study_name=study.study_name, direction=None, directions=directions, user_attrs={i.key: json.loads(i.value_json) for i in user_attrs}, system_attrs={i.key: json.loads(i.value_json) for i in system_attrs}, study_id=study.study_id, ) ) return frozen_studies def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int: return self._create_new_trial(study_id, template_trial)._trial_id def _create_new_trial( self, study_id: int, template_trial: Optional[FrozenTrial] = None ) -> FrozenTrial: """Create a new trial and returns a :class:`~optuna.trial.FrozenTrial`. Args: study_id: Study id. template_trial: A :class:`~optuna.trial.FrozenTrial` with default values for trial attributes. Returns: A :class:`~optuna.trial.FrozenTrial` instance. """ # Retry a couple of times. Deadlocks may occur in distributed environments. n_retries = 0 with _create_scoped_session(self.scoped_session) as session: while True: try: # Ensure that that study exists. # # Locking within a study is necessary since the creation of a trial is not an # atomic operation. More precisely, the trial number computed in # `_get_prepared_new_trial` is prone to race conditions without this lock. models.StudyModel.find_or_raise_by_id(study_id, session, for_update=True) trial = self._get_prepared_new_trial(study_id, template_trial, session) break # Successfully created trial. except sqlalchemy_exc.OperationalError: if n_retries > 2: raise n_retries += 1 if template_trial: frozen = copy.deepcopy(template_trial) frozen.number = trial.number frozen.datetime_start = trial.datetime_start frozen._trial_id = trial.trial_id else: frozen = FrozenTrial( number=trial.number, state=trial.state, value=None, values=None, datetime_start=trial.datetime_start, datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=trial.trial_id, ) return frozen def _get_prepared_new_trial( self, study_id: int, template_trial: Optional[FrozenTrial], session: "sqlalchemy_orm.Session", ) -> "models.TrialModel": if template_trial is None: trial = models.TrialModel( study_id=study_id, number=None, state=TrialState.RUNNING, datetime_start=datetime.now(), ) else: # Because only `RUNNING` trials can be updated, # we temporarily set the state of the new trial to `RUNNING`. # After all fields of the trial have been updated, # the state is set to `template_trial.state`. temp_state = TrialState.RUNNING trial = models.TrialModel( study_id=study_id, number=None, state=temp_state, datetime_start=template_trial.datetime_start, datetime_complete=template_trial.datetime_complete, ) session.add(trial) # Flush the session cache to reflect the above addition operation to # the current RDB transaction. # # Without flushing, the following operations (e.g, `_set_trial_param_without_commit`) # will fail because the target trial doesn't exist in the storage yet. session.flush() if template_trial is not None: if template_trial.values is not None and len(template_trial.values) > 1: for objective, value in enumerate(template_trial.values): self._set_trial_value_without_commit(session, trial.trial_id, objective, value) elif template_trial.value is not None: self._set_trial_value_without_commit( session, trial.trial_id, 0, template_trial.value ) for param_name, param_value in template_trial.params.items(): distribution = template_trial.distributions[param_name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) self._set_trial_param_without_commit( session, trial.trial_id, param_name, param_value_in_internal_repr, distribution ) for key, value in template_trial.user_attrs.items(): self._set_trial_user_attr_without_commit(session, trial.trial_id, key, value) for key, value in template_trial.system_attrs.items(): self._set_trial_system_attr_without_commit(session, trial.trial_id, key, value) for step, intermediate_value in template_trial.intermediate_values.items(): self._set_trial_intermediate_value_without_commit( session, trial.trial_id, step, intermediate_value ) trial.state = template_trial.state trial.number = trial.count_past_trials(session) session.add(trial) return trial def set_trial_param( self, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_param_without_commit( session, trial_id, param_name, param_value_internal, distribution ) def _set_trial_param_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) trial_param = models.TrialParamModel.find_by_trial_and_param_name( trial, param_name, session ) if trial_param is not None: # Raise error in case distribution is incompatible. distributions.check_distribution_compatibility( distributions.json_to_distribution(trial_param.distribution_json), distribution ) trial_param.param_value = param_value_internal trial_param.distribution_json = distributions.distribution_to_json(distribution) else: trial_param = models.TrialParamModel( trial_id=trial_id, param_name=param_name, param_value=param_value_internal, distribution_json=distributions.distribution_to_json(distribution), ) trial_param.check_and_add(session) def _check_and_set_param_distribution( self, study_id: int, trial_id: int, param_name: str, param_value_internal: float, distribution: distributions.BaseDistribution, ) -> None: with _create_scoped_session(self.scoped_session) as session: # Acquire lock. # # Assume that study exists. models.StudyModel.find_or_raise_by_id(study_id, session, for_update=True) models.TrialParamModel( trial_id=trial_id, param_name=param_name, param_value=param_value_internal, distribution_json=distributions.distribution_to_json(distribution), ).check_and_add(session) def get_trial_param(self, trial_id: int, param_name: str) -> float: with _create_scoped_session(self.scoped_session) as session: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) trial_param = models.TrialParamModel.find_or_raise_by_trial_and_param_name( trial, param_name, session ) param_value = trial_param.param_value return param_value def set_trial_state_values( self, trial_id: int, state: TrialState, values: Optional[Sequence[float]] = None ) -> bool: try: with _create_scoped_session(self.scoped_session) as session: trial = models.TrialModel.find_or_raise_by_id(trial_id, session, for_update=True) self.check_trial_is_updatable(trial_id, trial.state) if values is not None: for objective, v in enumerate(values): self._set_trial_value_without_commit(session, trial_id, objective, v) if state == TrialState.RUNNING and trial.state != TrialState.WAITING: return False trial.state = state if state == TrialState.RUNNING: trial.datetime_start = datetime.now() if state.is_finished(): trial.datetime_complete = datetime.now() except sqlalchemy_exc.IntegrityError: return False return True def _set_trial_value_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, objective: int, value: float ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) stored_value, value_type = TrialValueModel.value_to_stored_repr(value) trial_value = models.TrialValueModel.find_by_trial_and_objective(trial, objective, session) if trial_value is None: trial_value = models.TrialValueModel( trial_id=trial_id, objective=objective, value=stored_value, value_type=value_type ) session.add(trial_value) else: trial_value.value = stored_value trial_value.value_type = value_type def set_trial_intermediate_value( self, trial_id: int, step: int, intermediate_value: float ) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_intermediate_value_without_commit( session, trial_id, step, intermediate_value ) def _set_trial_intermediate_value_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, step: int, intermediate_value: float, ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) ( stored_value, value_type, ) = models.TrialIntermediateValueModel.intermediate_value_to_stored_repr( intermediate_value ) trial_intermediate_value = models.TrialIntermediateValueModel.find_by_trial_and_step( trial, step, session ) if trial_intermediate_value is None: trial_intermediate_value = models.TrialIntermediateValueModel( trial_id=trial_id, step=step, intermediate_value=stored_value, intermediate_value_type=value_type, ) session.add(trial_intermediate_value) else: trial_intermediate_value.intermediate_value = stored_value trial_intermediate_value.intermediate_value_type = value_type def set_trial_user_attr(self, trial_id: int, key: str, value: Any) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_user_attr_without_commit(session, trial_id, key, value) def _set_trial_user_attr_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, key: str, value: Any ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) attribute = models.TrialUserAttributeModel.find_by_trial_and_key(trial, key, session) if attribute is None: attribute = models.TrialUserAttributeModel( trial_id=trial_id, key=key, value_json=json.dumps(value) ) session.add(attribute) else: attribute.value_json = json.dumps(value) def set_trial_system_attr(self, trial_id: int, key: str, value: JSONSerializable) -> None: with _create_scoped_session(self.scoped_session, True) as session: self._set_trial_system_attr_without_commit(session, trial_id, key, value) def _set_trial_system_attr_without_commit( self, session: "sqlalchemy_orm.Session", trial_id: int, key: str, value: JSONSerializable ) -> None: trial = models.TrialModel.find_or_raise_by_id(trial_id, session) self.check_trial_is_updatable(trial_id, trial.state) attribute = models.TrialSystemAttributeModel.find_by_trial_and_key(trial, key, session) if attribute is None: attribute = models.TrialSystemAttributeModel( trial_id=trial_id, key=key, value_json=json.dumps(value) ) session.add(attribute) else: attribute.value_json = json.dumps(value) def get_trial_id_from_study_id_trial_number(self, study_id: int, trial_number: int) -> int: with _create_scoped_session(self.scoped_session) as session: trial_id = ( session.query(models.TrialModel.trial_id) .filter( models.TrialModel.number == trial_number, models.TrialModel.study_id == study_id, ) .one_or_none() ) if trial_id is None: raise KeyError( "No trial with trial number {} exists in study with study_id {}.".format( trial_number, study_id ) ) return trial_id[0] def get_trial(self, trial_id: int) -> FrozenTrial: with _create_scoped_session(self.scoped_session) as session: trial_model = models.TrialModel.find_or_raise_by_id(trial_id, session) frozen_trial = self._build_frozen_trial_from_trial_model(trial_model) return frozen_trial def get_all_trials( self, study_id: int, deepcopy: bool = True, states: Optional[Container[TrialState]] = None, ) -> List[FrozenTrial]: trials = self._get_trials(study_id, states, set()) return copy.deepcopy(trials) if deepcopy else trials def _get_trials( self, study_id: int, states: Optional[Container[TrialState]], excluded_trial_ids: Set[int], ) -> List[FrozenTrial]: with _create_scoped_session(self.scoped_session) as session: # Ensure that the study exists. models.StudyModel.find_or_raise_by_id(study_id, session) query = session.query(models.TrialModel.trial_id).filter( models.TrialModel.study_id == study_id ) if states is not None: # This assertion is for type checkers, since `states` is required to be Container # in the base class while `models.TrialModel.state.in_` requires Iterable. assert isinstance(states, Iterable) query = query.filter(models.TrialModel.state.in_(states)) trial_ids = query.all() trial_ids = set( trial_id_tuple[0] for trial_id_tuple in trial_ids if trial_id_tuple[0] not in excluded_trial_ids ) try: trial_models = ( session.query(models.TrialModel) .options(sqlalchemy_orm.selectinload(models.TrialModel.params)) .options(sqlalchemy_orm.selectinload(models.TrialModel.values)) .options(sqlalchemy_orm.selectinload(models.TrialModel.user_attributes)) .options(sqlalchemy_orm.selectinload(models.TrialModel.system_attributes)) .options(sqlalchemy_orm.selectinload(models.TrialModel.intermediate_values)) .filter( models.TrialModel.trial_id.in_(trial_ids), models.TrialModel.study_id == study_id, ) .order_by(models.TrialModel.trial_id) .all() ) except sqlalchemy_exc.OperationalError as e: # Likely exceeding the number of maximum allowed variables using IN. # This number differ between database dialects. For SQLite for instance, see # https://www.sqlite.org/limits.html and the section describing # SQLITE_MAX_VARIABLE_NUMBER. _logger.warning( "Caught an error from sqlalchemy: {}. Falling back to a slower alternative. " "".format(str(e)) ) trial_models = ( session.query(models.TrialModel) .options(sqlalchemy_orm.selectinload(models.TrialModel.params)) .options(sqlalchemy_orm.selectinload(models.TrialModel.values)) .options(sqlalchemy_orm.selectinload(models.TrialModel.user_attributes)) .options(sqlalchemy_orm.selectinload(models.TrialModel.system_attributes)) .options(sqlalchemy_orm.selectinload(models.TrialModel.intermediate_values)) .filter(models.TrialModel.study_id == study_id) .order_by(models.TrialModel.trial_id) .all() ) trial_models = [t for t in trial_models if t.trial_id in trial_ids] trials = [self._build_frozen_trial_from_trial_model(trial) for trial in trial_models] return trials def _build_frozen_trial_from_trial_model(self, trial: "models.TrialModel") -> FrozenTrial: values: Optional[List[float]] if trial.values: values = [0 for _ in trial.values] for value_model in trial.values: values[value_model.objective] = TrialValueModel.stored_repr_to_value( value_model.value, value_model.value_type ) else: values = None params = sorted(trial.params, key=lambda p: p.param_id) return FrozenTrial( number=trial.number, state=trial.state, value=None, values=values, datetime_start=trial.datetime_start, datetime_complete=trial.datetime_complete, params={ p.param_name: distributions.json_to_distribution( p.distribution_json ).to_external_repr(p.param_value) for p in params }, distributions={ p.param_name: distributions.json_to_distribution(p.distribution_json) for p in params }, user_attrs={attr.key: json.loads(attr.value_json) for attr in trial.user_attributes}, system_attrs={ attr.key: json.loads(attr.value_json) for attr in trial.system_attributes }, intermediate_values={ v.step: models.TrialIntermediateValueModel.stored_repr_to_intermediate_value( v.intermediate_value, v.intermediate_value_type ) for v in trial.intermediate_values }, trial_id=trial.trial_id, ) def get_best_trial(self, study_id: int) -> FrozenTrial: with _create_scoped_session(self.scoped_session) as session: _directions = self.get_study_directions(study_id) if len(_directions) > 1: raise RuntimeError( "Best trial can be obtained only for single-objective optimization." ) direction = _directions[0] if direction == StudyDirection.MAXIMIZE: trial = models.TrialModel.find_max_value_trial(study_id, 0, session) else: trial = models.TrialModel.find_min_value_trial(study_id, 0, session) trial_id = trial.trial_id return self.get_trial(trial_id) @staticmethod def _set_default_engine_kwargs_for_mysql(url: str, engine_kwargs: Dict[str, Any]) -> None: # Skip if RDB is not MySQL. if not url.startswith("mysql"): return # Do not overwrite value. if "pool_pre_ping" in engine_kwargs: return # If True, the connection pool checks liveness of connections at every checkout. # Without this option, trials that take longer than `wait_timeout` may cause connection # errors. For further details, please refer to the following document: # https://docs.sqlalchemy.org/en/13/core/pooling.html#pool-disconnects-pessimistic engine_kwargs["pool_pre_ping"] = True _logger.debug("pool_pre_ping=True was set to engine_kwargs to prevent connection timeout.") @staticmethod def _fill_storage_url_template(template: str) -> str: return template.format(SCHEMA_VERSION=models.SCHEMA_VERSION) def remove_session(self) -> None: """Removes the current session. A session is stored in SQLAlchemy's ThreadLocalRegistry for each thread. This method closes and removes the session which is associated to the current thread. Particularly, under multi-thread use cases, it is important to call this method *from each thread*. Otherwise, all sessions and their associated DB connections are destructed by a thread that occasionally invoked the garbage collector. By default, it is not allowed to touch a SQLite connection from threads other than the thread that created the connection. Therefore, we need to explicitly close the connection from each thread. """ self.scoped_session.remove() def upgrade(self) -> None: """Upgrade the storage schema.""" self._version_manager.upgrade() def get_current_version(self) -> str: """Return the schema version currently used by this storage.""" return self._version_manager.get_current_version() def get_head_version(self) -> str: """Return the latest schema version.""" return self._version_manager.get_head_version() def get_all_versions(self) -> List[str]: """Return the schema version list.""" return self._version_manager.get_all_versions() def record_heartbeat(self, trial_id: int) -> None: with _create_scoped_session(self.scoped_session, True) as session: heartbeat = models.TrialHeartbeatModel.where_trial_id(trial_id, session) if heartbeat is None: heartbeat = models.TrialHeartbeatModel(trial_id=trial_id) session.add(heartbeat) else: heartbeat.heartbeat = session.execute(sqlalchemy.func.now()).scalar() def _get_stale_trial_ids(self, study_id: int) -> List[int]: assert self.heartbeat_interval is not None if self.grace_period is None: grace_period = 2 * self.heartbeat_interval else: grace_period = self.grace_period stale_trial_ids = [] with _create_scoped_session(self.scoped_session, True) as session: current_heartbeat = session.execute(sqlalchemy.func.now()).scalar() assert current_heartbeat is not None # Added the following line to prevent mixing of timezone-aware and timezone-naive # `datetime` in PostgreSQL. See # https://github.com/optuna/optuna/pull/2190#issuecomment-766605088 for details current_heartbeat = current_heartbeat.replace(tzinfo=None) running_trials = ( session.query(models.TrialModel) .options(sqlalchemy_orm.selectinload(models.TrialModel.heartbeats)) .filter(models.TrialModel.state == TrialState.RUNNING) .filter(models.TrialModel.study_id == study_id) .all() ) for trial in running_trials: if len(trial.heartbeats) == 0: continue assert len(trial.heartbeats) == 1 heartbeat = trial.heartbeats[0].heartbeat if (current_heartbeat - heartbeat).seconds > grace_period: stale_trial_ids.append(trial.trial_id) return stale_trial_ids def get_heartbeat_interval(self) -> Optional[int]: return self.heartbeat_interval def get_failed_trial_callback( self, ) -> Optional[Callable[["optuna.study.Study", FrozenTrial], None]]: return self.failed_trial_callback class _VersionManager: def __init__( self, url: str, engine: "sqlalchemy.engine.Engine", scoped_session: "sqlalchemy_orm.scoped_session", ) -> None: self.url = url self.engine = engine self.scoped_session = scoped_session self._init_version_info_model() self._init_alembic() def _init_version_info_model(self) -> None: with _create_scoped_session(self.scoped_session, True) as session: version_info = models.VersionInfoModel.find(session) if version_info is not None: return version_info = models.VersionInfoModel( schema_version=models.SCHEMA_VERSION, library_version=version.__version__, ) session.add(version_info) def _init_alembic(self) -> None: logging.getLogger("alembic").setLevel(logging.WARN) with self.engine.connect() as connection: context = alembic_migration.MigrationContext.configure(connection) is_initialized = context.get_current_revision() is not None if is_initialized: # The `alembic_version` table already exists and is not empty. return if self._is_alembic_supported(): revision = self.get_head_version() else: # The storage has been created before alembic is introduced. revision = self._get_base_version() self._set_alembic_revision(revision) def _set_alembic_revision(self, revision: str) -> None: with self.engine.connect() as connection: context = alembic_migration.MigrationContext.configure(connection) with connection.begin(): script = self._create_alembic_script() context.stamp(script, revision) def check_table_schema_compatibility(self) -> None: with _create_scoped_session(self.scoped_session) as session: # NOTE: After invocation of `_init_version_info_model` method, # it is ensured that a `VersionInfoModel` entry exists. version_info = models.VersionInfoModel.find(session) assert version_info is not None current_version = self.get_current_version() head_version = self.get_head_version() if current_version == head_version: return message = ( "The runtime optuna version {} is no longer compatible with the table schema " "(set up by optuna {}). ".format(version.__version__, version_info.library_version) ) known_versions = self.get_all_versions() if current_version in known_versions: message += ( "Please execute `$ optuna storage upgrade --storage $STORAGE_URL` " "for upgrading the storage." ) else: message += ( "Please try updating optuna to the latest version by `$ pip install -U optuna`." ) raise RuntimeError(message) def get_current_version(self) -> str: with self.engine.connect() as connection: context = alembic_migration.MigrationContext.configure(connection) version = context.get_current_revision() assert version is not None return version def get_head_version(self) -> str: script = self._create_alembic_script() current_head = script.get_current_head() assert current_head is not None return current_head def _get_base_version(self) -> str: script = self._create_alembic_script() base = script.get_base() assert base is not None, "There should be exactly one base, i.e. v0.9.0.a." return base def get_all_versions(self) -> List[str]: script = self._create_alembic_script() return [r.revision for r in script.walk_revisions()] def upgrade(self) -> None: config = self._create_alembic_config() alembic_command.upgrade(config, "head") with _create_scoped_session(self.scoped_session, True) as session: version_info = models.VersionInfoModel.find(session) assert version_info is not None version_info.schema_version = models.SCHEMA_VERSION version_info.library_version = version.__version__ def _is_alembic_supported(self) -> bool: with _create_scoped_session(self.scoped_session) as session: version_info = models.VersionInfoModel.find(session) if version_info is None: # `None` means this storage was created just now. return True return version_info.schema_version == models.SCHEMA_VERSION def _create_alembic_script(self) -> "alembic_script.ScriptDirectory": config = self._create_alembic_config() script = alembic_script.ScriptDirectory.from_config(config) return script def _create_alembic_config(self) -> "alembic_config.Config": alembic_dir = os.path.join(os.path.dirname(__file__), "alembic") config = alembic_config.Config(os.path.join(os.path.dirname(__file__), "alembic.ini")) config.set_main_option("script_location", escape_alembic_config_value(alembic_dir)) config.set_main_option("sqlalchemy.url", escape_alembic_config_value(self.url)) return config def escape_alembic_config_value(value: str) -> str: # We must escape '%' in a value string because the character # is regarded as the trigger of variable expansion. # Please see the documentation of `configparser.BasicInterpolation` for more details. return value.replace("%", "%%") optuna-3.5.0/optuna/study/000077500000000000000000000000001453453102400155045ustar00rootroot00000000000000optuna-3.5.0/optuna/study/__init__.py000066400000000000000000000012771453453102400176240ustar00rootroot00000000000000from optuna._callbacks import MaxTrialsCallback from optuna.study._study_direction import StudyDirection from optuna.study._study_summary import StudySummary from optuna.study.study import copy_study from optuna.study.study import create_study from optuna.study.study import delete_study from optuna.study.study import get_all_study_names from optuna.study.study import get_all_study_summaries from optuna.study.study import load_study from optuna.study.study import Study __all__ = [ "MaxTrialsCallback", "StudyDirection", "StudySummary", "copy_study", "create_study", "delete_study", "get_all_study_names", "get_all_study_summaries", "load_study", "Study", ] optuna-3.5.0/optuna/study/_dataframe.py000066400000000000000000000102451453453102400201430ustar00rootroot00000000000000from __future__ import annotations import collections from typing import Any from typing import DefaultDict from typing import Set import optuna from optuna._imports import try_import from optuna.trial._state import TrialState with try_import() as _imports: # `Study.trials_dataframe` is disabled if pandas is not available. import pandas as pd # Required for type annotation in `Study.trials_dataframe`. if not _imports.is_successful(): pd = object # NOQA __all__ = ["pd"] def _create_records_and_aggregate_column( study: "optuna.Study", attrs: tuple[str, ...] ) -> tuple[list[dict[tuple[str, str], Any]], list[tuple[str, str]]]: attrs_to_df_columns: dict[str, str] = {} for attr in attrs: if attr.startswith("_"): # Python conventional underscores are omitted in the dataframe. df_column = attr[1:] else: df_column = attr attrs_to_df_columns[attr] = df_column # column_agg is an aggregator of column names. # Keys of column agg are attributes of `FrozenTrial` such as 'trial_id' and 'params'. # Values are dataframe columns such as ('trial_id', '') and ('params', 'n_layers'). column_agg: DefaultDict[str, Set] = collections.defaultdict(set) non_nested_attr = "" metric_names = study.metric_names records = [] for trial in study.get_trials(deepcopy=False): record = {} for attr, df_column in attrs_to_df_columns.items(): value = getattr(trial, attr) if isinstance(value, TrialState): value = value.name if isinstance(value, dict): for nested_attr, nested_value in value.items(): record[(df_column, nested_attr)] = nested_value column_agg[attr].add((df_column, nested_attr)) elif attr == "values": # Expand trial.values. # trial.values should be None when the trial's state is FAIL or PRUNED. trial_values = [None] * len(study.directions) if value is None else value iterator = ( enumerate(trial_values) if metric_names is None else zip(metric_names, trial_values) ) for nested_attr, nested_value in iterator: record[(df_column, nested_attr)] = nested_value column_agg[attr].add((df_column, nested_attr)) elif isinstance(value, list): for nested_attr, nested_value in enumerate(value): record[(df_column, nested_attr)] = nested_value column_agg[attr].add((df_column, nested_attr)) elif attr == "value": nested_attr = non_nested_attr if metric_names is None else metric_names[0] record[(df_column, nested_attr)] = value column_agg[attr].add((df_column, nested_attr)) else: record[(df_column, non_nested_attr)] = value column_agg[attr].add((df_column, non_nested_attr)) records.append(record) columns: list[tuple[str, str]] = sum( (sorted(column_agg[k]) for k in attrs if k in column_agg), [] ) return records, columns def _flatten_columns(columns: list[tuple[str, str]]) -> list[str]: # Flatten the `MultiIndex` columns where names are concatenated with underscores. # Filtering is required to omit non-nested columns avoiding unwanted trailing underscores. return ["_".join(filter(lambda c: c, map(lambda c: str(c), col))) for col in columns] def _trials_dataframe( study: "optuna.Study", attrs: tuple[str, ...], multi_index: bool ) -> "pd.DataFrame": _imports.check() # If no trials, return an empty dataframe. if len(study.get_trials(deepcopy=False)) == 0: return pd.DataFrame() if "value" in attrs and study._is_multi_objective(): attrs = tuple("values" if attr == "value" else attr for attr in attrs) records, columns = _create_records_and_aggregate_column(study, attrs) df = pd.DataFrame(records, columns=pd.MultiIndex.from_tuples(columns)) if not multi_index: df.columns = _flatten_columns(columns) return df optuna-3.5.0/optuna/study/_frozen.py000066400000000000000000000053401453453102400175220ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from typing import Any from optuna import logging from optuna.study._study_direction import StudyDirection _logger = logging.get_logger(__name__) class FrozenStudy: """Basic attributes of a :class:`~optuna.study.Study`. This class is private and not referenced by Optuna users. Attributes: study_name: Name of the :class:`~optuna.study.Study`. direction: :class:`~optuna.study.StudyDirection` of the :class:`~optuna.study.Study`. .. note:: This attribute is only available during single-objective optimization. directions: A list of :class:`~optuna.study.StudyDirection` objects. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` set with :func:`optuna.study.Study.set_user_attr`. system_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` internally set by Optuna. """ def __init__( self, study_name: str, direction: StudyDirection | None, user_attrs: dict[str, Any], system_attrs: dict[str, Any], study_id: int, *, directions: Sequence[StudyDirection] | None = None, ): self.study_name = study_name if direction is None and directions is None: raise ValueError("Specify one of `direction` and `directions`.") elif directions is not None: self._directions = list(directions) elif direction is not None: self._directions = [direction] else: raise ValueError("Specify only one of `direction` and `directions`.") self.user_attrs = user_attrs self.system_attrs = system_attrs self._study_id = study_id def __eq__(self, other: Any) -> bool: if not isinstance(other, FrozenStudy): return NotImplemented return other.__dict__ == self.__dict__ def __lt__(self, other: Any) -> bool: if not isinstance(other, FrozenStudy): return NotImplemented return self._study_id < other._study_id def __le__(self, other: Any) -> bool: if not isinstance(other, FrozenStudy): return NotImplemented return self._study_id <= other._study_id @property def direction(self) -> StudyDirection: if len(self._directions) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) return self._directions[0] @property def directions(self) -> list[StudyDirection]: return self._directions optuna-3.5.0/optuna/study/_multi_objective.py000066400000000000000000000065061453453102400214100ustar00rootroot00000000000000from typing import List from typing import Optional from typing import Sequence import optuna from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState def _get_pareto_front_trials_2d( trials: Sequence[FrozenTrial], directions: Sequence[StudyDirection] ) -> List[FrozenTrial]: trials = [trial for trial in trials if trial.state == TrialState.COMPLETE] n_trials = len(trials) if n_trials == 0: return [] trials.sort( key=lambda trial: ( _normalize_value(trial.values[0], directions[0]), _normalize_value(trial.values[1], directions[1]), ), ) last_nondominated_trial = trials[0] pareto_front = [last_nondominated_trial] for i in range(1, n_trials): trial = trials[i] if _dominates(last_nondominated_trial, trial, directions): continue pareto_front.append(trial) last_nondominated_trial = trial pareto_front.sort(key=lambda trial: trial.number) return pareto_front def _get_pareto_front_trials_nd( trials: Sequence[FrozenTrial], directions: Sequence[StudyDirection] ) -> List[FrozenTrial]: pareto_front = [] trials = [t for t in trials if t.state == TrialState.COMPLETE] # TODO(vincent): Optimize (use the fast non dominated sort defined in the NSGA-II paper). for trial in trials: dominated = False for other in trials: if _dominates(other, trial, directions): dominated = True break if not dominated: pareto_front.append(trial) return pareto_front def _get_pareto_front_trials_by_trials( trials: Sequence[FrozenTrial], directions: Sequence[StudyDirection] ) -> List[FrozenTrial]: if len(directions) == 2: return _get_pareto_front_trials_2d(trials, directions) # Log-linear in number of trials. return _get_pareto_front_trials_nd(trials, directions) # Quadratic in number of trials. def _get_pareto_front_trials(study: "optuna.study.Study") -> List[FrozenTrial]: return _get_pareto_front_trials_by_trials(study.trials, study.directions) def _dominates( trial0: FrozenTrial, trial1: FrozenTrial, directions: Sequence[StudyDirection] ) -> bool: values0 = trial0.values values1 = trial1.values assert values0 is not None assert values1 is not None if len(values0) != len(values1): raise ValueError("Trials with different numbers of objectives cannot be compared.") if len(values0) != len(directions): raise ValueError( "The number of the values and the number of the objectives are mismatched." ) if trial0.state != TrialState.COMPLETE: return False if trial1.state != TrialState.COMPLETE: return True normalized_values0 = [_normalize_value(v, d) for v, d in zip(values0, directions)] normalized_values1 = [_normalize_value(v, d) for v, d in zip(values1, directions)] if normalized_values0 == normalized_values1: return False return all(v0 <= v1 for v0, v1 in zip(normalized_values0, normalized_values1)) def _normalize_value(value: Optional[float], direction: StudyDirection) -> float: if value is None: value = float("inf") if direction is StudyDirection.MAXIMIZE: value = -value return value optuna-3.5.0/optuna/study/_optimize.py000066400000000000000000000214601453453102400200600ustar00rootroot00000000000000from concurrent.futures import FIRST_COMPLETED from concurrent.futures import Future from concurrent.futures import ThreadPoolExecutor from concurrent.futures import wait import datetime import gc import itertools import os import sys from typing import Any from typing import Callable from typing import List from typing import Optional from typing import Sequence from typing import Set from typing import Tuple from typing import Type from typing import Union import warnings import optuna from optuna import exceptions from optuna import logging from optuna import progress_bar as pbar_module from optuna import trial as trial_module from optuna.storages._heartbeat import get_heartbeat_thread from optuna.storages._heartbeat import is_heartbeat_enabled from optuna.study._tell import _tell_with_warning from optuna.study._tell import STUDY_TELL_WARNING_KEY from optuna.trial import FrozenTrial from optuna.trial import TrialState _logger = logging.get_logger(__name__) def _optimize( study: "optuna.Study", func: "optuna.study.study.ObjectiveFuncType", n_trials: Optional[int] = None, timeout: Optional[float] = None, n_jobs: int = 1, catch: Tuple[Type[Exception], ...] = (), callbacks: Optional[List[Callable[["optuna.Study", FrozenTrial], None]]] = None, gc_after_trial: bool = False, show_progress_bar: bool = False, ) -> None: if not isinstance(catch, tuple): raise TypeError( "The catch argument is of type '{}' but must be a tuple.".format(type(catch).__name__) ) if study._thread_local.in_optimize_loop: raise RuntimeError("Nested invocation of `Study.optimize` method isn't allowed.") if show_progress_bar and n_trials is None and timeout is not None and n_jobs != 1: warnings.warn("The timeout-based progress bar is not supported with n_jobs != 1.") show_progress_bar = False progress_bar = pbar_module._ProgressBar(show_progress_bar, n_trials, timeout) study._stop_flag = False try: if n_jobs == 1: _optimize_sequential( study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng=False, time_start=None, progress_bar=progress_bar, ) else: if n_jobs == -1: n_jobs = os.cpu_count() or 1 time_start = datetime.datetime.now() futures: Set[Future] = set() with ThreadPoolExecutor(max_workers=n_jobs) as executor: for n_submitted_trials in itertools.count(): if study._stop_flag: break if ( timeout is not None and (datetime.datetime.now() - time_start).total_seconds() > timeout ): break if n_trials is not None and n_submitted_trials >= n_trials: break if len(futures) >= n_jobs: completed, futures = wait(futures, return_when=FIRST_COMPLETED) # Raise if exception occurred in executing the completed futures. for f in completed: f.result() futures.add( executor.submit( _optimize_sequential, study, func, 1, timeout, catch, callbacks, gc_after_trial, True, time_start, progress_bar, ) ) finally: study._thread_local.in_optimize_loop = False progress_bar.close() def _optimize_sequential( study: "optuna.Study", func: "optuna.study.study.ObjectiveFuncType", n_trials: Optional[int], timeout: Optional[float], catch: Tuple[Type[Exception], ...], callbacks: Optional[List[Callable[["optuna.Study", FrozenTrial], None]]], gc_after_trial: bool, reseed_sampler_rng: bool, time_start: Optional[datetime.datetime], progress_bar: Optional[pbar_module._ProgressBar], ) -> None: # Here we set `in_optimize_loop = True`, not at the beginning of the `_optimize()` function. # Because it is a thread-local object and `n_jobs` option spawns new threads. study._thread_local.in_optimize_loop = True if reseed_sampler_rng: study.sampler.reseed_rng() i_trial = 0 if time_start is None: time_start = datetime.datetime.now() while True: if study._stop_flag: break if n_trials is not None: if i_trial >= n_trials: break i_trial += 1 if timeout is not None: elapsed_seconds = (datetime.datetime.now() - time_start).total_seconds() if elapsed_seconds >= timeout: break try: frozen_trial = _run_trial(study, func, catch) finally: # The following line mitigates memory problems that can be occurred in some # environments (e.g., services that use computing containers such as GitHub Actions). # Please refer to the following PR for further details: # https://github.com/optuna/optuna/pull/325. if gc_after_trial: gc.collect() if callbacks is not None: for callback in callbacks: callback(study, frozen_trial) if progress_bar is not None: elapsed_seconds = (datetime.datetime.now() - time_start).total_seconds() progress_bar.update(elapsed_seconds, study) study._storage.remove_session() def _run_trial( study: "optuna.Study", func: "optuna.study.study.ObjectiveFuncType", catch: Tuple[Type[Exception], ...], ) -> trial_module.FrozenTrial: if is_heartbeat_enabled(study._storage): optuna.storages.fail_stale_trials(study) trial = study.ask() state: Optional[TrialState] = None value_or_values: Optional[Union[float, Sequence[float]]] = None func_err: Optional[Union[Exception, KeyboardInterrupt]] = None func_err_fail_exc_info: Optional[Any] = None with get_heartbeat_thread(trial._trial_id, study._storage): try: value_or_values = func(trial) except exceptions.TrialPruned as e: # TODO(mamu): Handle multi-objective cases. state = TrialState.PRUNED func_err = e except (Exception, KeyboardInterrupt) as e: state = TrialState.FAIL func_err = e func_err_fail_exc_info = sys.exc_info() # `_tell_with_warning` may raise during trial post-processing. try: frozen_trial = _tell_with_warning( study=study, trial=trial, value_or_values=value_or_values, state=state, suppress_warning=True, ) except Exception: frozen_trial = study._storage.get_trial(trial._trial_id) raise finally: if frozen_trial.state == TrialState.COMPLETE: study._log_completed_trial(frozen_trial) elif frozen_trial.state == TrialState.PRUNED: _logger.info("Trial {} pruned. {}".format(frozen_trial.number, str(func_err))) elif frozen_trial.state == TrialState.FAIL: if func_err is not None: _log_failed_trial( frozen_trial, repr(func_err), exc_info=func_err_fail_exc_info, value_or_values=value_or_values, ) elif STUDY_TELL_WARNING_KEY in frozen_trial.system_attrs: _log_failed_trial( frozen_trial, frozen_trial.system_attrs[STUDY_TELL_WARNING_KEY], value_or_values=value_or_values, ) else: assert False, "Should not reach." else: assert False, "Should not reach." if ( frozen_trial.state == TrialState.FAIL and func_err is not None and not isinstance(func_err, catch) ): raise func_err return frozen_trial def _log_failed_trial( trial: FrozenTrial, message: Union[str, Warning], exc_info: Any = None, value_or_values: Any = None, ) -> None: _logger.warning( "Trial {} failed with parameters: {} because of the following error: {}.".format( trial.number, trial.params, message ), exc_info=exc_info, ) _logger.warning("Trial {} failed with value {}.".format(trial.number, repr(value_or_values))) optuna-3.5.0/optuna/study/_study_direction.py000066400000000000000000000006451453453102400214320ustar00rootroot00000000000000import enum class StudyDirection(enum.IntEnum): """Direction of a :class:`~optuna.study.Study`. Attributes: NOT_SET: Direction has not been set. MINIMIZE: :class:`~optuna.study.Study` minimizes the objective function. MAXIMIZE: :class:`~optuna.study.Study` maximizes the objective function. """ NOT_SET = 0 MINIMIZE = 1 MAXIMIZE = 2 optuna-3.5.0/optuna/study/_study_summary.py000066400000000000000000000101331453453102400211400ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import datetime from typing import Any import warnings from optuna import logging from optuna import trial from optuna.study._study_direction import StudyDirection _logger = logging.get_logger(__name__) class StudySummary: """Basic attributes and aggregated results of a :class:`~optuna.study.Study`. See also :func:`optuna.study.get_all_study_summaries`. Attributes: study_name: Name of the :class:`~optuna.study.Study`. direction: :class:`~optuna.study.StudyDirection` of the :class:`~optuna.study.Study`. .. note:: This attribute is only available during single-objective optimization. directions: A sequence of :class:`~optuna.study.StudyDirection` objects. best_trial: :class:`optuna.trial.FrozenTrial` with best objective value in the :class:`~optuna.study.Study`. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` set with :func:`optuna.study.Study.set_user_attr`. system_attrs: Dictionary that contains the attributes of the :class:`~optuna.study.Study` internally set by Optuna. .. warning:: Deprecated in v3.1.0. ``system_attrs`` argument will be removed in the future. The removal of this feature is currently scheduled for v5.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v3.1.0. n_trials: The number of trials ran in the :class:`~optuna.study.Study`. datetime_start: Datetime where the :class:`~optuna.study.Study` started. """ def __init__( self, study_name: str, direction: StudyDirection | None, best_trial: trial.FrozenTrial | None, user_attrs: dict[str, Any], system_attrs: dict[str, Any], n_trials: int, datetime_start: datetime.datetime | None, study_id: int, *, directions: Sequence[StudyDirection] | None = None, ): self.study_name = study_name if direction is None and directions is None: raise ValueError("Specify one of `direction` and `directions`.") elif directions is not None: self._directions = list(directions) elif direction is not None: self._directions = [direction] else: raise ValueError("Specify only one of `direction` and `directions`.") self.best_trial = best_trial self.user_attrs = user_attrs self._system_attrs = system_attrs self.n_trials = n_trials self.datetime_start = datetime_start self._study_id = study_id def __eq__(self, other: Any) -> bool: if not isinstance(other, StudySummary): return NotImplemented return other.__dict__ == self.__dict__ def __lt__(self, other: Any) -> bool: if not isinstance(other, StudySummary): return NotImplemented return self._study_id < other._study_id def __le__(self, other: Any) -> bool: if not isinstance(other, StudySummary): return NotImplemented return self._study_id <= other._study_id @property def direction(self) -> StudyDirection: if len(self._directions) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) return self._directions[0] @property def directions(self) -> Sequence[StudyDirection]: return self._directions @property def system_attrs(self) -> dict[str, Any]: warnings.warn( "`system_attrs` has been deprecated in v3.1.0. " "The removal of this feature is currently scheduled for v5.0.0, " "but this schedule is subject to change. " "See https://github.com/optuna/optuna/releases/tag/v3.1.0.", FutureWarning, ) return self._system_attrs optuna-3.5.0/optuna/study/_tell.py000066400000000000000000000151241453453102400171600ustar00rootroot00000000000000import copy import math from typing import Optional from typing import Sequence from typing import Union import warnings import optuna from optuna import logging from optuna import pruners from optuna import trial as trial_module from optuna.trial import FrozenTrial from optuna.trial import TrialState # This is used for propagating warning message to Study.optimize. STUDY_TELL_WARNING_KEY = "STUDY_TELL_WARNING" _logger = logging.get_logger(__name__) def _get_frozen_trial(study: "optuna.Study", trial: Union[trial_module.Trial, int]) -> FrozenTrial: if isinstance(trial, trial_module.Trial): trial_id = trial._trial_id elif isinstance(trial, int): trial_number = trial try: trial_id = study._storage.get_trial_id_from_study_id_trial_number( study._study_id, trial_number ) except KeyError as e: raise ValueError( f"Cannot tell for trial with number {trial_number} since it has not been " "created." ) from e else: raise TypeError("Trial must be a trial object or trial number.") return study._storage.get_trial(trial_id) def _check_state_and_values( state: Optional[TrialState], values: Optional[Union[float, Sequence[float]]] ) -> None: if state == TrialState.COMPLETE: if values is None: raise ValueError( "No values were told. Values are required when state is TrialState.COMPLETE." ) elif state in (TrialState.PRUNED, TrialState.FAIL): if values is not None: raise ValueError( "Values were told. Values cannot be specified when state is " "TrialState.PRUNED or TrialState.FAIL." ) elif state is not None: raise ValueError(f"Cannot tell with state {state}.") def _check_values_are_feasible(study: "optuna.Study", values: Sequence[float]) -> Optional[str]: for v in values: # TODO(Imamura): Construct error message taking into account all values and do not early # return `value` is assumed to be ignored on failure so we can set it to any value. try: float(v) except (ValueError, TypeError): return f"The value {repr(v)} could not be cast to float" if math.isnan(v): return f"The value {v} is not acceptable" if len(study.directions) != len(values): return ( f"The number of the values {len(values)} did not match the number of the objectives " f"{len(study.directions)}" ) return None def _tell_with_warning( study: "optuna.Study", trial: Union[trial_module.Trial, int], value_or_values: Optional[Union[float, Sequence[float]]] = None, state: Optional[TrialState] = None, skip_if_finished: bool = False, suppress_warning: bool = False, ) -> FrozenTrial: """Internal method of :func:`~optuna.study.Study.tell`. Refer to the document for :func:`~optuna.study.Study.tell` for the reference. This method has one additional parameter ``suppress_warning``. Args: suppress_warning: If :obj:`True`, tell will not show warnings when tell receives an invalid values. This flag is expected to be :obj:`True` only when it is invoked by Study.optimize. """ # We must invalidate all trials cache here as it is only valid within a trial. study._thread_local.cached_all_trials = None # Validate the trial argument. frozen_trial = _get_frozen_trial(study, trial) if frozen_trial.state.is_finished() and skip_if_finished: _logger.info( f"Skipped telling trial {frozen_trial.number} with values " f"{value_or_values} and state {state} since trial was already finished. " f"Finished trial has values {frozen_trial.values} and state {frozen_trial.state}." ) return copy.deepcopy(frozen_trial) elif frozen_trial.state != TrialState.RUNNING: raise ValueError(f"Cannot tell a {frozen_trial.state.name} trial.") # Validate the state and values arguments. values: Optional[Sequence[float]] if value_or_values is None: values = None elif isinstance(value_or_values, Sequence): values = value_or_values else: values = [value_or_values] _check_state_and_values(state, values) warning_message = None if state == TrialState.COMPLETE: assert values is not None values_conversion_failure_message = _check_values_are_feasible(study, values) if values_conversion_failure_message is not None: raise ValueError(values_conversion_failure_message) elif state == TrialState.PRUNED: # Register the last intermediate value if present as the value of the trial. # TODO(hvy): Whether a pruned trials should have an actual value can be discussed. assert values is None last_step = frozen_trial.last_step if last_step is not None: last_intermediate_value = frozen_trial.intermediate_values[last_step] # intermediate_values can be unacceptable value, i.e., NaN. if _check_values_are_feasible(study, [last_intermediate_value]) is None: values = [last_intermediate_value] elif state is None: if values is None: values_conversion_failure_message = "The value None could not be cast to float." else: values_conversion_failure_message = _check_values_are_feasible(study, values) if values_conversion_failure_message is None: state = TrialState.COMPLETE else: state = TrialState.FAIL values = None if not suppress_warning: warnings.warn(values_conversion_failure_message) else: warning_message = values_conversion_failure_message assert state is not None # Cast values to list of floats. if values is not None: # values have beed checked to be castable to floats in _check_values_are_feasible. values = [float(value) for value in values] # Post-processing and storing the trial. try: # Sampler defined trial post-processing. study = pruners._filter_study(study, frozen_trial) study.sampler.after_trial(study, frozen_trial, state, values) finally: study._storage.set_trial_state_values(frozen_trial._trial_id, state, values) frozen_trial = copy.deepcopy(study._storage.get_trial(frozen_trial._trial_id)) if warning_message is not None: frozen_trial._system_attrs[STUDY_TELL_WARNING_KEY] = warning_message return frozen_trial optuna-3.5.0/optuna/study/study.py000066400000000000000000001564511453453102400172420ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Container from collections.abc import Iterable from collections.abc import Mapping import copy from numbers import Real import threading from typing import Any from typing import Callable from typing import cast from typing import Sequence from typing import TYPE_CHECKING from typing import Union import warnings import numpy as np from optuna import exceptions from optuna import logging from optuna import pruners from optuna import samplers from optuna import storages from optuna import trial as trial_module from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna._experimental import experimental_func from optuna._imports import _LazyImport from optuna._typing import JSONSerializable from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.distributions import BaseDistribution from optuna.storages._heartbeat import is_heartbeat_enabled from optuna.study._multi_objective import _get_pareto_front_trials from optuna.study._optimize import _optimize from optuna.study._study_direction import StudyDirection from optuna.study._study_summary import StudySummary # NOQA from optuna.study._tell import _tell_with_warning from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState _dataframe = _LazyImport("optuna.study._dataframe") if TYPE_CHECKING: from optuna.study._dataframe import pd ObjectiveFuncType = Callable[[trial_module.Trial], Union[float, Sequence[float]]] _SYSTEM_ATTR_METRIC_NAMES = "study:metric_names" _logger = logging.get_logger(__name__) class _ThreadLocalStudyAttribute(threading.local): in_optimize_loop: bool = False cached_all_trials: list["FrozenTrial"] | None = None class Study: """A study corresponds to an optimization task, i.e., a set of trials. This object provides interfaces to run a new :class:`~optuna.trial.Trial`, access trials' history, set/get user-defined attributes of the study itself. Note that the direct use of this constructor is not recommended. To create and load a study, please refer to the documentation of :func:`~optuna.study.create_study` and :func:`~optuna.study.load_study` respectively. """ def __init__( self, study_name: str, storage: str | storages.BaseStorage, sampler: "samplers.BaseSampler" | None = None, pruner: pruners.BasePruner | None = None, ) -> None: self.study_name = study_name storage = storages.get_storage(storage) study_id = storage.get_study_id_from_name(study_name) self._study_id = study_id self._storage = storage self._directions = storage.get_study_directions(study_id) self.sampler = sampler or samplers.TPESampler() self.pruner = pruner or pruners.MedianPruner() self._thread_local = _ThreadLocalStudyAttribute() self._stop_flag = False def __getstate__(self) -> dict[Any, Any]: state = self.__dict__.copy() del state["_thread_local"] return state def __setstate__(self, state: dict[Any, Any]) -> None: self.__dict__.update(state) self._thread_local = _ThreadLocalStudyAttribute() @property def best_params(self) -> dict[str, Any]: """Return parameters of the best trial in the study. .. note:: This feature can only be used for single-objective optimization. Returns: A dictionary containing parameters of the best trial. """ return self.best_trial.params @property def best_value(self) -> float: """Return the best objective value in the study. .. note:: This feature can only be used for single-objective optimization. Returns: A float representing the best objective value. """ best_value = self.best_trial.value assert best_value is not None return best_value @property def best_trial(self) -> FrozenTrial: """Return the best trial in the study. .. note:: This feature can only be used for single-objective optimization. If your study is multi-objective, use :attr:`~optuna.study.Study.best_trials` instead. Returns: A :class:`~optuna.trial.FrozenTrial` object of the best trial. .. seealso:: The :ref:`reuse_best_trial` tutorial provides a detailed example of how to use this method. """ if self._is_multi_objective(): raise RuntimeError( "A single best trial cannot be retrieved from a multi-objective study. Consider " "using Study.best_trials to retrieve a list containing the best trials." ) return copy.deepcopy(self._storage.get_best_trial(self._study_id)) @property def best_trials(self) -> list[FrozenTrial]: """Return trials located at the Pareto front in the study. A trial is located at the Pareto front if there are no trials that dominate the trial. It's called that a trial ``t0`` dominates another trial ``t1`` if ``all(v0 <= v1) for v0, v1 in zip(t0.values, t1.values)`` and ``any(v0 < v1) for v0, v1 in zip(t0.values, t1.values)`` are held. Returns: A list of :class:`~optuna.trial.FrozenTrial` objects. """ return _get_pareto_front_trials(self) @property def direction(self) -> StudyDirection: """Return the direction of the study. .. note:: This feature can only be used for single-objective optimization. If your study is multi-objective, use :attr:`~optuna.study.Study.directions` instead. Returns: A :class:`~optuna.study.StudyDirection` object. """ if self._is_multi_objective(): raise RuntimeError( "A single direction cannot be retrieved from a multi-objective study. Consider " "using Study.directions to retrieve a list containing all directions." ) return self.directions[0] @property def directions(self) -> list[StudyDirection]: """Return the directions of the study. Returns: A list of :class:`~optuna.study.StudyDirection` objects. """ return self._directions @property def trials(self) -> list[FrozenTrial]: """Return all trials in the study. The returned trials are ordered by trial number. This is a short form of ``self.get_trials(deepcopy=True, states=None)``. Returns: A list of :class:`~optuna.trial.FrozenTrial` objects. .. seealso:: See :func:`~optuna.study.Study.get_trials` for related method. """ return self.get_trials(deepcopy=True, states=None) def get_trials( self, deepcopy: bool = True, states: Container[TrialState] | None = None, ) -> list[FrozenTrial]: """Return all trials in the study. The returned trials are ordered by trial number. .. seealso:: See :attr:`~optuna.study.Study.trials` for related property. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) trials = study.get_trials() assert len(trials) == 3 Args: deepcopy: Flag to control whether to apply ``copy.deepcopy()`` to the trials. Note that if you set the flag to :obj:`False`, you shouldn't mutate any fields of the returned trial. Otherwise the internal state of the study may corrupt and unexpected behavior may happen. states: Trial states to filter on. If :obj:`None`, include all states. Returns: A list of :class:`~optuna.trial.FrozenTrial` objects. """ return self._get_trials(deepcopy, states, use_cache=False) def _get_trials( self, deepcopy: bool = True, states: Container[TrialState] | None = None, use_cache: bool = False, ) -> list[FrozenTrial]: if use_cache: if self._thread_local.cached_all_trials is None: self._thread_local.cached_all_trials = self._storage.get_all_trials( self._study_id, deepcopy=False ) trials = self._thread_local.cached_all_trials if states is not None: filtered_trials = [t for t in trials if t.state in states] else: filtered_trials = trials return copy.deepcopy(filtered_trials) if deepcopy else filtered_trials return self._storage.get_all_trials(self._study_id, deepcopy=deepcopy, states=states) @property def user_attrs(self) -> dict[str, Any]: """Return user attributes. .. seealso:: See :func:`~optuna.study.Study.set_user_attr` for related method. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x**2 + y**2 study = optuna.create_study() study.set_user_attr("objective function", "quadratic function") study.set_user_attr("dimensions", 2) study.set_user_attr("contributors", ["Akiba", "Sano"]) assert study.user_attrs == { "objective function": "quadratic function", "dimensions": 2, "contributors": ["Akiba", "Sano"], } Returns: A dictionary containing all user attributes. """ return copy.deepcopy(self._storage.get_study_user_attrs(self._study_id)) @property @deprecated_func("3.1.0", "5.0.0") def system_attrs(self) -> dict[str, Any]: """Return system attributes. Returns: A dictionary containing all system attributes. """ return copy.deepcopy(self._storage.get_study_system_attrs(self._study_id)) @property def metric_names(self) -> list[str] | None: """Return metric names. .. note:: Use :meth:`~optuna.study.Study.set_metric_names` to set the metric names first. Returns: A list with names for each dimension of the returned values of the objective function. """ return self._storage.get_study_system_attrs(self._study_id).get(_SYSTEM_ATTR_METRIC_NAMES) def optimize( self, func: ObjectiveFuncType, n_trials: int | None = None, timeout: float | None = None, n_jobs: int = 1, catch: Iterable[type[Exception]] | type[Exception] = (), callbacks: list[Callable[["Study", FrozenTrial], None]] | None = None, gc_after_trial: bool = False, show_progress_bar: bool = False, ) -> None: """Optimize an objective function. Optimization is done by choosing a suitable set of hyperparameter values from a given range. Uses a sampler which implements the task of value suggestion based on a specified distribution. The sampler is specified in :func:`~optuna.study.create_study` and the default choice for the sampler is TPE. See also :class:`~optuna.samplers.TPESampler` for more details on 'TPE'. Optimization will be stopped when receiving a termination signal such as SIGINT and SIGTERM. Unlike other signals, a trial is automatically and cleanly failed when receiving SIGINT (Ctrl+C). If ``n_jobs`` is greater than one or if another signal than SIGINT is used, the interrupted trial state won't be properly updated. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) Args: func: A callable that implements objective function. n_trials: The number of trials for each process. :obj:`None` represents no limit in terms of the number of trials. The study continues to create trials until the number of trials reaches ``n_trials``, ``timeout`` period elapses, :func:`~optuna.study.Study.stop` is called, or a termination signal such as SIGTERM or Ctrl+C is received. .. seealso:: :class:`optuna.study.MaxTrialsCallback` can ensure how many times trials will be performed across all processes. timeout: Stop study after the given number of second(s). :obj:`None` represents no limit in terms of elapsed time. The study continues to create trials until the number of trials reaches ``n_trials``, ``timeout`` period elapses, :func:`~optuna.study.Study.stop` is called or, a termination signal such as SIGTERM or Ctrl+C is received. n_jobs: The number of parallel jobs. If this argument is set to ``-1``, the number is set to CPU count. .. note:: ``n_jobs`` allows parallelization using :obj:`threading` and may suffer from `Python's GIL `_. It is recommended to use :ref:`process-based parallelization` if ``func`` is CPU bound. catch: A study continues to run even when a trial raises one of the exceptions specified in this argument. Default is an empty tuple, i.e. the study will stop for any exception except for :class:`~optuna.exceptions.TrialPruned`. callbacks: List of callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial`. .. seealso:: See the tutorial of :ref:`optuna_callback` for how to use and implement callback functions. gc_after_trial: Flag to determine whether to automatically run garbage collection after each trial. Set to :obj:`True` to run the garbage collection, :obj:`False` otherwise. When it runs, it runs a full collection by internally calling :func:`gc.collect`. If you see an increase in memory consumption over several trials, try setting this flag to :obj:`True`. .. seealso:: :ref:`out-of-memory-gc-collect` show_progress_bar: Flag to show progress bars or not. To disable progress bar, set this :obj:`False`. Currently, progress bar is experimental feature and disabled when ``n_trials`` is :obj:`None`, ``timeout`` is not :obj:`None`, and ``n_jobs`` :math:`\\ne 1`. Raises: RuntimeError: If nested invocation of this method occurs. """ _optimize( study=self, func=func, n_trials=n_trials, timeout=timeout, n_jobs=n_jobs, catch=tuple(catch) if isinstance(catch, Iterable) else (catch,), callbacks=callbacks, gc_after_trial=gc_after_trial, show_progress_bar=show_progress_bar, ) def ask( self, fixed_distributions: dict[str, BaseDistribution] | None = None ) -> trial_module.Trial: """Create a new trial from which hyperparameters can be suggested. This method is part of an alternative to :func:`~optuna.study.Study.optimize` that allows controlling the lifetime of a trial outside the scope of ``func``. Each call to this method should be followed by a call to :func:`~optuna.study.Study.tell` to finish the created trial. .. seealso:: The :ref:`ask_and_tell` tutorial provides use-cases with examples. Example: Getting the trial object with the :func:`~optuna.study.Study.ask` method. .. testcode:: import optuna study = optuna.create_study() trial = study.ask() x = trial.suggest_float("x", -1, 1) study.tell(trial, x**2) Example: Passing previously defined distributions to the :func:`~optuna.study.Study.ask` method. .. testcode:: import optuna study = optuna.create_study() distributions = { "optimizer": optuna.distributions.CategoricalDistribution(["adam", "sgd"]), "lr": optuna.distributions.FloatDistribution(0.0001, 0.1, log=True), } # You can pass the distributions previously defined. trial = study.ask(fixed_distributions=distributions) # `optimizer` and `lr` are already suggested and accessible with `trial.params`. assert "optimizer" in trial.params assert "lr" in trial.params Args: fixed_distributions: A dictionary containing the parameter names and parameter's distributions. Each parameter in this dictionary is automatically suggested for the returned trial, even when the suggest method is not explicitly invoked by the user. If this argument is set to :obj:`None`, no parameter is automatically suggested. Returns: A :class:`~optuna.trial.Trial`. """ if not self._thread_local.in_optimize_loop and is_heartbeat_enabled(self._storage): warnings.warn("Heartbeat of storage is supposed to be used with Study.optimize.") fixed_distributions = fixed_distributions or {} fixed_distributions = { key: _convert_old_distribution_to_new_distribution(dist) for key, dist in fixed_distributions.items() } # Sync storage once every trial. self._thread_local.cached_all_trials = None trial_id = self._pop_waiting_trial_id() if trial_id is None: trial_id = self._storage.create_new_trial(self._study_id) trial = trial_module.Trial(self, trial_id) for name, param in fixed_distributions.items(): trial._suggest(name, param) return trial def tell( self, trial: trial_module.Trial | int, values: float | Sequence[float] | None = None, state: TrialState | None = None, skip_if_finished: bool = False, ) -> FrozenTrial: """Finish a trial created with :func:`~optuna.study.Study.ask`. .. seealso:: The :ref:`ask_and_tell` tutorial provides use-cases with examples. Example: .. testcode:: import optuna from optuna.trial import TrialState def f(x): return (x - 2) ** 2 def df(x): return 2 * x - 4 study = optuna.create_study() n_trials = 30 for _ in range(n_trials): trial = study.ask() lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) # Iterative gradient descent objective function. x = 3 # Initial value. for step in range(128): y = f(x) trial.report(y, step=step) if trial.should_prune(): # Finish the trial with the pruned state. study.tell(trial, state=TrialState.PRUNED) break gy = df(x) x -= gy * lr else: # Finish the trial with the final value after all iterations. study.tell(trial, y) Args: trial: A :class:`~optuna.trial.Trial` object or a trial number. values: Optional objective value or a sequence of such values in case the study is used for multi-objective optimization. Argument must be provided if ``state`` is :class:`~optuna.trial.TrialState.COMPLETE` and should be :obj:`None` if ``state`` is :class:`~optuna.trial.TrialState.FAIL` or :class:`~optuna.trial.TrialState.PRUNED`. state: State to be reported. Must be :obj:`None`, :class:`~optuna.trial.TrialState.COMPLETE`, :class:`~optuna.trial.TrialState.FAIL` or :class:`~optuna.trial.TrialState.PRUNED`. If ``state`` is :obj:`None`, it will be updated to :class:`~optuna.trial.TrialState.COMPLETE` or :class:`~optuna.trial.TrialState.FAIL` depending on whether validation for ``values`` reported succeed or not. skip_if_finished: Flag to control whether exception should be raised when values for already finished trial are told. If :obj:`True`, tell is skipped without any error when the trial is already finished. Returns: A :class:`~optuna.trial.FrozenTrial` representing the resulting trial. A returned trial is deep copied thus user can modify it as needed. """ return _tell_with_warning( study=self, trial=trial, value_or_values=values, state=state, skip_if_finished=skip_if_finished, ) def set_user_attr(self, key: str, value: Any) -> None: """Set a user attribute to the study. .. seealso:: See :attr:`~optuna.study.Study.user_attrs` for related attribute. .. seealso:: See the recipe on :ref:`attributes`. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x**2 + y**2 study = optuna.create_study() study.set_user_attr("objective function", "quadratic function") study.set_user_attr("dimensions", 2) study.set_user_attr("contributors", ["Akiba", "Sano"]) assert study.user_attrs == { "objective function": "quadratic function", "dimensions": 2, "contributors": ["Akiba", "Sano"], } Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self._storage.set_study_user_attr(self._study_id, key, value) @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: """Set a system attribute to the study. Note that Optuna internally uses this method to save system messages. Please use :func:`~optuna.study.Study.set_user_attr` to set users' attributes. Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self._storage.set_study_system_attr(self._study_id, key, value) def trials_dataframe( self, attrs: tuple[str, ...] = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "system_attrs", "state", ), multi_index: bool = False, ) -> "pd.DataFrame": """Export trials as a pandas DataFrame_. The DataFrame_ provides various features to analyze studies. It is also useful to draw a histogram of objective values and to export trials as a CSV file. If there are no trials, an empty DataFrame_ is returned. Example: .. testcode:: import optuna import pandas def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) # Create a dataframe from the study. df = study.trials_dataframe() assert isinstance(df, pandas.DataFrame) assert df.shape[0] == 3 # n_trials. Args: attrs: Specifies field names of :class:`~optuna.trial.FrozenTrial` to include them to a DataFrame of trials. multi_index: Specifies whether the returned DataFrame_ employs MultiIndex_ or not. Columns that are hierarchical by nature such as ``(params, x)`` will be flattened to ``params_x`` when set to :obj:`False`. Returns: A pandas DataFrame_ of trials in the :class:`~optuna.study.Study`. .. _DataFrame: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html .. _MultiIndex: https://pandas.pydata.org/pandas-docs/stable/advanced.html Note: If ``value`` is in ``attrs`` during multi-objective optimization, it is implicitly replaced with ``values``. Note: If :meth:`~optuna.study.Study.set_metric_names` is called, the ``value`` or ``values`` is implicitly replaced with the dictionary with the objective name as key and the objective value as value. """ return _dataframe._trials_dataframe(self, attrs, multi_index) def stop(self) -> None: """Exit from the current optimization loop after the running trials finish. This method lets the running :meth:`~optuna.study.Study.optimize` method return immediately after all trials which the :meth:`~optuna.study.Study.optimize` method spawned finishes. This method does not affect any behaviors of parallel or successive study processes. This method only works when it is called inside an objective function or callback. Example: .. testcode:: import optuna def objective(trial): if trial.number == 4: trial.study.stop() x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=10) assert len(study.trials) == 5 """ if not self._thread_local.in_optimize_loop: raise RuntimeError( "`Study.stop` is supposed to be invoked inside an objective function or a " "callback." ) self._stop_flag = True def enqueue_trial( self, params: dict[str, Any], user_attrs: dict[str, Any] | None = None, skip_if_exists: bool = False, ) -> None: """Enqueue a trial with given parameter values. You can fix the next sampling parameters which will be evaluated in your objective function. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.enqueue_trial({"x": 5}) study.enqueue_trial({"x": 0}, user_attrs={"memo": "optimal"}) study.optimize(objective, n_trials=2) assert study.trials[0].params == {"x": 5} assert study.trials[1].params == {"x": 0} assert study.trials[1].user_attrs == {"memo": "optimal"} Args: params: Parameter values to pass your objective function. user_attrs: A dictionary of user-specific attributes other than ``params``. skip_if_exists: When :obj:`True`, prevents duplicate trials from being enqueued again. .. note:: This method might produce duplicated trials if called simultaneously by multiple processes at the same time with same ``params`` dict. .. seealso:: Please refer to :ref:`enqueue_trial_tutorial` for the tutorial of specifying hyperparameters manually. """ if skip_if_exists and self._should_skip_enqueue(params): _logger.info(f"Trial with params {params} already exists. Skipping enqueue.") return self.add_trial( create_trial( state=TrialState.WAITING, system_attrs={"fixed_params": params}, user_attrs=user_attrs, ) ) def add_trial(self, trial: FrozenTrial) -> None: """Add trial to study. The trial is validated before being added. Example: .. testcode:: import optuna from optuna.distributions import FloatDistribution def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() assert len(study.trials) == 0 trial = optuna.trial.create_trial( params={"x": 2.0}, distributions={"x": FloatDistribution(0, 10)}, value=4.0, ) study.add_trial(trial) assert len(study.trials) == 1 study.optimize(objective, n_trials=3) assert len(study.trials) == 4 other_study = optuna.create_study() for trial in study.trials: other_study.add_trial(trial) assert len(other_study.trials) == len(study.trials) other_study.optimize(objective, n_trials=2) assert len(other_study.trials) == len(study.trials) + 2 .. seealso:: This method should in general be used to add already evaluated trials (``trial.state.is_finished() == True``). To queue trials for evaluation, please refer to :func:`~optuna.study.Study.enqueue_trial`. .. seealso:: See :func:`~optuna.trial.create_trial` for how to create trials. .. seealso:: Please refer to :ref:`add_trial_tutorial` for the tutorial of specifying hyperparameters with the evaluated value manually. Args: trial: Trial to add. """ trial._validate() if trial.values is not None and len(self.directions) != len(trial.values): raise ValueError( f"The added trial has {len(trial.values)} values, which is different from the " f"number of objectives {len(self.directions)} in the study (determined by " "Study.directions)." ) self._storage.create_new_trial(self._study_id, template_trial=trial) def add_trials(self, trials: Iterable[FrozenTrial]) -> None: """Add trials to study. The trials are validated before being added. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) assert len(study.trials) == 3 other_study = optuna.create_study() other_study.add_trials(study.trials) assert len(other_study.trials) == len(study.trials) other_study.optimize(objective, n_trials=2) assert len(other_study.trials) == len(study.trials) + 2 .. seealso:: See :func:`~optuna.study.Study.add_trial` for addition of each trial. Args: trials: Trials to add. """ for trial in trials: self.add_trial(trial) @experimental_func("3.2.0") def set_metric_names(self, metric_names: list[str]) -> None: """Set metric names. This method names each dimension of the returned values of the objective function. It is particularly useful in multi-objective optimization. The metric names are mainly referenced by the visualization functions. Example: .. testcode:: import optuna import pandas def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2, x + 1 study = optuna.create_study(directions=["minimize", "minimize"]) study.set_metric_names(["x**2", "x+1"]) study.optimize(objective, n_trials=3) df = study.trials_dataframe(multi_index=True) assert isinstance(df, pandas.DataFrame) assert list(df.get("values").keys()) == ["x**2", "x+1"] .. seealso:: The names set by this method are used in :meth:`~optuna.study.Study.trials_dataframe` and :func:`~optuna.visualization.plot_pareto_front`. Args: metric_names: A list of metric names for the objective function. """ if len(self.directions) != len(metric_names): raise ValueError("The number of objectives must match the length of the metric names.") self._storage.set_study_system_attr( self._study_id, _SYSTEM_ATTR_METRIC_NAMES, metric_names ) def _is_multi_objective(self) -> bool: """Return :obj:`True` if the study has multiple objectives. Returns: A boolean value indicates if `self.directions` has more than 1 element or not. """ return len(self.directions) > 1 def _pop_waiting_trial_id(self) -> int | None: for trial in self._storage.get_all_trials( self._study_id, deepcopy=False, states=(TrialState.WAITING,) ): if not self._storage.set_trial_state_values(trial._trial_id, state=TrialState.RUNNING): continue _logger.debug("Trial {} popped from the trial queue.".format(trial.number)) return trial._trial_id return None def _should_skip_enqueue(self, params: Mapping[str, JSONSerializable]) -> bool: for trial in self.get_trials(deepcopy=False): trial_params = trial.system_attrs.get("fixed_params", trial.params) if trial_params.keys() != params.keys(): # Can't have repeated trials if different params are suggested. continue repeated_params: list[bool] = [] for param_name, param_value in params.items(): existing_param = trial_params[param_name] if not isinstance(param_value, type(existing_param)): # Enqueued param has distribution that does not match existing param # (e.g. trying to enqueue categorical to float param). # We are not doing anything about it here, since sanitization should # be handled regardless if `skip_if_exists` is `True`. repeated_params.append(False) continue is_repeated = ( np.isnan(float(param_value)) or np.isclose(float(param_value), float(existing_param), atol=0.0) if isinstance(param_value, Real) else param_value == existing_param ) repeated_params.append(is_repeated) if all(repeated_params): return True return False @deprecated_func("2.5.0", "4.0.0") def _ask(self) -> trial_module.Trial: return self.ask() @deprecated_func("2.5.0", "4.0.0") def _tell( self, trial: trial_module.Trial, state: TrialState, values: list[float] | None ) -> None: self.tell(trial, values, state) def _log_completed_trial(self, trial: trial_module.FrozenTrial) -> None: if not _logger.isEnabledFor(logging.INFO): return metric_names = self.metric_names if len(trial.values) > 1: trial_values: list[float] | dict[str, float] if metric_names is None: trial_values = trial.values else: trial_values = {name: value for name, value in zip(metric_names, trial.values)} _logger.info( "Trial {} finished with values: {} and parameters: {}. ".format( trial.number, trial_values, trial.params ) ) elif len(trial.values) == 1: best_trial = self.best_trial trial_value: float | dict[str, float] if metric_names is None: trial_value = trial.values[0] else: trial_value = {metric_names[0]: trial.values[0]} _logger.info( "Trial {} finished with value: {} and parameters: {}. " "Best is trial {} with value: {}.".format( trial.number, trial_value, trial.params, best_trial.number, best_trial.value, ) ) else: assert False, "Should not reach." @convert_positional_args( previous_positional_arg_names=[ "storage", "sampler", "pruner", "study_name", "direction", "load_if_exists", ], ) def create_study( *, storage: str | storages.BaseStorage | None = None, sampler: "samplers.BaseSampler" | None = None, pruner: pruners.BasePruner | None = None, study_name: str | None = None, direction: str | StudyDirection | None = None, load_if_exists: bool = False, directions: Sequence[str | StudyDirection] | None = None, ) -> Study: """Create a new :class:`~optuna.study.Study`. Example: .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) Args: storage: Database URL. If this argument is set to None, in-memory storage is used, and the :class:`~optuna.study.Study` will not be persistent. .. note:: When a database URL is passed, Optuna internally uses `SQLAlchemy`_ to handle the database. Please refer to `SQLAlchemy's document`_ for further details. If you want to specify non-default options to `SQLAlchemy Engine`_, you can instantiate :class:`~optuna.storages.RDBStorage` with your desired options and pass it to the ``storage`` argument instead of a URL. .. _SQLAlchemy: https://www.sqlalchemy.org/ .. _SQLAlchemy's document: https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls .. _SQLAlchemy Engine: https://docs.sqlalchemy.org/en/latest/core/engines.html sampler: A sampler object that implements background algorithm for value suggestion. If :obj:`None` is specified, :class:`~optuna.samplers.TPESampler` is used during single-objective optimization and :class:`~optuna.samplers.NSGAIISampler` during multi-objective optimization. See also :class:`~optuna.samplers`. pruner: A pruner object that decides early stopping of unpromising trials. If :obj:`None` is specified, :class:`~optuna.pruners.MedianPruner` is used as the default. See also :class:`~optuna.pruners`. study_name: Study's name. If this argument is set to None, a unique name is generated automatically. direction: Direction of optimization. Set ``minimize`` for minimization and ``maximize`` for maximization. You can also pass the corresponding :class:`~optuna.study.StudyDirection` object. ``direction`` and ``directions`` must not be specified at the same time. .. note:: If none of `direction` and `directions` are specified, the direction of the study is set to "minimize". load_if_exists: Flag to control the behavior to handle a conflict of study names. In the case where a study named ``study_name`` already exists in the ``storage``, a :class:`~optuna.exceptions.DuplicatedStudyError` is raised if ``load_if_exists`` is set to :obj:`False`. Otherwise, the creation of the study is skipped, and the existing one is returned. directions: A sequence of directions during multi-objective optimization. ``direction`` and ``directions`` must not be specified at the same time. Returns: A :class:`~optuna.study.Study` object. See also: :func:`optuna.create_study` is an alias of :func:`optuna.study.create_study`. See also: The :ref:`rdb` tutorial provides concrete examples to save and resume optimization using RDB. """ if direction is None and directions is None: directions = ["minimize"] elif direction is not None and directions is not None: raise ValueError("Specify only one of `direction` and `directions`.") elif direction is not None: directions = [direction] elif directions is not None: directions = list(directions) else: assert False if len(directions) < 1: raise ValueError("The number of objectives must be greater than 0.") elif any( d not in ["minimize", "maximize", StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] for d in directions ): raise ValueError( "Please set either 'minimize' or 'maximize' to direction. You can also set the " "corresponding `StudyDirection` member." ) direction_objects = [ d if isinstance(d, StudyDirection) else StudyDirection[d.upper()] for d in directions ] storage = storages.get_storage(storage) try: study_id = storage.create_new_study(direction_objects, study_name) except exceptions.DuplicatedStudyError: if load_if_exists: assert study_name is not None _logger.info( "Using an existing study with name '{}' instead of " "creating a new one.".format(study_name) ) study_id = storage.get_study_id_from_name(study_name) else: raise if sampler is None and len(direction_objects) > 1: sampler = samplers.NSGAIISampler() study_name = storage.get_study_name_from_id(study_id) study = Study(study_name=study_name, storage=storage, sampler=sampler, pruner=pruner) return study @convert_positional_args( previous_positional_arg_names=[ "study_name", "storage", "sampler", "pruner", ], ) def load_study( *, study_name: str | None, storage: str | storages.BaseStorage, sampler: "samplers.BaseSampler" | None = None, pruner: pruners.BasePruner | None = None, ) -> Study: """Load the existing :class:`~optuna.study.Study` that has the specified name. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 10) return x**2 study = optuna.create_study(storage="sqlite:///example.db", study_name="my_study") study.optimize(objective, n_trials=3) loaded_study = optuna.load_study(study_name="my_study", storage="sqlite:///example.db") assert len(loaded_study.trials) == len(study.trials) .. testcleanup:: os.remove("example.db") Args: study_name: Study's name. Each study has a unique name as an identifier. If :obj:`None`, checks whether the storage contains a single study, and if so loads that study. ``study_name`` is required if there are multiple studies in the storage. storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. sampler: A sampler object that implements background algorithm for value suggestion. If :obj:`None` is specified, :class:`~optuna.samplers.TPESampler` is used as the default. See also :class:`~optuna.samplers`. pruner: A pruner object that decides early stopping of unpromising trials. If :obj:`None` is specified, :class:`~optuna.pruners.MedianPruner` is used as the default. See also :class:`~optuna.pruners`. See also: :func:`optuna.load_study` is an alias of :func:`optuna.study.load_study`. """ if study_name is None: study_names = get_all_study_names(storage) if len(study_names) != 1: raise ValueError( f"Could not determine the study name since the storage {storage} does not " "contain exactly 1 study. Specify `study_name`." ) study_name = study_names[0] _logger.info( f"Study name was omitted but trying to load '{study_name}' because that was the only " "study found in the storage." ) return Study(study_name=study_name, storage=storage, sampler=sampler, pruner=pruner) @convert_positional_args( previous_positional_arg_names=[ "study_name", "storage", ], ) def delete_study( *, study_name: str, storage: str | storages.BaseStorage, ) -> None: """Delete a :class:`~optuna.study.Study` object. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db") study.optimize(objective, n_trials=3) optuna.delete_study(study_name="example-study", storage="sqlite:///example.db") .. testcleanup:: os.remove("example.db") Args: study_name: Study's name. storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. See also: :func:`optuna.delete_study` is an alias of :func:`optuna.study.delete_study`. """ storage = storages.get_storage(storage) study_id = storage.get_study_id_from_name(study_name) storage.delete_study(study_id) @convert_positional_args( previous_positional_arg_names=[ "from_study_name", "from_storage", "to_storage", "to_study_name", ], warning_stacklevel=3, ) def copy_study( *, from_study_name: str, from_storage: str | storages.BaseStorage, to_storage: str | storages.BaseStorage, to_study_name: str | None = None, ) -> None: """Copy study from one storage to another. The direction(s) of the objective(s) in the study, trials, user attributes and system attributes are copied. .. note:: :func:`~optuna.copy_study` copies a study even if the optimization is working on. It means users will get a copied study that contains a trial that is not finished. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") if os.path.exists("example_copy.db"): raise RuntimeError("'example_copy.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study( study_name="example-study", storage="sqlite:///example.db", ) study.optimize(objective, n_trials=3) optuna.copy_study( from_study_name="example-study", from_storage="sqlite:///example.db", to_storage="sqlite:///example_copy.db", ) study = optuna.load_study( study_name=None, storage="sqlite:///example_copy.db", ) .. testcleanup:: os.remove("example.db") os.remove("example_copy.db") Args: from_study_name: Name of study. from_storage: Source database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. to_storage: Destination database URL. to_study_name: Name of the created study. If omitted, ``from_study_name`` is used. Raises: :class:`~optuna.exceptions.DuplicatedStudyError`: If a study with a conflicting name already exists in the destination storage. """ from_study = load_study(study_name=from_study_name, storage=from_storage) to_study = create_study( study_name=to_study_name or from_study_name, storage=to_storage, directions=from_study.directions, load_if_exists=False, ) for key, value in from_study._storage.get_study_system_attrs(from_study._study_id).items(): to_study._storage.set_study_system_attr(to_study._study_id, key, value) for key, value in from_study.user_attrs.items(): to_study.set_user_attr(key, value) # Trials are deep copied on `add_trials`. to_study.add_trials(from_study.get_trials(deepcopy=False)) def get_all_study_summaries( storage: str | storages.BaseStorage, include_best_trial: bool = True ) -> list[StudySummary]: """Get all history of studies stored in a specified storage. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db") study.optimize(objective, n_trials=3) study_summaries = optuna.study.get_all_study_summaries(storage="sqlite:///example.db") assert len(study_summaries) == 1 study_summary = study_summaries[0] assert study_summary.study_name == "example-study" .. testcleanup:: os.remove("example.db") Args: storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. include_best_trial: Include the best trials if exist. It potentially increases the number of queries and may take longer to fetch summaries depending on the storage. Returns: List of study history summarized as :class:`~optuna.study.StudySummary` objects. See also: :func:`optuna.get_all_study_summaries` is an alias of :func:`optuna.study.get_all_study_summaries`. """ storage = storages.get_storage(storage) frozen_studies = storage.get_all_studies() study_summaries = [] for s in frozen_studies: all_trials = storage.get_all_trials(s._study_id) completed_trials = [t for t in all_trials if t.state == TrialState.COMPLETE] n_trials = len(all_trials) if len(s.directions) == 1: direction = s.direction directions = None if include_best_trial and len(completed_trials) != 0: if direction == StudyDirection.MAXIMIZE: best_trial = max(completed_trials, key=lambda t: cast(float, t.value)) else: best_trial = min(completed_trials, key=lambda t: cast(float, t.value)) else: best_trial = None else: direction = None directions = s.directions best_trial = None datetime_start = min( [t.datetime_start for t in all_trials if t.datetime_start is not None], default=None ) study_summaries.append( StudySummary( study_name=s.study_name, direction=direction, best_trial=best_trial, user_attrs=s.user_attrs, system_attrs=s.system_attrs, n_trials=n_trials, datetime_start=datetime_start, study_id=s._study_id, directions=directions, ) ) return study_summaries def get_all_study_names(storage: str | storages.BaseStorage) -> list[str]: """Get all study names stored in a specified storage. Example: .. testsetup:: import os if os.path.exists("example.db"): raise RuntimeError("'example.db' already exists. Please remove it.") .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db") study.optimize(objective, n_trials=3) study_names = optuna.study.get_all_study_names(storage="sqlite:///example.db") assert len(study_names) == 1 assert study_names[0] == "example-study" .. testcleanup:: os.remove("example.db") Args: storage: Database URL such as ``sqlite:///example.db``. Please see also the documentation of :func:`~optuna.study.create_study` for further details. Returns: List of all study names in the storage. See also: :func:`optuna.get_all_study_names` is an alias of :func:`optuna.study.get_all_study_names`. """ storage = storages.get_storage(storage) study_names = [study.study_name for study in storage.get_all_studies()] return study_names optuna-3.5.0/optuna/terminator/000077500000000000000000000000001453453102400165205ustar00rootroot00000000000000optuna-3.5.0/optuna/terminator/__init__.py000066400000000000000000000017051453453102400206340ustar00rootroot00000000000000from optuna.terminator.callback import TerminatorCallback from optuna.terminator.erroreval import BaseErrorEvaluator from optuna.terminator.erroreval import CrossValidationErrorEvaluator from optuna.terminator.erroreval import report_cross_validation_scores from optuna.terminator.erroreval import StaticErrorEvaluator from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.terminator.improvement.evaluator import RegretBoundEvaluator from optuna.terminator.terminator import BaseTerminator from optuna.terminator.terminator import Terminator __all__ = [ "TerminatorCallback", "BaseErrorEvaluator", "CrossValidationErrorEvaluator", "report_cross_validation_scores", "StaticErrorEvaluator", "BaseImprovementEvaluator", "BestValueStagnationEvaluator", "RegretBoundEvaluator", "BaseTerminator", "Terminator", ] optuna-3.5.0/optuna/terminator/callback.py000066400000000000000000000053271453453102400206350ustar00rootroot00000000000000from __future__ import annotations from typing import Optional from optuna._experimental import experimental_class from optuna.logging import get_logger from optuna.study.study import Study from optuna.terminator.terminator import BaseTerminator from optuna.terminator.terminator import Terminator from optuna.trial import FrozenTrial _logger = get_logger(__name__) @experimental_class("3.2.0") class TerminatorCallback: """A callback that terminates the optimization using Terminator. This class implements a callback which wraps :class:`~optuna.terminator.Terminator` so that it can be used with the :func:`~optuna.study.Study.optimize` method. Args: terminator: A terminator object which determines whether to terminate the optimization by assessing the room for optimization and statistical error. Defaults to a :class:`~optuna.terminator.Terminator` object with default ``improvement_evaluator`` and ``error_evaluator``. Example: .. testcode:: from sklearn.datasets import load_wine from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import TerminatorCallback from optuna.terminator import report_cross_validation_scores def objective(trial): X, y = load_wine(return_X_y=True) clf = RandomForestClassifier( max_depth=trial.suggest_int("max_depth", 2, 32), min_samples_split=trial.suggest_float("min_samples_split", 0, 1), criterion=trial.suggest_categorical("criterion", ("gini", "entropy")), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) return scores.mean() study = optuna.create_study(direction="maximize") terminator = TerminatorCallback() study.optimize(objective, n_trials=50, callbacks=[terminator]) .. seealso:: Please refer to :class:`~optuna.terminator.Terminator` for the details of the terminator mechanism. """ def __init__( self, terminator: Optional[BaseTerminator] = None, ) -> None: self._terminator = terminator or Terminator() def __call__(self, study: Study, trial: FrozenTrial) -> None: should_terminate = self._terminator.should_terminate(study=study) if should_terminate: _logger.info("The study has been stopped by the terminator.") study.stop() optuna-3.5.0/optuna/terminator/erroreval.py000066400000000000000000000100611453453102400210710ustar00rootroot00000000000000from __future__ import annotations import abc from typing import cast import numpy as np from optuna._experimental import experimental_class from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial._state import TrialState _CROSS_VALIDATION_SCORES_KEY = "terminator:cv_scores" class BaseErrorEvaluator(metaclass=abc.ABCMeta): """Base class for error evaluators.""" @abc.abstractmethod def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: pass @experimental_class("3.2.0") class CrossValidationErrorEvaluator(BaseErrorEvaluator): """An error evaluator for objective functions based on cross-validation. This evaluator evaluates the objective function's statistical error, which comes from the randomness of dataset. This evaluator assumes that the objective function is the average of the cross-validation and uses the scaled variance of the cross-validation scores in the best trial at the moment as the statistical error. """ def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: """Evaluate the statistical error of the objective function based on cross-validation. Args: trials: A list of trials to consider. The best trial in ``trials`` is used to compute the statistical error. study_direction: The direction of the study. Returns: A float representing the statistical error of the objective function. """ trials = [trial for trial in trials if trial.state == TrialState.COMPLETE] assert len(trials) > 0 if study_direction == StudyDirection.MAXIMIZE: best_trial = max(trials, key=lambda t: cast(float, t.value)) else: best_trial = min(trials, key=lambda t: cast(float, t.value)) best_trial_attrs = best_trial.system_attrs if _CROSS_VALIDATION_SCORES_KEY in best_trial_attrs: cv_scores = best_trial_attrs[_CROSS_VALIDATION_SCORES_KEY] else: raise ValueError( "Cross-validation scores have not been reported. Please call " "`report_cross_validation_scores(trial, scores)` during a trial and pass the " "list of scores as `scores`." ) k = len(cv_scores) assert k > 1, "Should be guaranteed by `report_cross_validation_scores`." scale = 1 / k + 1 / (k - 1) var = scale * np.var(cv_scores) std = np.sqrt(var) return float(std) @experimental_class("3.2.0") def report_cross_validation_scores(trial: Trial, scores: list[float]) -> None: """A function to report cross-validation scores of a trial. This function should be called within the objective function to report the cross-validation scores. The reported scores are used to evaluate the statistical error for termination judgement. Args: trial: A :class:`~optuna.trial.Trial` object to report the cross-validation scores. scores: The cross-validation scores of the trial. """ if len(scores) <= 1: raise ValueError("The length of `scores` is expected to be greater than one.") trial.storage.set_trial_system_attr(trial._trial_id, _CROSS_VALIDATION_SCORES_KEY, scores) @experimental_class("3.2.0") class StaticErrorEvaluator(BaseErrorEvaluator): """An error evaluator that always returns a constant value. This evaluator can be used to terminate the optimization when the evaluated improvement potential is below the fixed threshold. Args: constant: A user-specified constant value to always return as an error estimate. """ def __init__(self, constant: float) -> None: self._constant = constant def evaluate( self, trials: list[FrozenTrial], study_direction: StudyDirection, ) -> float: return self._constant optuna-3.5.0/optuna/terminator/improvement/000077500000000000000000000000001453453102400210655ustar00rootroot00000000000000optuna-3.5.0/optuna/terminator/improvement/__init__.py000066400000000000000000000000001453453102400231640ustar00rootroot00000000000000optuna-3.5.0/optuna/terminator/improvement/_preprocessing.py000066400000000000000000000170401453453102400244630ustar00rootroot00000000000000import abc from typing import Any from typing import cast from typing import Dict from typing import List from typing import Optional from typing import Tuple import numpy as np import optuna from optuna._transform import _SearchSpaceTransform from optuna.distributions import _is_distribution_log from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.search_space import intersection_search_space from optuna.trial._state import TrialState class BasePreprocessing(metaclass=abc.ABCMeta): @abc.abstractmethod def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: pass class PreprocessingPipeline(BasePreprocessing): def __init__(self, processes: List[BasePreprocessing]) -> None: self._processes = processes def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: for p in self._processes: trials = p.apply(trials, study_direction) return trials class NullPreprocessing(BasePreprocessing): def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: return trials class UnscaleLog(BasePreprocessing): def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: mapped_trials = [] for trial in trials: assert trial.state == optuna.trial.TrialState.COMPLETE params, distributions = {}, {} for param_name in trial.params.keys(): ( param_value, param_distribution, ) = self._convert_param_value_distribution( trial.params[param_name], trial.distributions[param_name] ) params[param_name] = param_value distributions[param_name] = param_distribution trial = optuna.create_trial( value=trial.value, params=params, distributions=distributions, user_attrs=trial.user_attrs, system_attrs=trial.system_attrs, ) mapped_trials.append(trial) return mapped_trials @staticmethod def _convert_param_value_distribution( value: Any, distribution: BaseDistribution ) -> Tuple[Any, BaseDistribution]: if isinstance(distribution, (IntDistribution, FloatDistribution)): if _is_distribution_log(distribution): value = np.log(value) low = np.log(distribution.low) high = np.log(distribution.high) distribution = FloatDistribution(low=low, high=high) return value, distribution return value, distribution class SelectTopTrials(BasePreprocessing): def __init__( self, top_trials_ratio: float, min_n_trials: int, ) -> None: self._top_trials_ratio = top_trials_ratio self._min_n_trials = min_n_trials def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: trials = [trial for trial in trials if trial.state == optuna.trial.TrialState.COMPLETE] trials = sorted(trials, key=lambda t: cast(float, t.value)) if study_direction == optuna.study.StudyDirection.MAXIMIZE: trials = list(reversed(trials)) top_n = int(len(trials) * self._top_trials_ratio) top_n = max(top_n, self._min_n_trials) top_n = min(top_n, len(trials)) return trials[:top_n] class ToMinimize(BasePreprocessing): def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: mapped_trials = [] for trial in trials: if study_direction == optuna.study.StudyDirection.MAXIMIZE: value = None if trial.value is None else -trial.value else: value = trial.value trial = optuna.create_trial( value=value, params=trial.params, distributions=trial.distributions, user_attrs=trial.user_attrs, system_attrs=trial.system_attrs, state=trial.state, ) mapped_trials.append(trial) return mapped_trials class OneToHot(BasePreprocessing): def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: mapped_trials = [] for trial in trials: params = {} distributions: Dict[str, BaseDistribution] = {} for param, distribution in trial.distributions.items(): if isinstance(distribution, CategoricalDistribution): ir = distribution.to_internal_repr(trial.params[param]) values = [1.0 if i == ir else 0.0 for i in range(len(distribution.choices))] for i, v in enumerate(values): key = f"i{i}_{param}" params[key] = v distributions[key] = FloatDistribution(0.0, 1.0) else: key = f"i0_{param}" params[key] = trial.params[param] distributions[key] = distribution trial = optuna.create_trial( value=trial.value, params=params, distributions=distributions, user_attrs=trial.user_attrs, system_attrs=trial.system_attrs, state=trial.state, ) mapped_trials.append(trial) return mapped_trials class AddRandomInputs(BasePreprocessing): def __init__( self, n_additional_trials: int, dummy_value: float = np.nan, ) -> None: self._n_additional_trials = n_additional_trials self._dummy_value = dummy_value self._rng = np.random.RandomState() def apply( self, trials: List[optuna.trial.FrozenTrial], study_direction: Optional[optuna.study.StudyDirection], ) -> List[optuna.trial.FrozenTrial]: search_space = intersection_search_space(trials) additional_trials = [] for _ in range(self._n_additional_trials): params = {} for param_name, distribution in search_space.items(): trans = _SearchSpaceTransform({param_name: distribution}) trans_params = self._rng.uniform(trans.bounds[:, 0], trans.bounds[:, 1]) param_value = trans.untransform(trans_params)[param_name] params[param_name] = param_value trial = optuna.create_trial( value=self._dummy_value, params=params, distributions=search_space, state=TrialState.COMPLETE, ) additional_trials.append(trial) return trials + additional_trials optuna-3.5.0/optuna/terminator/improvement/evaluator.py000066400000000000000000000157331453453102400234520ustar00rootroot00000000000000import abc from typing import Dict from typing import List from typing import Optional from optuna._experimental import experimental_class from optuna.distributions import BaseDistribution from optuna.search_space import intersection_search_space from optuna.study import StudyDirection from optuna.terminator.improvement._preprocessing import AddRandomInputs from optuna.terminator.improvement._preprocessing import BasePreprocessing from optuna.terminator.improvement._preprocessing import OneToHot from optuna.terminator.improvement._preprocessing import PreprocessingPipeline from optuna.terminator.improvement._preprocessing import SelectTopTrials from optuna.terminator.improvement._preprocessing import ToMinimize from optuna.terminator.improvement._preprocessing import UnscaleLog from optuna.terminator.improvement.gp.base import _min_lcb from optuna.terminator.improvement.gp.base import _min_ucb from optuna.terminator.improvement.gp.base import BaseGaussianProcess from optuna.terminator.improvement.gp.botorch import _BoTorchGaussianProcess from optuna.trial import FrozenTrial from optuna.trial import TrialState DEFAULT_TOP_TRIALS_RATIO = 0.5 DEFAULT_MIN_N_TRIALS = 20 @experimental_class("3.2.0") class BaseImprovementEvaluator(metaclass=abc.ABCMeta): """Base class for improvement evaluators.""" @abc.abstractmethod def evaluate( self, trials: List[FrozenTrial], study_direction: StudyDirection, ) -> float: pass @experimental_class("3.2.0") class RegretBoundEvaluator(BaseImprovementEvaluator): """An error evaluator for upper bound on the regret with high-probability confidence. This evaluator evaluates the regret of current best solution, which defined as the difference between the objective value of the best solution and of the global optimum. To be specific, this evaluator calculates the upper bound on the regret based on the fact that empirical estimator of the objective function is bounded by lower and upper confidence bounds with high probability under the Gaussian process model assumption. Args: gp: A Gaussian process model on which evaluation base. If not specified, the default Gaussian process model is used. top_trials_ratio: A ratio of top trials to be considered when estimating the regret. Default to 0.5. min_n_trials: A minimum number of complete trials to estimate the regret. Default to 20. min_lcb_n_additional_samples: A minimum number of additional samples to estimate the lower confidence bound. Default to 2000. """ def __init__( self, gp: Optional[BaseGaussianProcess] = None, top_trials_ratio: float = DEFAULT_TOP_TRIALS_RATIO, min_n_trials: int = DEFAULT_MIN_N_TRIALS, min_lcb_n_additional_samples: int = 2000, ) -> None: self._gp = gp or _BoTorchGaussianProcess() self._top_trials_ratio = top_trials_ratio self._min_n_trials = min_n_trials self._min_lcb_n_additional_samples = min_lcb_n_additional_samples def get_preprocessing(self, add_random_inputs: bool = False) -> BasePreprocessing: processes = [ SelectTopTrials( top_trials_ratio=self._top_trials_ratio, min_n_trials=self._min_n_trials, ), UnscaleLog(), ToMinimize(), ] if add_random_inputs: processes += [AddRandomInputs(self._min_lcb_n_additional_samples)] processes += [OneToHot()] return PreprocessingPipeline(processes) def evaluate( self, trials: List[FrozenTrial], study_direction: StudyDirection, ) -> float: search_space = intersection_search_space(trials) self._validate_input(trials, search_space) fit_trials = self.get_preprocessing().apply(trials, study_direction) lcb_trials = self.get_preprocessing(add_random_inputs=True).apply(trials, study_direction) n_params = len(search_space) n_trials = len(fit_trials) self._gp.fit(fit_trials) ucb = _min_ucb(trials=fit_trials, gp=self._gp, n_params=n_params, n_trials=n_trials) lcb = _min_lcb(trials=lcb_trials, gp=self._gp, n_params=n_params, n_trials=n_trials) regret_bound = ucb - lcb return regret_bound @classmethod def _validate_input( cls, trials: List[FrozenTrial], search_space: Dict[str, BaseDistribution] ) -> None: if len([t for t in trials if t.state == TrialState.COMPLETE]) == 0: raise ValueError( "Because no trial has been completed yet, the regret bound cannot be evaluated." ) if len(search_space) == 0: raise ValueError( "The intersection search space is empty. This condition is not supported by " f"{cls.__name__}." ) @experimental_class("3.4.0") class BestValueStagnationEvaluator(BaseImprovementEvaluator): """Evaluates the stagnation period of the best value in an optimization process. This class is initialized with a maximum stagnation period (`max_stagnation_trials`) and is designed to evaluate the remaining trials before reaching this maximum period of allowed stagnation. If this remaining trials reach zero, the trial terminates. Therefore, the default error evaluator is instantiated by StaticErrorEvaluator(const=0). Args: max_stagnation_trials: The maximum number of trials allowed for stagnation. """ def __init__( self, max_stagnation_trials: int = 30, ) -> None: if max_stagnation_trials < 0: raise ValueError("The maximum number of stagnant trials must not be negative.") self._max_stagnation_trials = max_stagnation_trials def evaluate( self, trials: List[FrozenTrial], study_direction: StudyDirection, ) -> float: self._validate_input(trials) is_maximize_direction = True if (study_direction == StudyDirection.MAXIMIZE) else False trials = [t for t in trials if t.state == TrialState.COMPLETE] current_step = len(trials) - 1 best_step = 0 for i, trial in enumerate(trials): best_value = trials[best_step].value current_value = trial.value assert best_value is not None assert current_value is not None if is_maximize_direction and (best_value < current_value): best_step = i elif (not is_maximize_direction) and (best_value > current_value): best_step = i return self._max_stagnation_trials - (current_step - best_step) @classmethod def _validate_input( cls, trials: List[FrozenTrial], ) -> None: if len([t for t in trials if t.state == TrialState.COMPLETE]) == 0: raise ValueError( "Because no trial has been completed yet, the improvement cannot be evaluated." ) optuna-3.5.0/optuna/terminator/improvement/gp/000077500000000000000000000000001453453102400214735ustar00rootroot00000000000000optuna-3.5.0/optuna/terminator/improvement/gp/__init__.py000066400000000000000000000000001453453102400235720ustar00rootroot00000000000000optuna-3.5.0/optuna/terminator/improvement/gp/base.py000066400000000000000000000026361453453102400227660ustar00rootroot00000000000000import abc from typing import List from typing import Tuple import numpy as np from optuna._experimental import experimental_class from optuna.trial._frozen import FrozenTrial @experimental_class("3.2.0") class BaseGaussianProcess(metaclass=abc.ABCMeta): @abc.abstractmethod def fit( self, trials: List[FrozenTrial], ) -> None: pass @abc.abstractmethod def predict_mean_std( self, trials: List[FrozenTrial], ) -> Tuple[np.ndarray, np.ndarray]: pass def _min_ucb( trials: List[FrozenTrial], gp: BaseGaussianProcess, n_params: int, n_trials: int, ) -> float: mean, std = gp.predict_mean_std(trials) upper = mean + std * np.sqrt(_get_beta(n_params=n_params, n_trials=n_trials)) return float(min(upper)) def _min_lcb( trials: List[FrozenTrial], gp: BaseGaussianProcess, n_params: int, n_trials: int, ) -> float: mean, std = gp.predict_mean_std(trials) lower = mean - std * np.sqrt(_get_beta(n_params=n_params, n_trials=n_trials)) return float(min(lower)) def _get_beta(n_params: int, n_trials: int, delta: float = 0.1) -> float: beta = 2 * np.log(n_params * n_trials**2 * np.pi**2 / 6 / delta) # The following div is according to the original paper: "We then further scale it down # by a factor of 5 as defined in the experiments in Srinivas et al. (2010)" beta /= 5 return beta optuna-3.5.0/optuna/terminator/improvement/gp/botorch.py000066400000000000000000000073511453453102400235130ustar00rootroot00000000000000from __future__ import annotations from typing import Optional import numpy as np from packaging import version from optuna._imports import try_import from optuna.distributions import _is_distribution_log from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.search_space import intersection_search_space from optuna.terminator.improvement.gp.base import BaseGaussianProcess from optuna.trial._frozen import FrozenTrial from optuna.trial._state import TrialState with try_import() as _imports: import botorch from botorch.models import SingleTaskGP from botorch.models.transforms import Normalize from botorch.models.transforms import Standardize import gpytorch import torch if version.parse(botorch.version.version) < version.parse("0.8.0"): from botorch.fit import fit_gpytorch_model as fit_gpytorch_mll else: from botorch.fit import fit_gpytorch_mll __all__ = [ "fit_gpytorch_mll", "SingleTaskGP", "Normalize", "Standardize", "gpytorch", "torch", ] class _BoTorchGaussianProcess(BaseGaussianProcess): def __init__(self) -> None: _imports.check() self._gp: Optional[SingleTaskGP] = None def fit( self, trials: list[FrozenTrial], ) -> None: self._trials = trials x, bounds = _convert_trials_to_tensors(trials) n_params = x.shape[1] y = torch.tensor([trial.value for trial in trials], dtype=torch.float64) y = torch.unsqueeze(y, 1) self._gp = SingleTaskGP( x, y, input_transform=Normalize(d=n_params, bounds=bounds), outcome_transform=Standardize(m=1), ) mll = gpytorch.mlls.ExactMarginalLogLikelihood(self._gp.likelihood, self._gp) fit_gpytorch_mll(mll) def predict_mean_std( self, trials: list[FrozenTrial], ) -> tuple[np.ndarray, np.ndarray]: assert self._gp is not None x, _ = _convert_trials_to_tensors(trials) with torch.no_grad(), gpytorch.settings.fast_pred_var(): posterior = self._gp.posterior(x) mean = posterior.mean variance = posterior.variance std = variance.sqrt() return mean.detach().numpy().squeeze(-1), std.detach().numpy().squeeze(-1) def _convert_trials_to_tensors(trials: list[FrozenTrial]) -> tuple[torch.Tensor, torch.Tensor]: """Convert a list of FrozenTrial objects to tensors inputs and bounds. This function assumes the following condition for input trials: - any categorical param is converted to a float or int one; - log is unscaled for any float/int distribution; - the state is COMPLETE for any trial; - direction is MINIMIZE for any trial. """ search_space = intersection_search_space(trials) sorted_params = sorted(search_space.keys()) x = [] for trial in trials: assert trial.state == TrialState.COMPLETE x_row = [] for param in sorted_params: distribution = search_space[param] assert not _is_distribution_log(distribution) assert not isinstance(distribution, CategoricalDistribution) param_value = float(trial.params[param]) x_row.append(param_value) x.append(x_row) min_bounds = [] max_bounds = [] for param, distribution in search_space.items(): assert isinstance(distribution, (FloatDistribution, IntDistribution)) min_bounds.append(distribution.low) max_bounds.append(distribution.high) bounds = [min_bounds, max_bounds] return torch.tensor(x, dtype=torch.float64), torch.tensor(bounds, dtype=torch.float64) optuna-3.5.0/optuna/terminator/terminator.py000066400000000000000000000122251453453102400212600ustar00rootroot00000000000000from __future__ import annotations import abc from typing import Optional from optuna._experimental import experimental_class from optuna.study.study import Study from optuna.terminator.erroreval import BaseErrorEvaluator from optuna.terminator.erroreval import CrossValidationErrorEvaluator from optuna.terminator.erroreval import StaticErrorEvaluator from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.terminator.improvement.evaluator import DEFAULT_MIN_N_TRIALS from optuna.terminator.improvement.evaluator import RegretBoundEvaluator from optuna.trial import TrialState class BaseTerminator(metaclass=abc.ABCMeta): """Base class for terminators.""" @abc.abstractmethod def should_terminate(self, study: Study) -> bool: pass @experimental_class("3.2.0") class Terminator(BaseTerminator): """Automatic stopping mechanism for Optuna studies. This class implements an automatic stopping mechanism for Optuna studies, aiming to prevent unnecessary computation. The study is terminated when the statistical error, e.g. cross-validation error, exceeds the room left for optimization. For further information about the algorithm, please refer to the following paper: - `A. Makarova et al. Automatic termination for hyperparameter optimization. `_ Args: improvement_evaluator: An evaluator object for assessing the room left for optimization. Defaults to a :class:`~optuna.terminator.improvement.evaluator.RegretBoundEvaluator` object. error_evaluator: An evaluator for calculating the statistical error, e.g. cross-validation error. Defaults to a :class:`~optuna.terminator.CrossValidationErrorEvaluator` object. min_n_trials: The minimum number of trials before termination is considered. Defaults to ``20``. Raises: ValueError: If ``min_n_trials`` is not a positive integer. Example: .. testcode:: import logging import sys from sklearn.datasets import load_wine from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import Terminator from optuna.terminator import report_cross_validation_scores study = optuna.create_study(direction="maximize") terminator = Terminator() min_n_trials = 20 while True: trial = study.ask() X, y = load_wine(return_X_y=True) clf = RandomForestClassifier( max_depth=trial.suggest_int("max_depth", 2, 32), min_samples_split=trial.suggest_float("min_samples_split", 0, 1), criterion=trial.suggest_categorical("criterion", ("gini", "entropy")), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) value = scores.mean() logging.info(f"Trial #{trial.number} finished with value {value}.") study.tell(trial, value) if trial.number > min_n_trials and terminator.should_terminate(study): logging.info("Terminated by Optuna Terminator!") break .. seealso:: Please refer to :class:`~optuna.terminator.TerminatorCallback` for how to use the terminator mechanism with the :func:`~optuna.study.Study.optimize` method. """ def __init__( self, improvement_evaluator: Optional[BaseImprovementEvaluator] = None, error_evaluator: Optional[BaseErrorEvaluator] = None, min_n_trials: int = DEFAULT_MIN_N_TRIALS, ) -> None: if min_n_trials <= 0: raise ValueError("`min_n_trials` is expected to be a positive integer.") self._improvement_evaluator = improvement_evaluator or RegretBoundEvaluator() self._error_evaluator = error_evaluator or self._initialize_error_evalutor() self._min_n_trials = min_n_trials def _initialize_error_evalutor(self) -> BaseErrorEvaluator: if isinstance(self._improvement_evaluator, BestValueStagnationEvaluator): return StaticErrorEvaluator(constant=0) return CrossValidationErrorEvaluator() def should_terminate(self, study: Study) -> bool: """Judge whether the study should be terminated based on the reported values.""" trials = study.get_trials(states=[TrialState.COMPLETE]) if len(trials) < self._min_n_trials: return False improvement = self._improvement_evaluator.evaluate( trials=study.trials, study_direction=study.direction, ) error = self._error_evaluator.evaluate( trials=study.trials, study_direction=study.direction ) should_terminate = improvement < error return should_terminate optuna-3.5.0/optuna/testing/000077500000000000000000000000001453453102400160115ustar00rootroot00000000000000optuna-3.5.0/optuna/testing/__init__.py000066400000000000000000000000001453453102400201100ustar00rootroot00000000000000optuna-3.5.0/optuna/testing/distributions.py000066400000000000000000000007411453453102400212670ustar00rootroot00000000000000from __future__ import annotations from typing import Any from optuna.distributions import BaseDistribution class UnsupportedDistribution(BaseDistribution): def single(self) -> bool: return False def _contains(self, param_value_in_internal_repr: float) -> bool: return True def _asdict(self) -> dict: return {} def to_internal_repr(self, param_value_in_external_repr: Any) -> float: return float(param_value_in_external_repr) optuna-3.5.0/optuna/testing/objectives.py000066400000000000000000000003051453453102400205160ustar00rootroot00000000000000from optuna import TrialPruned from optuna.trial import Trial def fail_objective(_: Trial) -> float: raise ValueError() def pruned_objective(trial: Trial) -> float: raise TrialPruned() optuna-3.5.0/optuna/testing/pruners.py000066400000000000000000000004761453453102400200700ustar00rootroot00000000000000from __future__ import annotations import optuna class DeterministicPruner(optuna.pruners.BasePruner): def __init__(self, is_pruning: bool) -> None: self.is_pruning = is_pruning def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: return self.is_pruning optuna-3.5.0/optuna/testing/samplers.py000066400000000000000000000037261453453102400202210ustar00rootroot00000000000000from __future__ import annotations from typing import Any import optuna from optuna.distributions import BaseDistribution class DeterministicSampler(optuna.samplers.BaseSampler): def __init__(self, params: dict[str, Any]) -> None: self.params = params def infer_relative_search_space( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial" ) -> dict[str, BaseDistribution]: return {} def sample_relative( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", search_space: dict[str, BaseDistribution], ) -> dict[str, Any]: return {} def sample_independent( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: BaseDistribution, ) -> Any: param_value = self.params[param_name] assert param_distribution._contains(param_distribution.to_internal_repr(param_value)) return param_value class FirstTrialOnlyRandomSampler(optuna.samplers.RandomSampler): def sample_relative( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", search_space: dict[str, BaseDistribution], ) -> dict[str, float]: if len(study.trials) > 1: raise RuntimeError("`FirstTrialOnlyRandomSampler` only works on the first trial.") return super(FirstTrialOnlyRandomSampler, self).sample_relative(study, trial, search_space) def sample_independent( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: BaseDistribution, ) -> float: if len(study.trials) > 1: raise RuntimeError("`FirstTrialOnlyRandomSampler` only works on the first trial.") return super(FirstTrialOnlyRandomSampler, self).sample_independent( study, trial, param_name, param_distribution ) optuna-3.5.0/optuna/testing/storages.py000066400000000000000000000065061453453102400202210ustar00rootroot00000000000000from __future__ import annotations import sys from types import TracebackType from typing import Any from typing import IO import fakeredis import pytest import optuna from optuna.storages import JournalFileStorage from optuna.testing.tempfile_pool import NamedTemporaryFilePool try: import distributed except ImportError: pass STORAGE_MODES: list[Any] = [ "inmemory", "sqlite", "cached_sqlite", "journal", "journal_redis", pytest.param( "dask", marks=[ pytest.mark.integration, pytest.mark.skipif( sys.version_info[:2] >= (3, 11), reason="distributed doesn't yet support Python 3.11", ), ], ), ] STORAGE_MODES_HEARTBEAT = [ "sqlite", "cached_sqlite", ] SQLITE3_TIMEOUT = 300 class StorageSupplier: def __init__(self, storage_specifier: str, **kwargs: Any) -> None: self.storage_specifier = storage_specifier self.extra_args = kwargs self.dask_client: "distributed.Client" | None = None self.tempfile: IO[Any] | None = None def __enter__( self, ) -> ( optuna.storages.InMemoryStorage | optuna.storages._CachedStorage | optuna.storages.RDBStorage | optuna.storages.JournalStorage | "optuna.integration.DaskStorage" ): if self.storage_specifier == "inmemory": if len(self.extra_args) > 0: raise ValueError("InMemoryStorage does not accept any arguments!") return optuna.storages.InMemoryStorage() elif "sqlite" in self.storage_specifier: self.tempfile = NamedTemporaryFilePool().tempfile() url = "sqlite:///{}".format(self.tempfile.name) rdb_storage = optuna.storages.RDBStorage( url, engine_kwargs={"connect_args": {"timeout": SQLITE3_TIMEOUT}}, **self.extra_args, ) return ( optuna.storages._CachedStorage(rdb_storage) if "cached" in self.storage_specifier else rdb_storage ) elif self.storage_specifier == "journal_redis": journal_redis_storage = optuna.storages.JournalRedisStorage("redis://localhost") journal_redis_storage._redis = self.extra_args.get( "redis", fakeredis.FakeStrictRedis() # type: ignore[no-untyped-call] ) return optuna.storages.JournalStorage(journal_redis_storage) elif "journal" in self.storage_specifier: self.tempfile = NamedTemporaryFilePool().tempfile() file_storage = JournalFileStorage(self.tempfile.name) return optuna.storages.JournalStorage(file_storage) elif self.storage_specifier == "dask": self.dask_client = distributed.Client() # type: ignore[no-untyped-call] return optuna.integration.DaskStorage(client=self.dask_client, **self.extra_args) else: assert False def __exit__( self, exc_type: type[BaseException], exc_val: BaseException, exc_tb: TracebackType ) -> None: if self.tempfile: self.tempfile.close() if self.dask_client: self.dask_client.shutdown() # type: ignore[no-untyped-call] self.dask_client.close() # type: ignore[no-untyped-call] optuna-3.5.0/optuna/testing/tempfile_pool.py000066400000000000000000000023721453453102400212250ustar00rootroot00000000000000# On Windows, temporary file shold delete "after" storage was deleted # NamedTemporaryFilePool ensures tempfile delete after tests. from __future__ import annotations import atexit import gc import os import tempfile from types import TracebackType from typing import Any from typing import IO class NamedTemporaryFilePool: tempfile_pool: list[IO[Any]] = [] def __new__(cls, **kwargs: Any) -> "NamedTemporaryFilePool": if not hasattr(cls, "_instance"): cls._instance = super(NamedTemporaryFilePool, cls).__new__(cls) atexit.register(cls._instance.cleanup) return cls._instance def __init__(self, **kwargs: Any) -> None: self.kwargs = kwargs def tempfile(self) -> IO[Any]: self._tempfile = tempfile.NamedTemporaryFile(delete=False, **self.kwargs) self.tempfile_pool.append(self._tempfile) return self._tempfile def cleanup(self) -> None: gc.collect() for i in self.tempfile_pool: os.unlink(i.name) def __enter__(self) -> IO[Any]: return self.tempfile() def __exit__( self, exc_type: type[BaseException], exc_val: BaseException, exc_tb: TracebackType, ) -> None: self._tempfile.close() optuna-3.5.0/optuna/testing/threading.py000066400000000000000000000011561453453102400203330ustar00rootroot00000000000000from __future__ import annotations import threading from typing import Any from typing import Callable class _TestableThread(threading.Thread): def __init__(self, target: Callable[..., Any], args: tuple): threading.Thread.__init__(self, target=target, args=args) self.exc: BaseException | None = None def run(self) -> None: try: threading.Thread.run(self) except BaseException as e: self.exc = e def join(self, timeout: float | None = None) -> None: super(_TestableThread, self).join(timeout) if self.exc: raise self.exc optuna-3.5.0/optuna/testing/visualization.py000066400000000000000000000046431453453102400212730ustar00rootroot00000000000000from optuna import Study from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.trial import create_trial def prepare_study_with_trials( n_objectives: int = 1, direction: str = "minimize", value_for_first_trial: float = 0.0, ) -> Study: """Return a dummy study object for tests. This function is added to reduce the code to set up dummy study object in each test case. However, you can only use this function for unit tests that are loosely coupled with the dummy study object. Unit tests that are tightly coupled with the study become difficult to read because of `Mystery Guest `_ and/or `Eager Test `_ anti-patterns. Args: n_objectives: Number of objective values. direction: Study's optimization direction. value_for_first_trial: Objective value in first trial. This value will be broadcasted to all objectives in multi-objective optimization. Returns: :class:`~optuna.study.Study` """ study = create_study(directions=[direction] * n_objectives) study.add_trial( create_trial( values=[value_for_first_trial] * n_objectives, params={"param_a": 1.0, "param_b": 2.0, "param_c": 3.0, "param_d": 4.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), "param_c": FloatDistribution(2.0, 5.0), "param_d": FloatDistribution(2.0, 5.0), }, ) ) study.add_trial( create_trial( values=[2.0] * n_objectives, params={"param_b": 0.0, "param_d": 4.0}, distributions={ "param_b": FloatDistribution(0.0, 3.0), "param_d": FloatDistribution(2.0, 5.0), }, ) ) study.add_trial( create_trial( values=[1.0] * n_objectives, params={"param_a": 2.5, "param_b": 1.0, "param_c": 4.5, "param_d": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), "param_c": FloatDistribution(2.0, 5.0), "param_d": FloatDistribution(2.0, 5.0), }, ) ) return study optuna-3.5.0/optuna/trial/000077500000000000000000000000001453453102400154475ustar00rootroot00000000000000optuna-3.5.0/optuna/trial/__init__.py000066400000000000000000000005711453453102400175630ustar00rootroot00000000000000from optuna.trial._base import BaseTrial from optuna.trial._fixed import FixedTrial from optuna.trial._frozen import create_trial from optuna.trial._frozen import FrozenTrial from optuna.trial._state import TrialState from optuna.trial._trial import Trial __all__ = [ "BaseTrial", "FixedTrial", "FrozenTrial", "Trial", "TrialState", "create_trial", ] optuna-3.5.0/optuna/trial/_base.py000066400000000000000000000071651453453102400171030ustar00rootroot00000000000000import abc import datetime from typing import Any from typing import Dict from typing import Optional from typing import overload from typing import Sequence from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType _SUGGEST_INT_POSITIONAL_ARGS = ["self", "name", "low", "high", "step", "log"] class BaseTrial(abc.ABC): """Base class for trials. Note that this class is not supposed to be directly accessed by library users. """ @abc.abstractmethod def suggest_float( self, name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False, ) -> float: raise NotImplementedError @deprecated_func("3.0.0", "6.0.0") @abc.abstractmethod def suggest_uniform(self, name: str, low: float, high: float) -> float: raise NotImplementedError @deprecated_func("3.0.0", "6.0.0") @abc.abstractmethod def suggest_loguniform(self, name: str, low: float, high: float) -> float: raise NotImplementedError @deprecated_func("3.0.0", "6.0.0") @abc.abstractmethod def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: raise NotImplementedError @abc.abstractmethod def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: raise NotImplementedError @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload @abc.abstractmethod def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload @abc.abstractmethod def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... @abc.abstractmethod def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: raise NotImplementedError @abc.abstractmethod def report(self, value: float, step: int) -> None: raise NotImplementedError @abc.abstractmethod def should_prune(self) -> bool: raise NotImplementedError @abc.abstractmethod def set_user_attr(self, key: str, value: Any) -> None: raise NotImplementedError @abc.abstractmethod @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: raise NotImplementedError @property @abc.abstractmethod def params(self) -> Dict[str, Any]: raise NotImplementedError @property @abc.abstractmethod def distributions(self) -> Dict[str, BaseDistribution]: raise NotImplementedError @property @abc.abstractmethod def user_attrs(self) -> Dict[str, Any]: raise NotImplementedError @property @abc.abstractmethod def system_attrs(self) -> Dict[str, Any]: raise NotImplementedError @property @abc.abstractmethod def datetime_start(self) -> Optional[datetime.datetime]: raise NotImplementedError @property def number(self) -> int: raise NotImplementedError optuna-3.5.0/optuna/trial/_fixed.py000066400000000000000000000143451453453102400172660ustar00rootroot00000000000000import datetime from typing import Any from typing import Dict from typing import Optional from typing import overload from typing import Sequence import warnings from optuna import distributions from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS from optuna.trial._base import BaseTrial _suggest_deprecated_msg = "Use suggest_float{args} instead." class FixedTrial(BaseTrial): """A trial class which suggests a fixed value for each parameter. This object has the same methods as :class:`~optuna.trial.Trial`, and it suggests pre-defined parameter values. The parameter values can be determined at the construction of the :class:`~optuna.trial.FixedTrial` object. In contrast to :class:`~optuna.trial.Trial`, :class:`~optuna.trial.FixedTrial` does not depend on :class:`~optuna.study.Study`, and it is useful for deploying optimization results. Example: Evaluate an objective function with parameter values given by a user. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y assert objective(optuna.trial.FixedTrial({"x": 1, "y": 0})) == 1 .. note:: Please refer to :class:`~optuna.trial.Trial` for details of methods and properties. Args: params: A dictionary containing all parameters. number: A trial number. Defaults to ``0``. """ def __init__(self, params: Dict[str, Any], number: int = 0) -> None: self._params = params self._suggested_params: Dict[str, Any] = {} self._distributions: Dict[str, BaseDistribution] = {} self._user_attrs: Dict[str, Any] = {} self._system_attrs: Dict[str, Any] = {} self._datetime_start = datetime.datetime.now() self._number = number def suggest_float( self, name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False, ) -> float: return self._suggest(name, FloatDistribution(low, high, log=log, step=step)) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: return self.suggest_float(name, low, high, step=q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: return int(self._suggest(name, IntDistribution(low, high, log=log, step=step))) @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: return self._suggest(name, CategoricalDistribution(choices=choices)) def report(self, value: float, step: int) -> None: pass def should_prune(self) -> bool: return False def set_user_attr(self, key: str, value: Any) -> None: self._user_attrs[key] = value @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: self._system_attrs[key] = value def _suggest(self, name: str, distribution: BaseDistribution) -> Any: if name not in self._params: raise ValueError( "The value of the parameter '{}' is not found. Please set it at " "the construction of the FixedTrial object.".format(name) ) value = self._params[name] param_value_in_internal_repr = distribution.to_internal_repr(value) if not distribution._contains(param_value_in_internal_repr): warnings.warn( "The value {} of the parameter '{}' is out of " "the range of the distribution {}.".format(value, name, distribution) ) if name in self._distributions: distributions.check_distribution_compatibility(self._distributions[name], distribution) self._suggested_params[name] = value self._distributions[name] = distribution return value @property def params(self) -> Dict[str, Any]: return self._suggested_params @property def distributions(self) -> Dict[str, BaseDistribution]: return self._distributions @property def user_attrs(self) -> Dict[str, Any]: return self._user_attrs @property def system_attrs(self) -> Dict[str, Any]: return self._system_attrs @property def datetime_start(self) -> Optional[datetime.datetime]: return self._datetime_start @property def number(self) -> int: return self._number optuna-3.5.0/optuna/trial/_frozen.py000066400000000000000000000473271453453102400175000ustar00rootroot00000000000000import datetime from typing import Any from typing import cast from typing import Dict from typing import List from typing import Mapping from typing import Optional from typing import overload from typing import Sequence import warnings from optuna import distributions from optuna import logging from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna._typing import JSONSerializable from optuna.distributions import _convert_old_distribution_to_new_distribution from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS from optuna.trial._base import BaseTrial from optuna.trial._state import TrialState _logger = logging.get_logger(__name__) _suggest_deprecated_msg = "Use suggest_float{args} instead." class FrozenTrial(BaseTrial): """Status and results of a :class:`~optuna.trial.Trial`. An object of this class has the same methods as :class:`~optuna.trial.Trial`, but is not associated with, nor has any references to a :class:`~optuna.study.Study`. It is therefore not possible to make persistent changes to a storage from this object by itself, for instance by using :func:`~optuna.trial.FrozenTrial.set_user_attr`. It will suggest the parameter values stored in :attr:`params` and will not sample values from any distributions. It can be passed to objective functions (see :func:`~optuna.study.Study.optimize`) and is useful for deploying optimization results. Example: Re-evaluate an objective function with parameter values optimized study. .. testcode:: import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) assert objective(study.best_trial) == study.best_value .. note:: Instances are mutable, despite the name. For instance, :func:`~optuna.trial.FrozenTrial.set_user_attr` will update user attributes of objects in-place. Example: Overwritten attributes. .. testcode:: import copy import datetime import optuna def objective(trial): x = trial.suggest_float("x", -1, 1) # this user attribute always differs trial.set_user_attr("evaluation time", datetime.datetime.now()) return x**2 study = optuna.create_study() study.optimize(objective, n_trials=3) best_trial = study.best_trial best_trial_copy = copy.deepcopy(best_trial) # re-evaluate objective(best_trial) # the user attribute is overwritten by re-evaluation assert best_trial.user_attrs != best_trial_copy.user_attrs .. note:: Please refer to :class:`~optuna.trial.Trial` for details of methods and properties. Attributes: number: Unique and consecutive number of :class:`~optuna.trial.Trial` for each :class:`~optuna.study.Study`. Note that this field uses zero-based numbering. state: :class:`TrialState` of the :class:`~optuna.trial.Trial`. value: Objective value of the :class:`~optuna.trial.Trial`. ``value`` and ``values`` must not be specified at the same time. values: Sequence of objective values of the :class:`~optuna.trial.Trial`. The length is greater than 1 if the problem is multi-objective optimization. ``value`` and ``values`` must not be specified at the same time. datetime_start: Datetime where the :class:`~optuna.trial.Trial` started. datetime_complete: Datetime where the :class:`~optuna.trial.Trial` finished. params: Dictionary that contains suggested parameters. distributions: Dictionary that contains the distributions of :attr:`params`. user_attrs: Dictionary that contains the attributes of the :class:`~optuna.trial.Trial` set with :func:`optuna.trial.Trial.set_user_attr`. system_attrs: Dictionary that contains the attributes of the :class:`~optuna.trial.Trial` set with :func:`optuna.trial.Trial.set_system_attr`. intermediate_values: Intermediate objective values set with :func:`optuna.trial.Trial.report`. """ def __init__( self, number: int, state: TrialState, value: Optional[float], datetime_start: Optional[datetime.datetime], datetime_complete: Optional[datetime.datetime], params: Dict[str, Any], distributions: Dict[str, BaseDistribution], user_attrs: Dict[str, Any], system_attrs: Dict[str, Any], intermediate_values: Dict[int, float], trial_id: int, *, values: Optional[Sequence[float]] = None, ) -> None: self._number = number self.state = state self._values: Optional[List[float]] = None if value is not None and values is not None: raise ValueError("Specify only one of `value` and `values`.") elif value is not None: self._values = [value] elif values is not None: self._values = list(values) self._datetime_start = datetime_start self.datetime_complete = datetime_complete self._params = params self._user_attrs = user_attrs self._system_attrs = system_attrs self.intermediate_values = intermediate_values self._distributions = distributions self._trial_id = trial_id def __eq__(self, other: Any) -> bool: if not isinstance(other, FrozenTrial): return NotImplemented return other.__dict__ == self.__dict__ def __lt__(self, other: Any) -> bool: if not isinstance(other, FrozenTrial): return NotImplemented return self.number < other.number def __le__(self, other: Any) -> bool: if not isinstance(other, FrozenTrial): return NotImplemented return self.number <= other.number def __hash__(self) -> int: return hash(tuple(getattr(self, field) for field in self.__dict__)) def __repr__(self) -> str: return "{cls}({kwargs})".format( cls=self.__class__.__name__, kwargs=", ".join( "{field}={value}".format( field=field if not field.startswith("_") else field[1:], value=repr(getattr(self, field)), ) for field in self.__dict__ ) + ", value=None", ) def suggest_float( self, name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False, ) -> float: return self._suggest(name, FloatDistribution(low, high, log=log, step=step)) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: return self.suggest_float(name, low, high, step=q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: return int(self._suggest(name, IntDistribution(low, high, log=log, step=step))) @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: return self._suggest(name, CategoricalDistribution(choices=choices)) def report(self, value: float, step: int) -> None: """Interface of report function. Since :class:`~optuna.trial.FrozenTrial` is not pruned, this report function does nothing. .. seealso:: Please refer to :func:`~optuna.trial.FrozenTrial.should_prune`. Args: value: A value returned from the objective function. step: Step of the trial (e.g., Epoch of neural network training). Note that pruners assume that ``step`` starts at zero. For example, :class:`~optuna.pruners.MedianPruner` simply checks if ``step`` is less than ``n_warmup_steps`` as the warmup mechanism. """ pass def should_prune(self) -> bool: """Suggest whether the trial should be pruned or not. The suggestion is always :obj:`False` regardless of a pruning algorithm. .. note:: :class:`~optuna.trial.FrozenTrial` only samples one combination of parameters. Returns: :obj:`False`. """ return False def set_user_attr(self, key: str, value: Any) -> None: self._user_attrs[key] = value @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: self._system_attrs[key] = value def _validate(self) -> None: if self.state != TrialState.WAITING and self.datetime_start is None: raise ValueError( "`datetime_start` is supposed to be set when the trial state is not waiting." ) if self.state.is_finished(): if self.datetime_complete is None: raise ValueError("`datetime_complete` is supposed to be set for a finished trial.") else: if self.datetime_complete is not None: raise ValueError( "`datetime_complete` is supposed to be None for an unfinished trial." ) if self.state == TrialState.COMPLETE and self._values is None: raise ValueError("`value` is supposed to be set for a complete trial.") if set(self.params.keys()) != set(self.distributions.keys()): raise ValueError( "Inconsistent parameters {} and distributions {}.".format( set(self.params.keys()), set(self.distributions.keys()) ) ) for param_name, param_value in self.params.items(): distribution = self.distributions[param_name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) if not distribution._contains(param_value_in_internal_repr): raise ValueError( "The value {} of parameter '{}' isn't contained in the distribution " "{}.".format(param_value, param_name, distribution) ) def _suggest(self, name: str, distribution: BaseDistribution) -> Any: if name not in self._params: raise ValueError( "The value of the parameter '{}' is not found. Please set it at " "the construction of the FrozenTrial object.".format(name) ) value = self._params[name] param_value_in_internal_repr = distribution.to_internal_repr(value) if not distribution._contains(param_value_in_internal_repr): warnings.warn( "The value {} of the parameter '{}' is out of " "the range of the distribution {}.".format(value, name, distribution) ) if name in self._distributions: distributions.check_distribution_compatibility(self._distributions[name], distribution) self._distributions[name] = distribution return value @property def number(self) -> int: return self._number @number.setter def number(self, value: int) -> None: self._number = value @property def value(self) -> Optional[float]: if self._values is not None: if len(self._values) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) return self._values[0] return None @value.setter def value(self, v: Optional[float]) -> None: if self._values is not None: if len(self._values) > 1: raise RuntimeError( "This attribute is not available during multi-objective optimization." ) if v is not None: self._values = [v] else: self._values = None # These `_get_values`, `_set_values`, and `values = property(_get_values, _set_values)` are # defined to pass the mypy. # See https://github.com/python/mypy/issues/3004#issuecomment-726022329. def _get_values(self) -> Optional[List[float]]: return self._values def _set_values(self, v: Optional[Sequence[float]]) -> None: if v is not None: self._values = list(v) else: self._values = None values = property(_get_values, _set_values) @property def datetime_start(self) -> Optional[datetime.datetime]: return self._datetime_start @datetime_start.setter def datetime_start(self, value: Optional[datetime.datetime]) -> None: self._datetime_start = value @property def params(self) -> Dict[str, Any]: return self._params @params.setter def params(self, params: Dict[str, Any]) -> None: self._params = params @property def distributions(self) -> Dict[str, BaseDistribution]: return self._distributions @distributions.setter def distributions(self, value: Dict[str, BaseDistribution]) -> None: self._distributions = value @property def user_attrs(self) -> Dict[str, Any]: return self._user_attrs @user_attrs.setter def user_attrs(self, value: Dict[str, Any]) -> None: self._user_attrs = value @property def system_attrs(self) -> Dict[str, Any]: return self._system_attrs @system_attrs.setter def system_attrs(self, value: Mapping[str, JSONSerializable]) -> None: self._system_attrs = cast(Dict[str, Any], value) @property def last_step(self) -> Optional[int]: """Return the maximum step of :attr:`intermediate_values` in the trial. Returns: The maximum step of intermediates. """ if len(self.intermediate_values) == 0: return None else: return max(self.intermediate_values.keys()) @property def duration(self) -> Optional[datetime.timedelta]: """Return the elapsed time taken to complete the trial. Returns: The duration. """ if self.datetime_start and self.datetime_complete: return self.datetime_complete - self.datetime_start else: return None def create_trial( *, state: TrialState = TrialState.COMPLETE, value: Optional[float] = None, values: Optional[Sequence[float]] = None, params: Optional[Dict[str, Any]] = None, distributions: Optional[Dict[str, BaseDistribution]] = None, user_attrs: Optional[Dict[str, Any]] = None, system_attrs: Optional[Dict[str, Any]] = None, intermediate_values: Optional[Dict[int, float]] = None, ) -> FrozenTrial: """Create a new :class:`~optuna.trial.FrozenTrial`. Example: .. testcode:: import optuna from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution trial = optuna.trial.create_trial( params={"x": 1.0, "y": 0}, distributions={ "x": FloatDistribution(0, 10), "y": CategoricalDistribution([-1, 0, 1]), }, value=5.0, ) assert isinstance(trial, optuna.trial.FrozenTrial) assert trial.value == 5.0 assert trial.params == {"x": 1.0, "y": 0} .. seealso:: See :func:`~optuna.study.Study.add_trial` for how this function can be used to create a study from existing trials. .. note:: Please note that this is a low-level API. In general, trials that are passed to objective functions are created inside :func:`~optuna.study.Study.optimize`. .. note:: When ``state`` is :class:`TrialState.COMPLETE`, the following parameters are required: * ``params`` * ``distributions`` * ``value`` or ``values`` Args: state: Trial state. value: Trial objective value. Must be specified if ``state`` is :class:`TrialState.COMPLETE`. ``value`` and ``values`` must not be specified at the same time. values: Sequence of the trial objective values. The length is greater than 1 if the problem is multi-objective optimization. Must be specified if ``state`` is :class:`TrialState.COMPLETE`. ``value`` and ``values`` must not be specified at the same time. params: Dictionary with suggested parameters of the trial. distributions: Dictionary with parameter distributions of the trial. user_attrs: Dictionary with user attributes. system_attrs: Dictionary with system attributes. Should not have to be used for most users. intermediate_values: Dictionary with intermediate objective values of the trial. Returns: Created trial. """ params = params or {} distributions = distributions or {} distributions = { key: _convert_old_distribution_to_new_distribution(dist) for key, dist in distributions.items() } user_attrs = user_attrs or {} system_attrs = system_attrs or {} intermediate_values = intermediate_values or {} if state == TrialState.WAITING: datetime_start = None else: datetime_start = datetime.datetime.now() if state.is_finished(): datetime_complete: Optional[datetime.datetime] = datetime_start else: datetime_complete = None trial = FrozenTrial( number=-1, trial_id=-1, state=state, value=value, values=values, datetime_start=datetime_start, datetime_complete=datetime_complete, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) trial._validate() return trial optuna-3.5.0/optuna/trial/_state.py000066400000000000000000000020041453453102400172740ustar00rootroot00000000000000import enum class TrialState(enum.IntEnum): """State of a :class:`~optuna.trial.Trial`. Attributes: RUNNING: The :class:`~optuna.trial.Trial` is running. WAITING: The :class:`~optuna.trial.Trial` is waiting and unfinished. COMPLETE: The :class:`~optuna.trial.Trial` has been finished without any error. PRUNED: The :class:`~optuna.trial.Trial` has been pruned with :class:`~optuna.exceptions.TrialPruned`. FAIL: The :class:`~optuna.trial.Trial` has failed due to an uncaught error. """ RUNNING = 0 COMPLETE = 1 PRUNED = 2 FAIL = 3 WAITING = 4 def __repr__(self) -> str: return str(self) def is_finished(self) -> bool: """Return a bool value to represent whether the trial state is unfinished or not. The unfinished state is either ``RUNNING`` or ``WAITING``. """ return self != TrialState.RUNNING and self != TrialState.WAITING optuna-3.5.0/optuna/trial/_trial.py000066400000000000000000000721301453453102400172760ustar00rootroot00000000000000from __future__ import annotations from collections import UserDict import copy import datetime from typing import Any from typing import Dict from typing import Optional from typing import overload from typing import Sequence import warnings import optuna from optuna import distributions from optuna import logging from optuna import pruners from optuna._convert_positional_args import convert_positional_args from optuna._deprecated import deprecated_func from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial import FrozenTrial from optuna.trial._base import _SUGGEST_INT_POSITIONAL_ARGS from optuna.trial._base import BaseTrial _logger = logging.get_logger(__name__) _suggest_deprecated_msg = "Use suggest_float{args} instead." class Trial(BaseTrial): """A trial is a process of evaluating an objective function. This object is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial's state, and set/get user-defined attributes of the trial. Note that the direct use of this constructor is not recommended. This object is seamlessly instantiated and passed to the objective function behind the :func:`optuna.study.Study.optimize()` method; hence library users do not care about instantiation of this object. Args: study: A :class:`~optuna.study.Study` object. trial_id: A trial ID that is automatically generated. """ def __init__(self, study: "optuna.study.Study", trial_id: int) -> None: self.study = study self._trial_id = trial_id self.storage = self.study._storage self._cached_frozen_trial = self.storage.get_trial(self._trial_id) study = pruners._filter_study(self.study, self._cached_frozen_trial) self.study.sampler.before_trial(study, self._cached_frozen_trial) self.relative_search_space = self.study.sampler.infer_relative_search_space( study, self._cached_frozen_trial ) self._relative_params: Optional[Dict[str, Any]] = None self._fixed_params = self._cached_frozen_trial.system_attrs.get("fixed_params", {}) @property def relative_params(self) -> Dict[str, Any]: if self._relative_params is None: study = pruners._filter_study(self.study, self._cached_frozen_trial) self._relative_params = self.study.sampler.sample_relative( study, self._cached_frozen_trial, self.relative_search_space ) return self._relative_params def suggest_float( self, name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False, ) -> float: """Suggest a value for the floating point parameter. Example: Suggest a momentum, learning rate and scaling factor of learning rate for neural network training. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0) def objective(trial): momentum = trial.suggest_float("momentum", 0.0, 1.0) learning_rate_init = trial.suggest_float( "learning_rate_init", 1e-5, 1e-3, log=True ) power_t = trial.suggest_float("power_t", 0.2, 0.8, step=0.1) clf = MLPClassifier( hidden_layer_sizes=(100, 50), momentum=momentum, learning_rate_init=learning_rate_init, solver="sgd", random_state=0, power_t=power_t, ) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than 0. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A step of discretization. .. note:: The ``step`` and ``log`` arguments cannot be used at the same time. To set the ``step`` argument to a float number, set the ``log`` argument to :obj:`False`. log: A flag to sample the value from the log domain or not. If ``log`` is true, the value is sampled from the range in the log domain. Otherwise, the value is sampled from the range in the linear domain. .. note:: The ``step`` and ``log`` arguments cannot be used at the same time. To set the ``log`` argument to :obj:`True`, set the ``step`` argument to :obj:`None`. Returns: A suggested float value. .. seealso:: :ref:`configurations` tutorial describes more details and flexible usages. """ distribution = FloatDistribution(low, high, log=log, step=step) suggested_value = self._suggest(name, distribution) self._check_distribution(name, distribution) return suggested_value @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="")) def suggest_uniform(self, name: str, low: float, high: float) -> float: """Suggest a value for the continuous parameter. The value is sampled from the range :math:`[\\mathsf{low}, \\mathsf{high})` in the linear domain. When :math:`\\mathsf{low} = \\mathsf{high}`, the value of :math:`\\mathsf{low}` will be returned. Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. Returns: A suggested float value. """ return self.suggest_float(name, low, high) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., log=True)")) def suggest_loguniform(self, name: str, low: float, high: float) -> float: """Suggest a value for the continuous parameter. The value is sampled from the range :math:`[\\mathsf{low}, \\mathsf{high})` in the log domain. When :math:`\\mathsf{low} = \\mathsf{high}`, the value of :math:`\\mathsf{low}` will be returned. Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. Returns: A suggested float value. """ return self.suggest_float(name, low, high, log=True) @deprecated_func("3.0.0", "6.0.0", text=_suggest_deprecated_msg.format(args="(..., step=...)")) def suggest_discrete_uniform(self, name: str, low: float, high: float, q: float) -> float: """Suggest a value for the discrete parameter. The value is sampled from the range :math:`[\\mathsf{low}, \\mathsf{high}]`, and the step of discretization is :math:`q`. More specifically, this method returns one of the values in the sequence :math:`\\mathsf{low}, \\mathsf{low} + q, \\mathsf{low} + 2 q, \\dots, \\mathsf{low} + k q \\le \\mathsf{high}`, where :math:`k` denotes an integer. Note that :math:`high` may be changed due to round-off errors if :math:`q` is not an integer. Please check warning messages to find the changed values. Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. q: A step of discretization. Returns: A suggested float value. """ return self.suggest_float(name, low, high, step=q) @convert_positional_args(previous_positional_arg_names=_SUGGEST_INT_POSITIONAL_ARGS) def suggest_int( self, name: str, low: int, high: int, *, step: int = 1, log: bool = False ) -> int: """Suggest a value for the integer parameter. The value is sampled from the integers in :math:`[\\mathsf{low}, \\mathsf{high}]`. Example: Suggest the number of trees in `RandomForestClassifier `_. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) def objective(trial): n_estimators = trial.suggest_int("n_estimators", 50, 400) clf = RandomForestClassifier(n_estimators=n_estimators, random_state=0) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: name: A parameter name. low: Lower endpoint of the range of suggested values. ``low`` is included in the range. ``low`` must be less than or equal to ``high``. If ``log`` is :obj:`True`, ``low`` must be larger than 0. high: Upper endpoint of the range of suggested values. ``high`` is included in the range. ``high`` must be greater than or equal to ``low``. step: A step of discretization. .. note:: Note that :math:`\\mathsf{high}` is modified if the range is not divisible by :math:`\\mathsf{step}`. Please check the warning messages to find the changed values. .. note:: The method returns one of the values in the sequence :math:`\\mathsf{low}, \\mathsf{low} + \\mathsf{step}, \\mathsf{low} + 2 * \\mathsf{step}, \\dots, \\mathsf{low} + k * \\mathsf{step} \\le \\mathsf{high}`, where :math:`k` denotes an integer. .. note:: The ``step != 1`` and ``log`` arguments cannot be used at the same time. To set the ``step`` argument :math:`\\mathsf{step} \\ge 2`, set the ``log`` argument to :obj:`False`. log: A flag to sample the value from the log domain or not. .. note:: If ``log`` is true, at first, the range of suggested values is divided into grid points of width 1. The range of suggested values is then converted to a log domain, from which a value is sampled. The uniformly sampled value is re-converted to the original domain and rounded to the nearest grid point that we just split, and the suggested value is determined. For example, if `low = 2` and `high = 8`, then the range of suggested values is `[2, 3, 4, 5, 6, 7, 8]` and lower values tend to be more sampled than higher values. .. note:: The ``step != 1`` and ``log`` arguments cannot be used at the same time. To set the ``log`` argument to :obj:`True`, set the ``step`` argument to 1. .. seealso:: :ref:`configurations` tutorial describes more details and flexible usages. """ distribution = IntDistribution(low=low, high=high, log=log, step=step) suggested_value = int(self._suggest(name, distribution)) self._check_distribution(name, distribution) return suggested_value @overload def suggest_categorical(self, name: str, choices: Sequence[None]) -> None: ... @overload def suggest_categorical(self, name: str, choices: Sequence[bool]) -> bool: ... @overload def suggest_categorical(self, name: str, choices: Sequence[int]) -> int: ... @overload def suggest_categorical(self, name: str, choices: Sequence[float]) -> float: ... @overload def suggest_categorical(self, name: str, choices: Sequence[str]) -> str: ... @overload def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: ... def suggest_categorical( self, name: str, choices: Sequence[CategoricalChoiceType] ) -> CategoricalChoiceType: """Suggest a value for the categorical parameter. The value is sampled from ``choices``. Example: Suggest a kernel function of `SVC `_. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.svm import SVC import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) def objective(trial): kernel = trial.suggest_categorical("kernel", ["linear", "poly", "rbf"]) clf = SVC(kernel=kernel, gamma="scale", random_state=0) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: name: A parameter name. choices: Parameter value candidates. .. seealso:: :class:`~optuna.distributions.CategoricalDistribution`. Returns: A suggested value. .. seealso:: :ref:`configurations` tutorial describes more details and flexible usages. """ # There is no need to call self._check_distribution because # CategoricalDistribution does not support dynamic value space. return self._suggest(name, CategoricalDistribution(choices=choices)) def report(self, value: float, step: int) -> None: """Report an objective function value for a given step. The reported values are used by the pruners to determine whether this trial should be pruned. .. seealso:: Please refer to :class:`~optuna.pruners.BasePruner`. .. note:: The reported value is converted to ``float`` type by applying ``float()`` function internally. Thus, it accepts all float-like types (e.g., ``numpy.float32``). If the conversion fails, a ``TypeError`` is raised. .. note:: If this method is called multiple times at the same ``step`` in a trial, the reported ``value`` only the first time is stored and the reported values from the second time are ignored. .. note:: :func:`~optuna.trial.Trial.report` does not support multi-objective optimization. Example: Report intermediate scores of `SGDClassifier `_ training. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.linear_model import SGDClassifier from sklearn.model_selection import train_test_split import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y) def objective(trial): clf = SGDClassifier(random_state=0) for step in range(100): clf.partial_fit(X_train, y_train, np.unique(y)) intermediate_value = clf.score(X_valid, y_valid) trial.report(intermediate_value, step=step) if trial.should_prune(): raise optuna.TrialPruned() return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) Args: value: A value returned from the objective function. step: Step of the trial (e.g., Epoch of neural network training). Note that pruners assume that ``step`` starts at zero. For example, :class:`~optuna.pruners.MedianPruner` simply checks if ``step`` is less than ``n_warmup_steps`` as the warmup mechanism. ``step`` must be a positive integer. """ if len(self.study.directions) > 1: raise NotImplementedError( "Trial.report is not supported for multi-objective optimization." ) try: # For convenience, we allow users to report a value that can be cast to `float`. value = float(value) except (TypeError, ValueError): message = "The `value` argument is of type '{}' but supposed to be a float.".format( type(value).__name__ ) raise TypeError(message) from None if step < 0: raise ValueError("The `step` argument is {} but cannot be negative.".format(step)) if step in self._cached_frozen_trial.intermediate_values: # Do nothing if already reported. warnings.warn( "The reported value is ignored because this `step` {} is already reported.".format( step ) ) return self.storage.set_trial_intermediate_value(self._trial_id, step, value) self._cached_frozen_trial.intermediate_values[step] = value def should_prune(self) -> bool: """Suggest whether the trial should be pruned or not. The suggestion is made by a pruning algorithm associated with the trial and is based on previously reported values. The algorithm can be specified when constructing a :class:`~optuna.study.Study`. .. note:: If no values have been reported, the algorithm cannot make meaningful suggestions. Similarly, if this method is called multiple times with the exact same set of reported values, the suggestions will be the same. .. seealso:: Please refer to the example code in :func:`optuna.trial.Trial.report`. .. note:: :func:`~optuna.trial.Trial.should_prune` does not support multi-objective optimization. Returns: A boolean value. If :obj:`True`, the trial should be pruned according to the configured pruning algorithm. Otherwise, the trial should continue. """ if len(self.study.directions) > 1: raise NotImplementedError( "Trial.should_prune is not supported for multi-objective optimization." ) trial = self._get_latest_trial() return self.study.pruner.prune(self.study, trial) def set_user_attr(self, key: str, value: Any) -> None: """Set user attributes to the trial. The user attributes in the trial can be access via :func:`optuna.trial.Trial.user_attrs`. .. seealso:: See the recipe on :ref:`attributes`. Example: Save fixed hyperparameters of neural network training. .. testcode:: import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neural_network import MLPClassifier import optuna X, y = load_iris(return_X_y=True) X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0) def objective(trial): trial.set_user_attr("BATCHSIZE", 128) momentum = trial.suggest_float("momentum", 0, 1.0) clf = MLPClassifier( hidden_layer_sizes=(100, 50), batch_size=trial.user_attrs["BATCHSIZE"], momentum=momentum, solver="sgd", random_state=0, ) clf.fit(X_train, y_train) return clf.score(X_valid, y_valid) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=3) assert "BATCHSIZE" in study.best_trial.user_attrs.keys() assert study.best_trial.user_attrs["BATCHSIZE"] == 128 Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self.storage.set_trial_user_attr(self._trial_id, key, value) self._cached_frozen_trial.user_attrs[key] = value @deprecated_func("3.1.0", "5.0.0") def set_system_attr(self, key: str, value: Any) -> None: """Set system attributes to the trial. Note that Optuna internally uses this method to save system messages such as failure reason of trials. Please use :func:`~optuna.trial.Trial.set_user_attr` to set users' attributes. Args: key: A key string of the attribute. value: A value of the attribute. The value should be JSON serializable. """ self.storage.set_trial_system_attr(self._trial_id, key, value) self._cached_frozen_trial.system_attrs[key] = value def _suggest(self, name: str, distribution: BaseDistribution) -> Any: storage = self.storage trial_id = self._trial_id trial = self._get_latest_trial() if name in trial.distributions: # No need to sample if already suggested. distributions.check_distribution_compatibility(trial.distributions[name], distribution) param_value = trial.params[name] else: if self._is_fixed_param(name, distribution): param_value = self._fixed_params[name] elif distribution.single(): param_value = distributions._get_single_value(distribution) elif self._is_relative_param(name, distribution): param_value = self.relative_params[name] else: study = pruners._filter_study(self.study, trial) param_value = self.study.sampler.sample_independent( study, trial, name, distribution ) # `param_value` is validated here (invalid value like `np.nan` raises ValueError). param_value_in_internal_repr = distribution.to_internal_repr(param_value) storage.set_trial_param(trial_id, name, param_value_in_internal_repr, distribution) self._cached_frozen_trial.distributions[name] = distribution self._cached_frozen_trial.params[name] = param_value return param_value def _is_fixed_param(self, name: str, distribution: BaseDistribution) -> bool: if name not in self._fixed_params: return False param_value = self._fixed_params[name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) contained = distribution._contains(param_value_in_internal_repr) if not contained: warnings.warn( "Fixed parameter '{}' with value {} is out of range " "for distribution {}.".format(name, param_value, distribution) ) return True def _is_relative_param(self, name: str, distribution: BaseDistribution) -> bool: if name not in self.relative_params: return False if name not in self.relative_search_space: raise ValueError( "The parameter '{}' was sampled by `sample_relative` method " "but it is not contained in the relative search space.".format(name) ) relative_distribution = self.relative_search_space[name] distributions.check_distribution_compatibility(relative_distribution, distribution) param_value = self.relative_params[name] param_value_in_internal_repr = distribution.to_internal_repr(param_value) return distribution._contains(param_value_in_internal_repr) def _check_distribution(self, name: str, distribution: BaseDistribution) -> None: old_distribution = self._cached_frozen_trial.distributions.get(name, distribution) if old_distribution != distribution: warnings.warn( 'Inconsistent parameter values for distribution with name "{}"! ' "This might be a configuration mistake. " "Optuna allows to call the same distribution with the same " "name more than once in a trial. " "When the parameter values are inconsistent optuna only " "uses the values of the first call and ignores all following. " "Using these values: {}".format(name, old_distribution._asdict()), RuntimeWarning, ) def _get_latest_trial(self) -> FrozenTrial: # TODO(eukaryo): Remove this method after `system_attrs` property is removed. latest_trial = copy.copy(self._cached_frozen_trial) latest_trial.system_attrs = _LazyTrialSystemAttrs( # type: ignore[assignment] self._trial_id, self.storage ) return latest_trial @property def params(self) -> Dict[str, Any]: """Return parameters to be optimized. Returns: A dictionary containing all parameters. """ return copy.deepcopy(self._cached_frozen_trial.params) @property def distributions(self) -> Dict[str, BaseDistribution]: """Return distributions of parameters to be optimized. Returns: A dictionary containing all distributions. """ return copy.deepcopy(self._cached_frozen_trial.distributions) @property def user_attrs(self) -> Dict[str, Any]: """Return user attributes. Returns: A dictionary containing all user attributes. """ return copy.deepcopy(self._cached_frozen_trial.user_attrs) @property @deprecated_func("3.1.0", "5.0.0") def system_attrs(self) -> Dict[str, Any]: """Return system attributes. Returns: A dictionary containing all system attributes. """ return copy.deepcopy(self.storage.get_trial_system_attrs(self._trial_id)) @property def datetime_start(self) -> Optional[datetime.datetime]: """Return start datetime. Returns: Datetime where the :class:`~optuna.trial.Trial` started. """ return self._cached_frozen_trial.datetime_start @property def number(self) -> int: """Return trial's number which is consecutive and unique in a study. Returns: A trial number. """ return self._cached_frozen_trial.number class _LazyTrialSystemAttrs(UserDict): def __init__(self, trial_id: int, storage: optuna.storages.BaseStorage) -> None: super().__init__() self._trial_id = trial_id self._storage = storage self._initialized = False def __getattribute__(self, key: str) -> Any: if key == "data": if not self._initialized: self._initialized = True super().update(self._storage.get_trial_system_attrs(self._trial_id)) return super().__getattribute__(key) optuna-3.5.0/optuna/version.py000066400000000000000000000000261453453102400163710ustar00rootroot00000000000000__version__ = "3.5.0" optuna-3.5.0/optuna/visualization/000077500000000000000000000000001453453102400172355ustar00rootroot00000000000000optuna-3.5.0/optuna/visualization/__init__.py000066400000000000000000000023601453453102400213470ustar00rootroot00000000000000from optuna.visualization import matplotlib from optuna.visualization._contour import plot_contour from optuna.visualization._edf import plot_edf from optuna.visualization._hypervolume_history import plot_hypervolume_history from optuna.visualization._intermediate_values import plot_intermediate_values from optuna.visualization._optimization_history import plot_optimization_history from optuna.visualization._parallel_coordinate import plot_parallel_coordinate from optuna.visualization._param_importances import plot_param_importances from optuna.visualization._pareto_front import plot_pareto_front from optuna.visualization._rank import plot_rank from optuna.visualization._slice import plot_slice from optuna.visualization._terminator_improvement import plot_terminator_improvement from optuna.visualization._timeline import plot_timeline from optuna.visualization._utils import is_available __all__ = [ "is_available", "matplotlib", "plot_contour", "plot_edf", "plot_hypervolume_history", "plot_intermediate_values", "plot_optimization_history", "plot_parallel_coordinate", "plot_param_importances", "plot_pareto_front", "plot_slice", "plot_rank", "plot_terminator_improvement", "plot_timeline", ] optuna-3.5.0/optuna/visualization/_contour.py000066400000000000000000000355071453453102400214510ustar00rootroot00000000000000from __future__ import annotations import math from typing import Any from typing import Callable from typing import NamedTuple from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _is_numerical from optuna.visualization._utils import _is_reverse_scale if _imports.is_successful(): from optuna.visualization._plotly_imports import Contour from optuna.visualization._plotly_imports import go from optuna.visualization._plotly_imports import make_subplots from optuna.visualization._plotly_imports import Scatter from optuna.visualization._utils import COLOR_SCALE _logger = get_logger(__name__) PADDING_RATIO = 0.05 class _AxisInfo(NamedTuple): name: str range: tuple[float, float] is_log: bool is_cat: bool indices: list[str | int | float] values: list[str | float | None] class _SubContourInfo(NamedTuple): xaxis: _AxisInfo yaxis: _AxisInfo z_values: dict[tuple[int, int], float] constraints: list[bool] = [] class _ContourInfo(NamedTuple): sorted_params: list[str] sub_plot_infos: list[list[_SubContourInfo]] reverse_scale: bool target_name: str class _PlotValues(NamedTuple): x: list[Any] y: list[Any] def plot_contour( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the parameter relationship as contour plot in a study. Note that, if a parameter contains missing values, a trial with missing values is not plotted. Example: The following code snippet shows how to plot the parameter relationship as contour plot. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) fig = optuna.visualization.plot_contour(study, params=["x", "y"]) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`plotly.graph_objs.Figure` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() info = _get_contour_info(study, params, target, target_name) return _get_contour_plot(info) def _get_contour_plot(info: _ContourInfo) -> "go.Figure": layout = go.Layout(title="Contour Plot") sorted_params = info.sorted_params sub_plot_infos = info.sub_plot_infos reverse_scale = info.reverse_scale target_name = info.target_name if len(sorted_params) <= 1: return go.Figure(data=[], layout=layout) if len(sorted_params) == 2: x_param = sorted_params[0] y_param = sorted_params[1] sub_plot_info = sub_plot_infos[0][0] sub_plots = _get_contour_subplot(sub_plot_info, reverse_scale, target_name) figure = go.Figure(data=sub_plots, layout=layout) figure.update_xaxes(title_text=x_param, range=sub_plot_info.xaxis.range) figure.update_yaxes(title_text=y_param, range=sub_plot_info.yaxis.range) if sub_plot_info.xaxis.is_cat: figure.update_xaxes(type="category") if sub_plot_info.yaxis.is_cat: figure.update_yaxes(type="category") if sub_plot_info.xaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.xaxis.range] figure.update_xaxes(range=log_range, type="log") if sub_plot_info.yaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.yaxis.range] figure.update_yaxes(range=log_range, type="log") else: figure = make_subplots( rows=len(sorted_params), cols=len(sorted_params), shared_xaxes=True, shared_yaxes=True ) figure.update_layout(layout) showscale = True # showscale option only needs to be specified once. for x_i, x_param in enumerate(sorted_params): for y_i, y_param in enumerate(sorted_params): if x_param == y_param: figure.add_trace(go.Scatter(), row=y_i + 1, col=x_i + 1) else: sub_plots = _get_contour_subplot( sub_plot_infos[y_i][x_i], reverse_scale, target_name ) contour = sub_plots[0] scatter = sub_plots[1] contour.update(showscale=showscale) # showscale's default is True. if showscale: showscale = False figure.add_trace(contour, row=y_i + 1, col=x_i + 1) figure.add_trace(scatter, row=y_i + 1, col=x_i + 1) xaxis = sub_plot_infos[y_i][x_i].xaxis yaxis = sub_plot_infos[y_i][x_i].yaxis figure.update_xaxes(range=xaxis.range, row=y_i + 1, col=x_i + 1) figure.update_yaxes(range=yaxis.range, row=y_i + 1, col=x_i + 1) if xaxis.is_cat: figure.update_xaxes(type="category", row=y_i + 1, col=x_i + 1) if yaxis.is_cat: figure.update_yaxes(type="category", row=y_i + 1, col=x_i + 1) if xaxis.is_log: log_range = [math.log10(p) for p in xaxis.range] figure.update_xaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if yaxis.is_log: log_range = [math.log10(p) for p in yaxis.range] figure.update_yaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if x_i == 0: figure.update_yaxes(title_text=y_param, row=y_i + 1, col=x_i + 1) if y_i == len(sorted_params) - 1: figure.update_xaxes(title_text=x_param, row=y_i + 1, col=x_i + 1) return figure def _get_contour_subplot( info: _SubContourInfo, reverse_scale: bool, target_name: str = "Objective Value", ) -> tuple["Contour", "Scatter", "Scatter"]: x_indices = info.xaxis.indices y_indices = info.yaxis.indices feasible = _PlotValues([], []) infeasible = _PlotValues([], []) for x_value, y_value, c in zip(info.xaxis.values, info.yaxis.values, info.constraints): if x_value is not None and y_value is not None: if c: feasible.x.append(x_value) feasible.y.append(y_value) else: infeasible.x.append(x_value) infeasible.y.append(y_value) z_values = [ [float("nan") for _ in range(len(info.xaxis.indices))] for _ in range(len(info.yaxis.indices)) ] for (x_i, y_i), z_value in info.z_values.items(): z_values[y_i][x_i] = z_value if len(x_indices) < 2 or len(y_indices) < 2: return go.Contour(), go.Scatter(), go.Scatter() contour = go.Contour( x=x_indices, y=y_indices, z=z_values, colorbar={"title": target_name}, colorscale=COLOR_SCALE, connectgaps=True, contours_coloring="heatmap", hoverinfo="none", line_smoothing=1.3, reversescale=reverse_scale, ) return ( contour, _create_scatter(feasible.x, feasible.y, is_feasible=True), _create_scatter(infeasible.x, infeasible.y, is_feasible=False), ) def _create_scatter(x: list[Any], y: list[Any], is_feasible: bool) -> Scatter: edge_color = "Gray" marker_color = "black" if is_feasible else "#cccccc" name = "Feasible Trial" if is_feasible else "Infeasible Trial" return go.Scatter( x=x, y=y, marker={ "line": {"width": 2.0, "color": edge_color}, "color": marker_color, }, mode="markers", name=name, showlegend=False, ) def _get_contour_info( study: Study, params: list[str] | None = None, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> _ContourInfo: _check_plot_args(study, target, target_name) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) all_params = {p_name for t in trials for p_name in t.params.keys()} if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") sorted_params = [] elif params is None: sorted_params = sorted(all_params) else: if len(params) <= 1: _logger.warning("The length of params must be greater than 1.") for input_p_name in params: if input_p_name not in all_params: raise ValueError("Parameter {} does not exist in your study.".format(input_p_name)) sorted_params = sorted(set(params)) sub_plot_infos: list[list[_SubContourInfo]] if len(sorted_params) == 2: x_param = sorted_params[0] y_param = sorted_params[1] sub_plot_info = _get_contour_subplot_info(study, trials, x_param, y_param, target) sub_plot_infos = [[sub_plot_info]] else: sub_plot_infos = [] for i, y_param in enumerate(sorted_params): sub_plot_infos.append([]) for x_param in sorted_params: sub_plot_info = _get_contour_subplot_info(study, trials, x_param, y_param, target) sub_plot_infos[i].append(sub_plot_info) reverse_scale = _is_reverse_scale(study, target) return _ContourInfo( sorted_params=sorted_params, sub_plot_infos=sub_plot_infos, reverse_scale=reverse_scale, target_name=target_name, ) def _get_contour_subplot_info( study: Study, trials: list[FrozenTrial], x_param: str, y_param: str, target: Callable[[FrozenTrial], float] | None, ) -> _SubContourInfo: xaxis = _get_axis_info(trials, x_param) yaxis = _get_axis_info(trials, y_param) if x_param == y_param: return _SubContourInfo(xaxis=xaxis, yaxis=yaxis, z_values={}) if len(xaxis.indices) < 2: _logger.warning("Param {} unique value length is less than 2.".format(x_param)) return _SubContourInfo(xaxis=xaxis, yaxis=yaxis, z_values={}) if len(yaxis.indices) < 2: _logger.warning("Param {} unique value length is less than 2.".format(y_param)) return _SubContourInfo(xaxis=xaxis, yaxis=yaxis, z_values={}) z_values: dict[tuple[int, int], float] = {} for i, trial in enumerate(trials): if x_param not in trial.params or y_param not in trial.params: continue x_value = xaxis.values[i] y_value = yaxis.values[i] assert x_value is not None assert y_value is not None x_i = xaxis.indices.index(x_value) y_i = yaxis.indices.index(y_value) if target is None: value = trial.value else: value = target(trial) assert value is not None existing = z_values.get((x_i, y_i)) if existing is None or target is not None: # When target function is present, we can't be sure what the z-value # represents and therefore we don't know how to select the best one. z_values[(x_i, y_i)] = value else: z_values[(x_i, y_i)] = ( min(existing, value) if study.direction is StudyDirection.MINIMIZE else max(existing, value) ) return _SubContourInfo( xaxis=xaxis, yaxis=yaxis, z_values=z_values, constraints=[_satisfy_constraints(t) for t in trials], ) def _satisfy_constraints(trial: FrozenTrial) -> bool: constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) return constraints is None or all([x <= 0.0 for x in constraints]) def _get_axis_info(trials: list[FrozenTrial], param_name: str) -> _AxisInfo: values: list[str | float | None] if _is_numerical(trials, param_name): values = [t.params.get(param_name) for t in trials] else: values = [ str(t.params.get(param_name)) if param_name in t.params else None for t in trials ] min_value = min([v for v in values if v is not None]) max_value = max([v for v in values if v is not None]) if _is_log_scale(trials, param_name): min_value = float(min_value) max_value = float(max_value) padding = (math.log10(max_value) - math.log10(min_value)) * PADDING_RATIO min_value = math.pow(10, math.log10(min_value) - padding) max_value = math.pow(10, math.log10(max_value) + padding) is_log = True is_cat = False elif _is_numerical(trials, param_name): min_value = float(min_value) max_value = float(max_value) padding = (max_value - min_value) * PADDING_RATIO min_value = min_value - padding max_value = max_value + padding is_log = False is_cat = False else: unique_values = set(values) span = len(unique_values) - 1 if None in unique_values: span -= 1 padding = span * PADDING_RATIO min_value = -padding max_value = span + padding is_log = False is_cat = True indices = sorted(set([v for v in values if v is not None])) if len(indices) < 2: return _AxisInfo( name=param_name, range=(min_value, max_value), is_log=is_log, is_cat=is_cat, indices=indices, values=values, ) if _is_numerical(trials, param_name): indices.insert(0, min_value) indices.append(max_value) return _AxisInfo( name=param_name, range=(min_value, max_value), is_log=is_log, is_cat=is_cat, indices=indices, values=values, ) optuna-3.5.0/optuna/visualization/_edf.py000066400000000000000000000133051453453102400205060ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import cast from typing import NamedTuple import numpy as np from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) NUM_SAMPLES_X_AXIS = 100 class _EDFLineInfo(NamedTuple): study_name: str y_values: np.ndarray class _EDFInfo(NamedTuple): lines: list[_EDFLineInfo] x_values: np.ndarray def plot_edf( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the objective value EDF (empirical distribution function) of a study. Note that only the complete trials are considered when plotting the EDF. .. note:: EDF is useful to analyze and improve search spaces. For instance, you can see a practical use case of EDF in the paper `Designing Network Design Spaces `_. .. note:: The plotted EDF assumes that the value of the objective function is in accordance with the uniform distribution over the objective space. Example: The following code snippet shows how to plot EDF. .. plotly:: import math import optuna def ackley(x, y): a = 20 * math.exp(-0.2 * math.sqrt(0.5 * (x ** 2 + y ** 2))) b = math.exp(0.5 * (math.cos(2 * math.pi * x) + math.cos(2 * math.pi * y))) return -a - b + math.e + 20 def objective(trial, low, high): x = trial.suggest_float("x", low, high) y = trial.suggest_float("y", low, high) return ackley(x, y) sampler = optuna.samplers.RandomSampler(seed=10) # Widest search space. study0 = optuna.create_study(study_name="x=[0,5), y=[0,5)", sampler=sampler) study0.optimize(lambda t: objective(t, 0, 5), n_trials=500) # Narrower search space. study1 = optuna.create_study(study_name="x=[0,4), y=[0,4)", sampler=sampler) study1.optimize(lambda t: objective(t, 0, 4), n_trials=500) # Narrowest search space but it doesn't include the global optimum point. study2 = optuna.create_study(study_name="x=[1,3), y=[1,3)", sampler=sampler) study2.optimize(lambda t: objective(t, 1, 3), n_trials=500) fig = optuna.visualization.plot_edf([study0, study1, study2]) fig.show() Args: study: A target :class:`~optuna.study.Study` object. You can pass multiple studies if you want to compare those EDFs. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() layout = go.Layout( title="Empirical Distribution Function Plot", xaxis={"title": target_name}, yaxis={"title": "Cumulative Probability"}, ) info = _get_edf_info(study, target, target_name) edf_lines = info.lines if len(edf_lines) == 0: return go.Figure(data=[], layout=layout) traces = [] for study_name, y_values in edf_lines: traces.append(go.Scatter(x=info.x_values, y=y_values, name=study_name, mode="lines")) figure = go.Figure(data=traces, layout=layout) figure.update_yaxes(range=[0, 1]) return figure def _get_edf_info( study: Study | Sequence[Study], target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> _EDFInfo: if isinstance(study, Study): studies = [study] else: studies = list(study) _check_plot_args(studies, target, target_name) if len(studies) == 0: _logger.warning("There are no studies.") return _EDFInfo(lines=[], x_values=np.array([])) if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target study_names = [] all_values: list[np.ndarray] = [] for study in studies: trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) values = np.array([target(trial) for trial in trials]) all_values.append(values) study_names.append(study.study_name) if all(len(values) == 0 for values in all_values): _logger.warning("There are no complete trials.") return _EDFInfo(lines=[], x_values=np.array([])) min_x_value = np.min(np.concatenate(all_values)) max_x_value = np.max(np.concatenate(all_values)) x_values = np.linspace(min_x_value, max_x_value, NUM_SAMPLES_X_AXIS) edf_line_info_list = [] for study_name, values in zip(study_names, all_values): y_values = np.sum(values[:, np.newaxis] <= x_values, axis=0) / values.size edf_line_info_list.append(_EDFLineInfo(study_name=study_name, y_values=y_values)) return _EDFInfo(lines=edf_line_info_list, x_values=x_values) optuna-3.5.0/optuna/visualization/_hypervolume_history.py000066400000000000000000000121131453453102400241040ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence from typing import NamedTuple import numpy as np from optuna._experimental import experimental_func from optuna._hypervolume import WFG from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study._multi_objective import _dominates from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _HypervolumeHistoryInfo(NamedTuple): trial_numbers: list[int] values: list[float] @experimental_func("3.3.0") def plot_hypervolume_history( study: Study, reference_point: Sequence[float], ) -> "go.Figure": """Plot hypervolume history of all trials in a study. Example: The following code snippet shows how to plot optimization history. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) reference_point=[100., 50.] fig = optuna.visualization.plot_hypervolume_history(study, reference_point) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their hypervolumes. The number of objectives must be 2 or more. reference_point: A reference point to use for hypervolume computation. The dimension of the reference point must be the same as the number of objectives. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() if not study._is_multi_objective(): raise ValueError( "Study must be multi-objective. For single-objective optimization, " "please use plot_optimization_history instead." ) if len(reference_point) != len(study.directions): raise ValueError( "The dimension of the reference point must be the same as the number of objectives." ) info = _get_hypervolume_history_info(study, np.asarray(reference_point, dtype=np.float64)) return _get_hypervolume_history_plot(info) def _get_hypervolume_history_plot( info: _HypervolumeHistoryInfo, ) -> "go.Figure": layout = go.Layout( title="Hypervolume History Plot", xaxis={"title": "Trial"}, yaxis={"title": "Hypervolume"}, ) data = go.Scatter( x=info.trial_numbers, y=info.values, mode="lines+markers", ) return go.Figure(data=data, layout=layout) def _get_hypervolume_history_info( study: Study, reference_point: np.ndarray, ) -> _HypervolumeHistoryInfo: completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) if len(completed_trials) == 0: _logger.warning("Your study does not have any completed trials.") # Our hypervolume computation module assumes that all objectives are minimized. # Here we transform the objective values and the reference point. signs = np.asarray([1 if d == StudyDirection.MINIMIZE else -1 for d in study.directions]) minimization_reference_point = signs * reference_point # Only feasible trials are considered in hypervolume computation. trial_numbers = [] values = [] best_trials: list[FrozenTrial] = [] hypervolume = 0.0 for trial in completed_trials: trial_numbers.append(trial.number) has_constraints = _CONSTRAINTS_KEY in trial.system_attrs if has_constraints: constraints_values = trial.system_attrs[_CONSTRAINTS_KEY] if any(map(lambda x: x > 0.0, constraints_values)): # The trial is infeasible. values.append(hypervolume) continue if any(map(lambda t: _dominates(t, trial, study.directions), best_trials)): # The trial is not on the Pareto front. values.append(hypervolume) continue best_trials = list( filter(lambda t: not _dominates(trial, t, study.directions), best_trials) ) + [trial] solution_set = np.asarray( list( filter( lambda v: (v <= minimization_reference_point).all(), [signs * trial.values for trial in best_trials], ) ) ) if solution_set.size > 0: hypervolume = WFG().compute(solution_set, minimization_reference_point) values.append(hypervolume) if len(best_trials) == 0: _logger.warning("Your study does not have any feasible trials.") return _HypervolumeHistoryInfo(trial_numbers, values) optuna-3.5.0/optuna/visualization/_intermediate_values.py000066400000000000000000000074471453453102400240130ustar00rootroot00000000000000from __future__ import annotations from typing import NamedTuple from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _TrialInfo(NamedTuple): trial_number: int sorted_intermediate_values: list[tuple[int, float]] feasible: bool class _IntermediatePlotInfo(NamedTuple): trial_infos: list[_TrialInfo] def _get_intermediate_plot_info(study: Study) -> _IntermediatePlotInfo: trials = study.get_trials( deepcopy=False, states=(TrialState.PRUNED, TrialState.COMPLETE, TrialState.RUNNING) ) def _satisfies_constraints(trial: FrozenTrial) -> bool: constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) return constraints is None or all([x <= 0.0 for x in constraints]) trial_infos = [ _TrialInfo( trial.number, sorted(trial.intermediate_values.items()), _satisfies_constraints(trial) ) for trial in trials if len(trial.intermediate_values) > 0 ] if len(trials) == 0: _logger.warning("Study instance does not contain trials.") elif len(trial_infos) == 0: _logger.warning( "You need to set up the pruning feature to utilize `plot_intermediate_values()`" ) return _IntermediatePlotInfo(trial_infos) def plot_intermediate_values(study: Study) -> "go.Figure": """Plot intermediate values of all trials in a study. Example: The following code snippet shows how to plot intermediate values. .. plotly:: import optuna def f(x): return (x - 2) ** 2 def df(x): return 2 * x - 4 def objective(trial): lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) x = 3 for step in range(128): y = f(x) trial.report(y, step=step) if trial.should_prune(): raise optuna.TrialPruned() gy = df(x) x -= gy * lr return y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=16) fig = optuna.visualization.plot_intermediate_values(study) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their intermediate values. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() return _get_intermediate_plot(_get_intermediate_plot_info(study)) def _get_intermediate_plot(info: _IntermediatePlotInfo) -> "go.Figure": layout = go.Layout( title="Intermediate Values Plot", xaxis={"title": "Step"}, yaxis={"title": "Intermediate Value"}, showlegend=False, ) trial_infos = info.trial_infos if len(trial_infos) == 0: return go.Figure(data=[], layout=layout) default_marker = {"maxdisplayed": 10} traces = [ go.Scatter( x=tuple((x for x, _ in tinfo.sorted_intermediate_values)), y=tuple((y for _, y in tinfo.sorted_intermediate_values)), mode="lines+markers", marker=default_marker if tinfo.feasible else {**default_marker, "color": "#CCCCCC"}, # type: ignore[dict-item] name="Trial{}".format(tinfo.trial_number), ) for tinfo in trial_infos ] return go.Figure(data=traces, layout=layout) optuna-3.5.0/optuna/visualization/_optimization_history.py000066400000000000000000000271201453453102400242570ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from enum import Enum import math from typing import cast from typing import NamedTuple import numpy as np from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _ValueState(Enum): Feasible = 0 Infeasible = 1 Incomplete = 2 class _ValuesInfo(NamedTuple): values: list[float] stds: list[float] | None label_name: str states: list[_ValueState] class _OptimizationHistoryInfo(NamedTuple): trial_numbers: list[int] values_info: _ValuesInfo best_values_info: _ValuesInfo | None def _get_optimization_history_info_list( study: Study | Sequence[Study], target: Callable[[FrozenTrial], float] | None, target_name: str, error_bar: bool, ) -> list[_OptimizationHistoryInfo]: _check_plot_args(study, target, target_name) if isinstance(study, Study): studies = [study] else: studies = list(study) info_list: list[_OptimizationHistoryInfo] = [] for study in studies: trials = study.get_trials() label_name = target_name if len(studies) == 1 else f"{target_name} of {study.study_name}" values = [] value_states = [] for trial in trials: if trial.state != TrialState.COMPLETE: values.append(float("nan")) value_states.append(_ValueState.Incomplete) continue constraints = trial.system_attrs.get(_CONSTRAINTS_KEY) if constraints is None or all([x <= 0.0 for x in constraints]): value_states.append(_ValueState.Feasible) else: value_states.append(_ValueState.Infeasible) if target is not None: values.append(target(trial)) else: values.append(cast(float, trial.value)) if target is not None: # We don't calculate best for user-defined target function since we cannot tell # which direction is better. best_values_info: _ValuesInfo | None = None else: feasible_best_values = [] if study.direction == StudyDirection.MINIMIZE: feasible_best_values = [ v if s == _ValueState.Feasible else float("inf") for v, s in zip(values, value_states) ] best_values = list(np.minimum.accumulate(feasible_best_values)) else: feasible_best_values = [ v if s == _ValueState.Feasible else -float("inf") for v, s in zip(values, value_states) ] best_values = list(np.maximum.accumulate(feasible_best_values)) best_label_name = ( "Best Value" if len(studies) == 1 else f"Best Value of {study.study_name}" ) best_values_info = _ValuesInfo(best_values, None, best_label_name, value_states) info_list.append( _OptimizationHistoryInfo( trial_numbers=[t.number for t in trials], values_info=_ValuesInfo(values, None, label_name, value_states), best_values_info=best_values_info, ) ) if len(info_list) == 0: _logger.warning("There are no studies.") feasible_trial_count = sum( info.values_info.states.count(_ValueState.Feasible) for info in info_list ) infeasible_trial_count = sum( info.values_info.states.count(_ValueState.Infeasible) for info in info_list ) if feasible_trial_count + infeasible_trial_count == 0: _logger.warning("There are no complete trials.") info_list.clear() if not error_bar: return info_list # When error_bar=True, a list of 0 or 1 element is returned. if len(info_list) == 0: return [] if feasible_trial_count == 0: _logger.warning("There are no feasible trials.") return [] all_trial_numbers = [number for info in info_list for number in info.trial_numbers] max_num_trial = max(all_trial_numbers) + 1 def _aggregate(label_name: str, use_best_value: bool) -> tuple[list[int], _ValuesInfo]: # Calculate mean and std of values for each trial number. values: list[list[float]] = [[] for _ in range(max_num_trial)] states: list[list[_ValueState]] = [[] for _ in range(max_num_trial)] assert info_list is not None for trial_numbers, values_info, best_values_info in info_list: if use_best_value: assert best_values_info is not None values_info = best_values_info for n, v, s in zip(trial_numbers, values_info.values, values_info.states): if not math.isinf(v): if not use_best_value and s == _ValueState.Feasible: values[n].append(v) elif use_best_value: values[n].append(v) states[n].append(s) trial_numbers_union: list[int] = [] value_states: list[_ValueState] = [] value_means: list[float] = [] value_stds: list[float] = [] for i in range(max_num_trial): if len(states[i]) > 0 and _ValueState.Feasible in states[i]: value_states.append(_ValueState.Feasible) trial_numbers_union.append(i) value_means.append(np.mean(values[i]).item()) value_stds.append(np.std(values[i]).item()) else: value_states.append(_ValueState.Infeasible) return trial_numbers_union, _ValuesInfo(value_means, value_stds, label_name, value_states) eb_trial_numbers, eb_values_info = _aggregate(target_name, False) eb_best_values_info: _ValuesInfo | None = None if target is None: _, eb_best_values_info = _aggregate("Best Value", True) return [_OptimizationHistoryInfo(eb_trial_numbers, eb_values_info, eb_best_values_info)] def plot_optimization_history( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", error_bar: bool = False, ) -> "go.Figure": """Plot optimization history of all trials in a study. Example: The following code snippet shows how to plot optimization history. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) fig = optuna.visualization.plot_optimization_history(study) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. You can pass multiple studies if you want to compare those optimization histories. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. error_bar: A flag to show the error bar. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() info_list = _get_optimization_history_info_list(study, target, target_name, error_bar) return _get_optimization_history_plot(info_list, target_name) def _get_optimization_history_plot( info_list: list[_OptimizationHistoryInfo], target_name: str, ) -> "go.Figure": layout = go.Layout( title="Optimization History Plot", xaxis={"title": "Trial"}, yaxis={"title": target_name}, ) traces = [] for trial_numbers, values_info, best_values_info in info_list: infeasible_trial_numbers = [ n for n, s in zip(trial_numbers, values_info.states) if s == _ValueState.Infeasible ] if values_info.stds is None: error_y = None feasible_trial_numbers = [ num for num, s in zip(trial_numbers, values_info.states) if s == _ValueState.Feasible ] feasible_trial_values = [] for num in feasible_trial_numbers: feasible_trial_values.append(values_info.values[num]) infeasible_trial_values = [] for num in infeasible_trial_numbers: infeasible_trial_values.append(values_info.values[num]) else: if ( _ValueState.Infeasible in values_info.states or _ValueState.Incomplete in values_info.states ): _logger.warning( "Your study contains infeasible trials. " "In optimization history plot, " "error bars are calculated for only feasible trial values." ) error_y = {"type": "data", "array": values_info.stds, "visible": True} feasible_trial_numbers = trial_numbers feasible_trial_values = values_info.values infeasible_trial_values = [] traces.append( go.Scatter( x=feasible_trial_numbers, y=feasible_trial_values, error_y=error_y, mode="markers", name=values_info.label_name, ) ) if best_values_info is not None: traces.append( go.Scatter( x=trial_numbers, y=best_values_info.values, name=best_values_info.label_name, mode="lines", ) ) if best_values_info.stds is not None: upper = np.array(best_values_info.values) + np.array(best_values_info.stds) traces.append( go.Scatter( x=trial_numbers, y=upper, mode="lines", line=dict(width=0.01), showlegend=False, ) ) lower = np.array(best_values_info.values) - np.array(best_values_info.stds) traces.append( go.Scatter( x=trial_numbers, y=lower, mode="none", showlegend=False, fill="tonexty", fillcolor="rgba(255,0,0,0.2)", ) ) traces.append( go.Scatter( x=infeasible_trial_numbers, y=infeasible_trial_values, error_y=error_y, mode="markers", name="Infeasible Trial", marker={"color": "#cccccc"}, showlegend=False, ) ) return go.Figure(data=traces, layout=layout) optuna-3.5.0/optuna/visualization/_parallel_coordinate.py000066400000000000000000000250661453453102400237620ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict import math from typing import Any from typing import Callable from typing import cast from typing import NamedTuple import numpy as np from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _get_skipped_trial_numbers from optuna.visualization._utils import _is_categorical from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _is_numerical from optuna.visualization._utils import _is_reverse_scale if _imports.is_successful(): from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE _logger = get_logger(__name__) class _DimensionInfo(NamedTuple): label: str values: tuple[float, ...] range: tuple[float, float] is_log: bool is_cat: bool tickvals: list[int | float] ticktext: list[str] class _ParallelCoordinateInfo(NamedTuple): dim_objective: _DimensionInfo dims_params: list[_DimensionInfo] reverse_scale: bool target_name: str def plot_parallel_coordinate( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the high-dimensional parameter relationships in a study. Note that, if a parameter contains missing values, a trial with missing values is not plotted. Example: The following code snippet shows how to plot the high-dimensional parameter relationships. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) fig = optuna.visualization.plot_parallel_coordinate(study, params=["x", "y"]) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. Returns: A :class:`plotly.graph_objs.Figure` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() info = _get_parallel_coordinate_info(study, params, target, target_name) return _get_parallel_coordinate_plot(info) def _get_parallel_coordinate_plot(info: _ParallelCoordinateInfo) -> "go.Figure": layout = go.Layout(title="Parallel Coordinate Plot") if len(info.dims_params) == 0 or len(info.dim_objective.values) == 0: return go.Figure(data=[], layout=layout) dims = _get_dims_from_info(info) reverse_scale = info.reverse_scale target_name = info.target_name traces = [ go.Parcoords( dimensions=dims, labelangle=30, labelside="bottom", line={ "color": dims[0]["values"], "colorscale": COLOR_SCALE, "colorbar": {"title": target_name}, "showscale": True, "reversescale": reverse_scale, }, ) ] figure = go.Figure(data=traces, layout=layout) return figure def _get_parallel_coordinate_info( study: Study, params: list[str] | None = None, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> _ParallelCoordinateInfo: _check_plot_args(study, target, target_name) reverse_scale = _is_reverse_scale(study, target) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) all_params = {p_name for t in trials for p_name in t.params.keys()} if params is not None: for input_p_name in params: if input_p_name not in all_params: raise ValueError("Parameter {} does not exist in your study.".format(input_p_name)) all_params = set(params) sorted_params = sorted(all_params) if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target skipped_trial_numbers = _get_skipped_trial_numbers(trials, sorted_params) objectives = tuple([target(t) for t in trials if t.number not in skipped_trial_numbers]) # The value of (0, 0) is a dummy range. It is ignored when we plot. objective_range = (min(objectives), max(objectives)) if len(objectives) > 0 else (0, 0) dim_objective = _DimensionInfo( label=target_name, values=objectives, range=objective_range, is_log=False, is_cat=False, tickvals=[], ticktext=[], ) if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") return _ParallelCoordinateInfo( dim_objective=dim_objective, dims_params=[], reverse_scale=reverse_scale, target_name=target_name, ) if len(objectives) == 0: _logger.warning("Your study has only completed trials with missing parameters.") return _ParallelCoordinateInfo( dim_objective=dim_objective, dims_params=[], reverse_scale=reverse_scale, target_name=target_name, ) numeric_cat_params_indices: list[int] = [] dims = [] for dim_index, p_name in enumerate(sorted_params, start=1): values = [] for t in trials: if t.number in skipped_trial_numbers: continue if p_name in t.params: values.append(t.params[p_name]) if _is_log_scale(trials, p_name): values = [math.log10(v) for v in values] min_value = min(values) max_value = max(values) tickvals: list[int | float] = list( range(math.ceil(min_value), math.floor(max_value) + 1) ) if min_value not in tickvals: tickvals = [min_value] + tickvals if max_value not in tickvals: tickvals = tickvals + [max_value] dim = _DimensionInfo( label=_truncate_label(p_name), values=tuple(values), range=(min_value, max_value), is_log=True, is_cat=False, tickvals=tickvals, ticktext=["{:.3g}".format(math.pow(10, x)) for x in tickvals], ) elif _is_categorical(trials, p_name): vocab: defaultdict[int | str, int] = defaultdict(lambda: len(vocab)) ticktext: list[str] if _is_numerical(trials, p_name): _ = [vocab[v] for v in sorted(values)] values = [vocab[v] for v in values] ticktext = [str(v) for v in list(sorted(vocab.keys()))] numeric_cat_params_indices.append(dim_index) else: values = [vocab[v] for v in values] ticktext = [str(v) for v in list(sorted(vocab.keys(), key=lambda x: vocab[x]))] dim = _DimensionInfo( label=_truncate_label(p_name), values=tuple(values), range=(min(values), max(values)), is_log=False, is_cat=True, tickvals=list(range(len(vocab))), ticktext=ticktext, ) else: dim = _DimensionInfo( label=_truncate_label(p_name), values=tuple(values), range=(min(values), max(values)), is_log=False, is_cat=False, tickvals=[], ticktext=[], ) dims.append(dim) if numeric_cat_params_indices: dims.insert(0, dim_objective) # np.lexsort consumes the sort keys the order from back to front. # So the values of parameters have to be reversed the order. idx = np.lexsort([dims[index].values for index in numeric_cat_params_indices][::-1]) updated_dims = [] for dim in dims: # Since the values are mapped to other categories by the index, # the index will be swapped according to the sorted index of numeric params. updated_dims.append( _DimensionInfo( label=dim.label, values=tuple(np.array(dim.values)[idx]), range=dim.range, is_log=dim.is_log, is_cat=dim.is_cat, tickvals=dim.tickvals, ticktext=dim.ticktext, ) ) dim_objective = updated_dims[0] dims = updated_dims[1:] return _ParallelCoordinateInfo( dim_objective=dim_objective, dims_params=dims, reverse_scale=reverse_scale, target_name=target_name, ) def _get_dims_from_info(info: _ParallelCoordinateInfo) -> list[dict[str, Any]]: dims = [ { "label": info.dim_objective.label, "values": info.dim_objective.values, "range": info.dim_objective.range, } ] for dim in info.dims_params: if dim.is_log or dim.is_cat: dims.append( { "label": dim.label, "values": dim.values, "range": dim.range, "tickvals": dim.tickvals, "ticktext": dim.ticktext, } ) else: dims.append({"label": dim.label, "values": dim.values, "range": dim.range}) return dims def _truncate_label(label: str) -> str: return label if len(label) < 20 else "{}...".format(label[:17]) optuna-3.5.0/optuna/visualization/_param_importances.py000066400000000000000000000176101453453102400234570ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from typing import NamedTuple import optuna from optuna.distributions import BaseDistribution from optuna.importance._base import BaseImportanceEvaluator from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite if _imports.is_successful(): from optuna.visualization._plotly_imports import go logger = get_logger(__name__) class _ImportancesInfo(NamedTuple): importance_values: list[float] param_names: list[str] importance_labels: list[str] target_name: str def _get_importances_info( study: Study, evaluator: BaseImportanceEvaluator | None, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> _ImportancesInfo: _check_plot_args(study, target, target_name) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) if len(trials) == 0: logger.warning("Study instance does not contain completed trials.") return _ImportancesInfo( importance_values=[], param_names=[], importance_labels=[], target_name=target_name, ) importances = optuna.importance.get_param_importances( study, evaluator=evaluator, params=params, target=target ) importances = dict(reversed(list(importances.items()))) importance_values = list(importances.values()) param_names = list(importances.keys()) importance_labels = [f"{val:.2f}" if val >= 0.01 else "<0.01" for val in importance_values] return _ImportancesInfo( importance_values=importance_values, param_names=param_names, importance_labels=importance_labels, target_name=target_name, ) def _get_importances_infos( study: Study, evaluator: BaseImportanceEvaluator | None, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> tuple[_ImportancesInfo, ...]: metric_names = study.metric_names if target or not study._is_multi_objective(): target_name = metric_names[0] if metric_names is not None and not target else target_name importances_infos: tuple[_ImportancesInfo, ...] = ( _get_importances_info( study, evaluator, params, target=target, target_name=target_name, ), ) else: n_objectives = len(study.directions) target_names = ( metric_names if metric_names is not None else (f"{target_name} {objective_id}" for objective_id in range(n_objectives)) ) importances_infos = tuple( _get_importances_info( study, evaluator, params, target=lambda t: t.values[objective_id], target_name=target_name, ) for objective_id, target_name in enumerate(target_names) ) return importances_infos def plot_param_importances( study: Study, evaluator: BaseImportanceEvaluator | None = None, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot hyperparameter importances. Example: The following code snippet shows how to plot hyperparameter importances. .. plotly:: import optuna def objective(trial): x = trial.suggest_int("x", 0, 2) y = trial.suggest_float("y", -1.0, 1.0) z = trial.suggest_float("z", 0.0, 1.5) return x ** 2 + y ** 3 - z ** 4 sampler = optuna.samplers.RandomSampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) fig = optuna.visualization.plot_param_importances(study) fig.show() .. seealso:: This function visualizes the results of :func:`optuna.importance.get_param_importances`. Args: study: An optimized study. evaluator: An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to :class:`~optuna.importance.FanovaImportanceEvaluator`. .. note:: :class:`~optuna.importance.FanovaImportanceEvaluator` takes over 1 minute when given a study that contains 1000+ trials. We published `optuna-fast-fanova `_ library, that is a Cython accelerated fANOVA implementation. By using it, you can get hyperparameter importances within a few seconds. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. For multi-objective optimization, all objectives will be plotted if ``target`` is :obj:`None`. .. note:: This argument can be used to specify which objective to plot if ``study`` is being used for multi-objective optimization. For example, to get only the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. target_name: Target's name to display on the legend. Names set via :meth:`~optuna.study.Study.set_metric_names` will be used if ``target`` is :obj:`None`, overriding this argument. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() importances_infos = _get_importances_infos(study, evaluator, params, target, target_name) return _get_importances_plot(importances_infos, study) def _get_importances_plot(infos: tuple[_ImportancesInfo, ...], study: Study) -> "go.Figure": layout = go.Layout( title="Hyperparameter Importances", xaxis={"title": "Hyperparameter Importance"}, yaxis={"title": "Hyperparameter"}, ) data: list[go.Bar] = [] for info in infos: if not info.importance_values: continue data.append( go.Bar( x=info.importance_values, y=info.param_names, name=info.target_name, text=info.importance_labels, textposition="outside", cliponaxis=False, # Ensure text is not clipped. hovertemplate=_get_hover_template(info, study), orientation="h", ) ) return go.Figure(data, layout) def _get_distribution(param_name: str, study: Study) -> BaseDistribution: for trial in study.trials: if param_name in trial.distributions: return trial.distributions[param_name] assert False def _make_hovertext(param_name: str, importance: float, study: Study) -> str: return "{} ({}): {}".format( param_name, _get_distribution(param_name, study).__class__.__name__, importance ) def _get_hover_template(importances_info: _ImportancesInfo, study: Study) -> list[str]: return [ _make_hovertext(param_name, importance, study) for param_name, importance in zip( importances_info.param_names, importances_info.importance_values ) ] optuna-3.5.0/optuna/visualization/_pareto_front.py000066400000000000000000000425261453453102400224610ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from typing import Any from typing import NamedTuple import warnings import optuna from optuna.exceptions import ExperimentalWarning from optuna.study import Study from optuna.study._multi_objective import _get_pareto_front_trials_by_trials from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _make_hovertext if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = optuna.logging.get_logger(__name__) class _ParetoFrontInfo(NamedTuple): n_targets: int target_names: list[str] best_trials_with_values: list[tuple[FrozenTrial, list[float]]] non_best_trials_with_values: list[tuple[FrozenTrial, list[float]]] infeasible_trials_with_values: list[tuple[FrozenTrial, list[float]]] axis_order: list[int] include_dominated_trials: bool has_constraints_func: bool def plot_pareto_front( study: Study, *, target_names: list[str] | None = None, include_dominated_trials: bool = True, axis_order: list[int] | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, targets: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> "go.Figure": """Plot the Pareto front of a study. .. seealso:: Please refer to :ref:`multi_objective` for the tutorial of the Pareto front visualization. Example: The following code snippet shows how to plot the Pareto front of a study. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) fig = optuna.visualization.plot_pareto_front(study) fig.show() Example: The following code snippet shows how to plot a 2-dimensional Pareto front of a 3-dimensional study. This example is scalable, e.g., for plotting a 2- or 3-dimensional Pareto front of a 4-dimensional study and so on. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 5 * x ** 2 + 3 * y ** 2 v1 = (x - 10) ** 2 + (y - 10) ** 2 v2 = x + y return v0, v1, v2 study = optuna.create_study(directions=["minimize", "minimize", "minimize"]) study.optimize(objective, n_trials=100) fig = optuna.visualization.plot_pareto_front( study, targets=lambda t: (t.values[0], t.values[1]), target_names=["Objective 0", "Objective 1"], ) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their objective values. The number of objectives must be either 2 or 3 when ``targets`` is :obj:`None`. target_names: Objective name list used as the axis titles. If :obj:`None` is specified, "Objective {objective_index}" is used instead. If ``targets`` is specified for a study that does not contain any completed trial, ``target_name`` must be specified. include_dominated_trials: A flag to include all dominated trial's objective values. axis_order: A list of indices indicating the axis order. If :obj:`None` is specified, default order is used. ``axis_order`` and ``targets`` cannot be used at the same time. .. warning:: Deprecated in v3.0.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v5.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v3.0.0. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraint is violated. A value equal to or smaller than 0 is considered feasible. This specification is the same as in, for example, :class:`~optuna.samplers.NSGAIISampler`. If given, trials are classified into three categories: feasible and best, feasible but non-best, and infeasible. Categories are shown in different colors. Here, whether a trial is best (on Pareto front) or not is determined ignoring all infeasible trials. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. targets: A function that returns targets values to display. The argument to this function is :class:`~optuna.trial.FrozenTrial`. ``axis_order`` and ``targets`` cannot be used at the same time. If ``study.n_objectives`` is neither 2 nor 3, ``targets`` must be specified. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() info = _get_pareto_front_info( study, target_names, include_dominated_trials, axis_order, constraints_func, targets ) return _get_pareto_front_plot(info) def _get_pareto_front_plot(info: _ParetoFrontInfo) -> "go.Figure": include_dominated_trials = info.include_dominated_trials has_constraints_func = info.has_constraints_func if not has_constraints_func: data = [ _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.non_best_trials_with_values, hovertemplate="%{text}Trial", dominated_trials=True, ), _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.best_trials_with_values, hovertemplate="%{text}Best Trial", dominated_trials=False, ), ] else: data = [ _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.infeasible_trials_with_values, hovertemplate="%{text}Infeasible Trial", infeasible=True, ), _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.non_best_trials_with_values, hovertemplate="%{text}Feasible Trial", dominated_trials=True, ), _make_scatter_object( info.n_targets, info.axis_order, include_dominated_trials, info.best_trials_with_values, hovertemplate="%{text}Best Trial", dominated_trials=False, ), ] if info.n_targets == 2: layout = go.Layout( title="Pareto-front Plot", xaxis_title=info.target_names[info.axis_order[0]], yaxis_title=info.target_names[info.axis_order[1]], ) else: layout = go.Layout( title="Pareto-front Plot", scene={ "xaxis_title": info.target_names[info.axis_order[0]], "yaxis_title": info.target_names[info.axis_order[1]], "zaxis_title": info.target_names[info.axis_order[2]], }, ) return go.Figure(data=data, layout=layout) def _get_pareto_front_info( study: Study, target_names: list[str] | None = None, include_dominated_trials: bool = True, axis_order: list[int] | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, targets: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> _ParetoFrontInfo: if axis_order is not None: warnings.warn( "`axis_order` has been deprecated in v3.0.0. " "This feature will be removed in v5.0.0. " "See https://github.com/optuna/optuna/releases/tag/v3.0.0.", FutureWarning, ) if targets is not None and axis_order is not None: raise ValueError( "Using both `targets` and `axis_order` is not supported. " "Use either `targets` or `axis_order`." ) if constraints_func is not None: warnings.warn( "``constraints_func`` argument is an experimental feature." " The interface can change in the future.", ExperimentalWarning, ) feasible_trials = [] infeasible_trials = [] for trial in study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)): if all(map(lambda x: x <= 0.0, constraints_func(trial))): feasible_trials.append(trial) else: infeasible_trials.append(trial) best_trials = _get_pareto_front_trials_by_trials(feasible_trials, study.directions) if include_dominated_trials: non_best_trials = _get_non_pareto_front_trials(feasible_trials, best_trials) else: non_best_trials = [] if len(best_trials) == 0: _logger.warning("Your study does not have any completed and feasible trials.") else: all_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) best_trials = _get_pareto_front_trials_by_trials(all_trials, study.directions) if len(best_trials) == 0: _logger.warning("Your study does not have any completed trials.") if include_dominated_trials: non_best_trials = _get_non_pareto_front_trials(all_trials, best_trials) else: non_best_trials = [] infeasible_trials = [] _targets = targets if _targets is None: if len(study.directions) in (2, 3): _targets = _targets_default else: raise ValueError( "`plot_pareto_front` function only supports 2 or 3 objective" " studies when using `targets` is `None`. Please use `targets`" " if your objective studies have more than 3 objectives." ) def _make_trials_with_values( trials: list[FrozenTrial], targets: Callable[[FrozenTrial], Sequence[float]], ) -> list[tuple[FrozenTrial, list[float]]]: target_values = [targets(trial) for trial in trials] for v in target_values: if not isinstance(v, Sequence): raise ValueError( "`targets` should return a sequence of target values." " your `targets` returns {}".format(type(v)) ) return [(trial, list(v)) for trial, v in zip(trials, target_values)] best_trials_with_values = _make_trials_with_values(best_trials, _targets) non_best_trials_with_values = _make_trials_with_values(non_best_trials, _targets) infeasible_trials_with_values = _make_trials_with_values(infeasible_trials, _targets) def _infer_n_targets( trials_with_values: Sequence[tuple[FrozenTrial, Sequence[float]]] ) -> int | None: if len(trials_with_values) > 0: return len(trials_with_values[0][1]) return None # Check for `non_best_trials_with_values` can be skipped, because if `best_trials_with_values` # is empty, then `non_best_trials_with_values` will also be empty. n_targets = _infer_n_targets(best_trials_with_values) or _infer_n_targets( infeasible_trials_with_values ) if n_targets is None: if target_names is not None: n_targets = len(target_names) elif targets is None: n_targets = len(study.directions) else: raise ValueError( "If `targets` is specified for empty studies, `target_names` must be specified." ) if n_targets not in (2, 3): raise ValueError( "`plot_pareto_front` function only supports 2 or 3 targets." " you used {} targets now.".format(n_targets) ) if target_names is None: metric_names = study.metric_names if metric_names is None: target_names = [f"Objective {i}" for i in range(n_targets)] else: target_names = metric_names elif len(target_names) != n_targets: raise ValueError(f"The length of `target_names` is supposed to be {n_targets}.") if axis_order is None: axis_order = list(range(n_targets)) else: if len(axis_order) != n_targets: raise ValueError( f"Size of `axis_order` {axis_order}. Expect: {n_targets}, " f"Actual: {len(axis_order)}." ) if len(set(axis_order)) != n_targets: raise ValueError(f"Elements of given `axis_order` {axis_order} are not unique!.") if max(axis_order) > n_targets - 1: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {max(axis_order)} " f"higher than {n_targets - 1}." ) if min(axis_order) < 0: raise ValueError( f"Given `axis_order` {axis_order} contains invalid index {min(axis_order)} " "lower than 0." ) return _ParetoFrontInfo( n_targets=n_targets, target_names=target_names, best_trials_with_values=best_trials_with_values, non_best_trials_with_values=non_best_trials_with_values, infeasible_trials_with_values=infeasible_trials_with_values, axis_order=axis_order, include_dominated_trials=include_dominated_trials, has_constraints_func=constraints_func is not None, ) def _targets_default(trial: FrozenTrial) -> Sequence[float]: return trial.values def _get_non_pareto_front_trials( trials: list[FrozenTrial], pareto_trials: list[FrozenTrial] ) -> list[FrozenTrial]: non_pareto_trials = [] for trial in trials: if trial not in pareto_trials: non_pareto_trials.append(trial) return non_pareto_trials def _make_scatter_object( n_targets: int, axis_order: Sequence[int], include_dominated_trials: bool, trials_with_values: Sequence[tuple[FrozenTrial, Sequence[float]]], hovertemplate: str, infeasible: bool = False, dominated_trials: bool = False, ) -> "go.Scatter" | "go.Scatter3d": trials_with_values = trials_with_values or [] marker = _make_marker( [trial for trial, _ in trials_with_values], include_dominated_trials, dominated_trials=dominated_trials, infeasible=infeasible, ) if n_targets == 2: return go.Scatter( x=[values[axis_order[0]] for _, values in trials_with_values], y=[values[axis_order[1]] for _, values in trials_with_values], text=[_make_hovertext(trial) for trial, _ in trials_with_values], mode="markers", hovertemplate=hovertemplate, marker=marker, showlegend=False, ) elif n_targets == 3: return go.Scatter3d( x=[values[axis_order[0]] for _, values in trials_with_values], y=[values[axis_order[1]] for _, values in trials_with_values], z=[values[axis_order[2]] for _, values in trials_with_values], text=[_make_hovertext(trial) for trial, _ in trials_with_values], mode="markers", hovertemplate=hovertemplate, marker=marker, showlegend=False, ) else: assert False, "Must not reach here" def _make_marker( trials: Sequence[FrozenTrial], include_dominated_trials: bool, dominated_trials: bool = False, infeasible: bool = False, ) -> dict[str, Any]: if dominated_trials and not include_dominated_trials: assert len(trials) == 0 if infeasible: return { "color": "#cccccc", } elif dominated_trials: return { "line": {"width": 0.5, "color": "Grey"}, "color": [t.number for t in trials], "colorscale": "Blues", "colorbar": { "title": "Trial", }, } else: return { "line": {"width": 0.5, "color": "Grey"}, "color": [t.number for t in trials], "colorscale": "Reds", "colorbar": { "title": "Best Trial", "x": 1.1 if include_dominated_trials else 1, "xpad": 40, }, } optuna-3.5.0/optuna/visualization/_plotly_imports.py000066400000000000000000000015231453453102400230470ustar00rootroot00000000000000from packaging import version from optuna._imports import try_import with try_import() as _imports: import plotly from plotly import __version__ as plotly_version import plotly.graph_objs as go from plotly.graph_objs import Contour from plotly.graph_objs import Scatter from plotly.subplots import make_subplots if version.parse(plotly_version) < version.parse("4.0.0"): raise ImportError( "Your version of Plotly is " + plotly_version + " . " "Please install plotly version 4.0.0 or higher. " "Plotly can be installed by executing `$ pip install -U plotly>=4.0.0`. " "For further information, please refer to the installation guide of plotly. ", name="plotly", ) __all__ = ["_imports", "plotly", "go", "Contour", "Scatter", "make_subplots"] optuna-3.5.0/optuna/visualization/_rank.py000066400000000000000000000345051453453102400207100ustar00rootroot00000000000000from __future__ import annotations import math import typing from typing import Any from typing import Callable from typing import NamedTuple import numpy as np from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _is_numerical from optuna.visualization.matplotlib._matplotlib_imports import _imports as matplotlib_imports plotly_is_available = _imports.is_successful() if plotly_is_available: from optuna.visualization._plotly_imports import go from optuna.visualization._plotly_imports import make_subplots from optuna.visualization._plotly_imports import plotly from optuna.visualization._plotly_imports import Scatter if matplotlib_imports.is_successful(): # TODO(c-bata): Refactor to remove matplotlib and plotly dependencies in `_get_rank_info()`. # See https://github.com/optuna/optuna/pull/5133#discussion_r1414761672 for the discussion. from optuna.visualization.matplotlib._matplotlib_imports import plt as matplotlib_plt _logger = get_logger(__name__) PADDING_RATIO = 0.05 class _AxisInfo(NamedTuple): name: str range: tuple[float, float] is_log: bool is_cat: bool class _RankSubplotInfo(NamedTuple): xaxis: _AxisInfo yaxis: _AxisInfo xs: list[Any] ys: list[Any] trials: list[FrozenTrial] zs: np.ndarray colors: np.ndarray class _RankPlotInfo(NamedTuple): params: list[str] sub_plot_infos: list[list[_RankSubplotInfo]] target_name: str zs: np.ndarray colors: np.ndarray has_custom_target: bool @experimental_func("3.2.0") def plot_rank( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot parameter relations as scatter plots with colors indicating ranks of target value. Note that trials missing the specified parameters will not be plotted. Example: The following code snippet shows how to plot the parameter relationship as a rank plot. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) c0 = 400 - (x + y)**2 trial.set_user_attr("constraint", [c0]) return x ** 2 + y def constraints(trial): return trial.user_attrs["constraint"] sampler = optuna.samplers.TPESampler(seed=10, constraints_func=constraints) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) fig = optuna.visualization.plot_rank(study, params=["x", "y"]) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`plotly.graph_objs.Figure` object. .. note:: This function requires plotly >= 5.0.0. """ _imports.check() info = _get_rank_info(study, params, target, target_name) return _get_rank_plot(info) def _get_order_with_same_order_averaging(data: np.ndarray) -> np.ndarray: order = np.zeros_like(data, dtype=float) data_sorted = np.sort(data) for i, d in enumerate(data): indices = np.where(data_sorted == d)[0] order[i] = sum(indices) / len(indices) return order def _get_rank_info( study: Study, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> _RankPlotInfo: _check_plot_args(study, target, target_name) trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) all_params = {p_name for t in trials for p_name in t.params.keys()} if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") params = [] elif params is None: params = sorted(all_params) else: for input_p_name in params: if input_p_name not in all_params: raise ValueError("Parameter {} does not exist in your study.".format(input_p_name)) if len(params) == 0: _logger.warning("params is an empty list.") has_custom_target = True if target is None: def target(trial: FrozenTrial) -> float: return typing.cast(float, trial.value) has_custom_target = False target_values = np.array([target(trial) for trial in trials]) raw_ranks = _get_order_with_same_order_averaging(target_values) color_idxs = raw_ranks / (len(trials) - 1) if len(trials) >= 2 else np.array([0.5]) colors = _convert_color_idxs_to_scaled_rgb_colors(color_idxs) sub_plot_infos: list[list[_RankSubplotInfo]] if len(params) == 2: x_param = params[0] y_param = params[1] sub_plot_info = _get_rank_subplot_info(trials, target_values, colors, x_param, y_param) sub_plot_infos = [[sub_plot_info]] else: sub_plot_infos = [ [ _get_rank_subplot_info(trials, target_values, colors, x_param, y_param) for x_param in params ] for y_param in params ] return _RankPlotInfo( params=params, sub_plot_infos=sub_plot_infos, target_name=target_name, zs=target_values, colors=colors, has_custom_target=has_custom_target, ) def _get_rank_subplot_info( trials: list[FrozenTrial], target_values: np.ndarray, colors: np.ndarray, x_param: str, y_param: str, ) -> _RankSubplotInfo: xaxis = _get_axis_info(trials, x_param) yaxis = _get_axis_info(trials, y_param) infeasible_trial_ids = [] for i in range(len(trials)): constraints = trials[i].system_attrs.get(_CONSTRAINTS_KEY) if constraints is not None and any([x > 0.0 for x in constraints]): infeasible_trial_ids.append(i) colors[infeasible_trial_ids] = (204, 204, 204) # equal to "#CCCCCC" filtered_ids = [ i for i in range(len(trials)) if x_param in trials[i].params and y_param in trials[i].params ] filtered_trials = [trials[i] for i in filtered_ids] xs = [trial.params[x_param] for trial in filtered_trials] ys = [trial.params[y_param] for trial in filtered_trials] zs = target_values[filtered_ids] colors = colors[filtered_ids] return _RankSubplotInfo( xaxis=xaxis, yaxis=yaxis, xs=xs, ys=ys, trials=filtered_trials, zs=np.array(zs), colors=colors, ) def _get_axis_info(trials: list[FrozenTrial], param_name: str) -> _AxisInfo: values: list[str | float | None] is_numerical = _is_numerical(trials, param_name) if is_numerical: values = [t.params.get(param_name) for t in trials] else: values = [ str(t.params.get(param_name)) if param_name in t.params else None for t in trials ] min_value = min([v for v in values if v is not None]) max_value = max([v for v in values if v is not None]) if _is_log_scale(trials, param_name): min_value = float(min_value) max_value = float(max_value) padding = (math.log10(max_value) - math.log10(min_value)) * PADDING_RATIO min_value = math.pow(10, math.log10(min_value) - padding) max_value = math.pow(10, math.log10(max_value) + padding) is_log = True is_cat = False elif is_numerical: min_value = float(min_value) max_value = float(max_value) padding = (max_value - min_value) * PADDING_RATIO min_value = min_value - padding max_value = max_value + padding is_log = False is_cat = False else: unique_values = set(values) span = len(unique_values) - 1 if None in unique_values: span -= 1 padding = span * PADDING_RATIO min_value = -padding max_value = span + padding is_log = False is_cat = True return _AxisInfo( name=param_name, range=(min_value, max_value), is_log=is_log, is_cat=is_cat, ) def _get_rank_subplot( info: _RankSubplotInfo, target_name: str, print_raw_objectives: bool ) -> "Scatter": def get_hover_text(trial: FrozenTrial, target_value: float) -> str: lines = [f"Trial #{trial.number}"] lines += [f"{k}: {v}" for k, v in trial.params.items()] lines += [f"{target_name}: {target_value}"] if print_raw_objectives: lines += [f"Objective #{i}: {v}" for i, v in enumerate(trial.values)] return "
".join(lines) scatter = go.Scatter( x=info.xs, y=info.ys, marker={ "color": list(map(plotly.colors.label_rgb, info.colors)), "line": {"width": 0.5, "color": "Grey"}, }, mode="markers", showlegend=False, hovertemplate="%{hovertext}", hovertext=[ get_hover_text(trial, target_value) for trial, target_value in zip(info.trials, info.zs) ], ) return scatter class _TickInfo(NamedTuple): coloridxs: list[float] text: list[str] def _get_tick_info(target_values: np.ndarray) -> _TickInfo: sorted_target_values = np.sort(target_values) coloridxs = [0, 0.25, 0.5, 0.75, 1] values = np.quantile(sorted_target_values, coloridxs) rank_text = ["min.", "25%", "50%", "75%", "max."] text = [f"{rank_text[i]} ({values[i]:3g})" for i in range(len(values))] return _TickInfo(coloridxs=coloridxs, text=text) def _get_rank_plot( info: _RankPlotInfo, ) -> "go.Figure": params = info.params sub_plot_infos = info.sub_plot_infos layout = go.Layout(title=f"Rank ({info.target_name})") if len(params) == 0: return go.Figure(data=[], layout=layout) if len(params) == 2: x_param = params[0] y_param = params[1] sub_plot_info = sub_plot_infos[0][0] sub_plots = _get_rank_subplot(sub_plot_info, info.target_name, info.has_custom_target) figure = go.Figure(data=sub_plots, layout=layout) figure.update_xaxes(title_text=x_param, range=sub_plot_info.xaxis.range) figure.update_yaxes(title_text=y_param, range=sub_plot_info.yaxis.range) if sub_plot_info.xaxis.is_cat: figure.update_xaxes(type="category") if sub_plot_info.yaxis.is_cat: figure.update_yaxes(type="category") if sub_plot_info.xaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.xaxis.range] figure.update_xaxes(range=log_range, type="log") if sub_plot_info.yaxis.is_log: log_range = [math.log10(p) for p in sub_plot_info.yaxis.range] figure.update_yaxes(range=log_range, type="log") else: figure = make_subplots( rows=len(params), cols=len(params), shared_xaxes=True, shared_yaxes=True, horizontal_spacing=0.08 / len(params), vertical_spacing=0.08 / len(params), ) figure.update_layout(layout) for x_i, x_param in enumerate(params): for y_i, y_param in enumerate(params): scatter = _get_rank_subplot( sub_plot_infos[y_i][x_i], info.target_name, info.has_custom_target ) figure.add_trace(scatter, row=y_i + 1, col=x_i + 1) xaxis = sub_plot_infos[y_i][x_i].xaxis yaxis = sub_plot_infos[y_i][x_i].yaxis figure.update_xaxes(range=xaxis.range, row=y_i + 1, col=x_i + 1) figure.update_yaxes(range=yaxis.range, row=y_i + 1, col=x_i + 1) if xaxis.is_cat: figure.update_xaxes(type="category", row=y_i + 1, col=x_i + 1) if yaxis.is_cat: figure.update_yaxes(type="category", row=y_i + 1, col=x_i + 1) if xaxis.is_log: log_range = [math.log10(p) for p in xaxis.range] figure.update_xaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if yaxis.is_log: log_range = [math.log10(p) for p in yaxis.range] figure.update_yaxes(range=log_range, type="log", row=y_i + 1, col=x_i + 1) if x_i == 0: figure.update_yaxes(title_text=y_param, row=y_i + 1, col=x_i + 1) if y_i == len(params) - 1: figure.update_xaxes(title_text=x_param, row=y_i + 1, col=x_i + 1) tick_info = _get_tick_info(info.zs) colormap = "RdYlBu_r" colorbar_trace = go.Scatter( x=[None], y=[None], mode="markers", marker=dict( colorscale=colormap, showscale=True, cmin=0, cmax=1, colorbar=dict(thickness=10, tickvals=tick_info.coloridxs, ticktext=tick_info.text), ), hoverinfo="none", showlegend=False, ) figure.add_trace(colorbar_trace) return figure def _convert_color_idxs_to_scaled_rgb_colors(color_idxs: np.ndarray) -> np.ndarray: colormap = "RdYlBu_r" if plotly_is_available: # sample_colorscale requires plotly >= 5.0.0. labeled_colors = plotly.colors.sample_colorscale(colormap, color_idxs) scaled_rgb_colors = np.array([plotly.colors.unlabel_rgb(cl) for cl in labeled_colors]) return scaled_rgb_colors else: cmap = matplotlib_plt.get_cmap(colormap) colors = cmap(color_idxs)[:, :3] # Drop alpha values. rgb_colors = np.asarray(colors * 255, dtype=int) return rgb_colors optuna-3.5.0/optuna/visualization/_slice.py000066400000000000000000000226731453453102400210570ustar00rootroot00000000000000from __future__ import annotations from typing import Any from typing import Callable from typing import cast from typing import NamedTuple from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _is_log_scale if _imports.is_successful(): from optuna.visualization._plotly_imports import go from optuna.visualization._plotly_imports import make_subplots from optuna.visualization._plotly_imports import Scatter from optuna.visualization._utils import COLOR_SCALE _logger = get_logger(__name__) class _SliceSubplotInfo(NamedTuple): param_name: str x: list[Any] y: list[float] trial_numbers: list[int] is_log: bool is_numerical: bool constraints: list[bool] x_labels: tuple[CategoricalChoiceType, ...] | None class _SlicePlotInfo(NamedTuple): target_name: str subplots: list[_SliceSubplotInfo] class _PlotValues(NamedTuple): x: list[Any] y: list[float] trial_numbers: list[int] def _get_slice_subplot_info( trials: list[FrozenTrial], param: str, target: Callable[[FrozenTrial], float] | None, log_scale: bool, numerical: bool, x_labels: tuple[CategoricalChoiceType, ...] | None, ) -> _SliceSubplotInfo: if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target plot_info = _SliceSubplotInfo( param_name=param, x=[], y=[], trial_numbers=[], is_log=log_scale, is_numerical=numerical, x_labels=x_labels, constraints=[], ) for t in trials: if param not in t.params: continue plot_info.x.append(t.params[param]) plot_info.y.append(target(t)) plot_info.trial_numbers.append(t.number) constraints = t.system_attrs.get(_CONSTRAINTS_KEY) plot_info.constraints.append(constraints is None or all([x <= 0.0 for x in constraints])) return plot_info def _get_slice_plot_info( study: Study, params: list[str] | None, target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> _SlicePlotInfo: _check_plot_args(study, target, target_name) trials = _filter_nonfinite( study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)), target=target ) if len(trials) == 0: _logger.warning("Your study does not have any completed trials.") return _SlicePlotInfo(target_name, []) all_params = {p_name for t in trials for p_name in t.params.keys()} distributions = {} for trial in trials: for param_name, distribution in trial.distributions.items(): if param_name not in distributions: distributions[param_name] = distribution x_labels = {} for param_name, distribution in distributions.items(): if isinstance(distribution, CategoricalDistribution): x_labels[param_name] = distribution.choices if params is None: sorted_params = sorted(all_params) else: for input_p_name in params: if input_p_name not in all_params: raise ValueError(f"Parameter {input_p_name} does not exist in your study.") sorted_params = sorted(set(params)) return _SlicePlotInfo( target_name=target_name, subplots=[ _get_slice_subplot_info( trials=trials, param=param, target=target, log_scale=_is_log_scale(trials, param), numerical=not isinstance(distributions[param], CategoricalDistribution), x_labels=x_labels.get(param), ) for param in sorted_params ], ) def plot_slice( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "go.Figure": """Plot the parameter relationship as slice plot in a study. Note that, if a parameter contains missing values, a trial with missing values is not plotted. Example: The following code snippet shows how to plot the parameter relationship as slice plot. .. plotly:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) fig = optuna.visualization.plot_slice(study, params=["x", "y"]) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() return _get_slice_plot(_get_slice_plot_info(study, params, target, target_name)) def _get_slice_plot(info: _SlicePlotInfo) -> "go.Figure": layout = go.Layout(title="Slice Plot") if len(info.subplots) == 0: return go.Figure(data=[], layout=layout) elif len(info.subplots) == 1: figure = go.Figure(data=_generate_slice_subplot(info.subplots[0]), layout=layout) figure.update_xaxes(title_text=info.subplots[0].param_name) figure.update_yaxes(title_text=info.target_name) if not info.subplots[0].is_numerical: figure.update_xaxes( type="category", categoryorder="array", categoryarray=info.subplots[0].x_labels ) elif info.subplots[0].is_log: figure.update_xaxes(type="log") else: figure = make_subplots(rows=1, cols=len(info.subplots), shared_yaxes=True) figure.update_layout(layout) showscale = True # showscale option only needs to be specified once. for column_index, subplot_info in enumerate(info.subplots, start=1): trace = _generate_slice_subplot(subplot_info) trace[0].update(marker={"showscale": showscale}) # showscale's default is True. if showscale: showscale = False for t in trace: figure.add_trace(t, row=1, col=column_index) figure.update_xaxes(title_text=subplot_info.param_name, row=1, col=column_index) if column_index == 1: figure.update_yaxes(title_text=info.target_name, row=1, col=column_index) if not subplot_info.is_numerical: figure.update_xaxes( type="category", categoryorder="array", categoryarray=subplot_info.x_labels, row=1, col=column_index, ) elif subplot_info.is_log: figure.update_xaxes(type="log", row=1, col=column_index) if len(info.subplots) > 3: # Ensure that each subplot has a minimum width without relying on autusizing. figure.update_layout(width=300 * len(info.subplots)) return figure def _generate_slice_subplot(subplot_info: _SliceSubplotInfo) -> list[Scatter]: trace = [] feasible = _PlotValues([], [], []) infeasible = _PlotValues([], [], []) for x, y, num, c in zip( subplot_info.x, subplot_info.y, subplot_info.trial_numbers, subplot_info.constraints ): if x is not None or x != "None" or y is not None or y != "None": if c: feasible.x.append(x) feasible.y.append(y) feasible.trial_numbers.append(num) else: infeasible.x.append(x) infeasible.y.append(y) trace.append( go.Scatter( x=feasible.x, y=feasible.y, mode="markers", name="Feasible Trial", marker={ "line": {"width": 0.5, "color": "Grey"}, "color": feasible.trial_numbers, "colorscale": COLOR_SCALE, "colorbar": { "title": "Trial", "x": 1.0, # Offset the colorbar position with a fixed width `xpad`. "xpad": 40, }, }, showlegend=False, ) ) if len(infeasible.x) > 0: trace.append( go.Scatter( x=infeasible.x, y=infeasible.y, mode="markers", name="Infeasible Trial", marker={ "color": "#cccccc", }, showlegend=False, ) ) return trace optuna-3.5.0/optuna/visualization/_terminator_improvement.py000066400000000000000000000211361453453102400245620ustar00rootroot00000000000000from __future__ import annotations from typing import NamedTuple import tqdm import optuna from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study.study import Study from optuna.terminator import BaseErrorEvaluator from optuna.terminator import BaseImprovementEvaluator from optuna.terminator import CrossValidationErrorEvaluator from optuna.terminator import RegretBoundEvaluator from optuna.terminator.improvement.evaluator import DEFAULT_MIN_N_TRIALS from optuna.visualization._plotly_imports import _imports if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) PADDING_RATIO_Y = 0.05 OPACITY = 0.25 class _ImprovementInfo(NamedTuple): trial_numbers: list[int] improvements: list[float] errors: list[float] | None @experimental_func("3.2.0") def plot_terminator_improvement( study: Study, plot_error: bool = False, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, min_n_trials: int = DEFAULT_MIN_N_TRIALS, ) -> "go.Figure": """Plot the potentials for future objective improvement. This function visualizes the objective improvement potentials, evaluated with ``improvement_evaluator``. It helps to determine whether we should continue the optimization or not. You can also plot the error evaluated with ``error_evaluator`` if the ``plot_error`` argument is set to :obj:`True`. Note that this function may take some time to compute the improvement potentials. Example: The following code snippet shows how to plot improvement potentials, together with cross-validation errors. .. plotly:: from lightgbm import LGBMClassifier from sklearn.datasets import load_wine from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import report_cross_validation_scores from optuna.visualization import plot_terminator_improvement def objective(trial): X, y = load_wine(return_X_y=True) clf = LGBMClassifier( reg_alpha=trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True), reg_lambda=trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True), num_leaves=trial.suggest_int("num_leaves", 2, 256), colsample_bytree=trial.suggest_float("colsample_bytree", 0.4, 1.0), subsample=trial.suggest_float("subsample", 0.4, 1.0), subsample_freq=trial.suggest_int("subsample_freq", 1, 7), min_child_samples=trial.suggest_int("min_child_samples", 5, 100), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) return scores.mean() study = optuna.create_study() study.optimize(objective, n_trials=30) fig = plot_terminator_improvement(study, plot_error=True) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their improvement. plot_error: A flag to show the error. If it is set to :obj:`True`, errors evaluated by ``error_evaluator`` are also plotted as line graph. Defaults to :obj:`False`. improvement_evaluator: An object that evaluates the improvement of the objective function. Defaults to :class:`~optuna.terminator.RegretBoundEvaluator`. error_evaluator: An object that evaluates the error inherent in the objective function. Defaults to :class:`~optuna.terminator.CrossValidationErrorEvaluator`. min_n_trials: The minimum number of trials before termination is considered. Terminator improvements for trials below this value are shown in a lighter color. Defaults to ``20``. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() info = _get_improvement_info(study, plot_error, improvement_evaluator, error_evaluator) return _get_improvement_plot(info, min_n_trials) def _get_improvement_info( study: Study, get_error: bool = False, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, ) -> _ImprovementInfo: if study._is_multi_objective(): raise ValueError("This function does not support multi-objective optimization study.") if improvement_evaluator is None: improvement_evaluator = RegretBoundEvaluator() if error_evaluator is None: error_evaluator = CrossValidationErrorEvaluator() trial_numbers = [] completed_trials = [] improvements = [] errors = [] for trial in tqdm.tqdm(study.trials): if trial.state == optuna.trial.TrialState.COMPLETE: completed_trials.append(trial) if len(completed_trials) == 0: continue trial_numbers.append(trial.number) improvement = improvement_evaluator.evaluate( trials=completed_trials, study_direction=study.direction ) improvements.append(improvement) if get_error: error = error_evaluator.evaluate( trials=completed_trials, study_direction=study.direction ) errors.append(error) if len(errors) == 0: return _ImprovementInfo( trial_numbers=trial_numbers, improvements=improvements, errors=None ) else: return _ImprovementInfo( trial_numbers=trial_numbers, improvements=improvements, errors=errors ) def _get_improvement_scatter( trial_numbers: list[int], improvements: list[float], opacity: float = 1.0, showlegend: bool = True, ) -> "go.Scatter": plotly_blue_with_opacity = f"rgba(99, 110, 250, {opacity})" return go.Scatter( x=trial_numbers, y=improvements, mode="markers+lines", marker=dict(color=plotly_blue_with_opacity), line=dict(color=plotly_blue_with_opacity), name="Terminator Improvement", showlegend=showlegend, legendgroup="improvement", ) def _get_error_scatter( trial_numbers: list[int], errors: list[float] | None, ) -> "go.Scatter": if errors is None: return go.Scatter() plotly_red = "rgb(239, 85, 59)" return go.Scatter( x=trial_numbers, y=errors, mode="markers+lines", name="Error", marker=dict(color=plotly_red), line=dict(color=plotly_red), ) def _get_y_range(info: _ImprovementInfo, min_n_trials: int) -> tuple[float, float]: min_value = min(info.improvements) if info.errors is not None: min_value = min(min_value, min(info.errors)) # Determine the display range based on trials after min_n_trials. if len(info.trial_numbers) > min_n_trials: max_value = max(info.improvements[min_n_trials:]) # If there are no trials after min_trials, determine the display range based on all trials. else: max_value = max(info.improvements) if info.errors is not None: max_value = max(max_value, max(info.errors)) padding = (max_value - min_value) * PADDING_RATIO_Y return (min_value - padding, max_value + padding) def _get_improvement_plot(info: _ImprovementInfo, min_n_trials: int) -> "go.Figure": n_trials = len(info.trial_numbers) fig = go.Figure( layout=go.Layout( title="Terminator Improvement Plot", xaxis=dict(title="Trial"), yaxis=dict(title="Terminator Improvement"), ) ) if n_trials == 0: _logger.warning("There are no complete trials.") return fig fig.add_trace( _get_improvement_scatter( info.trial_numbers[: min_n_trials + 1], info.improvements[: min_n_trials + 1], # Plot line with a lighter color until the number of trials reaches min_n_trials. OPACITY, n_trials <= min_n_trials, # Avoid showing legend twice. ) ) if n_trials > min_n_trials: fig.add_trace( _get_improvement_scatter( info.trial_numbers[min_n_trials:], info.improvements[min_n_trials:], ) ) fig.add_trace(_get_error_scatter(info.trial_numbers, info.errors)) fig.update_yaxes(range=_get_y_range(info, min_n_trials)) return fig optuna-3.5.0/optuna/visualization/_timeline.py000066400000000000000000000114011453453102400215510ustar00rootroot00000000000000from __future__ import annotations import datetime from typing import NamedTuple from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import Study from optuna.trial import TrialState from optuna.visualization._plotly_imports import _imports from optuna.visualization._utils import _make_hovertext if _imports.is_successful(): from optuna.visualization._plotly_imports import go _logger = get_logger(__name__) class _TimelineBarInfo(NamedTuple): number: int start: datetime.datetime complete: datetime.datetime state: TrialState hovertext: str infeasible: bool class _TimelineInfo(NamedTuple): bars: list[_TimelineBarInfo] @experimental_func("3.2.0") def plot_timeline(study: Study) -> "go.Figure": """Plot the timeline of a study. Example: The following code snippet shows how to plot the timeline of a study. Timeline plot can visualize trials with overlapping execution time (e.g., in distributed environments). .. plotly:: import time import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) time.sleep(x * 0.1) if x > 0.8: raise ValueError() if x > 0.4: raise optuna.TrialPruned() return x ** 2 study = optuna.create_study(direction="minimize") study.optimize( objective, n_trials=50, n_jobs=2, catch=(ValueError,) ) fig = optuna.visualization.plot_timeline(study) fig.show() Args: study: A :class:`~optuna.study.Study` object whose trials are plotted with their lifetime. Returns: A :class:`plotly.graph_objs.Figure` object. """ _imports.check() info = _get_timeline_info(study) return _get_timeline_plot(info) def _get_timeline_info(study: Study) -> _TimelineInfo: bars = [] for t in study.get_trials(deepcopy=False): date_complete = t.datetime_complete or datetime.datetime.now() date_start = t.datetime_start or date_complete infeasible = ( False if _CONSTRAINTS_KEY not in t.system_attrs else any([x > 0 for x in t.system_attrs[_CONSTRAINTS_KEY]]) ) if date_complete < date_start: _logger.warning( ( f"The start and end times for Trial {t.number} seem to be reversed. " f"The start time is {date_start} and the end time is {date_complete}." ) ) bars.append( _TimelineBarInfo( number=t.number, start=date_start, complete=date_complete, state=t.state, hovertext=_make_hovertext(t), infeasible=infeasible, ) ) if len(bars) == 0: _logger.warning("Your study does not have any trials.") return _TimelineInfo(bars) def _get_timeline_plot(info: _TimelineInfo) -> "go.Figure": _cm = { "COMPLETE": "blue", "FAIL": "red", "PRUNED": "orange", "RUNNING": "green", "WAITING": "gray", } fig = go.Figure() for s in sorted(TrialState, key=lambda x: x.name): if s.name == "COMPLETE": infeasible_bars = [b for b in info.bars if b.state == s and b.infeasible] feasible_bars = [b for b in info.bars if b.state == s and not b.infeasible] _plot_bars(infeasible_bars, "#cccccc", "INFEASIBLE", fig) _plot_bars(feasible_bars, _cm[s.name], s.name, fig) else: bars = [b for b in info.bars if b.state == s] _plot_bars(bars, _cm[s.name], s.name, fig) fig.update_xaxes(type="date") fig.update_layout( go.Layout( title="Timeline Plot", xaxis={"title": "Datetime"}, yaxis={"title": "Trial"}, ) ) fig.update_layout(showlegend=True) # Draw a legend even if all TrialStates are the same. return fig def _plot_bars(bars: list[_TimelineBarInfo], color: str, name: str, fig: go.Figure) -> None: if len(bars) == 0: return fig.add_trace( go.Bar( name=name, x=[(b.complete - b.start).total_seconds() * 1000 for b in bars], y=[b.number for b in bars], base=[b.start.isoformat() for b in bars], text=[b.hovertext for b in bars], hovertemplate="%{text}" + name + "", orientation="h", marker=dict(color=color), textposition="none", # Avoid drawing hovertext in a bar. ) ) optuna-3.5.0/optuna/visualization/_utils.py000066400000000000000000000137351453453102400211170ustar00rootroot00000000000000from __future__ import annotations import json from typing import Any from typing import Callable from typing import cast from typing import Sequence import warnings import numpy as np import optuna from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.study import Study from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial from optuna.visualization import _plotly_imports __all__ = ["is_available"] _logger = optuna.logging.get_logger(__name__) def is_available() -> bool: """Returns whether visualization with plotly is available or not. .. note:: :mod:`~optuna.visualization` module depends on plotly version 4.0.0 or higher. If a supported version of plotly isn't installed in your environment, this function will return :obj:`False`. In such case, please execute ``$ pip install -U plotly>=4.0.0`` to install plotly. Returns: :obj:`True` if visualization with plotly is available, :obj:`False` otherwise. """ return _plotly_imports._imports.is_successful() if is_available(): import plotly.colors COLOR_SCALE = plotly.colors.sequential.Blues def _check_plot_args( study: Study | Sequence[Study], target: Callable[[FrozenTrial], float] | None, target_name: str, ) -> None: studies: Sequence[Study] if isinstance(study, Study): studies = [study] else: studies = study if target is None and any(study._is_multi_objective() for study in studies): raise ValueError( "If the `study` is being used for multi-objective optimization, " "please specify the `target`." ) if target is not None and target_name == "Objective Value": warnings.warn( "`target` is specified, but `target_name` is the default value, 'Objective Value'." ) def _is_log_scale(trials: list[FrozenTrial], param: str) -> bool: for trial in trials: if param in trial.params: dist = trial.distributions[param] if isinstance(dist, (FloatDistribution, IntDistribution)): if dist.log: return True return False def _is_categorical(trials: list[FrozenTrial], param: str) -> bool: return any( isinstance(t.distributions[param], CategoricalDistribution) for t in trials if param in t.params ) def _is_numerical(trials: list[FrozenTrial], param: str) -> bool: return all( (isinstance(t.params[param], int) or isinstance(t.params[param], float)) and not isinstance(t.params[param], bool) for t in trials if param in t.params ) def _get_param_values(trials: list[FrozenTrial], p_name: str) -> list[Any]: values = [t.params[p_name] for t in trials if p_name in t.params] if _is_numerical(trials, p_name): return values return list(map(str, values)) def _get_skipped_trial_numbers( trials: list[FrozenTrial], used_param_names: Sequence[str] ) -> set[int]: """Utility function for ``plot_parallel_coordinate``. If trial's parameters do not contain a parameter in ``used_param_names``, ``plot_parallel_coordinate`` methods do not use such trials. Args: trials: List of ``FrozenTrial``s. used_param_names: The parameter names used in ``plot_parallel_coordinate``. Returns: A set of invalid trial numbers. """ skipped_trial_numbers = set() for trial in trials: for used_param in used_param_names: if used_param not in trial.params.keys(): skipped_trial_numbers.add(trial.number) break return skipped_trial_numbers def _filter_nonfinite( trials: list[FrozenTrial], target: Callable[[FrozenTrial], float] | None = None, with_message: bool = True, ) -> list[FrozenTrial]: # For multi-objective optimization target must be specified to select # one of objective values to filter trials by (and plot by later on). # This function is not raising when target is missing, since we're # assuming plot args have been sanitized before. if target is None: def _target(t: FrozenTrial) -> float: return cast(float, t.value) target = _target filtered_trials: list[FrozenTrial] = [] for trial in trials: value = target(trial) try: value = float(value) except ( ValueError, TypeError, ): warnings.warn( f"Trial{trial.number}'s target value {repr(value)} could not be cast to float." ) raise # Not a Number, positive infinity and negative infinity are considered to be non-finite. if not np.isfinite(value): if with_message: _logger.warning( f"Trial {trial.number} is omitted in visualization " "because its objective value is inf or nan." ) else: filtered_trials.append(trial) return filtered_trials def _is_reverse_scale(study: Study, target: Callable[[FrozenTrial], float] | None) -> bool: return target is not None or study.direction == StudyDirection.MINIMIZE def _make_json_compatible(value: Any) -> Any: try: json.dumps(value) return value except TypeError: # The value can't be converted to JSON directly, so return a string representation. return str(value) def _make_hovertext(trial: FrozenTrial) -> str: user_attrs = {key: _make_json_compatible(value) for key, value in trial.user_attrs.items()} user_attrs_dict = {"user_attrs": user_attrs} if user_attrs else {} text = json.dumps( { "number": trial.number, "values": trial.values, "params": trial.params, **user_attrs_dict, }, indent=2, ) return text.replace("\n", "
") optuna-3.5.0/optuna/visualization/matplotlib/000077500000000000000000000000001453453102400214045ustar00rootroot00000000000000optuna-3.5.0/optuna/visualization/matplotlib/__init__.py000066400000000000000000000025011453453102400235130ustar00rootroot00000000000000from optuna.visualization.matplotlib._contour import plot_contour from optuna.visualization.matplotlib._edf import plot_edf from optuna.visualization.matplotlib._hypervolume_history import plot_hypervolume_history from optuna.visualization.matplotlib._intermediate_values import plot_intermediate_values from optuna.visualization.matplotlib._optimization_history import plot_optimization_history from optuna.visualization.matplotlib._parallel_coordinate import plot_parallel_coordinate from optuna.visualization.matplotlib._param_importances import plot_param_importances from optuna.visualization.matplotlib._pareto_front import plot_pareto_front from optuna.visualization.matplotlib._rank import plot_rank from optuna.visualization.matplotlib._slice import plot_slice from optuna.visualization.matplotlib._terminator_improvement import plot_terminator_improvement from optuna.visualization.matplotlib._timeline import plot_timeline from optuna.visualization.matplotlib._utils import is_available __all__ = [ "is_available", "plot_contour", "plot_edf", "plot_intermediate_values", "plot_hypervolume_history", "plot_optimization_history", "plot_parallel_coordinate", "plot_param_importances", "plot_pareto_front", "plot_rank", "plot_slice", "plot_terminator_improvement", "plot_timeline", ] optuna-3.5.0/optuna/visualization/matplotlib/_contour.py000066400000000000000000000331211453453102400236060ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from typing import Sequence import numpy as np from optuna._experimental import experimental_func from optuna._imports import try_import from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._contour import _AxisInfo from optuna.visualization._contour import _ContourInfo from optuna.visualization._contour import _get_contour_info from optuna.visualization._contour import _PlotValues from optuna.visualization._contour import _SubContourInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports with try_import() as _optuna_imports: import scipy if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import Colormap from optuna.visualization.matplotlib._matplotlib_imports import ContourSet from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) CONTOUR_POINT_NUM = 100 @experimental_func("2.2.0") def plot_contour( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the parameter relationship as contour plot in a study with Matplotlib. Note that, if a parameter contains missing values, a trial with missing values is not plotted. .. seealso:: Please refer to :func:`optuna.visualization.plot_contour` for an example. Warnings: Output figures of this Matplotlib-based :func:`~optuna.visualization.matplotlib.plot_contour` function would be different from those of the Plotly-based :func:`~optuna.visualization.plot_contour`. Example: The following code snippet shows how to plot the parameter relationship as contour plot. .. plot:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) optuna.visualization.matplotlib.plot_contour(study, params=["x", "y"]) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`matplotlib.axes.Axes` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() _logger.warning( "Output figures of this Matplotlib-based `plot_contour` function would be different from " "those of the Plotly-based `plot_contour`." ) info = _get_contour_info(study, params, target, target_name) return _get_contour_plot(info) def _get_contour_plot(info: _ContourInfo) -> "Axes": sorted_params = info.sorted_params sub_plot_infos = info.sub_plot_infos reverse_scale = info.reverse_scale target_name = info.target_name if len(sorted_params) <= 1: _, ax = plt.subplots() return ax n_params = len(sorted_params) plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. if n_params == 2: # Set up the graph style. fig, axs = plt.subplots() axs.set_title("Contour Plot") cmap = _set_cmap(reverse_scale) cs = _generate_contour_subplot(sub_plot_infos[0][0], axs, cmap) if isinstance(cs, ContourSet): axcb = fig.colorbar(cs) axcb.set_label(target_name) else: # Set up the graph style. fig, axs = plt.subplots(n_params, n_params) fig.suptitle("Contour Plot") cmap = _set_cmap(reverse_scale) # Prepare data and draw contour plots. cs_list = [] for x_i in range(len(sorted_params)): for y_i in range(len(sorted_params)): ax = axs[y_i, x_i] cs = _generate_contour_subplot(sub_plot_infos[y_i][x_i], ax, cmap) if isinstance(cs, ContourSet): cs_list.append(cs) if cs_list: axcb = fig.colorbar(cs_list[0], ax=axs) axcb.set_label(target_name) return axs def _set_cmap(reverse_scale: bool) -> "Colormap": cmap = "Blues_r" if not reverse_scale else "Blues" return plt.get_cmap(cmap) class _LabelEncoder: def __init__(self) -> None: self.labels: list[str] = [] def fit(self, labels: list[str]) -> "_LabelEncoder": self.labels = sorted(set(labels)) return self def transform(self, labels: list[str]) -> list[int]: return [self.labels.index(label) for label in labels] def fit_transform(self, labels: list[str]) -> list[int]: return self.fit(labels).transform(labels) def get_labels(self) -> list[str]: return self.labels def get_indices(self) -> list[int]: return list(range(len(self.labels))) def _calculate_griddata( info: _SubContourInfo, ) -> tuple[ np.ndarray, np.ndarray, np.ndarray, list[int], list[str], list[int], list[str], _PlotValues, _PlotValues, ]: xaxis = info.xaxis yaxis = info.yaxis z_values_dict = info.z_values x_values = [] y_values = [] z_values = [] for x_value, y_value in zip(xaxis.values, yaxis.values): if x_value is not None and y_value is not None: x_values.append(x_value) y_values.append(y_value) x_i = xaxis.indices.index(x_value) y_i = yaxis.indices.index(y_value) z_values.append(z_values_dict[(x_i, y_i)]) # Return empty values when x or y has no value. if len(x_values) == 0 or len(y_values) == 0: return ( np.array([]), np.array([]), np.array([]), [], [], [], [], _PlotValues([], []), _PlotValues([], []), ) def _calculate_axis_data( axis: _AxisInfo, values: Sequence[str | float], ) -> tuple[np.ndarray, list[str], list[int], list[int | float]]: # Convert categorical values to int. cat_param_labels: list[str] = [] cat_param_pos: list[int] = [] returned_values: Sequence[int | float] if axis.is_cat: enc = _LabelEncoder() returned_values = enc.fit_transform(list(map(str, values))) cat_param_labels = enc.get_labels() cat_param_pos = enc.get_indices() else: returned_values = list(map(lambda x: float(x), values)) # For x and y, create 1-D array of evenly spaced coordinates on linear or log scale. if axis.is_log: ci = np.logspace(np.log10(axis.range[0]), np.log10(axis.range[1]), CONTOUR_POINT_NUM) else: ci = np.linspace(axis.range[0], axis.range[1], CONTOUR_POINT_NUM) return ci, cat_param_labels, cat_param_pos, list(returned_values) xi, cat_param_labels_x, cat_param_pos_x, transformed_x_values = _calculate_axis_data( xaxis, x_values, ) yi, cat_param_labels_y, cat_param_pos_y, transformed_y_values = _calculate_axis_data( yaxis, y_values, ) # Calculate grid data points. zi: np.ndarray = np.array([]) # Create irregularly spaced map of trial values # and interpolate it with Plotly's interpolation formulation. if xaxis.name != yaxis.name: zmap = _create_zmap(transformed_x_values, transformed_y_values, z_values, xi, yi) zi = _interpolate_zmap(zmap, CONTOUR_POINT_NUM) # categorize by constraints feasible = _PlotValues([], []) infeasible = _PlotValues([], []) for x_value, y_value, c in zip(transformed_x_values, transformed_y_values, info.constraints): if c: feasible.x.append(x_value) feasible.y.append(y_value) else: infeasible.x.append(x_value) infeasible.y.append(y_value) return ( xi, yi, zi, cat_param_pos_x, cat_param_labels_x, cat_param_pos_y, cat_param_labels_y, feasible, infeasible, ) def _generate_contour_subplot(info: _SubContourInfo, ax: "Axes", cmap: "Colormap") -> "ContourSet": if len(info.xaxis.indices) < 2 or len(info.yaxis.indices) < 2: ax.label_outer() return ax ax.set(xlabel=info.xaxis.name, ylabel=info.yaxis.name) ax.set_xlim(info.xaxis.range[0], info.xaxis.range[1]) ax.set_ylim(info.yaxis.range[0], info.yaxis.range[1]) if info.xaxis.name == info.yaxis.name: ax.label_outer() return ax ( xi, yi, zi, x_cat_param_pos, x_cat_param_label, y_cat_param_pos, y_cat_param_label, feasible_plot_values, infeasible_plot_values, ) = _calculate_griddata(info) cs = None if len(zi) > 0: if info.xaxis.is_log: ax.set_xscale("log") if info.yaxis.is_log: ax.set_yscale("log") if info.xaxis.name != info.yaxis.name: # Contour the gridded data. ax.contour(xi, yi, zi, 15, linewidths=0.5, colors="k") cs = ax.contourf(xi, yi, zi, 15, cmap=cmap.reversed()) # Plot data points. ax.scatter( feasible_plot_values.x, feasible_plot_values.y, marker="o", c="black", s=20, edgecolors="grey", linewidth=2.0, ) ax.scatter( infeasible_plot_values.x, infeasible_plot_values.y, marker="o", c="#cccccc", s=20, edgecolors="grey", linewidth=2.0, ) if info.xaxis.is_cat: ax.set_xticks(x_cat_param_pos) ax.set_xticklabels(x_cat_param_label) if info.yaxis.is_cat: ax.set_yticks(y_cat_param_pos) ax.set_yticklabels(y_cat_param_label) ax.label_outer() return cs def _create_zmap( x_values: list[int | float], y_values: list[int | float], z_values: list[float], xi: np.ndarray, yi: np.ndarray, ) -> dict[tuple[int, int], float]: # Creates z-map from trial values and params. # z-map is represented by hashmap of coordinate and trial value pairs. # # Coordinates are represented by tuple of integers, where the first item # indicates x-axis index and the second item indicates y-axis index # and refer to a position of trial value on irregular param grid. # # Since params were resampled either with linspace or logspace # original params might not be on the x and y axes anymore # so we are going with close approximations of trial value positions. zmap = dict() for x, y, z in zip(x_values, y_values, z_values): xindex = int(np.argmin(np.abs(xi - x))) yindex = int(np.argmin(np.abs(yi - y))) zmap[(xindex, yindex)] = z return zmap def _interpolate_zmap(zmap: dict[tuple[int, int], float], contour_plot_num: int) -> np.ndarray: # Implements interpolation formulation used in Plotly # to interpolate heatmaps and contour plots # https://github.com/plotly/plotly.js/blob/95b3bd1bb19d8dc226627442f8f66bce9576def8/src/traces/heatmap/interp2d.js#L15-L20 # citing their doc: # # > Fill in missing data from a 2D array using an iterative # > poisson equation solver with zero-derivative BC at edges. # > Amazingly, this just amounts to repeatedly averaging all the existing # > nearest neighbors # # Plotly's algorithm is equivalent to solve the following linear simultaneous equation. # It is discretization form of the Poisson equation. # # z[x, y] = zmap[(x, y)] (if zmap[(x, y)] is given) # 4 * z[x, y] = z[x-1, y] + z[x+1, y] + z[x, y-1] + z[x, y+1] (if zmap[(x, y)] is not given) a_data = [] a_row = [] a_col = [] b = np.zeros(contour_plot_num**2) for x in range(contour_plot_num): for y in range(contour_plot_num): grid_index = y * contour_plot_num + x if (x, y) in zmap: a_data.append(1) a_row.append(grid_index) a_col.append(grid_index) b[grid_index] = zmap[(x, y)] else: for dx, dy in ((-1, 0), (1, 0), (0, -1), (0, 1)): if 0 <= x + dx < contour_plot_num and 0 <= y + dy < contour_plot_num: a_data.append(1) a_row.append(grid_index) a_col.append(grid_index) a_data.append(-1) a_row.append(grid_index) a_col.append(grid_index + dy * contour_plot_num + dx) z = scipy.sparse.linalg.spsolve(scipy.sparse.csc_matrix((a_data, (a_row, a_col))), b) return z.reshape((contour_plot_num, contour_plot_num)) optuna-3.5.0/optuna/visualization/matplotlib/_edf.py000066400000000000000000000077441453453102400226670ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._edf import _get_edf_info from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("2.2.0") def plot_edf( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the objective value EDF (empirical distribution function) of a study with Matplotlib. Note that only the complete trials are considered when plotting the EDF. .. seealso:: Please refer to :func:`optuna.visualization.plot_edf` for an example, where this function can be replaced with it. .. note:: Please refer to `matplotlib.pyplot.legend `_ to adjust the style of the generated legend. Example: The following code snippet shows how to plot EDF. .. plot:: import math import optuna def ackley(x, y): a = 20 * math.exp(-0.2 * math.sqrt(0.5 * (x ** 2 + y ** 2))) b = math.exp(0.5 * (math.cos(2 * math.pi * x) + math.cos(2 * math.pi * y))) return -a - b + math.e + 20 def objective(trial, low, high): x = trial.suggest_float("x", low, high) y = trial.suggest_float("y", low, high) return ackley(x, y) sampler = optuna.samplers.RandomSampler(seed=10) # Widest search space. study0 = optuna.create_study(study_name="x=[0,5), y=[0,5)", sampler=sampler) study0.optimize(lambda t: objective(t, 0, 5), n_trials=500) # Narrower search space. study1 = optuna.create_study(study_name="x=[0,4), y=[0,4)", sampler=sampler) study1.optimize(lambda t: objective(t, 0, 4), n_trials=500) # Narrowest search space but it doesn't include the global optimum point. study2 = optuna.create_study(study_name="x=[1,3), y=[1,3)", sampler=sampler) study2.optimize(lambda t: objective(t, 1, 3), n_trials=500) optuna.visualization.matplotlib.plot_edf([study0, study1, study2]) Args: study: A target :class:`~optuna.study.Study` object. You can pass multiple studies if you want to compare those EDFs. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Empirical Distribution Function Plot") ax.set_xlabel(target_name) ax.set_ylabel("Cumulative Probability") ax.set_ylim(0, 1) cmap = plt.get_cmap("tab20") # Use tab20 colormap for multiple line plots. info = _get_edf_info(study, target, target_name) edf_lines = info.lines if len(edf_lines) == 0: return ax for i, (study_name, y_values) in enumerate(edf_lines): ax.plot(info.x_values, y_values, color=cmap(i), alpha=0.7, label=study_name) if len(edf_lines) >= 2: ax.legend() return ax optuna-3.5.0/optuna/visualization/matplotlib/_hypervolume_history.py000066400000000000000000000063131453453102400262600ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import numpy as np from optuna._experimental import experimental_func from optuna.study import Study from optuna.visualization._hypervolume_history import _get_hypervolume_history_info from optuna.visualization._hypervolume_history import _HypervolumeHistoryInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("3.3.0") def plot_hypervolume_history( study: Study, reference_point: Sequence[float], ) -> "Axes": """Plot hypervolume history of all trials in a study with Matplotlib. Example: The following code snippet shows how to plot optimization history. .. plot:: import optuna import matplotlib.pyplot as plt def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) reference_point=[100, 50] optuna.visualization.matplotlib.plot_hypervolume_history(study, reference_point) plt.tight_layout() .. note:: You need to adjust the size of the plot by yourself using ``plt.tight_layout()`` or ``plt.savefig(IMAGE_NAME, bbox_inches='tight')``. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their hypervolumes. The number of objectives must be 2 or more. reference_point: A reference point to use for hypervolume computation. The dimension of the reference point must be the same as the number of objectives. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() if not study._is_multi_objective(): raise ValueError( "Study must be multi-objective. For single-objective optimization, " "please use plot_optimization_history instead." ) if len(reference_point) != len(study.directions): raise ValueError( "The dimension of the reference point must be the same as the number of objectives." ) info = _get_hypervolume_history_info(study, np.asarray(reference_point, dtype=np.float64)) return _get_hypervolume_history_plot(info) def _get_hypervolume_history_plot( info: _HypervolumeHistoryInfo, ) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Hypervolume History Plot") ax.set_xlabel("Trial") ax.set_ylabel("Hypervolume") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. ax.plot( info.trial_numbers, info.values, marker="o", color=cmap(0), alpha=0.5, ) return ax optuna-3.5.0/optuna/visualization/matplotlib/_intermediate_values.py000066400000000000000000000062261453453102400261540ustar00rootroot00000000000000from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.visualization._intermediate_values import _get_intermediate_plot_info from optuna.visualization._intermediate_values import _IntermediatePlotInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("2.2.0") def plot_intermediate_values(study: Study) -> "Axes": """Plot intermediate values of all trials in a study with Matplotlib. .. note:: Please refer to `matplotlib.pyplot.legend `_ to adjust the style of the generated legend. Example: The following code snippet shows how to plot intermediate values. .. plot:: import optuna def f(x): return (x - 2) ** 2 def df(x): return 2 * x - 4 def objective(trial): lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True) x = 3 for step in range(128): y = f(x) trial.report(y, step=step) if trial.should_prune(): raise optuna.TrialPruned() gy = df(x) x -= gy * lr return y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=16) optuna.visualization.matplotlib.plot_intermediate_values(study) .. seealso:: Please refer to :func:`optuna.visualization.plot_intermediate_values` for an example. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their intermediate values. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() return _get_intermediate_plot(_get_intermediate_plot_info(study)) def _get_intermediate_plot(info: _IntermediatePlotInfo) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots(tight_layout=True) ax.set_title("Intermediate Values Plot") ax.set_xlabel("Step") ax.set_ylabel("Intermediate Value") cmap = plt.get_cmap("tab20") # Use tab20 colormap for multiple line plots. trial_infos = info.trial_infos for i, tinfo in enumerate(trial_infos): ax.plot( tuple((x for x, _ in tinfo.sorted_intermediate_values)), tuple((y for _, y in tinfo.sorted_intermediate_values)), color=cmap(i) if tinfo.feasible else "#CCCCCC", marker=".", alpha=0.7, label="Trial{}".format(tinfo.trial_number), ) if len(trial_infos) >= 2: ax.legend(bbox_to_anchor=(1.05, 1), loc="upper left", borderaxespad=0.0) return ax optuna-3.5.0/optuna/visualization/matplotlib/_matplotlib_imports.py000066400000000000000000000023361453453102400260450ustar00rootroot00000000000000from packaging import version from optuna._imports import try_import with try_import() as _imports: # TODO(ytknzw): Add specific imports. import matplotlib from matplotlib import __version__ as matplotlib_version from matplotlib import pyplot as plt from matplotlib.axes._axes import Axes from matplotlib.collections import LineCollection from matplotlib.collections import PathCollection from matplotlib.colors import Colormap from matplotlib.contour import ContourSet from matplotlib.figure import Figure # TODO(ytknzw): Set precise version. if version.parse(matplotlib_version) < version.parse("3.0.0"): raise ImportError( "Your version of Matplotlib is " + matplotlib_version + " . " "Please install Matplotlib version 3.0.0 or higher. " "Matplotlib can be installed by executing `$ pip install -U matplotlib>=3.0.0`. " "For further information, please refer to the installation guide of Matplotlib. ", name="matplotlib", ) __all__ = [ "_imports", "matplotlib", "matplotlib_version", "plt", "Axes", "LineCollection", "PathCollection", "Colormap", "ContourSet", "Figure", ] optuna-3.5.0/optuna/visualization/matplotlib/_optimization_history.py000066400000000000000000000142531453453102400264310ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import numpy as np from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._optimization_history import _get_optimization_history_info_list from optuna.visualization._optimization_history import _OptimizationHistoryInfo from optuna.visualization._optimization_history import _ValueState from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("2.2.0") def plot_optimization_history( study: Study | Sequence[Study], *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", error_bar: bool = False, ) -> "Axes": """Plot optimization history of all trials in a study with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_optimization_history` for an example. Example: The following code snippet shows how to plot optimization history. .. plot:: import optuna import matplotlib.pyplot as plt def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) optuna.visualization.matplotlib.plot_optimization_history(study) plt.tight_layout() .. note:: You need to adjust the size of the plot by yourself using ``plt.tight_layout()`` or ``plt.savefig(IMAGE_NAME, bbox_inches='tight')``. Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. You can pass multiple studies if you want to compare those optimization histories. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. error_bar: A flag to show the error bar. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info_list = _get_optimization_history_info_list(study, target, target_name, error_bar) return _get_optimization_history_plot(info_list, target_name) def _get_optimization_history_plot( info_list: list[_OptimizationHistoryInfo], target_name: str, ) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Optimization History Plot") ax.set_xlabel("Trial") ax.set_ylabel(target_name) cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. for i, (trial_numbers, values_info, best_values_info) in enumerate(info_list): if values_info.stds is not None: if ( _ValueState.Infeasible in values_info.states or _ValueState.Incomplete in values_info.states ): _logger.warning( "Your study contains infeasible trials. " "In optimization history plot, " "error bars are calculated for only feasible trial values." ) feasible_trial_numbers = trial_numbers feasible_trial_values = values_info.values plt.errorbar( x=feasible_trial_numbers, y=feasible_trial_values, yerr=values_info.stds, capsize=5, fmt="o", color="tab:blue", ) infeasible_trial_numbers: list[int] = [] infeasible_trial_values: list[float] = [] else: feasible_trial_numbers = [ n for n, s in zip(trial_numbers, values_info.states) if s == _ValueState.Feasible ] infeasible_trial_numbers = [ n for n, s in zip(trial_numbers, values_info.states) if s == _ValueState.Infeasible ] feasible_trial_values = [] for num in feasible_trial_numbers: feasible_trial_values.append(values_info.values[num]) infeasible_trial_values = [] for num in infeasible_trial_numbers: infeasible_trial_values.append(values_info.values[num]) ax.scatter( x=feasible_trial_numbers, y=feasible_trial_values, color=cmap(0) if len(info_list) == 1 else cmap(2 * i), alpha=1, label=values_info.label_name, ) if best_values_info is not None: ax.plot( trial_numbers, best_values_info.values, color=cmap(3) if len(info_list) == 1 else cmap(2 * i + 1), alpha=0.5, label=best_values_info.label_name, ) if best_values_info.stds is not None: lower = np.array(best_values_info.values) - np.array(best_values_info.stds) upper = np.array(best_values_info.values) + np.array(best_values_info.stds) ax.fill_between( x=trial_numbers, y1=lower, y2=upper, color="tab:red", alpha=0.4, ) ax.legend() ax.scatter( x=infeasible_trial_numbers, y=infeasible_trial_values, color="#cccccc", ) plt.legend(bbox_to_anchor=(1.05, 1.0), loc="upper left") return ax optuna-3.5.0/optuna/visualization/matplotlib/_parallel_coordinate.py000066400000000000000000000122671453453102400261300ustar00rootroot00000000000000from __future__ import annotations from typing import Callable import numpy as np from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._parallel_coordinate import _get_parallel_coordinate_info from optuna.visualization._parallel_coordinate import _ParallelCoordinateInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import LineCollection from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("2.2.0") def plot_parallel_coordinate( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the high-dimensional parameter relationships in a study with Matplotlib. Note that, if a parameter contains missing values, a trial with missing values is not plotted. .. seealso:: Please refer to :func:`optuna.visualization.plot_parallel_coordinate` for an example. Example: The following code snippet shows how to plot the high-dimensional parameter relationships. .. plot:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) optuna.visualization.matplotlib.plot_parallel_coordinate(study, params=["x", "y"]) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label and the legend. Returns: A :class:`matplotlib.axes.Axes` object. .. note:: The colormap is reversed when the ``target`` argument isn't :obj:`None` or ``direction`` of :class:`~optuna.study.Study` is ``minimize``. """ _imports.check() info = _get_parallel_coordinate_info(study, params, target, target_name) return _get_parallel_coordinate_plot(info) def _get_parallel_coordinate_plot(info: _ParallelCoordinateInfo) -> "Axes": reversescale = info.reverse_scale target_name = info.target_name # Set up the graph style. fig, ax = plt.subplots() cmap = plt.get_cmap("Blues_r" if reversescale else "Blues") ax.set_title("Parallel Coordinate Plot") ax.spines["top"].set_visible(False) ax.spines["bottom"].set_visible(False) # Prepare data for plotting. if len(info.dims_params) == 0 or len(info.dim_objective.values) == 0: return ax obj_min = info.dim_objective.range[0] obj_max = info.dim_objective.range[1] obj_w = obj_max - obj_min dims_obj_base = [[o] for o in info.dim_objective.values] for dim in info.dims_params: p_min = dim.range[0] p_max = dim.range[1] p_w = p_max - p_min if p_w == 0.0: center = obj_w / 2 + obj_min for i in range(len(dim.values)): dims_obj_base[i].append(center) else: for i, v in enumerate(dim.values): dims_obj_base[i].append((v - p_min) / p_w * obj_w + obj_min) # Draw multiple line plots and axes. # Ref: https://stackoverflow.com/a/50029441 n_params = len(info.dims_params) ax.set_xlim(0, n_params) ax.set_ylim(info.dim_objective.range[0], info.dim_objective.range[1]) xs = [range(n_params + 1) for _ in range(len(dims_obj_base))] segments = [np.column_stack([x, y]) for x, y in zip(xs, dims_obj_base)] lc = LineCollection(segments, cmap=cmap) lc.set_array(np.asarray(info.dim_objective.values)) axcb = fig.colorbar(lc, pad=0.1, ax=ax) axcb.set_label(target_name) var_names = [info.dim_objective.label] + [dim.label for dim in info.dims_params] plt.xticks(range(n_params + 1), var_names, rotation=330) for i, dim in enumerate(info.dims_params): ax2 = ax.twinx() if dim.is_log: ax2.set_ylim(np.power(10, dim.range[0]), np.power(10, dim.range[1])) ax2.set_yscale("log") else: ax2.set_ylim(dim.range[0], dim.range[1]) ax2.spines["top"].set_visible(False) ax2.spines["bottom"].set_visible(False) ax2.xaxis.set_visible(False) ax2.spines["right"].set_position(("axes", (i + 1) / n_params)) if dim.is_cat: ax2.set_yticks(dim.tickvals) ax2.set_yticklabels(dim.ticktext) ax.add_collection(lc) return ax optuna-3.5.0/optuna/visualization/matplotlib/_param_importances.py000066400000000000000000000123061453453102400256230ustar00rootroot00000000000000from __future__ import annotations from typing import Callable import numpy as np from optuna._experimental import experimental_func from optuna.importance._base import BaseImportanceEvaluator from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._param_importances import _get_importances_infos from optuna.visualization._param_importances import _ImportancesInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import Figure from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) AXES_PADDING_RATIO = 1.05 @experimental_func("2.2.0") def plot_param_importances( study: Study, evaluator: BaseImportanceEvaluator | None = None, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot hyperparameter importances with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_param_importances` for an example. Example: The following code snippet shows how to plot hyperparameter importances. .. plot:: import optuna def objective(trial): x = trial.suggest_int("x", 0, 2) y = trial.suggest_float("y", -1.0, 1.0) z = trial.suggest_float("z", 0.0, 1.5) return x ** 2 + y ** 3 - z ** 4 sampler = optuna.samplers.RandomSampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) optuna.visualization.matplotlib.plot_param_importances(study) Args: study: An optimized study. evaluator: An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to :class:`~optuna.importance.FanovaImportanceEvaluator`. params: A list of names of parameters to assess. If :obj:`None`, all parameters that are present in all of the completed trials are assessed. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. For multi-objective optimization, all objectives will be plotted if ``target`` is :obj:`None`. .. note:: This argument can be used to specify which objective to plot if ``study`` is being used for multi-objective optimization. For example, to get only the hyperparameter importance of the first objective, use ``target=lambda t: t.values[0]`` for the target parameter. target_name: Target's name to display on the axis label. Names set via :meth:`~optuna.study.Study.set_metric_names` will be used if ``target`` is :obj:`None`, overriding this argument. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() importances_infos = _get_importances_infos(study, evaluator, params, target, target_name) return _get_importances_plot(importances_infos) def _get_importances_plot(infos: tuple[_ImportancesInfo, ...]) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. fig, ax = plt.subplots() ax.set_title("Hyperparameter Importances", loc="left") ax.set_xlabel("Hyperparameter Importance") ax.set_ylabel("Hyperparameter") height = 0.8 / len(infos) # Default height split between objectives. for objective_id, info in enumerate(infos): param_names = info.param_names pos = np.arange(len(param_names)) offset = height * objective_id importance_values = info.importance_values if not importance_values: continue # Draw horizontal bars. ax.barh( pos + offset, importance_values, height=height, align="center", label=info.target_name, color=plt.get_cmap("tab20c")(objective_id), ) _set_bar_labels(info, fig, ax, offset) ax.set_yticks(pos + offset / 2, param_names) ax.legend(loc="best") return ax def _set_bar_labels(info: _ImportancesInfo, fig: "Figure", ax: "Axes", offset: float) -> None: renderer = fig.canvas.get_renderer() for idx, (val, label) in enumerate(zip(info.importance_values, info.importance_labels)): text = ax.text(val, idx + offset, label, va="center") # Sometimes horizontal axis needs to be re-scaled # to avoid text going over plot area. bbox = text.get_window_extent(renderer) bbox = bbox.transformed(ax.transData.inverted()) _, plot_xmax = ax.get_xlim() bbox_xmax = bbox.xmax if bbox_xmax > plot_xmax: ax.set_xlim(xmax=AXES_PADDING_RATIO * bbox_xmax) optuna-3.5.0/optuna/visualization/matplotlib/_pareto_front.py000066400000000000000000000206101453453102400246160ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from typing import Sequence from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._pareto_front import _get_pareto_front_info from optuna.visualization._pareto_front import _ParetoFrontInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("2.8.0") def plot_pareto_front( study: Study, *, target_names: list[str] | None = None, include_dominated_trials: bool = True, axis_order: list[int] | None = None, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None = None, targets: Callable[[FrozenTrial], Sequence[float]] | None = None, ) -> "Axes": """Plot the Pareto front of a study. .. seealso:: Please refer to :func:`optuna.visualization.plot_pareto_front` for an example. Example: The following code snippet shows how to plot the Pareto front of a study. .. plot:: import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 return v0, v1 study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(objective, n_trials=50) optuna.visualization.matplotlib.plot_pareto_front(study) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their objective values. ``study.n_objectives`` must be either 2 or 3 when ``targets`` is :obj:`None`. target_names: Objective name list used as the axis titles. If :obj:`None` is specified, "Objective {objective_index}" is used instead. If ``targets`` is specified for a study that does not contain any completed trial, ``target_name`` must be specified. include_dominated_trials: A flag to include all dominated trial's objective values. axis_order: A list of indices indicating the axis order. If :obj:`None` is specified, default order is used. ``axis_order`` and ``targets`` cannot be used at the same time. .. warning:: Deprecated in v3.0.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v5.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v3.0.0. constraints_func: An optional function that computes the objective constraints. It must take a :class:`~optuna.trial.FrozenTrial` and return the constraints. The return value must be a sequence of :obj:`float` s. A value strictly larger than 0 means that a constraint is violated. A value equal to or smaller than 0 is considered feasible. This specification is the same as in, for example, :class:`~optuna.samplers.NSGAIISampler`. If given, trials are classified into three categories: feasible and best, feasible but non-best, and infeasible. Categories are shown in different colors. Here, whether a trial is best (on Pareto front) or not is determined ignoring all infeasible trials. targets: A function that returns a tuple of target values to display. The argument to this function is :class:`~optuna.trial.FrozenTrial`. ``targets`` must be :obj:`None` or return 2 or 3 values. ``axis_order`` and ``targets`` cannot be used at the same time. If the number of objectives is neither 2 nor 3, ``targets`` must be specified. .. note:: Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info = _get_pareto_front_info( study, target_names, include_dominated_trials, axis_order, constraints_func, targets ) return _get_pareto_front_plot(info) def _get_pareto_front_plot(info: _ParetoFrontInfo) -> "Axes": if info.n_targets == 2: return _get_pareto_front_2d(info) elif info.n_targets == 3: return _get_pareto_front_3d(info) else: assert False, "Must not reach here" def _get_pareto_front_2d(info: _ParetoFrontInfo) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Pareto-front Plot") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. ax.set_xlabel(info.target_names[info.axis_order[0]]) ax.set_ylabel(info.target_names[info.axis_order[1]]) trial_label: str = "Trial" if len(info.infeasible_trials_with_values) > 0: ax.scatter( x=[values[info.axis_order[0]] for _, values in info.infeasible_trials_with_values], y=[values[info.axis_order[1]] for _, values in info.infeasible_trials_with_values], color="#cccccc", label="Infeasible Trial", ) trial_label = "Feasible Trial" if len(info.non_best_trials_with_values) > 0: ax.scatter( x=[values[info.axis_order[0]] for _, values in info.non_best_trials_with_values], y=[values[info.axis_order[1]] for _, values in info.non_best_trials_with_values], color=cmap(0), label=trial_label, ) if len(info.best_trials_with_values) > 0: ax.scatter( x=[values[info.axis_order[0]] for _, values in info.best_trials_with_values], y=[values[info.axis_order[1]] for _, values in info.best_trials_with_values], color=cmap(3), label="Best Trial", ) if info.non_best_trials_with_values is not None and ax.has_data(): ax.legend() return ax def _get_pareto_front_3d(info: _ParetoFrontInfo) -> "Axes": # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. fig = plt.figure() ax = fig.add_subplot(projection="3d") ax.set_title("Pareto-front Plot") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. ax.set_xlabel(info.target_names[info.axis_order[0]]) ax.set_ylabel(info.target_names[info.axis_order[1]]) ax.set_zlabel(info.target_names[info.axis_order[2]]) trial_label: str = "Trial" if ( info.infeasible_trials_with_values is not None and len(info.infeasible_trials_with_values) > 0 ): ax.scatter( xs=[values[info.axis_order[0]] for _, values in info.infeasible_trials_with_values], ys=[values[info.axis_order[1]] for _, values in info.infeasible_trials_with_values], zs=[values[info.axis_order[2]] for _, values in info.infeasible_trials_with_values], color="#cccccc", label="Infeasible Trial", ) trial_label = "Feasible Trial" if info.non_best_trials_with_values is not None and len(info.non_best_trials_with_values) > 0: ax.scatter( xs=[values[info.axis_order[0]] for _, values in info.non_best_trials_with_values], ys=[values[info.axis_order[1]] for _, values in info.non_best_trials_with_values], zs=[values[info.axis_order[2]] for _, values in info.non_best_trials_with_values], color=cmap(0), label=trial_label, ) if info.best_trials_with_values is not None and len(info.best_trials_with_values): ax.scatter( xs=[values[info.axis_order[0]] for _, values in info.best_trials_with_values], ys=[values[info.axis_order[1]] for _, values in info.best_trials_with_values], zs=[values[info.axis_order[2]] for _, values in info.best_trials_with_values], color=cmap(3), label="Best Trial", ) if info.non_best_trials_with_values is not None and ax.has_data(): ax.legend() return ax optuna-3.5.0/optuna/visualization/matplotlib/_rank.py000066400000000000000000000122101453453102400230440ustar00rootroot00000000000000from __future__ import annotations from typing import Callable from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._rank import _get_rank_info from optuna.visualization._rank import _get_tick_info from optuna.visualization._rank import _RankPlotInfo from optuna.visualization._rank import _RankSubplotInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import PathCollection from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) @experimental_func("3.2.0") def plot_rank( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot parameter relations as scatter plots with colors indicating ranks of target value. Note that trials missing the specified parameters will not be plotted. .. seealso:: Please refer to :func:`optuna.visualization.plot_rank` for an example. Warnings: Output figures of this Matplotlib-based :func:`~optuna.visualization.matplotlib.plot_rank` function would be different from those of the Plotly-based :func:`~optuna.visualization.plot_rank`. Example: The following code snippet shows how to plot the parameter relationship as a rank plot. .. plot:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) c0 = 400 - (x + y)**2 trial.set_user_attr("constraint", [c0]) return x ** 2 + y def constraints(trial): return trial.user_attrs["constraint"] sampler = optuna.samplers.TPESampler(seed=10, constraints_func=constraints) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30) optuna.visualization.matplotlib.plot_rank(study, params=["x", "y"]) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the color bar. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() _logger.warning( "Output figures of this Matplotlib-based `plot_rank` function would be different from " "those of the Plotly-based `plot_rank`." ) info = _get_rank_info(study, params, target, target_name) return _get_rank_plot(info) def _get_rank_plot( info: _RankPlotInfo, ) -> "Axes": params = info.params sub_plot_infos = info.sub_plot_infos plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. title = f"Rank ({info.target_name})" n_params = len(params) if n_params == 0: _, ax = plt.subplots() ax.set_title(title) return ax if n_params == 1 or n_params == 2: fig, axs = plt.subplots() axs.set_title(title) pc = _add_rank_subplot(axs, sub_plot_infos[0][0]) else: fig, axs = plt.subplots(n_params, n_params) fig.suptitle(title) for x_i in range(n_params): for y_i in range(n_params): ax = axs[x_i, y_i] # Set the x or y label only if the subplot is in the edge of the overall figure. pc = _add_rank_subplot( ax, sub_plot_infos[x_i][y_i], set_x_label=x_i == (n_params - 1), set_y_label=y_i == 0, ) tick_info = _get_tick_info(info.zs) pc.set_cmap(plt.get_cmap("RdYlBu_r")) cbar = fig.colorbar(pc, ax=axs, ticks=tick_info.coloridxs) cbar.ax.set_yticklabels(tick_info.text) cbar.outline.set_edgecolor("gray") return axs def _add_rank_subplot( ax: "Axes", info: _RankSubplotInfo, set_x_label: bool = True, set_y_label: bool = True ) -> "PathCollection": if set_x_label: ax.set_xlabel(info.xaxis.name) if set_y_label: ax.set_ylabel(info.yaxis.name) if not info.xaxis.is_cat: ax.set_xlim(info.xaxis.range[0], info.xaxis.range[1]) if not info.yaxis.is_cat: ax.set_ylim(info.yaxis.range[0], info.yaxis.range[1]) if info.xaxis.is_log: ax.set_xscale("log") if info.yaxis.is_log: ax.set_yscale("log") return ax.scatter(x=info.xs, y=info.ys, c=info.colors / 255, edgecolors="grey") optuna-3.5.0/optuna/visualization/matplotlib/_slice.py000066400000000000000000000156631453453102400232270ustar00rootroot00000000000000from __future__ import annotations from collections import defaultdict import math from typing import Any from typing import Callable from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import FrozenTrial from optuna.visualization._slice import _get_slice_plot_info from optuna.visualization._slice import _PlotValues from optuna.visualization._slice import _SlicePlotInfo from optuna.visualization._slice import _SliceSubplotInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import Colormap from optuna.visualization.matplotlib._matplotlib_imports import matplotlib from optuna.visualization.matplotlib._matplotlib_imports import PathCollection from optuna.visualization.matplotlib._matplotlib_imports import plt @experimental_func("2.2.0") def plot_slice( study: Study, params: list[str] | None = None, *, target: Callable[[FrozenTrial], float] | None = None, target_name: str = "Objective Value", ) -> "Axes": """Plot the parameter relationship as slice plot in a study with Matplotlib. .. seealso:: Please refer to :func:`optuna.visualization.plot_slice` for an example. Example: The following code snippet shows how to plot the parameter relationship as slice plot. .. plot:: import optuna def objective(trial): x = trial.suggest_float("x", -100, 100) y = trial.suggest_categorical("y", [-1, 0, 1]) return x ** 2 + y sampler = optuna.samplers.TPESampler(seed=10) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) optuna.visualization.matplotlib.plot_slice(study, params=["x", "y"]) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their target values. params: Parameter list to visualize. The default is all parameters. target: A function to specify the value to display. If it is :obj:`None` and ``study`` is being used for single-objective optimization, the objective values are plotted. .. note:: Specify this argument if ``study`` is being used for multi-objective optimization. target_name: Target's name to display on the axis label. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() return _get_slice_plot(_get_slice_plot_info(study, params, target, target_name)) def _get_slice_plot(info: _SlicePlotInfo) -> "Axes": if len(info.subplots) == 0: _, ax = plt.subplots() return ax # Set up the graph style. cmap = plt.get_cmap("Blues") padding_ratio = 0.05 plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. if len(info.subplots) == 1: # Set up the graph style. fig, axs = plt.subplots() axs.set_title("Slice Plot") # Draw a scatter plot. sc = _generate_slice_subplot(info.subplots[0], axs, cmap, padding_ratio, info.target_name) else: # Set up the graph style. min_figwidth = matplotlib.rcParams["figure.figsize"][0] / 2 fighight = matplotlib.rcParams["figure.figsize"][1] # Ensure that each subplot has a minimum width without relying on auto-sizing. fig, axs = plt.subplots( 1, len(info.subplots), sharey=True, figsize=(min_figwidth * len(info.subplots), fighight), ) fig.suptitle("Slice Plot") # Draw scatter plots. for i, subplot in enumerate(info.subplots): ax = axs[i] sc = _generate_slice_subplot(subplot, ax, cmap, padding_ratio, info.target_name) axcb = fig.colorbar(sc, ax=axs) axcb.set_label("Trial") return axs def _generate_slice_subplot( subplot_info: _SliceSubplotInfo, ax: "Axes", cmap: "Colormap", padding_ratio: float, target_name: str, ) -> "PathCollection": ax.set(xlabel=subplot_info.param_name, ylabel=target_name) scale = None feasible = _PlotValues([], [], []) infeasible = _PlotValues([], [], []) for x, y, num, c in zip( subplot_info.x, subplot_info.y, subplot_info.trial_numbers, subplot_info.constraints ): if x is not None or x != "None" or y is not None or y != "None": if c: feasible.x.append(x) feasible.y.append(y) feasible.trial_numbers.append(num) else: infeasible.x.append(x) infeasible.y.append(y) infeasible.trial_numbers.append(num) if subplot_info.is_log: ax.set_xscale("log") scale = "log" if subplot_info.is_numerical: feasible_x = feasible.x feasible_y = feasible.y feasible_c = feasible.trial_numbers infeasible_x = infeasible.x infeasible_y = infeasible.y else: feasible_x, feasible_y, feasible_c = _get_categorical_plot_values(subplot_info, feasible) infeasible_x, infeasible_y, _ = _get_categorical_plot_values(subplot_info, infeasible) scale = "categorical" xlim = _calc_lim_with_padding(feasible_x + infeasible_x, padding_ratio, scale) ax.set_xlim(xlim[0], xlim[1]) sc = ax.scatter(feasible_x, feasible_y, c=feasible_c, cmap=cmap, edgecolors="grey") ax.scatter(infeasible_x, infeasible_y, c="#cccccc", label="Infeasible Trial") ax.label_outer() return sc def _get_categorical_plot_values( subplot_info: _SliceSubplotInfo, values: _PlotValues ) -> tuple[list[Any], list[float], list[int]]: assert subplot_info.x_labels is not None value_x = [] value_y = [] value_c = [] points_dict = defaultdict(list) for x, y, number in zip(values.x, values.y, values.trial_numbers): points_dict[x].append((y, number)) for x_label in subplot_info.x_labels: for y, number in points_dict[x_label]: value_x.append(str(x_label)) value_y.append(y) value_c.append(number) return value_x, value_y, value_c def _calc_lim_with_padding( values: list[Any], padding_ratio: float, scale: str | None ) -> tuple[float, float]: value_max = max(values) value_min = min(values) if scale == "log": padding = (math.log10(value_max) - math.log10(value_min)) * padding_ratio return ( math.pow(10, math.log10(value_min) - padding), math.pow(10, math.log10(value_max) + padding), ) elif scale == "categorical": width = len(set(values)) - 1 padding = width * padding_ratio return -padding, width + padding else: padding = (value_max - value_min) * padding_ratio return value_min - padding, value_max + padding optuna-3.5.0/optuna/visualization/matplotlib/_terminator_improvement.py000066400000000000000000000133601453453102400267310ustar00rootroot00000000000000from __future__ import annotations from optuna._experimental import experimental_func from optuna.logging import get_logger from optuna.study.study import Study from optuna.terminator import BaseErrorEvaluator from optuna.terminator import BaseImprovementEvaluator from optuna.terminator.improvement.evaluator import DEFAULT_MIN_N_TRIALS from optuna.visualization._terminator_improvement import _get_improvement_info from optuna.visualization._terminator_improvement import _get_y_range from optuna.visualization._terminator_improvement import _ImprovementInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt _logger = get_logger(__name__) PADDING_RATIO_Y = 0.05 ALPHA = 0.25 @experimental_func("3.2.0") def plot_terminator_improvement( study: Study, plot_error: bool = False, improvement_evaluator: BaseImprovementEvaluator | None = None, error_evaluator: BaseErrorEvaluator | None = None, min_n_trials: int = DEFAULT_MIN_N_TRIALS, ) -> "Axes": """Plot the potentials for future objective improvement. This function visualizes the objective improvement potentials, evaluated with ``improvement_evaluator``. It helps to determine whether we should continue the optimization or not. You can also plot the error evaluated with ``error_evaluator`` if the ``plot_error`` argument is set to :obj:`True`. Note that this function may take some time to compute the improvement potentials. .. seealso:: Please refer to :func:`optuna.visualization.plot_terminator_improvement`. Example: The following code snippet shows how to plot improvement potentials, together with cross-validation errors. .. plot:: from lightgbm import LGBMClassifier from sklearn.datasets import load_wine from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold import optuna from optuna.terminator import report_cross_validation_scores from optuna.visualization.matplotlib import plot_terminator_improvement def objective(trial): X, y = load_wine(return_X_y=True) clf = LGBMClassifier( reg_alpha=trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True), reg_lambda=trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True), num_leaves=trial.suggest_int("num_leaves", 2, 256), colsample_bytree=trial.suggest_float("colsample_bytree", 0.4, 1.0), subsample=trial.suggest_float("subsample", 0.4, 1.0), subsample_freq=trial.suggest_int("subsample_freq", 1, 7), min_child_samples=trial.suggest_int("min_child_samples", 5, 100), ) scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True)) report_cross_validation_scores(trial, scores) return scores.mean() study = optuna.create_study() study.optimize(objective, n_trials=30) plot_terminator_improvement(study, plot_error=True) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted for their improvement. plot_error: A flag to show the error. If it is set to :obj:`True`, errors evaluated by ``error_evaluator`` are also plotted as line graph. Defaults to :obj:`False`. improvement_evaluator: An object that evaluates the improvement of the objective function. Default to :class:`~optuna.terminator.RegretBoundEvaluator`. error_evaluator: An object that evaluates the error inherent in the objective function. Default to :class:`~optuna.terminator.CrossValidationErrorEvaluator`. min_n_trials: The minimum number of trials before termination is considered. Terminator improvements for trials below this value are shown in a lighter color. Defaults to ``20``. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info = _get_improvement_info(study, plot_error, improvement_evaluator, error_evaluator) return _get_improvement_plot(info, min_n_trials) def _get_improvement_plot(info: _ImprovementInfo, min_n_trials: int) -> "Axes": n_trials = len(info.trial_numbers) # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. _, ax = plt.subplots() ax.set_title("Terminator Improvement Plot") ax.set_xlabel("Trial") ax.set_ylabel("Terminator Improvement") cmap = plt.get_cmap("tab10") # Use tab10 colormap for similar outputs to plotly. if n_trials == 0: _logger.warning("There are no complete trials.") return ax ax.plot( info.trial_numbers[: min_n_trials + 1], info.improvements[: min_n_trials + 1], marker="o", color=cmap(0), alpha=ALPHA, label="Terminator Improvement" if n_trials <= min_n_trials else None, ) if n_trials > min_n_trials: ax.plot( info.trial_numbers[min_n_trials:], info.improvements[min_n_trials:], marker="o", color=cmap(0), label="Terminator Improvement", ) if info.errors is not None: ax.plot( info.trial_numbers, info.errors, marker="o", color=cmap(3), label="Error", ) ax.legend() ax.set_ylim(_get_y_range(info, min_n_trials)) return ax optuna-3.5.0/optuna/visualization/matplotlib/_timeline.py000066400000000000000000000073551453453102400237350ustar00rootroot00000000000000from optuna._experimental import experimental_func from optuna.study import Study from optuna.trial import TrialState from optuna.visualization._timeline import _get_timeline_info from optuna.visualization._timeline import _TimelineBarInfo from optuna.visualization._timeline import _TimelineInfo from optuna.visualization.matplotlib._matplotlib_imports import _imports if _imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import matplotlib from optuna.visualization.matplotlib._matplotlib_imports import plt _INFEASIBLE_KEY = "INFEASIBLE" @experimental_func("3.2.0") def plot_timeline(study: Study) -> "Axes": """Plot the timeline of a study. .. seealso:: Please refer to :func:`optuna.visualization.plot_timeline` for an example. Example: The following code snippet shows how to plot the timeline of a study. .. plot:: import time import optuna def objective(trial): x = trial.suggest_float("x", 0, 1) time.sleep(x * 0.1) if x > 0.8: raise ValueError() if x > 0.4: raise optuna.TrialPruned() return x ** 2 study = optuna.create_study(direction="minimize") study.optimize( objective, n_trials=50, n_jobs=2, catch=(ValueError,) ) optuna.visualization.matplotlib.plot_timeline(study) Args: study: A :class:`~optuna.study.Study` object whose trials are plotted with their lifetime. Returns: A :class:`matplotlib.axes.Axes` object. """ _imports.check() info = _get_timeline_info(study) return _get_timeline_plot(info) def _get_state_name(bar_info: _TimelineBarInfo) -> str: if bar_info.state == TrialState.COMPLETE and bar_info.infeasible: return _INFEASIBLE_KEY else: return bar_info.state.name def _get_timeline_plot(info: _TimelineInfo) -> "Axes": _cm = { TrialState.COMPLETE.name: "tab:blue", TrialState.FAIL.name: "tab:red", TrialState.PRUNED.name: "tab:orange", _INFEASIBLE_KEY: "#CCCCCC", TrialState.RUNNING.name: "tab:green", TrialState.WAITING.name: "tab:gray", } # Set up the graph style. plt.style.use("ggplot") # Use ggplot style sheet for similar outputs to plotly. fig, ax = plt.subplots() ax.set_title("Timeline Plot") ax.set_xlabel("Datetime") ax.set_ylabel("Trial") if len(info.bars) == 0: return ax ax.barh( y=[b.number for b in info.bars], width=[b.complete - b.start for b in info.bars], left=[b.start for b in info.bars], color=[_cm[_get_state_name(b)] for b in info.bars], ) # There are 5 types of TrialState in total. # However, the legend depicts only types present in the arguments. legend_handles = [] for state_name, color in _cm.items(): if any(_get_state_name(b) == state_name for b in info.bars): legend_handles.append(matplotlib.patches.Patch(color=color, label=state_name)) ax.legend(handles=legend_handles, loc="upper left", bbox_to_anchor=(1.05, 1.0)) fig.tight_layout() assert len(info.bars) > 0 start_time = min([b.start for b in info.bars]) complete_time = max([b.complete for b in info.bars]) margin = (complete_time - start_time) * 0.05 ax.set_xlim(right=complete_time + margin, left=start_time - margin) ax.yaxis.set_major_locator(matplotlib.ticker.MaxNLocator(integer=True)) ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter("%H:%M:%S")) plt.gcf().autofmt_xdate() return ax optuna-3.5.0/optuna/visualization/matplotlib/_utils.py000066400000000000000000000034541453453102400232630ustar00rootroot00000000000000from __future__ import annotations from optuna._experimental import experimental_func from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.trial import FrozenTrial from optuna.visualization.matplotlib import _matplotlib_imports __all__ = ["is_available"] @experimental_func("2.2.0") def is_available() -> bool: """Returns whether visualization with Matplotlib is available or not. .. note:: :mod:`~optuna.visualization.matplotlib` module depends on Matplotlib version 3.0.0 or higher. If a supported version of Matplotlib isn't installed in your environment, this function will return :obj:`False`. In such a case, please execute ``$ pip install -U matplotlib>=3.0.0`` to install Matplotlib. Returns: :obj:`True` if visualization with Matplotlib is available, :obj:`False` otherwise. """ return _matplotlib_imports._imports.is_successful() def _is_log_scale(trials: list[FrozenTrial], param: str) -> bool: for trial in trials: if param in trial.params: dist = trial.distributions[param] if isinstance(dist, (FloatDistribution, IntDistribution)): if dist.log: return True return False def _is_categorical(trials: list[FrozenTrial], param: str) -> bool: return any( isinstance(t.distributions[param], CategoricalDistribution) for t in trials if param in t.params ) def _is_numerical(trials: list[FrozenTrial], param: str) -> bool: return all( (isinstance(t.params[param], int) or isinstance(t.params[param], float)) and not isinstance(t.params[param], bool) for t in trials if param in t.params ) optuna-3.5.0/pyproject.toml000066400000000000000000000107701453453102400157470ustar00rootroot00000000000000[build-system] requires = ["setuptools >= 61.1.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "optuna" description = "A hyperparameter optimization framework" readme = "README.md" license = {file = "LICENSE"} authors = [ {name = "Takuya Akiba"} ] classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Science/Research", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3 :: Only", "Topic :: Scientific/Engineering", "Topic :: Scientific/Engineering :: Mathematics", "Topic :: Scientific/Engineering :: Artificial Intelligence", "Topic :: Software Development", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Python Modules", ] requires-python = ">=3.7" dependencies = [ "alembic>=1.5.0", "colorlog", "numpy", "packaging>=20.0", "sqlalchemy>=1.3.0", "tqdm", "PyYAML", # Only used in `optuna/cli.py`. ] dynamic = ["version"] [project.optional-dependencies] benchmark = [ "asv>=0.5.0", "botorch", "cma", "scikit-optimize", "virtualenv" ] checking = [ "black", "blackdoc", "flake8", "isort", "mypy", "mypy_boto3_s3", "types-PyYAML", "types-redis", "types-setuptools", "types-tqdm", "typing_extensions>=3.10.0.0", ] document = [ "ase", "cma", "cmaes>=0.10.0", # optuna/samplers/_cmaes.py. "botorch", "distributed", "fvcore", "lightgbm", "matplotlib!=3.6.0", "mlflow", "pandas", "pillow", "plotly>=4.9.0", # optuna/visualization. "scikit-learn", "scikit-optimize", "sphinx", "sphinx-copybutton", "sphinx-gallery", "sphinx-plotly-directive", "sphinx_rtd_theme>=1.2.0", "torch", "torchaudio", "torchvision", ] integration = [ "botorch>=0.4.0", "catboost>=0.26; sys_platform!='darwin'", "catboost>=0.26,<1.2; sys_platform=='darwin'", "cma", "distributed", "lightgbm", "lightning", "mlflow", "pandas", "pytorch-ignite", "scikit-learn>=0.24.2", "scikit-optimize", "shap", "tensorflow", "torch", "torchaudio", "torchvision", "wandb", "xgboost", ] optional = [ "boto3", # optuna/artifacts/_boto3.py. "botorch; python_version<='3.11'", # optuna/terminator/gp/botorch. "cmaes>=0.10.0", # optuna/samplers/_cmaes.py. "google-cloud-storage", # optuna/artifacts/_gcs.py. "matplotlib!=3.6.0", # optuna/visualization/matplotlib. "pandas", # optuna/study.py. "plotly>=4.9.0", # optuna/visualization. "redis", # optuna/storages/redis.py. "scikit-learn>=0.24.2", # optuna/visualization/param_importances.py. ] test = [ "coverage", "fakeredis[lua]", "kaleido", "moto", "pytest", "scipy>=1.9.2; python_version>='3.8'", ] [project.urls] homepage = "https://optuna.org/" repository = "https://github.com/optuna/optuna" documentation = "https://optuna.readthedocs.io" bugtracker = "https://github.com/optuna/optuna/issues" [project.scripts] optuna = "optuna.cli:main" [tool.setuptools.packages.find] include = ["optuna*"] [tool.setuptools.dynamic] version = {attr = "optuna.version.__version__"} [tool.setuptools.package-data] "optuna" = [ "storages/_rdb/alembic.ini", "storages/_rdb/alembic/*.*", "storages/_rdb/alembic/versions/*.*", "py.typed", ] [tool.black] line-length = 99 target-version = ['py38'] exclude = ''' /( \.eggs | \.git | \.hg | \.mypy_cache | \.venv | venv | _build | buck-out | build | dist | docs )/ ''' [tool.isort] profile = 'black' src_paths = ['optuna', 'tests', 'docs', 'benchmarks'] skip_glob = ['docs/source/conf.py', '**/alembic/versions/*.py', 'tutorial/**/*.py'] line_length = 99 lines_after_imports = 2 force_single_line = 'True' force_sort_within_sections = 'True' order_by_type = 'False' [tool.pytest.ini_options] addopts = "--color=yes" filterwarnings = 'ignore::optuna.exceptions.ExperimentalWarning' markers = [ "skip_coverage: marks tests are skipped when calculating the coverage", "slow: marks tests as slow (deselect with '-m \"not slow\"')", "integration: marks tests are related to integration", ] optuna-3.5.0/setup.cfg000066400000000000000000000011231453453102400146440ustar00rootroot00000000000000# This section is for flake8. [flake8] ignore = E203, W503 max-line-length = 99 statistics = True exclude = .venv,venv,build,tutorial,.asv # This section is for mypy. [mypy] # Options configure mypy's strict mode. warn_unused_configs = True disallow_untyped_calls = True disallow_untyped_defs = True disallow_incomplete_defs = True check_untyped_defs = True no_implicit_optional = True warn_redundant_casts = True strict_equality = True extra_checks = True no_implicit_reexport = True ignore_missing_imports = True exclude = .venv|venv|build|docs|tutorial|optuna/storages/_rdb/alembic optuna-3.5.0/tests/000077500000000000000000000000001453453102400141705ustar00rootroot00000000000000optuna-3.5.0/tests/__init__.py000066400000000000000000000000001453453102400162670ustar00rootroot00000000000000optuna-3.5.0/tests/artifacts_tests/000077500000000000000000000000001453453102400173725ustar00rootroot00000000000000optuna-3.5.0/tests/artifacts_tests/__init__.py000066400000000000000000000000001453453102400214710ustar00rootroot00000000000000optuna-3.5.0/tests/artifacts_tests/stubs.py000066400000000000000000000031111453453102400211000ustar00rootroot00000000000000from __future__ import annotations import copy import io import shutil import threading from typing import TYPE_CHECKING from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from typing import BinaryIO class FailArtifactStore: def open_reader(self, artifact_id: str) -> BinaryIO: raise Exception("something error raised") def write(self, artifact_id: str, content_body: BinaryIO) -> None: raise Exception("something error raised") def remove(self, artifact_id: str) -> None: raise Exception("something error raised") class InMemoryArtifactStore: def __init__(self) -> None: self._data: dict[str, io.BytesIO] = {} self._lock = threading.Lock() def open_reader(self, artifact_id: str) -> BinaryIO: with self._lock: data = self._data.get(artifact_id) if data is None: raise ArtifactNotFound("not found") return copy.deepcopy(data) def write(self, artifact_id: str, content_body: BinaryIO) -> None: buf = io.BytesIO() shutil.copyfileobj(content_body, buf) buf.seek(0) with self._lock: self._data[artifact_id] = buf def remove(self, artifact_id: str) -> None: with self._lock: if artifact_id not in self._data: raise ArtifactNotFound("not found") del self._data[artifact_id] if TYPE_CHECKING: from optuna.artifacts._protocol import ArtifactStore _fail: ArtifactStore = FailArtifactStore() _inmemory: ArtifactStore = InMemoryArtifactStore() optuna-3.5.0/tests/artifacts_tests/test_backoff.py000066400000000000000000000015331453453102400224000ustar00rootroot00000000000000import io import uuid from optuna.artifacts import Backoff from .stubs import FailArtifactStore from .stubs import InMemoryArtifactStore def test_backoff_time() -> None: backend = Backoff( backend=FailArtifactStore(), min_delay=0.1, multiplier=10, max_delay=10, ) assert backend._get_sleep_secs(0) == 0.1 assert backend._get_sleep_secs(1) == 1 assert backend._get_sleep_secs(2) == 10 def test_read_and_write() -> None: artifact_id = f"test-{uuid.uuid4()}" dummy_content = b"Hello World" backend = Backoff( backend=InMemoryArtifactStore(), min_delay=0.1, multiplier=10, max_delay=10, ) backend.write(artifact_id, io.BytesIO(dummy_content)) with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content optuna-3.5.0/tests/artifacts_tests/test_boto3.py000066400000000000000000000054371453453102400220420ustar00rootroot00000000000000from __future__ import annotations import io from typing import TYPE_CHECKING import boto3 from moto import mock_s3 import pytest from optuna.artifacts import Boto3ArtifactStore from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from collections.abc import Iterator from mypy_boto3_s3 import S3Client from typing_extensions import Annotated # TODO(Shinichi) import Annotated from typing after python 3.8 support is dropped. @pytest.fixture() def init_mock_client() -> Iterator[tuple[str, S3Client]]: with mock_s3(): # Runs before each test bucket_name = "moto-bucket" s3_client = boto3.client("s3") s3_client.create_bucket(Bucket=bucket_name) yield bucket_name, s3_client # Runs after each test objects = s3_client.list_objects(Bucket=bucket_name).get("Contents", []) if objects: s3_client.delete_objects( Bucket=bucket_name, Delete={"Objects": [{"Key": obj["Key"] for obj in objects}], "Quiet": True}, ) s3_client.delete_bucket(Bucket=bucket_name) @pytest.mark.parametrize("avoid_buf_copy", [True, False]) def test_upload_download( init_mock_client: Annotated[tuple[str, S3Client], pytest.fixture], avoid_buf_copy: bool, ) -> None: bucket_name, s3_client = init_mock_client backend = Boto3ArtifactStore(bucket_name, avoid_buf_copy=avoid_buf_copy) artifact_id = "dummy-uuid" dummy_content = b"Hello World" buf = io.BytesIO(dummy_content) backend.write(artifact_id, buf) assert len(s3_client.list_objects(Bucket=bucket_name)["Contents"]) == 1 obj = s3_client.get_object(Bucket=bucket_name, Key=artifact_id) assert obj["Body"].read() == dummy_content with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content if avoid_buf_copy is False: assert buf.closed is False def test_remove(init_mock_client: Annotated[tuple[str, S3Client], pytest.fixture]) -> None: bucket_name, s3_client = init_mock_client backend = Boto3ArtifactStore(bucket_name) artifact_id = "dummy-uuid" backend.write(artifact_id, io.BytesIO(b"Hello")) objects = s3_client.list_objects(Bucket=bucket_name)["Contents"] assert len([obj for obj in objects if obj["Key"] == artifact_id]) == 1 backend.remove(artifact_id) objects = s3_client.list_objects(Bucket=bucket_name).get("Contents", []) assert len([obj for obj in objects if obj["Key"] == artifact_id]) == 0 def test_file_not_found_exception( init_mock_client: Annotated[tuple[str, S3Client], pytest.fixture] ) -> None: bucket_name, _ = init_mock_client backend = Boto3ArtifactStore(bucket_name) with pytest.raises(ArtifactNotFound): backend.open_reader("not-found-id") optuna-3.5.0/tests/artifacts_tests/test_filesystem.py000066400000000000000000000023221453453102400231660ustar00rootroot00000000000000import io from pathlib import Path import pytest from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts.exceptions import ArtifactNotFound def test_upload_download(tmp_path: Path) -> None: artifact_id = "dummy-uuid" dummy_content = b"Hello World" backend = FileSystemArtifactStore(tmp_path) backend.write(artifact_id, io.BytesIO(dummy_content)) with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content def test_remove(tmp_path: Path) -> None: artifact_id = "dummy-uuid" dummy_content = b"Hello World" backend = FileSystemArtifactStore(tmp_path) backend.write(artifact_id, io.BytesIO(dummy_content)) objects = list(tmp_path.glob("*")) assert len([obj for obj in objects if obj.name == artifact_id]) == 1 backend.remove(artifact_id) objects = list(tmp_path.glob("*")) assert len([obj for obj in objects if obj.name == artifact_id]) == 0 def test_file_not_found(tmp_path: str) -> None: backend = FileSystemArtifactStore(tmp_path) with pytest.raises(ArtifactNotFound): backend.open_reader("not-found-id") with pytest.raises(ArtifactNotFound): backend.remove("not-found-id") optuna-3.5.0/tests/artifacts_tests/test_gcs.py000066400000000000000000000071701453453102400215640ustar00rootroot00000000000000from __future__ import annotations import contextlib import io import os from typing import Dict from typing import Optional from typing import TYPE_CHECKING from unittest.mock import patch import google.cloud.storage import pytest from optuna.artifacts import GCSArtifactStore from optuna.artifacts.exceptions import ArtifactNotFound if TYPE_CHECKING: from collections.abc import Iterator _MOCK_BUCKET_CONTENT: Dict[str, bytes] = dict() class MockBucket: def get_blob(self, blob_name: str) -> Optional["MockBlob"]: if blob_name in _MOCK_BUCKET_CONTENT: return MockBlob(blob_name) else: return None def blob(self, blob_name: str) -> "MockBlob": return MockBlob(blob_name) def delete_blob(self, blob_name: str) -> None: global _MOCK_BUCKET_CONTENT del _MOCK_BUCKET_CONTENT[blob_name] def list_blobs(self) -> Iterator["MockBlob"]: for blob_name in _MOCK_BUCKET_CONTENT.keys(): yield MockBlob(blob_name) class MockBlob: def __init__(self, blob_name: str) -> None: self.blob_name = blob_name def download_as_bytes(self) -> bytes: return _MOCK_BUCKET_CONTENT[self.blob_name] def upload_from_string(self, data: bytes) -> None: global _MOCK_BUCKET_CONTENT _MOCK_BUCKET_CONTENT[self.blob_name] = data @contextlib.contextmanager def init_mock_client() -> Iterator[None]: # In case we fail to patch `google.cloud.storage.Client`, we deliberately set an invalid # credential path so that we do not accidentally access GCS. # Note that this is not a perfect measure; it can become ineffective in future when the # mechanism for finding the default credential is changed in the Cloud Storage API. os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/dev/null" with patch("google.cloud.storage.Client") as MockClient: instance = MockClient.return_value def bucket(name: str) -> MockBucket: assert name == "mock-bucket" return MockBucket() instance.bucket.side_effect = bucket yield @pytest.mark.parametrize("explicit_client", [False, True]) def test_upload_download(explicit_client: bool) -> None: with init_mock_client(): bucket_name = "mock-bucket" if explicit_client: backend = GCSArtifactStore(bucket_name, google.cloud.storage.Client()) else: backend = GCSArtifactStore(bucket_name) artifact_id = "dummy-uuid" dummy_content = b"Hello World" buf = io.BytesIO(dummy_content) backend.write(artifact_id, buf) client = google.cloud.storage.Client() assert len(list(client.bucket(bucket_name).list_blobs())) == 1 blob = client.bucket(bucket_name).blob(artifact_id) assert blob.download_as_bytes() == dummy_content with backend.open_reader(artifact_id) as f: actual = f.read() assert actual == dummy_content def test_remove() -> None: with init_mock_client(): bucket_name = "mock-bucket" backend = GCSArtifactStore(bucket_name) client = google.cloud.storage.Client() artifact_id = "dummy-uuid" backend.write(artifact_id, io.BytesIO(b"Hello")) assert len(list(client.bucket(bucket_name).list_blobs())) == 1 backend.remove(artifact_id) assert len(list(client.bucket(bucket_name).list_blobs())) == 0 def test_file_not_found_exception() -> None: with init_mock_client(): bucket_name = "mock-bucket" backend = GCSArtifactStore(bucket_name) with pytest.raises(ArtifactNotFound): backend.open_reader("not-found-id") optuna-3.5.0/tests/artifacts_tests/test_upload_artifact.py000066400000000000000000000071561453453102400241550ustar00rootroot00000000000000from __future__ import annotations import json import pathlib import pytest import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact from optuna.artifacts._protocol import ArtifactStore from optuna.artifacts._upload import ArtifactMeta @pytest.fixture(params=["FileSystem"]) def artifact_store(tmp_path: pathlib.PurePath, request: pytest.FixtureRequest) -> ArtifactStore: if request.param == "FileSystem": return FileSystemArtifactStore(str(tmp_path)) assert False, f"Unknown artifact store: {request.param}" def test_upload_trial_artifact(tmp_path: pathlib.PurePath, artifact_store: ArtifactStore) -> None: file_path = str(tmp_path / "dummy.txt") with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) trial = study.ask() upload_artifact(trial, file_path, artifact_store) frozen_trial = study._storage.get_trial(trial._trial_id) with pytest.raises(ValueError): upload_artifact(frozen_trial, file_path, artifact_store) upload_artifact(frozen_trial, file_path, artifact_store, storage=trial.study._storage) system_attrs = storage.get_trial_system_attrs(frozen_trial._trial_id) artifact_items = [ ArtifactMeta(**json.loads(val)) for key, val in system_attrs.items() if key.startswith("artifacts:") ] assert len(artifact_items) == 2 assert artifact_items[0].artifact_id != artifact_items[1].artifact_id assert artifact_items[0].filename == "dummy.txt" assert artifact_items[0].mimetype == "text/plain" assert artifact_items[0].encoding is None def test_upload_study_artifact(tmp_path: pathlib.PurePath, artifact_store: ArtifactStore) -> None: file_path = str(tmp_path / "dummy.txt") with open(file_path, "w") as f: f.write("foo") storage = optuna.storages.InMemoryStorage() study = optuna.create_study(storage=storage) artifact_id = upload_artifact(study, file_path, artifact_store) system_attrs = storage.get_study_system_attrs(study._study_id) artifact_items = [ ArtifactMeta(**json.loads(val)) for key, val in system_attrs.items() if key.startswith("artifacts:") ] assert len(artifact_items) == 1 assert artifact_items[0].artifact_id == artifact_id assert artifact_items[0].filename == "dummy.txt" assert artifact_items[0].mimetype == "text/plain" assert artifact_items[0].encoding is None def test_upload_artifact_with_mimetype( tmp_path: pathlib.PurePath, artifact_store: ArtifactStore ) -> None: file_path = str(tmp_path / "dummy.obj") with open(file_path, "w") as f: f.write("foo") study = optuna.create_study() trial = study.ask() upload_artifact(trial, file_path, artifact_store, mimetype="model/obj", encoding="utf-8") frozen_trial = study._storage.get_trial(trial._trial_id) with pytest.raises(ValueError): upload_artifact(frozen_trial, file_path, artifact_store) upload_artifact(frozen_trial, file_path, artifact_store, storage=trial.study._storage) system_attrs = trial.study._storage.get_trial(frozen_trial._trial_id).system_attrs artifact_items = [ ArtifactMeta(**json.loads(val)) for key, val in system_attrs.items() if key.startswith("artifacts:") ] assert len(artifact_items) == 2 assert artifact_items[0].artifact_id != artifact_items[1].artifact_id assert artifact_items[0].filename == "dummy.obj" assert artifact_items[0].mimetype == "model/obj" assert artifact_items[0].encoding == "utf-8" optuna-3.5.0/tests/hypervolume_tests/000077500000000000000000000000001453453102400177715ustar00rootroot00000000000000optuna-3.5.0/tests/hypervolume_tests/__init__.py000066400000000000000000000000001453453102400220700ustar00rootroot00000000000000optuna-3.5.0/tests/hypervolume_tests/test_hssp.py000066400000000000000000000033601453453102400223610ustar00rootroot00000000000000import itertools from typing import Tuple import numpy as np import pytest import optuna def _compute_hssp_truth_and_approx(test_case: np.ndarray, subset_size: int) -> Tuple[float, float]: r = 1.1 * np.max(test_case, axis=0) truth = 0.0 for subset in itertools.permutations(test_case, subset_size): truth = max(truth, optuna._hypervolume.WFG().compute(np.asarray(subset), r)) indices = optuna._hypervolume.hssp._solve_hssp( test_case, np.arange(len(test_case)), subset_size, r ) approx = optuna._hypervolume.WFG().compute(test_case[indices], r) return truth, approx @pytest.mark.parametrize("dim", [2, 3]) def test_solve_hssp(dim: int) -> None: rng = np.random.RandomState(128) for i in range(1, 9): subset_size = np.random.randint(1, i + 1) test_case = rng.rand(8, dim) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert approx / truth > 0.6321 # 1 - 1/e @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_solve_hssp_infinite_loss() -> None: rng = np.random.RandomState(128) subset_size = 4 test_case = rng.rand(9, 2) test_case[-1].fill(float("inf")) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert np.isinf(truth) assert np.isinf(approx) test_case = rng.rand(9, 3) test_case[-1].fill(float("inf")) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert truth == 0 assert np.isnan(approx) for dim in range(2, 4): test_case = rng.rand(9, dim) test_case[-1].fill(-float("inf")) truth, approx = _compute_hssp_truth_and_approx(test_case, subset_size) assert np.isinf(truth) assert np.isinf(approx) optuna-3.5.0/tests/hypervolume_tests/test_utils.py000066400000000000000000000013311453453102400225400ustar00rootroot00000000000000import numpy as np import optuna def test_compute_2points_volume() -> None: p1 = np.ones(10) p2 = np.zeros(10) assert 1 == optuna._hypervolume._compute_2points_volume(p1, p2) assert 1 == optuna._hypervolume._compute_2points_volume(p2, p1) p1 = np.ones(10) * 2 p2 = np.ones(10) assert 1 == optuna._hypervolume._compute_2points_volume(p1, p2) def test_compute_2d() -> None: for n in range(2, 30): r = n * np.ones(2) s = np.asarray([[n - 1 - i, i] for i in range(n)]) for i in range(n + 1): s = np.vstack((s, np.asarray([i, n - i]))) np.random.shuffle(s) v = optuna._hypervolume._compute_2d(s, r) assert v == n * n - n * (n - 1) // 2 optuna-3.5.0/tests/hypervolume_tests/test_wfg.py000066400000000000000000000034341453453102400221710ustar00rootroot00000000000000import numpy as np import pytest import optuna def test_wfg_2d() -> None: for n in range(2, 30): r = n * np.ones(2) s = np.asarray([[n - 1 - i, i] for i in range(n)]) for i in range(n + 1): s = np.vstack((s, np.asarray([i, n - i]))) np.random.shuffle(s) v = optuna._hypervolume.WFG().compute(s, r) assert v == n * n - n * (n - 1) // 2 def test_wfg_3d() -> None: n = 3 r = 10 * np.ones(n) s = [np.hstack((np.zeros(i), [1], np.zeros(n - i - 1))) for i in range(n)] for _ in range(10): s.append(np.random.randint(1, 10, size=(n,))) o = np.asarray(s) np.random.shuffle(o) v = optuna._hypervolume.WFG().compute(o, r) assert v == 10**n - 1 def test_wfg_nd() -> None: for n in range(2, 10): r = 10 * np.ones(n) s = [np.hstack((np.zeros(i), [1], np.zeros(n - i - 1))) for i in range(n)] for _ in range(10): s.append(np.random.randint(1, 10, size=(n,))) o = np.asarray(s) np.random.shuffle(o) v = optuna._hypervolume.WFG().compute(o, r) assert v == 10**n - 1 def test_wfg_duplicate_points() -> None: n = 3 r = 10 * np.ones(n) s = [np.hstack((np.zeros(i), [1], np.zeros(n - i - 1))) for i in range(n)] for _ in range(10): s.append(np.random.randint(1, 10, size=(n,))) o = np.asarray(s) v = optuna._hypervolume.WFG().compute(o, r) # Add an already existing point. o = np.vstack([o, o[-1]]) np.random.shuffle(o) v_with_duplicate_point = optuna._hypervolume.WFG().compute(o, r) assert v == v_with_duplicate_point def test_invalid_input() -> None: r = np.ones(3) s = np.atleast_2d(2 * np.ones(3)) with pytest.raises(ValueError): _ = optuna._hypervolume.WFG().compute(s, r) optuna-3.5.0/tests/importance_tests/000077500000000000000000000000001453453102400175535ustar00rootroot00000000000000optuna-3.5.0/tests/importance_tests/__init__.py000066400000000000000000000000001453453102400216520ustar00rootroot00000000000000optuna-3.5.0/tests/importance_tests/fanova_tests/000077500000000000000000000000001453453102400222475ustar00rootroot00000000000000optuna-3.5.0/tests/importance_tests/fanova_tests/__init__.py000066400000000000000000000000001453453102400243460ustar00rootroot00000000000000optuna-3.5.0/tests/importance_tests/fanova_tests/test_tree.py000066400000000000000000000257411453453102400246300ustar00rootroot00000000000000import math from typing import Dict from typing import List from typing import Tuple from unittest.mock import Mock import numpy import pytest from optuna.importance._fanova._tree import _FanovaTree @pytest.fixture def tree() -> _FanovaTree: sklearn_tree = Mock() sklearn_tree.n_features = 3 sklearn_tree.node_count = 5 sklearn_tree.feature = [1, 2, -1, -1, -1] sklearn_tree.children_left = [1, 2, -1, -1, -1] sklearn_tree.children_right = [4, 3, -1, -1, -1] # value has the shape (node_count, n_output, max_n_classes) sklearn_tree.value = numpy.array([[[-1.0]], [[-1.0]], [[0.1]], [[0.2]], [[0.5]]]) sklearn_tree.threshold = [0.5, 1.5, -1.0, -1.0, -1.0] search_spaces = numpy.array([[0.0, 1.0], [0.0, 1.0], [0.0, 2.0]]) return _FanovaTree(tree=sklearn_tree, search_spaces=search_spaces) @pytest.fixture def expected_tree_statistics() -> List[Dict[str, List]]: # Statistics the each node in the tree. return [ {"values": [0.1, 0.2, 0.5], "weights": [0.75, 0.25, 1.0]}, {"values": [0.1, 0.2], "weights": [0.75, 0.25]}, {"values": [0.1], "weights": [0.75]}, {"values": [0.2], "weights": [0.25]}, {"values": [0.5], "weights": [1.0]}, ] def test_tree_variance(tree: _FanovaTree, expected_tree_statistics: List[Dict[str, List]]) -> None: # The root node at node index `0` holds the values and weights for all nodes in the tree. expected_statistics = expected_tree_statistics[0] expected_values = expected_statistics["values"] expected_weights = expected_statistics["weights"] expected_average_value = numpy.average(expected_values, weights=expected_weights) expected_variance = numpy.average( (expected_values - expected_average_value) ** 2, weights=expected_weights ) assert math.isclose(tree.variance, expected_variance) Size = float NodeIndex = int Cardinality = float @pytest.mark.parametrize( "features,expected", [ ([0], [([1.0], [(0, 1.0)])]), ([1], [([0.5], [(1, 0.5)]), ([0.5], [(4, 0.5)])]), ([2], [([1.5], [(2, 1.5), (4, 2.0)]), ([0.5], [(3, 0.5), (4, 2.0)])]), ([0, 1], [([1.0, 0.5], [(1, 0.5)]), ([1.0, 0.5], [(4, 0.5)])]), ([0, 2], [([1.0, 1.5], [(2, 1.5), (4, 2.0)]), ([1.0, 0.5], [(3, 0.5), (4, 2.0)])]), ( [1, 2], [ ([0.5, 1.5], [(2, 0.5 * 1.5)]), ([0.5, 1.5], [(4, 0.5 * 2.0)]), ([0.5, 0.5], [(3, 0.5 * 0.5)]), ([0.5, 0.5], [(4, 0.5 * 2.0)]), ], ), ( [0, 1, 2], [ ([1.0, 0.5, 1.5], [(2, 1.0 * 0.5 * 1.5)]), ([1.0, 0.5, 1.5], [(4, 1.0 * 0.5 * 2.0)]), ([1.0, 0.5, 0.5], [(3, 1.0 * 0.5 * 0.5)]), ([1.0, 0.5, 0.5], [(4, 1.0 * 0.5 * 2.0)]), ], ), ], ) def test_tree_get_marginal_variance( tree: _FanovaTree, features: List[int], expected: List[Tuple[List[Size], List[Tuple[NodeIndex, Cardinality]]]], expected_tree_statistics: List[Dict[str, List]], ) -> None: variance = tree.get_marginal_variance(numpy.array(features)) expected_values = [] expected_weights = [] for sizes, node_indices_and_corrections in expected: expected_split_values = [] expected_split_weights = [] for node_index, cardinality in node_indices_and_corrections: expected_statistics = expected_tree_statistics[node_index] expected_split_values.append(expected_statistics["values"]) expected_split_weights.append( [w / cardinality for w in expected_statistics["weights"]] ) expected_value = numpy.average(expected_split_values, weights=expected_split_weights) expected_weight = numpy.prod(numpy.array(sizes) * numpy.sum(expected_split_weights)) expected_values.append(expected_value) expected_weights.append(expected_weight) expected_average_value = numpy.average(expected_values, weights=expected_weights) expected_variance = numpy.average( (expected_values - expected_average_value) ** 2, weights=expected_weights ) assert math.isclose(variance, expected_variance) @pytest.mark.parametrize( "feature_vector,expected", [ ([0.5, float("nan"), float("nan")], [(0, 1.0)]), ([float("nan"), 0.25, float("nan")], [(1, 0.5)]), ([float("nan"), 0.75, float("nan")], [(4, 0.5)]), ([float("nan"), float("nan"), 0.75], [(2, 1.5), (4, 2.0)]), ([float("nan"), float("nan"), 1.75], [(3, 0.5), (4, 2.0)]), ([0.5, 0.25, float("nan")], [(1, 1.0 * 0.5)]), ([0.5, 0.75, float("nan")], [(4, 1.0 * 0.5)]), ([0.5, float("nan"), 0.75], [(2, 1.0 * 1.5), (4, 1.0 * 2.0)]), ([0.5, float("nan"), 1.75], [(3, 1.0 * 0.5), (4, 1.0 * 2.0)]), ([float("nan"), 0.25, 0.75], [(2, 0.5 * 1.5)]), ([float("nan"), 0.25, 1.75], [(3, 0.5 * 0.5)]), ([float("nan"), 0.75, 0.75], [(4, 0.5 * 2.0)]), ([float("nan"), 0.75, 1.75], [(4, 0.5 * 2.0)]), ([0.5, 0.25, 0.75], [(2, 1.0 * 0.5 * 1.5)]), ([0.5, 0.25, 1.75], [(3, 1.0 * 0.5 * 0.5)]), ([0.5, 0.75, 0.75], [(4, 1.0 * 0.5 * 2.0)]), ([0.5, 0.75, 1.75], [(4, 1.0 * 0.5 * 2.0)]), ], ) def test_tree_get_marginalized_statistics( tree: _FanovaTree, feature_vector: List[float], expected: List[Tuple[NodeIndex, Cardinality]], expected_tree_statistics: List[Dict[str, List]], ) -> None: value, weight = tree._get_marginalized_statistics(numpy.array(feature_vector)) expected_values = [] expected_weights = [] for node_index, cardinality in expected: expected_statistics = expected_tree_statistics[node_index] expected_values.append(expected_statistics["values"]) expected_weights.append([w / cardinality for w in expected_statistics["weights"]]) expected_value = numpy.average(expected_values, weights=expected_weights) expected_weight = numpy.sum(expected_weights) assert math.isclose(value, expected_value) assert math.isclose(weight, expected_weight) def test_tree_statistics( tree: _FanovaTree, expected_tree_statistics: List[Dict[str, List]] ) -> None: statistics = tree._statistics for statistic, expected_statistic in zip(statistics, expected_tree_statistics): value, weight = statistic expected_values = expected_statistic["values"] expected_weights = expected_statistic["weights"] expected_value = numpy.average(expected_values, weights=expected_weights) assert math.isclose(value, expected_value) assert math.isclose(weight, sum(expected_weights)) @pytest.mark.parametrize("node_index,expected", [(0, [0.5]), (1, [0.25, 0.75]), (2, [0.75, 1.75])]) def test_tree_split_midpoints( tree: _FanovaTree, node_index: NodeIndex, expected: List[float] ) -> None: numpy.testing.assert_equal(tree._split_midpoints[node_index], expected) @pytest.mark.parametrize("node_index,expected", [(0, [1.0]), (1, [0.5, 0.5]), (2, [1.5, 0.5])]) def test_tree_split_sizes(tree: _FanovaTree, node_index: NodeIndex, expected: List[float]) -> None: numpy.testing.assert_equal(tree._split_sizes[node_index], expected) @pytest.mark.parametrize( "node_index,expected", [ (0, [False, True, True]), (1, [False, False, True]), (2, [False, False, False]), (3, [False, False, False]), (4, [False, False, False]), ], ) def test_tree_subtree_active_features( tree: _FanovaTree, node_index: NodeIndex, expected: List[bool] ) -> None: active_features: numpy.ndarray = tree._subtree_active_features[node_index] == expected assert active_features.all() def test_tree_attrs(tree: _FanovaTree) -> None: assert tree._n_features == 3 assert tree._n_nodes == 5 assert not tree._is_node_leaf(0) assert not tree._is_node_leaf(1) assert tree._is_node_leaf(2) assert tree._is_node_leaf(3) assert tree._is_node_leaf(4) assert tree._get_node_left_child(0) == 1 assert tree._get_node_left_child(1) == 2 assert tree._get_node_left_child(2) == -1 assert tree._get_node_left_child(3) == -1 assert tree._get_node_left_child(4) == -1 assert tree._get_node_right_child(0) == 4 assert tree._get_node_right_child(1) == 3 assert tree._get_node_right_child(2) == -1 assert tree._get_node_right_child(3) == -1 assert tree._get_node_right_child(4) == -1 assert tree._get_node_children(0) == (1, 4) assert tree._get_node_children(1) == (2, 3) assert tree._get_node_children(2) == (-1, -1) assert tree._get_node_children(3) == (-1, -1) assert tree._get_node_children(4) == (-1, -1) assert tree._get_node_value(0) == -1.0 assert tree._get_node_value(1) == -1.0 assert tree._get_node_value(2) == 0.1 assert tree._get_node_value(3) == 0.2 assert tree._get_node_value(4) == 0.5 assert tree._get_node_split_threshold(0) == 0.5 assert tree._get_node_split_threshold(1) == 1.5 assert tree._get_node_split_threshold(2) == -1.0 assert tree._get_node_split_threshold(3) == -1.0 assert tree._get_node_split_threshold(4) == -1.0 assert tree._get_node_split_feature(0) == 1 assert tree._get_node_split_feature(1) == 2 def test_tree_get_node_subspaces(tree: _FanovaTree) -> None: search_spaces = numpy.array([[0.0, 1.0], [0.0, 1.0], [0.0, 2.0]]) search_spaces_copy = search_spaces.copy() # Test splitting on second feature, first node. expected_left_child_subspace = numpy.array([[0.0, 1.0], [0.0, 0.5], [0.0, 2.0]]) expected_right_child_subspace = numpy.array([[0.0, 1.0], [0.5, 1.0], [0.0, 2.0]]) numpy.testing.assert_array_equal( tree._get_node_left_child_subspaces(0, search_spaces), expected_left_child_subspace ) numpy.testing.assert_array_equal( tree._get_node_right_child_subspaces(0, search_spaces), expected_right_child_subspace ) numpy.testing.assert_array_equal( tree._get_node_children_subspaces(0, search_spaces)[0], expected_left_child_subspace ) numpy.testing.assert_array_equal( tree._get_node_children_subspaces(0, search_spaces)[1], expected_right_child_subspace ) numpy.testing.assert_array_equal(search_spaces, search_spaces_copy) # Test splitting on third feature, second node. expected_left_child_subspace = numpy.array([[0.0, 1.0], [0.0, 1.0], [0.0, 1.5]]) expected_right_child_subspace = numpy.array([[0.0, 1.0], [0.0, 1.0], [1.5, 2.0]]) numpy.testing.assert_array_equal( tree._get_node_left_child_subspaces(1, search_spaces), expected_left_child_subspace ) numpy.testing.assert_array_equal( tree._get_node_right_child_subspaces(1, search_spaces), expected_right_child_subspace ) numpy.testing.assert_array_equal( tree._get_node_children_subspaces(1, search_spaces)[0], expected_left_child_subspace ) numpy.testing.assert_array_equal( tree._get_node_children_subspaces(1, search_spaces)[1], expected_right_child_subspace ) numpy.testing.assert_array_equal(search_spaces, search_spaces_copy) optuna-3.5.0/tests/importance_tests/test_fanova.py000066400000000000000000000113051453453102400224360ustar00rootroot00000000000000from typing import Tuple import pytest from optuna import create_study from optuna import Trial from optuna.distributions import FloatDistribution from optuna.importance import FanovaImportanceEvaluator from optuna.samplers import RandomSampler from optuna.trial import create_trial def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 def multi_objective_function(trial: Trial) -> Tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1, x2 * x3 def test_fanova_importance_evaluator_n_trees() -> None: # Assumes that `seed` can be fixed to reproduce identical results. study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = FanovaImportanceEvaluator(n_trees=10, seed=0) param_importance = evaluator.evaluate(study) evaluator = FanovaImportanceEvaluator(n_trees=20, seed=0) param_importance_different_n_trees = evaluator.evaluate(study) assert param_importance != param_importance_different_n_trees def test_fanova_importance_evaluator_max_depth() -> None: # Assumes that `seed` can be fixed to reproduce identical results. study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = FanovaImportanceEvaluator(max_depth=1, seed=0) param_importance = evaluator.evaluate(study) evaluator = FanovaImportanceEvaluator(max_depth=2, seed=0) param_importance_different_max_depth = evaluator.evaluate(study) assert param_importance != param_importance_different_max_depth @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) def test_fanova_importance_evaluator_with_infinite(inf_value: float) -> None: # The test ensures that trials with infinite values are ignored to calculate importance scores. n_trial = 10 seed = 13 # Importance scores are calculated without a trial with an inf value. study = create_study(sampler=RandomSampler(seed=seed)) study.optimize(objective, n_trials=n_trial) evaluator = FanovaImportanceEvaluator(seed=seed) param_importance_without_inf = evaluator.evaluate(study) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( value=inf_value, params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Importance scores are calculated with a trial with an inf value. param_importance_with_inf = evaluator.evaluate(study) # Obtained importance scores should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert param_importance_with_inf == param_importance_without_inf @pytest.mark.parametrize("target_idx", [0, 1]) @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) def test_multi_objective_fanova_importance_evaluator_with_infinite( target_idx: int, inf_value: float ) -> None: # The test ensures that trials with infinite values are ignored to calculate importance scores. n_trial = 10 seed = 13 # Importance scores are calculated without a trial with an inf value. study = create_study(directions=["minimize", "minimize"], sampler=RandomSampler(seed=seed)) study.optimize(multi_objective_function, n_trials=n_trial) evaluator = FanovaImportanceEvaluator(seed=seed) param_importance_without_inf = evaluator.evaluate(study, target=lambda t: t.values[target_idx]) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( values=[inf_value, inf_value], params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Importance scores are calculated with a trial with an inf value. param_importance_with_inf = evaluator.evaluate(study, target=lambda t: t.values[target_idx]) # Obtained importance scores should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert param_importance_with_inf == param_importance_without_inf optuna-3.5.0/tests/importance_tests/test_init.py000066400000000000000000000304601453453102400221320ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from typing import Any from typing import Type import numpy as np import pytest import optuna from optuna import samplers from optuna.exceptions import ExperimentalWarning from optuna.importance import BaseImportanceEvaluator from optuna.importance import FanovaImportanceEvaluator from optuna.importance import get_param_importances from optuna.importance import MeanDecreaseImpurityImportanceEvaluator from optuna.samplers import RandomSampler from optuna.study import create_study from optuna.testing.objectives import pruned_objective from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import Trial evaluators: list[Type[BaseImportanceEvaluator]] = [ MeanDecreaseImpurityImportanceEvaluator, FanovaImportanceEvaluator, ] parametrize_evaluator = pytest.mark.parametrize("evaluator_init_func", evaluators) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_param_importance_target_is_none_and_study_is_multi_obj( storage_mode: str, evaluator_init_func: Callable[[], BaseImportanceEvaluator], ) -> None: def objective(trial: Trial) -> tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) x4 = trial.suggest_int("x4", -3, 3) x5 = trial.suggest_int("x5", 1, 5, log=True) x6 = trial.suggest_categorical("x6", [1.0, 1.1, 1.2]) if trial.number % 2 == 0: # Conditional parameters are ignored unless `params` is specified and is not `None`. x7 = trial.suggest_float("x7", 0.1, 3) value = x1**4 + x2 + x3 - x4**2 - x5 + x6 if trial.number % 2 == 0: value += x7 return value, 0.0 with StorageSupplier(storage_mode) as storage: study = create_study(directions=["minimize", "minimize"], storage=storage) study.optimize(objective, n_trials=3) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("normalize", [True, False]) def test_get_param_importances( storage_mode: str, evaluator_init_func: Callable[[], BaseImportanceEvaluator], normalize: bool ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) x4 = trial.suggest_int("x4", -3, 3) x5 = trial.suggest_int("x5", 1, 5, log=True) x6 = trial.suggest_categorical("x6", [1.0, 1.1, 1.2]) if trial.number % 2 == 0: # Conditional parameters are ignored unless `params` is specified and is not `None`. x7 = trial.suggest_float("x7", 0.1, 3) value = x1**4 + x2 + x3 - x4**2 - x5 + x6 if trial.number % 2 == 0: value += x7 return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=samplers.RandomSampler()) study.optimize(objective, n_trials=3) param_importance = get_param_importances( study, evaluator=evaluator_init_func(), normalize=normalize ) assert isinstance(param_importance, dict) assert len(param_importance) == 6 assert all( param_name in param_importance for param_name in ["x1", "x2", "x3", "x4", "x5", "x6"] ) prev_importance = float("inf") for param_name, importance in param_importance.items(): assert isinstance(param_name, str) assert isinstance(importance, float) assert importance <= prev_importance prev_importance = importance # Sanity check for param importances assert all(0 <= x < float("inf") for x in param_importance.values()) if normalize: assert np.isclose(sum(param_importance.values()), 1.0) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("params", [[], ["x1"], ["x1", "x3"], ["x1", "x4"]]) @pytest.mark.parametrize("normalize", [True, False]) def test_get_param_importances_with_params( storage_mode: str, params: list[str], evaluator_init_func: Callable[[], BaseImportanceEvaluator], normalize: bool, ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) if trial.number % 2 == 0: x4 = trial.suggest_float("x4", 0.1, 3) value = x1**4 + x2 + x3 if trial.number % 2 == 0: value += x4 return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=10) param_importance = get_param_importances( study, evaluator=evaluator_init_func(), params=params, normalize=normalize ) assert isinstance(param_importance, dict) assert len(param_importance) == len(params) assert all(param in param_importance for param in params) for param_name, importance in param_importance.items(): assert isinstance(param_name, str) assert isinstance(importance, float) # Sanity check for param importances assert all(0 <= x < float("inf") for x in param_importance.values()) if normalize: assert len(param_importance) == 0 or np.isclose(sum(param_importance.values()), 1.0) def test_get_param_importances_unnormalized_experimental() -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) return x1**2 study = create_study() study.optimize(objective, n_trials=4) with pytest.warns(ExperimentalWarning): get_param_importances(study, normalize=False) @parametrize_evaluator @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("normalize", [True, False]) def test_get_param_importances_with_target( storage_mode: str, evaluator_init_func: Callable[[], BaseImportanceEvaluator], normalize: bool ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 0, 3, step=1) if trial.number % 2 == 0: x4 = trial.suggest_float("x4", 0.1, 3) value = x1**4 + x2 + x3 if trial.number % 2 == 0: value += x4 return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=3) param_importance = get_param_importances( study, evaluator=evaluator_init_func(), target=lambda t: t.params["x1"] + t.params["x2"], normalize=normalize, ) assert isinstance(param_importance, dict) assert len(param_importance) == 3 assert all(param_name in param_importance for param_name in ["x1", "x2", "x3"]) prev_importance = float("inf") for param_name, importance in param_importance.items(): assert isinstance(param_name, str) assert isinstance(importance, float) assert importance <= prev_importance prev_importance = importance # Sanity check for param importances assert all(0 <= x < float("inf") for x in param_importance.values()) if normalize: assert np.isclose(sum(param_importance.values()), 1.0) @parametrize_evaluator def test_get_param_importances_invalid_empty_study( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: study = create_study() with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) study.optimize(pruned_objective, n_trials=3) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) @parametrize_evaluator def test_get_param_importances_invalid_single_trial( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) return x1**2 study = create_study() study.optimize(objective, n_trials=1) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func()) @parametrize_evaluator def test_get_param_importances_invalid_no_completed_trials_params( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) if trial.number % 2 == 0: _ = trial.suggest_float("x2", 0.1, 3, log=True) raise optuna.TrialPruned return x1**2 study = create_study() study.optimize(objective, n_trials=3) # None of the trials with `x2` are completed. with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x2"]) # None of the trials with `x2` are completed. Adding "x1" should not matter. with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x1", "x2"]) # None of the trials contain `x3`. with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x3"]) @parametrize_evaluator def test_get_param_importances_invalid_dynamic_search_space_params( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, trial.number + 0.1) return x1**2 study = create_study() study.optimize(objective, n_trials=3) with pytest.raises(ValueError): get_param_importances(study, evaluator=evaluator_init_func(), params=["x1"]) @parametrize_evaluator def test_get_param_importances_empty_search_space( evaluator_init_func: Callable[[], BaseImportanceEvaluator] ) -> None: def objective(trial: Trial) -> float: x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 1, 1) return 4 * x**2 + 4 * y**2 study = create_study() study.optimize(objective, n_trials=3) param_importance = get_param_importances(study, evaluator=evaluator_init_func()) assert len(param_importance) == 2 assert all([param in param_importance for param in ["x", "y"]]) assert param_importance["x"] > 0.0 assert param_importance["y"] == 0.0 @parametrize_evaluator def test_importance_evaluator_seed(evaluator_init_func: Any) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = evaluator_init_func(seed=2) param_importance = evaluator.evaluate(study) evaluator = evaluator_init_func(seed=2) param_importance_same_seed = evaluator.evaluate(study) assert param_importance == param_importance_same_seed evaluator = evaluator_init_func(seed=3) param_importance_different_seed = evaluator.evaluate(study) assert param_importance != param_importance_different_seed @parametrize_evaluator def test_importance_evaluator_with_target(evaluator_init_func: Any) -> None: def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 # Assumes that `seed` can be fixed to reproduce identical results. study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = evaluator_init_func(seed=0) param_importance = evaluator.evaluate(study) param_importance_with_target = evaluator.evaluate( study, target=lambda t: t.params["x1"] + t.params["x2"], ) assert param_importance != param_importance_with_target optuna-3.5.0/tests/importance_tests/test_mean_decrease_impurity.py000066400000000000000000000115471453453102400257110ustar00rootroot00000000000000from typing import Tuple import pytest from optuna import create_study from optuna import Trial from optuna.distributions import FloatDistribution from optuna.importance import MeanDecreaseImpurityImportanceEvaluator from optuna.samplers import RandomSampler from optuna.trial import create_trial def objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 def multi_objective_function(trial: Trial) -> Tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1, x2 * x3 def test_mean_decrease_impurity_importance_evaluator_n_trees() -> None: # Assumes that `seed` can be fixed to reproduce identical results. study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = MeanDecreaseImpurityImportanceEvaluator(n_trees=10, seed=0) param_importance = evaluator.evaluate(study) evaluator = MeanDecreaseImpurityImportanceEvaluator(n_trees=20, seed=0) param_importance_different_n_trees = evaluator.evaluate(study) assert param_importance != param_importance_different_n_trees def test_mean_decrease_impurity_importance_evaluator_max_depth() -> None: # Assumes that `seed` can be fixed to reproduce identical results. study = create_study(sampler=RandomSampler(seed=0)) study.optimize(objective, n_trials=3) evaluator = MeanDecreaseImpurityImportanceEvaluator(max_depth=1, seed=0) param_importance = evaluator.evaluate(study) evaluator = MeanDecreaseImpurityImportanceEvaluator(max_depth=2, seed=0) param_importance_different_max_depth = evaluator.evaluate(study) assert param_importance != param_importance_different_max_depth @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) def test_mean_decrease_impurity_importance_evaluator_with_infinite(inf_value: float) -> None: # The test ensures that trials with infinite values are ignored to calculate importance scores. n_trial = 10 seed = 13 # Importance scores are calculated without a trial with an inf value. study = create_study(sampler=RandomSampler(seed=seed)) study.optimize(objective, n_trials=n_trial) evaluator = MeanDecreaseImpurityImportanceEvaluator(seed=seed) param_importance_without_inf = evaluator.evaluate(study) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( value=inf_value, params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Importance scores are calculated with a trial with an inf value. param_importance_with_inf = evaluator.evaluate(study) # Obtained importance scores should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert param_importance_with_inf == param_importance_without_inf @pytest.mark.parametrize("target_idx", [0, 1]) @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) def test_multi_objective_mean_decrease_impurity_importance_evaluator_with_infinite( target_idx: int, inf_value: float ) -> None: # The test ensures that trials with infinite values are ignored to calculate importance scores. n_trial = 10 seed = 13 # Importance scores are calculated without a trial with an inf value. study = create_study(directions=["minimize", "minimize"], sampler=RandomSampler(seed=seed)) study.optimize(multi_objective_function, n_trials=n_trial) evaluator = MeanDecreaseImpurityImportanceEvaluator(seed=seed) param_importance_without_inf = evaluator.evaluate(study, target=lambda t: t.values[target_idx]) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( values=[inf_value, inf_value], params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Importance scores are calculated with a trial with an inf value. param_importance_with_inf = evaluator.evaluate(study, target=lambda t: t.values[target_idx]) # Obtained importance scores should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert param_importance_with_inf == param_importance_without_inf optuna-3.5.0/tests/integration_tests/000077500000000000000000000000001453453102400177355ustar00rootroot00000000000000optuna-3.5.0/tests/integration_tests/__init__.py000066400000000000000000000000001453453102400220340ustar00rootroot00000000000000optuna-3.5.0/tests/integration_tests/lightgbm_tuner_tests/000077500000000000000000000000001453453102400241715ustar00rootroot00000000000000optuna-3.5.0/tests/integration_tests/lightgbm_tuner_tests/__init__.py000066400000000000000000000000001453453102400262700ustar00rootroot00000000000000optuna-3.5.0/tests/integration_tests/lightgbm_tuner_tests/test_alias.py000066400000000000000000000075261453453102400267050ustar00rootroot00000000000000from typing import List import pytest from optuna.integration._lightgbm_tuner.alias import _handling_alias_metrics from optuna.integration._lightgbm_tuner.alias import _handling_alias_parameters pytestmark = pytest.mark.integration def test__handling_alias_parameters() -> None: params = {"reg_alpha": 0.1} _handling_alias_parameters(params) assert "reg_alpha" not in params assert "lambda_l1" in params def test_handling_alias_parameter_with_user_supplied_param() -> None: params = { "num_boost_round": 5, "early_stopping_rounds": 2, "eta": 0.5, } _handling_alias_parameters(params) assert "eta" not in params assert "learning_rate" in params assert params["learning_rate"] == 0.5 def test_handling_alias_parameter() -> None: params = { "num_boost_round": 5, "early_stopping_rounds": 2, "min_data": 0.2, } _handling_alias_parameters(params) assert "min_data" not in params assert "min_child_samples" in params assert params["min_child_samples"] == 0.2 def test_handling_alias_parameter_duplication() -> None: params = { "num_boost_round": 5, "early_stopping_rounds": 2, "min_data": 0.2, "min_child_samples": 0.3, "l1_regularization": 0.0, "l2_regularization": 0.0, "reg_alpha": 0.0, "reg_lambda": 0.0, } _handling_alias_parameters(params) # Here are the main alias names assert "min_child_samples" in params assert "lambda_l1" in params assert "lambda_l2" in params # None of them are the main alias names assert "min_data" not in params assert "l1_regularization" not in params assert "l2_regularization" not in params assert "reg_alpha" not in params assert "reg_lambda" not in params @pytest.mark.parametrize( "aliases, expect", [ ( [ "ndcg", "lambdarank", "rank_xendcg", "xendcg", "xe_ndcg", "xe_ndcg_mart", "xendcg_mart", ], "ndcg", ), (["mean_average_precision", "map"], "map"), (["rmse", "l2_root", "root_mean_squared_error"], "rmse"), (["l1", "regression_l1", "mean_absolute_error", "mae"], "l1"), (["l2", "regression", "regression_l2", "mean_squared_error", "mse"], "l2"), (["auc"], "auc"), (["binary_logloss", "binary"], "binary_logloss"), ( [ "multi_logloss", "multiclass", "softmax", "multiclassova", "multiclass_ova", "ova", "ovr", ], "multi_logloss", ), (["cross_entropy", "xentropy"], "cross_entropy"), (["cross_entropy_lambda", "xentlambda"], "cross_entropy_lambda"), (["kullback_leibler", "kldiv"], "kullback_leibler"), (["mape", "mean_absolute_percentage_error"], "mape"), (["auc_mu"], "auc_mu"), (["custom", "none", "null", "na"], "custom"), ([], None), # If "metric" not in lgbm_params.keys(): return None. ([["lambdarank"]], ["ndcg"]), ( [["lambdarank", "mean_average_precision", "root_mean_squared_error"]], ["ndcg", "map", "rmse"], ), ], ) def test_handling_alias_metrics(aliases: List[str], expect: str) -> None: if len(aliases) > 0: for alias in aliases: lgbm_params = {"metric": alias} _handling_alias_metrics(lgbm_params) assert lgbm_params["metric"] == expect else: lgbm_params = {} _handling_alias_metrics(lgbm_params) assert lgbm_params == {} def test_handling_unexpected_alias_metrics() -> None: with pytest.raises(ValueError): _handling_alias_metrics({"metric": 1}) optuna-3.5.0/tests/integration_tests/lightgbm_tuner_tests/test_optimize.py000066400000000000000000001166441453453102400274560ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Generator import contextlib from tempfile import TemporaryDirectory from typing import Any from typing import TYPE_CHECKING from unittest import mock import warnings import numpy as np import pytest import optuna from optuna._imports import try_import from optuna.integration._lightgbm_tuner.optimize import _BaseTuner from optuna.integration._lightgbm_tuner.optimize import _OptunaObjective from optuna.integration._lightgbm_tuner.optimize import _OptunaObjectiveCV from optuna.integration._lightgbm_tuner.optimize import LightGBMTuner from optuna.integration._lightgbm_tuner.optimize import LightGBMTunerCV import optuna.integration.lightgbm as lgb from optuna.study import Study with try_import(): from lightgbm import early_stopping from lightgbm import log_evaluation import sklearn.datasets from sklearn.model_selection import KFold from sklearn.model_selection import train_test_split pytestmark = pytest.mark.integration @contextlib.contextmanager def turnoff_train(metric: str = "binary_logloss") -> Generator[None, None, None]: unexpected_value = 0.5 dummy_num_iterations = 1234 class DummyBooster: def __init__(self) -> None: self.best_score = { "valid_0": {metric: unexpected_value}, } def current_iteration(self) -> int: return dummy_num_iterations dummy_booster = DummyBooster() with mock.patch("lightgbm.train", return_value=dummy_booster): yield @contextlib.contextmanager def turnoff_cv(metric: str = "binary_logloss") -> Generator[None, None, None]: unexpected_value = 0.5 dummy_results = {"valid {}-mean".format(metric): [unexpected_value]} with mock.patch("lightgbm.cv", return_value=dummy_results): yield class TestOptunaObjective: def test_init_(self) -> None: target_param_names = ["learning_rate"] # Invalid parameter name. with pytest.raises(NotImplementedError): dataset = mock.MagicMock(spec="lgb.Dataset") _OptunaObjective(target_param_names, {}, dataset, {}, 0, "tune_learning_rate", None) def test_call(self) -> None: target_param_names = ["lambda_l1"] lgbm_params: dict[str, Any] = {} train_set = lgb.Dataset(None) val_set = lgb.Dataset(None) lgbm_kwargs = {"valid_sets": val_set} best_score = -np.inf with turnoff_train(): objective = _OptunaObjective( target_param_names, lgbm_params, train_set, lgbm_kwargs, best_score, "tune_lambda_l1", None, ) study = optuna.create_study(direction="minimize") study.optimize(objective, n_trials=10) assert study.best_value == 0.5 class TestOptunaObjectiveCV: def test_call(self) -> None: target_param_names = ["lambda_l1"] lgbm_params: dict[str, Any] = {} train_set = lgb.Dataset(None) lgbm_kwargs: dict[str, Any] = {} best_score = -np.inf with turnoff_cv(): objective = _OptunaObjectiveCV( target_param_names, lgbm_params, train_set, lgbm_kwargs, best_score, "tune_lambda_l1", None, ) study = optuna.create_study(direction="minimize") study.optimize(objective, n_trials=10) assert study.best_value == 0.5 class TestBaseTuner: def test_get_booster_best_score(self) -> None: expected_value = 1.0 booster = mock.MagicMock( spec="lgb.Booster", best_score={"valid_0": {"binary_logloss": expected_value}} ) dummy_dataset = lgb.Dataset(None) tuner = _BaseTuner(lgbm_kwargs=dict(valid_sets=dummy_dataset)) val_score = tuner._get_booster_best_score(booster) assert val_score == expected_value def test_higher_is_better(self) -> None: for metric in [ "auc", "auc_mu", "ndcg", "lambdarank", "rank_xendcg", "xendcg", "xe_ndcg", "xe_ndcg_mart", "xendcg_mart", "map", "mean_average_precision", "average_precision", ]: tuner = _BaseTuner(lgbm_params={"metric": metric}) assert tuner.higher_is_better() for metric in [ "mae", "rmse", "quantile", "mape", "binary_logloss", "multi_logloss", "cross_entropy", ]: tuner = _BaseTuner(lgbm_params={"metric": metric}) assert not tuner.higher_is_better() def test_get_booster_best_score__using_valid_names_as_str(self) -> None: expected_value = 1.0 booster = mock.MagicMock( spec="lgb.Booster", best_score={"dev": {"binary_logloss": expected_value}} ) dummy_dataset = lgb.Dataset(None) tuner = _BaseTuner(lgbm_kwargs={"valid_names": "dev", "valid_sets": dummy_dataset}) val_score = tuner._get_booster_best_score(booster) assert val_score == expected_value def test_get_booster_best_score__using_valid_names_as_list(self) -> None: unexpected_value = 0.5 expected_value = 1.0 booster = mock.MagicMock( spec="lgb.Booster", best_score={ "train": {"binary_logloss": unexpected_value}, "val": {"binary_logloss": expected_value}, }, ) dummy_train_dataset = lgb.Dataset(None) dummy_val_dataset = lgb.Dataset(None) tuner = _BaseTuner( lgbm_kwargs={ "valid_names": ["train", "val"], "valid_sets": [dummy_train_dataset, dummy_val_dataset], } ) val_score = tuner._get_booster_best_score(booster) assert val_score == expected_value def test_compare_validation_metrics(self) -> None: for metric in [ "auc", "ndcg", "lambdarank", "rank_xendcg", "xendcg", "xe_ndcg", "xe_ndcg_mart", "xendcg_mart", "map", "mean_average_precision", ]: tuner = _BaseTuner(lgbm_params={"metric": metric}) assert tuner.compare_validation_metrics(0.5, 0.1) assert not tuner.compare_validation_metrics(0.5, 0.5) assert not tuner.compare_validation_metrics(0.1, 0.5) for metric in ["rmsle", "rmse", "binary_logloss"]: tuner = _BaseTuner(lgbm_params={"metric": metric}) assert not tuner.compare_validation_metrics(0.5, 0.1) assert not tuner.compare_validation_metrics(0.5, 0.5) assert tuner.compare_validation_metrics(0.1, 0.5) @pytest.mark.parametrize( "metric, eval_at_param, expected", [ ("auc", {"eval_at": 5}, "auc"), ("accuracy", {"eval_at": 5}, "accuracy"), ("rmsle", {"eval_at": 5}, "rmsle"), ("rmse", {"eval_at": 5}, "rmse"), ("binary_logloss", {"eval_at": 5}, "binary_logloss"), ("ndcg", {"eval_at": 5}, "ndcg@5"), ("ndcg", {"ndcg_at": 5}, "ndcg@5"), ("ndcg", {"ndcg_eval_at": 5}, "ndcg@5"), ("ndcg", {"eval_at": [20]}, "ndcg@20"), ("ndcg", {"eval_at": [10, 20]}, "ndcg@10"), ("ndcg", {}, "ndcg@1"), ("map", {"eval_at": 5}, "map@5"), ("map", {"eval_at": [20]}, "map@20"), ("map", {"eval_at": [10, 20]}, "map@10"), ("map", {}, "map@1"), ], ) def test_metric_with_eval_at( self, metric: str, eval_at_param: dict[str, int | list[int]], expected: str ) -> None: params: dict[str, str | int | list[int]] = {"metric": metric} params.update(eval_at_param) tuner = _BaseTuner(lgbm_params=params) assert tuner._metric_with_eval_at(metric) == expected def test_metric_with_eval_at_error(self) -> None: tuner = _BaseTuner(lgbm_params={"metric": "ndcg", "eval_at": "1"}) with pytest.raises(ValueError): tuner._metric_with_eval_at("ndcg") class TestLightGBMTuner: def _get_tuner_object( self, params: dict[str, Any] = {}, train_set: "lgb.Dataset" | None = None, kwargs_options: dict[str, Any] = {}, study: Study | None = None, ) -> lgb.LightGBMTuner: # Required keyword arguments. dummy_dataset = lgb.Dataset(None) train_set = train_set or mock.MagicMock(spec="lgb.Dataset") runner = lgb.LightGBMTuner( params, train_set, num_boost_round=5, valid_sets=dummy_dataset, callbacks=[early_stopping(stopping_rounds=2)], study=study, **kwargs_options, ) return runner def test_deprecated_args(self) -> None: dummy_dataset = lgb.Dataset(None) with pytest.warns(FutureWarning): LightGBMTuner({}, dummy_dataset, valid_sets=[dummy_dataset], verbosity=1) def test_no_eval_set_args(self) -> None: params: dict[str, Any] = {} train_set = lgb.Dataset(None) with pytest.raises(ValueError) as excinfo: lgb.LightGBMTuner( params, train_set, num_boost_round=5, callbacks=[early_stopping(stopping_rounds=2)], ) assert excinfo.type == ValueError assert str(excinfo.value) == "`valid_sets` is required." @pytest.mark.parametrize( "metric, study_direction", [ ("auc", "minimize"), ("mse", "maximize"), (None, "maximize"), # The default metric is binary_logloss. ], ) def test_inconsistent_study_direction(self, metric: str, study_direction: str) -> None: params: dict[str, Any] = {} if metric is not None: params["metric"] = metric train_set = lgb.Dataset(None) valid_set = lgb.Dataset(None) study = optuna.create_study(direction=study_direction) with pytest.raises(ValueError) as excinfo: lgb.LightGBMTuner( params, train_set, valid_sets=[train_set, valid_set], num_boost_round=5, callbacks=[early_stopping(stopping_rounds=2)], study=study, ) assert excinfo.type == ValueError assert str(excinfo.value).startswith("Study direction is inconsistent with the metric") def test_with_minimum_required_args(self) -> None: runner = self._get_tuner_object() assert "num_boost_round" in runner.lgbm_kwargs assert "num_boost_round" not in runner.auto_options assert runner.lgbm_kwargs["num_boost_round"] == 5 def test__parse_args_wrapper_args(self) -> None: params: dict[str, Any] = {} train_set = lgb.Dataset(None) val_set = lgb.Dataset(None) runner = lgb.LightGBMTuner( params, train_set, num_boost_round=12, callbacks=[early_stopping(stopping_rounds=10)], valid_sets=val_set, time_budget=600, sample_size=1000, ) new_args = ["time_budget", "time_budget", "sample_size"] for new_arg in new_args: assert new_arg not in runner.lgbm_kwargs assert new_arg in runner.auto_options @pytest.mark.parametrize( "metric, study_direction, expected", [("auc", "maximize", -np.inf), ("l2", "minimize", np.inf)], ) def test_best_score(self, metric: str, study_direction: str, expected: float) -> None: with turnoff_train(metric=metric): study = optuna.create_study(direction=study_direction) runner = self._get_tuner_object( params=dict(lambda_l1=0.0, metric=metric), kwargs_options={}, study=study ) assert runner.best_score == expected runner.tune_regularization_factors() assert runner.best_score == 0.5 def test_best_params(self) -> None: unexpected_value = 20 # out of scope. with turnoff_train(): study = optuna.create_study() runner = self._get_tuner_object( params=dict(lambda_l1=unexpected_value), kwargs_options={}, study=study ) assert runner.best_params["lambda_l1"] == unexpected_value runner.tune_regularization_factors() assert runner.best_params["lambda_l1"] != unexpected_value def test_sample_train_set(self) -> None: sample_size = 3 X_trn = np.random.uniform(10, size=50).reshape((10, 5)) y_trn = np.random.randint(2, size=10) train_dataset = lgb.Dataset(X_trn, label=y_trn) runner = self._get_tuner_object( train_set=train_dataset, kwargs_options=dict(sample_size=sample_size) ) runner.sample_train_set() # Workaround for mypy. if not TYPE_CHECKING: runner.train_subset.construct() # Cannot get label before construct `lgb.Dataset`. assert runner.train_subset.get_label().shape[0] == sample_size def test_time_budget(self) -> None: unexpected_value = 1.1 # out of scope. with turnoff_train(): runner = self._get_tuner_object( params=dict( feature_fraction=unexpected_value, # set default as unexpected value. ), kwargs_options=dict(time_budget=0), ) assert len(runner.study.trials) == 0 # No trials run because `time_budget` is set to zero. runner.tune_feature_fraction() assert runner.lgbm_params["feature_fraction"] == unexpected_value assert len(runner.study.trials) == 0 def test_tune_feature_fraction(self) -> None: unexpected_value = 1.1 # out of scope. with turnoff_train(): runner = self._get_tuner_object( params=dict( feature_fraction=unexpected_value, # set default as unexpected value. ), ) assert len(runner.study.trials) == 0 runner.tune_feature_fraction() assert runner.lgbm_params["feature_fraction"] != unexpected_value assert len(runner.study.trials) == 7 def test_tune_num_leaves(self) -> None: unexpected_value = 1 # out of scope. with turnoff_train(): runner = self._get_tuner_object(params=dict(num_leaves=unexpected_value)) assert len(runner.study.trials) == 0 runner.tune_num_leaves() assert runner.lgbm_params["num_leaves"] != unexpected_value assert len(runner.study.trials) == 20 def test_tune_num_leaves_negative_max_depth(self) -> None: params: dict[str, Any] = {"metric": "binary_logloss", "max_depth": -1, "verbose": -1} X_trn = np.random.uniform(10, size=(10, 5)) y_trn = np.random.randint(2, size=10) train_dataset = lgb.Dataset(X_trn, label=y_trn) valid_dataset = lgb.Dataset(X_trn, label=y_trn) runner = lgb.LightGBMTuner( params, train_dataset, num_boost_round=3, valid_sets=valid_dataset, callbacks=[early_stopping(stopping_rounds=2), log_evaluation(-1)], ) runner.tune_num_leaves() assert len(runner.study.trials) == 20 def test_tune_bagging(self) -> None: unexpected_value = 1 # out of scope. with turnoff_train(): runner = self._get_tuner_object(params=dict(bagging_fraction=unexpected_value)) assert len(runner.study.trials) == 0 runner.tune_bagging() assert runner.lgbm_params["bagging_fraction"] != unexpected_value assert len(runner.study.trials) == 10 def test_tune_feature_fraction_stage2(self) -> None: unexpected_value = 0.5 with turnoff_train(): runner = self._get_tuner_object(params=dict(feature_fraction=unexpected_value)) assert len(runner.study.trials) == 0 runner.tune_feature_fraction_stage2() assert runner.lgbm_params["feature_fraction"] != unexpected_value assert len(runner.study.trials) == 6 def test_tune_regularization_factors(self) -> None: unexpected_value = 20 # out of scope. with turnoff_train(): runner = self._get_tuner_object( params=dict(lambda_l1=unexpected_value) # set default as unexpected value. ) assert len(runner.study.trials) == 0 runner.tune_regularization_factors() assert runner.lgbm_params["lambda_l1"] != unexpected_value assert len(runner.study.trials) == 20 def test_tune_min_data_in_leaf(self) -> None: unexpected_value = 1 # out of scope. with turnoff_train(): runner = self._get_tuner_object( params=dict( min_child_samples=unexpected_value, # set default as unexpected value. ), ) assert len(runner.study.trials) == 0 runner.tune_min_data_in_leaf() assert runner.lgbm_params["min_child_samples"] != unexpected_value assert len(runner.study.trials) == 5 def test_when_a_step_does_not_improve_best_score(self) -> None: params: dict = {} valid_data = np.zeros((10, 10)) valid_sets = lgb.Dataset(valid_data) dataset = mock.MagicMock(spec="lgb.Dataset") tuner = LightGBMTuner(params, dataset, valid_sets=valid_sets) assert not tuner.higher_is_better() with mock.patch("lightgbm.train"), mock.patch.object( _BaseTuner, "_get_booster_best_score", return_value=0.9 ): tuner.tune_feature_fraction() assert "feature_fraction" in tuner.best_params assert tuner.best_score == 0.9 # Assume that tuning `num_leaves` doesn't improve the `best_score`. with mock.patch("lightgbm.train"), mock.patch.object( _BaseTuner, "_get_booster_best_score", return_value=1.1 ): tuner.tune_num_leaves() def test_resume_run(self) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() tuner = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, callbacks=[log_evaluation(-1)] ) with mock.patch.object(_BaseTuner, "_get_booster_best_score", return_value=1.0): tuner.tune_regularization_factors() n_trials = len(study.trials) assert n_trials == len(study.trials) tuner2 = LightGBMTuner(params, dataset, valid_sets=dataset, study=study) with mock.patch.object(_BaseTuner, "_get_booster_best_score", return_value=1.0): tuner2.tune_regularization_factors() assert n_trials == len(study.trials) @pytest.mark.parametrize( "verbosity, level", [ (None, optuna.logging.INFO), (-2, optuna.logging.CRITICAL), (-1, optuna.logging.CRITICAL), (0, optuna.logging.WARNING), (1, optuna.logging.INFO), (2, optuna.logging.DEBUG), ], ) def test_run_verbosity(self, verbosity: int, level: int) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.set_verbosity(optuna.logging.INFO) params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) tuner = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, verbosity=verbosity, callbacks=[log_evaluation(-1)], time_budget=1, ) with mock.patch.object(_BaseTuner, "_get_booster_best_score", return_value=1.0): tuner.run() assert optuna.logging.get_verbosity() == level assert tuner.lgbm_params["verbose"] == -1 @pytest.mark.parametrize("show_progress_bar, expected", [(True, 6), (False, 0)]) def test_run_show_progress_bar(self, show_progress_bar: bool, expected: int) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() tuner = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, callbacks=[log_evaluation(-1)], time_budget=1, show_progress_bar=show_progress_bar, ) with mock.patch.object( _BaseTuner, "_get_booster_best_score", return_value=1.0 ), mock.patch("tqdm.tqdm") as mock_tqdm: tuner.run() assert mock_tqdm.call_count == expected def test_get_best_booster(self) -> None: unexpected_value = 20 # out of scope. params: dict = {"verbose": -1, "lambda_l1": unexpected_value} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() tuner = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, callbacks=[log_evaluation(-1)] ) with pytest.raises(ValueError): tuner.get_best_booster() with mock.patch.object(_BaseTuner, "_get_booster_best_score", return_value=0.0): tuner.tune_regularization_factors() best_booster = tuner.get_best_booster() assert isinstance(best_booster.params, dict) assert best_booster.params["lambda_l1"] != unexpected_value tuner2 = LightGBMTuner(params, dataset, valid_sets=dataset, study=study) # Resumed study does not have the best booster. with pytest.raises(ValueError): tuner2.get_best_booster() @pytest.mark.parametrize("dir_exists, expected", [(False, True), (True, False)]) def test_model_dir(self, dir_exists: bool, expected: bool) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) with mock.patch("optuna.integration._lightgbm_tuner.optimize.os.mkdir") as m: with mock.patch("os.path.exists", return_value=dir_exists): LightGBMTuner(params, dataset, valid_sets=dataset, model_dir="./booster") assert m.called == expected def test_best_booster_with_model_dir(self) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() with TemporaryDirectory() as tmpdir: tuner = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, model_dir=tmpdir, callbacks=[log_evaluation(-1)], ) with mock.patch.object(_BaseTuner, "_get_booster_best_score", return_value=0.0): tuner.tune_regularization_factors() best_booster = tuner.get_best_booster() tuner2 = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, model_dir=tmpdir ) best_booster2 = tuner2.get_best_booster() assert best_booster.params == best_booster2.params @pytest.mark.parametrize("direction, overall_best", [("minimize", 1), ("maximize", 2)]) def test_create_stepwise_study(self, direction: str, overall_best: int) -> None: dataset = mock.MagicMock(spec="lgb.Dataset") tuner = LightGBMTuner({}, dataset, valid_sets=lgb.Dataset(np.zeros((10, 10)))) def objective(trial: optuna.trial.Trial, value: float) -> float: trial.storage.set_trial_system_attr( trial._trial_id, optuna.integration._lightgbm_tuner.optimize._STEP_NAME_KEY, "step{:.0f}".format(value), ) return trial.suggest_float("x", value, value) study = optuna.create_study(direction=direction) study_step1 = tuner._create_stepwise_study(study, "step1") with pytest.raises(ValueError): study_step1.best_trial study_step1.optimize(lambda t: objective(t, 1), n_trials=1) study_step2 = tuner._create_stepwise_study(study, "step2") # `study` has a trial, but `study_step2` has no trials. with pytest.raises(ValueError): study_step2.best_trial study_step2.optimize(lambda t: objective(t, 2), n_trials=2) assert len(study_step1.trials) == 1 assert len(study_step2.trials) == 2 assert len(study.trials) == 3 assert study_step1.best_trial.value == 1 assert study_step2.best_trial.value == 2 assert study.best_trial.value == overall_best def test_optuna_callback(self) -> None: params: dict[str, Any] = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) callback_mock = mock.MagicMock() study = optuna.create_study() tuner = LightGBMTuner( params, dataset, valid_sets=dataset, study=study, callbacks=[log_evaluation(-1)], optuna_callbacks=[callback_mock], ) with mock.patch.object(_BaseTuner, "_get_booster_best_score", return_value=1.0): tuner._tune_params(["num_leaves"], 10, optuna.samplers.TPESampler(), "num_leaves") assert callback_mock.call_count == 10 def test_tune_best_score_reproducibility(self) -> None: iris = sklearn.datasets.load_iris() X_trainval, X_test, y_trainval, y_test = train_test_split( iris.data, iris.target, random_state=0 ) train = lgb.Dataset(X_trainval, y_trainval) valid = lgb.Dataset(X_test, y_test) params = { "objective": "regression", "metric": "rmse", "random_seed": 0, "deterministic": True, "force_col_wise": True, "verbosity": -1, } tuner_first_try = lgb.LightGBMTuner( params, train, valid_sets=valid, callbacks=[early_stopping(stopping_rounds=3), log_evaluation(-1)], optuna_seed=10, ) tuner_first_try.run() best_score_first_try = tuner_first_try.best_score tuner_second_try = lgb.LightGBMTuner( params, train, valid_sets=valid, callbacks=[early_stopping(stopping_rounds=3), log_evaluation(-1)], optuna_seed=10, ) tuner_second_try.run() best_score_second_try = tuner_second_try.best_score assert best_score_second_try == best_score_first_try first_try_trials = tuner_first_try.study.trials second_try_trials = tuner_second_try.study.trials assert len(first_try_trials) == len(second_try_trials) for first_trial, second_trial in zip(first_try_trials, second_try_trials): assert first_trial.value == second_trial.value assert first_trial.params == second_trial.params class TestLightGBMTunerCV: def _get_tunercv_object( self, params: dict[str, Any] = {}, train_set: lgb.Dataset | None = None, kwargs_options: dict[str, Any] = {}, study: optuna.study.Study | None = None, ) -> LightGBMTunerCV: # Required keyword arguments. kwargs: dict[str, Any] = dict(num_boost_round=5, study=study) kwargs.update(kwargs_options) train_set = train_set or mock.MagicMock(spec="lgb.Dataset") runner = LightGBMTunerCV( params, train_set, callbacks=[early_stopping(stopping_rounds=2)], **kwargs ) return runner def test_deprecated_args(self) -> None: dummy_dataset = lgb.Dataset(None) with pytest.warns(FutureWarning): LightGBMTunerCV({}, dummy_dataset, verbosity=1) @pytest.mark.parametrize( "metric, study_direction", [ ("auc", "minimize"), ("mse", "maximize"), (None, "maximize"), # The default metric is binary_logloss. ], ) def test_inconsistent_study_direction(self, metric: str, study_direction: str) -> None: params: dict[str, Any] = {} if metric is not None: params["metric"] = metric train_set = lgb.Dataset(None) study = optuna.create_study(direction=study_direction) with pytest.raises(ValueError) as excinfo: LightGBMTunerCV( params, train_set, num_boost_round=5, callbacks=[early_stopping(stopping_rounds=2)], study=study, ) assert excinfo.type == ValueError assert str(excinfo.value).startswith("Study direction is inconsistent with the metric") def test_with_minimum_required_args(self) -> None: runner = self._get_tunercv_object() assert "num_boost_round" in runner.lgbm_kwargs assert "num_boost_round" not in runner.auto_options assert runner.lgbm_kwargs["num_boost_round"] == 5 def test_tune_feature_fraction(self) -> None: unexpected_value = 1.1 # out of scope. with turnoff_cv(): runner = self._get_tunercv_object( params=dict( feature_fraction=unexpected_value, # set default as unexpected value. ), ) assert len(runner.study.trials) == 0 runner.tune_feature_fraction() assert runner.lgbm_params["feature_fraction"] != unexpected_value assert len(runner.study.trials) == 7 def test_tune_num_leaves(self) -> None: unexpected_value = 1 # out of scope. with turnoff_cv(): runner = self._get_tunercv_object(params=dict(num_leaves=unexpected_value)) assert len(runner.study.trials) == 0 runner.tune_num_leaves() assert runner.lgbm_params["num_leaves"] != unexpected_value assert len(runner.study.trials) == 20 def test_tune_bagging(self) -> None: unexpected_value = 1 # out of scope. with turnoff_cv(): runner = self._get_tunercv_object(params=dict(bagging_fraction=unexpected_value)) assert len(runner.study.trials) == 0 runner.tune_bagging() assert runner.lgbm_params["bagging_fraction"] != unexpected_value assert len(runner.study.trials) == 10 def test_tune_feature_fraction_stage2(self) -> None: unexpected_value = 0.5 with turnoff_cv(): runner = self._get_tunercv_object(params=dict(feature_fraction=unexpected_value)) assert len(runner.study.trials) == 0 runner.tune_feature_fraction_stage2() assert runner.lgbm_params["feature_fraction"] != unexpected_value assert len(runner.study.trials) == 6 def test_tune_regularization_factors(self) -> None: unexpected_value = 20 # out of scope. with turnoff_cv(): runner = self._get_tunercv_object( params=dict(lambda_l1=unexpected_value) # set default as unexpected value. ) assert len(runner.study.trials) == 0 runner.tune_regularization_factors() assert runner.lgbm_params["lambda_l1"] != unexpected_value assert len(runner.study.trials) == 20 def test_tune_min_data_in_leaf(self) -> None: unexpected_value = 1 # out of scope. with turnoff_cv(): runner = self._get_tunercv_object( params=dict( min_child_samples=unexpected_value, # set default as unexpected value. ), ) assert len(runner.study.trials) == 0 runner.tune_min_data_in_leaf() assert runner.lgbm_params["min_child_samples"] != unexpected_value assert len(runner.study.trials) == 5 def test_resume_run(self) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() tuner = LightGBMTunerCV(params, dataset, study=study) with mock.patch.object(_OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0]): tuner.tune_regularization_factors() n_trials = len(study.trials) assert n_trials == len(study.trials) tuner2 = LightGBMTuner(params, dataset, valid_sets=dataset, study=study) with mock.patch.object(_OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0]): tuner2.tune_regularization_factors() assert n_trials == len(study.trials) @pytest.mark.parametrize( "verbosity, level", [ (None, optuna.logging.INFO), (-2, optuna.logging.CRITICAL), (-1, optuna.logging.CRITICAL), (0, optuna.logging.WARNING), (1, optuna.logging.INFO), (2, optuna.logging.DEBUG), ], ) def test_run_verbosity(self, verbosity: int, level: int) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.set_verbosity(optuna.logging.INFO) params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) tuner = LightGBMTunerCV( params, dataset, study=study, verbosity=verbosity, time_budget=1 ) with mock.patch.object(_OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0]): tuner.run() assert optuna.logging.get_verbosity() == level assert tuner.lgbm_params["verbose"] == -1 @pytest.mark.parametrize("show_progress_bar, expected", [(True, 6), (False, 0)]) def test_run_show_progress_bar(self, show_progress_bar: bool, expected: int) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() tuner = LightGBMTunerCV( params, dataset, study=study, time_budget=1, show_progress_bar=show_progress_bar ) with mock.patch.object( _OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0] ), mock.patch("tqdm.tqdm") as mock_tqdm: tuner.run() assert mock_tqdm.call_count == expected def test_optuna_callback(self) -> None: params: dict[str, Any] = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) callback_mock = mock.MagicMock() study = optuna.create_study() tuner = LightGBMTunerCV(params, dataset, study=study, optuna_callbacks=[callback_mock]) with mock.patch.object(_OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0]): tuner._tune_params(["num_leaves"], 10, optuna.samplers.TPESampler(), "num_leaves") assert callback_mock.call_count == 10 @pytest.mark.parametrize("dir_exists, expected", [(False, True), (True, False)]) def test_model_dir(self, dir_exists: bool, expected: bool) -> None: unexpected_value = 20 # out of scope. params: dict = {"verbose": -1, "lambda_l1": unexpected_value} dataset = lgb.Dataset(np.zeros((10, 10))) with mock.patch("os.mkdir") as m: with mock.patch("os.path.exists", return_value=dir_exists): LightGBMTunerCV(params, dataset, model_dir="./booster") assert m.called == expected def test_get_best_booster(self) -> None: unexpected_value = 20 # out of scope. params: dict = {"verbose": -1, "lambda_l1": unexpected_value} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() with TemporaryDirectory() as tmpdir: tuner = LightGBMTunerCV( params, dataset, study=study, model_dir=tmpdir, return_cvbooster=True ) with pytest.raises(ValueError): tuner.get_best_booster() with mock.patch.object(_OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0]): tuner.tune_regularization_factors() best_boosters = tuner.get_best_booster().boosters for booster in best_boosters: assert booster.params["lambda_l1"] != unexpected_value tuner2 = LightGBMTunerCV( params, dataset, study=study, model_dir=tmpdir, return_cvbooster=True ) best_boosters2 = tuner2.get_best_booster().boosters for booster, booster2 in zip(best_boosters, best_boosters2): assert booster.params == booster2.params def test_get_best_booster_with_error(self) -> None: params: dict = {"verbose": -1} dataset = lgb.Dataset(np.zeros((10, 10))) study = optuna.create_study() tuner = LightGBMTunerCV( params, dataset, study=study, model_dir=None, return_cvbooster=True ) # No trial is completed yet. with pytest.raises(ValueError): tuner.get_best_booster() with mock.patch.object(_OptunaObjectiveCV, "_get_cv_scores", return_value=[1.0]): tuner.tune_regularization_factors() tuner2 = LightGBMTunerCV( params, dataset, study=study, model_dir=None, return_cvbooster=True ) # Resumed the study does not have the best booster. with pytest.raises(ValueError): tuner2.get_best_booster() with TemporaryDirectory() as tmpdir: tuner3 = LightGBMTunerCV( params, dataset, study=study, model_dir=tmpdir, return_cvbooster=True ) # The booster was not saved hence not found in the `model_dir`. with pytest.raises(ValueError): tuner3.get_best_booster() def test_tune_best_score_reproducibility(self) -> None: iris = sklearn.datasets.load_iris() X_trainval, X_test, y_trainval, y_test = train_test_split( iris.data, iris.target, random_state=0 ) train = lgb.Dataset(X_trainval, y_trainval) params = { "objective": "regression", "metric": "rmse", "random_seed": 0, "deterministic": True, "force_col_wise": True, "verbosity": -1, } tuner_first_try = lgb.LightGBMTunerCV( params, train, callbacks=[early_stopping(stopping_rounds=3)], folds=KFold(n_splits=3), optuna_seed=10, ) tuner_first_try.run() best_score_first_try = tuner_first_try.best_score tuner_second_try = lgb.LightGBMTunerCV( params, train, callbacks=[early_stopping(stopping_rounds=3)], folds=KFold(n_splits=3), optuna_seed=10, ) tuner_second_try.run() best_score_second_try = tuner_second_try.best_score assert best_score_second_try == best_score_first_try first_try_trials = tuner_first_try.study.trials second_try_trials = tuner_second_try.study.trials assert len(first_try_trials) == len(second_try_trials) for first_trial, second_trial in zip(first_try_trials, second_try_trials): assert first_trial.value == second_trial.value assert first_trial.params == second_trial.params optuna-3.5.0/tests/integration_tests/test_botorch.py000066400000000000000000000452211453453102400230120ustar00rootroot00000000000000from typing import Any from typing import Optional from typing import Sequence from typing import Tuple from unittest.mock import patch import warnings from packaging import version import pytest import optuna from optuna import integration from optuna._imports import try_import from optuna.integration import BoTorchSampler from optuna.samplers import RandomSampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.storages import RDBStorage from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial import TrialState with try_import() as _imports: import botorch import torch if not _imports.is_successful(): from unittest.mock import MagicMock torch = MagicMock() # NOQA pytestmark = pytest.mark.integration @pytest.mark.parametrize("n_objectives", [1, 2, 4]) def test_botorch_candidates_func_none(n_objectives: int) -> None: if n_objectives == 1 and version.parse(botorch.version.version) < version.parse("0.8.1"): pytest.skip("botorch >=0.8.1 is required for logei_candidates_func.") n_trials = 3 n_startup_trials = 2 sampler = BoTorchSampler(n_startup_trials=n_startup_trials) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials # TODO(hvy): Do not check for the correct candidates function using private APIs. if n_objectives == 1: assert sampler._candidates_func is integration.botorch.logei_candidates_func elif n_objectives == 2: assert sampler._candidates_func is integration.botorch.qehvi_candidates_func elif n_objectives == 4: assert sampler._candidates_func is integration.botorch.ehvi_candidates_func else: assert False, "Should not reach." def test_botorch_candidates_func() -> None: candidates_func_call_count = 0 def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: assert train_con is None candidates = torch.rand(1) nonlocal candidates_func_call_count candidates_func_call_count += 1 return candidates n_trials = 3 n_startup_trials = 1 sampler = BoTorchSampler(candidates_func=candidates_func, n_startup_trials=n_startup_trials) study = optuna.create_study(direction="minimize", sampler=sampler) study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=n_trials) assert len(study.trials) == n_trials assert candidates_func_call_count == n_trials - n_startup_trials @pytest.mark.parametrize( "candidates_func, n_objectives", [ (integration.botorch.ehvi_candidates_func, 4), (integration.botorch.ehvi_candidates_func, 5), # alpha > 0 (integration.botorch.logei_candidates_func, 1), (integration.botorch.qei_candidates_func, 1), (integration.botorch.qnei_candidates_func, 1), (integration.botorch.qehvi_candidates_func, 2), (integration.botorch.qehvi_candidates_func, 7), # alpha > 0 (integration.botorch.qparego_candidates_func, 4), (integration.botorch.qnehvi_candidates_func, 2), (integration.botorch.qnehvi_candidates_func, 6), # alpha > 0 ], ) def test_botorch_specify_candidates_func(candidates_func: Any, n_objectives: int) -> None: if candidates_func == integration.botorch.logei_candidates_func and version.parse( botorch.version.version ) < version.parse("0.8.1"): pytest.skip("LogExpectedImprovement is not available in botorch <0.8.1.") n_trials = 4 n_startup_trials = 2 sampler = BoTorchSampler( candidates_func=candidates_func, n_startup_trials=n_startup_trials, ) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials @pytest.mark.parametrize( "candidates_func, n_objectives", [ (integration.botorch.logei_candidates_func, 1), (integration.botorch.qei_candidates_func, 1), (integration.botorch.qehvi_candidates_func, 2), (integration.botorch.qparego_candidates_func, 4), (integration.botorch.qnehvi_candidates_func, 2), (integration.botorch.qnehvi_candidates_func, 3), # alpha > 0 ], ) def test_botorch_specify_candidates_func_constrained( candidates_func: Any, n_objectives: int ) -> None: n_trials = 4 n_startup_trials = 2 constraints_func_call_count = 0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: xs = sum(trial.params[f"x{i}"] for i in range(n_objectives)) nonlocal constraints_func_call_count constraints_func_call_count += 1 return (xs - 0.5,) sampler = BoTorchSampler( constraints_func=constraints_func, candidates_func=candidates_func, n_startup_trials=n_startup_trials, ) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials assert constraints_func_call_count == n_trials def test_botorch_candidates_func_invalid_batch_size() -> None: def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: return torch.rand(2, 1) # Must have the batch size one, not two. sampler = BoTorchSampler(candidates_func=candidates_func, n_startup_trials=1) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(ValueError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=3) def test_botorch_candidates_func_invalid_dimensionality() -> None: def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: return torch.rand(1, 1, 1) # Must have one or two dimensions, not three. sampler = BoTorchSampler(candidates_func=candidates_func, n_startup_trials=1) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(ValueError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=3) def test_botorch_candidates_func_invalid_candidates_size() -> None: n_params = 3 def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: return torch.rand(n_params - 1) # Must return candidates for all parameters. sampler = BoTorchSampler(candidates_func=candidates_func, n_startup_trials=1) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(ValueError): study.optimize( lambda t: sum(t.suggest_float(f"x{i}", 0, 1) for i in range(n_params)), n_trials=3 ) def test_botorch_constraints_func_invalid_inconsistent_n_constraints() -> None: def constraints_func(trial: FrozenTrial) -> Sequence[float]: x0 = trial.params["x0"] return [x0 - 0.5] * trial.number # Number of constraints may not change. sampler = BoTorchSampler(constraints_func=constraints_func, n_startup_trials=1) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(RuntimeError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=3) def test_botorch_constraints_func_raises() -> None: def constraints_func(trial: FrozenTrial) -> Sequence[float]: if trial.number == 1: raise RuntimeError return (0.0,) sampler = BoTorchSampler(constraints_func=constraints_func) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(RuntimeError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=3) assert len(study.trials) == 2 for trial in study.trials: sys_con = trial.system_attrs[_CONSTRAINTS_KEY] expected_sys_con: Optional[Tuple[int]] if trial.number == 0: expected_sys_con = (0,) elif trial.number == 1: expected_sys_con = None else: assert False, "Should not reach." assert sys_con == expected_sys_con def test_botorch_constraints_func_nan_warning() -> None: def constraints_func(trial: FrozenTrial) -> Sequence[float]: if trial.number == 1: raise RuntimeError return (0.0,) last_trial_number_candidates_func = None def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: trial_number = train_x.size(0) assert train_con is not None if trial_number > 0: assert not train_con[0, :].isnan().any() if trial_number > 1: assert train_con[1, :].isnan().all() if trial_number > 2: assert not train_con[2, :].isnan().any() nonlocal last_trial_number_candidates_func last_trial_number_candidates_func = trial_number return torch.rand(1) sampler = BoTorchSampler( candidates_func=candidates_func, constraints_func=constraints_func, n_startup_trials=1, ) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(RuntimeError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=None) assert len(study.trials) == 2 # Warns when `train_con` contains NaN. with pytest.warns(UserWarning): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=2) assert len(study.trials) == 4 assert last_trial_number_candidates_func == study.trials[-1].number def test_botorch_constraints_func_none_warning() -> None: candidates_func_call_count = 0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: raise RuntimeError def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: # `train_con` should be `None` if `constraints_func` always fails. assert train_con is None nonlocal candidates_func_call_count candidates_func_call_count += 1 return torch.rand(1) sampler = BoTorchSampler( candidates_func=candidates_func, constraints_func=constraints_func, n_startup_trials=1, ) study = optuna.create_study(direction="minimize", sampler=sampler) with pytest.raises(RuntimeError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=None) assert len(study.trials) == 1 # Warns when `train_con` becomes `None`. with pytest.warns(UserWarning), pytest.raises(RuntimeError): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=1) assert len(study.trials) == 2 assert candidates_func_call_count == 1 def test_botorch_constraints_func_late() -> None: def constraints_func(trial: FrozenTrial) -> Sequence[float]: return (0,) last_trial_number_candidates_func = None def candidates_func( train_x: torch.Tensor, train_obj: torch.Tensor, train_con: Optional[torch.Tensor], bounds: torch.Tensor, running_x: Optional[torch.Tensor], ) -> torch.Tensor: trial_number = train_x.size(0) if trial_number < 3: assert train_con is None if trial_number == 3: assert train_con is not None assert train_con[:2, :].isnan().all() assert not train_con[2, :].isnan().any() nonlocal last_trial_number_candidates_func last_trial_number_candidates_func = trial_number return torch.rand(1) sampler = BoTorchSampler( candidates_func=candidates_func, n_startup_trials=1, ) study = optuna.create_study(direction="minimize", sampler=sampler) study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=2) assert len(study.trials) == 2 sampler = BoTorchSampler( candidates_func=candidates_func, constraints_func=constraints_func, n_startup_trials=1, ) study.sampler = sampler # Warns when `train_con` contains NaN. Should not raise but will with NaN for previous trials # that were not computed with constraints. with pytest.warns(UserWarning): study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=2) assert len(study.trials) == 4 assert last_trial_number_candidates_func == study.trials[-1].number def test_botorch_n_startup_trials() -> None: independent_sampler = RandomSampler() sampler = BoTorchSampler(n_startup_trials=2, independent_sampler=independent_sampler) study = optuna.create_study(directions=["minimize", "maximize"], sampler=sampler) with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_independent, patch.object( sampler, "sample_relative", wraps=sampler.sample_relative ) as mock_relative: study.optimize( lambda t: [t.suggest_float("x0", 0, 1), t.suggest_float("x1", 0, 1)], n_trials=3 ) assert mock_independent.call_count == 4 # The objective function has two parameters. assert mock_relative.call_count == 3 def test_botorch_distributions() -> None: def objective(trial: Trial) -> float: x0 = trial.suggest_float("x0", 0, 1) x1 = trial.suggest_float("x1", 0.1, 1, log=True) x2 = trial.suggest_float("x2", 0, 1, step=0.1) x3 = trial.suggest_int("x3", 0, 2) x4 = trial.suggest_int("x4", 2, 4, log=True) x5 = trial.suggest_int("x5", 0, 4, step=2) x6 = trial.suggest_categorical("x6", [0.1, 0.2, 0.3]) return x0 + x1 + x2 + x3 + x4 + x5 + x6 sampler = BoTorchSampler() study = optuna.create_study(direction="minimize", sampler=sampler) study.optimize(objective, n_trials=3) assert len(study.trials) == 3 def test_botorch_invalid_different_studies() -> None: # Using the same sampler with different studies should yield an error since the sampler is # stateful holding the computed constraints. Two studies are considered different if their # IDs differ. # We use the RDB storage since this check does not work for the in-memory storage where all # study IDs are identically 0. storage = RDBStorage("sqlite:///:memory:") sampler = BoTorchSampler() study = optuna.create_study(direction="minimize", sampler=sampler, storage=storage) study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=3) other_study = optuna.create_study(direction="minimize", sampler=sampler, storage=storage) with pytest.raises(RuntimeError): other_study.optimize(lambda t: t.suggest_float("x0", 0, 1), n_trials=3) def test_call_after_trial_of_independent_sampler() -> None: independent_sampler = optuna.samplers.RandomSampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = BoTorchSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) with patch.object( independent_sampler, "after_trial", wraps=independent_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @pytest.mark.parametrize("device", [None, torch.device("cpu"), torch.device("cuda:0")]) def test_device_argument(device: Optional[torch.device]) -> None: sampler = BoTorchSampler(device=device) if not torch.cuda.is_available() and sampler._device.type == "cuda": pytest.skip(reason="GPU is unavailable.") def objective(trial: Trial) -> float: return trial.suggest_float("x", 0.0, 1.0) def constraints_func(trial: FrozenTrial) -> Sequence[float]: x0 = trial.params["x"] return [x0 - 0.5] sampler = BoTorchSampler(constraints_func=constraints_func, n_startup_trials=1) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=3) @pytest.mark.parametrize( "candidates_func, n_objectives", [ (integration.botorch.qei_candidates_func, 1), (integration.botorch.qehvi_candidates_func, 2), (integration.botorch.qparego_candidates_func, 4), (integration.botorch.qnehvi_candidates_func, 2), (integration.botorch.qnehvi_candidates_func, 3), # alpha > 0 ], ) def test_botorch_consider_running_trials(candidates_func: Any, n_objectives: int) -> None: sampler = BoTorchSampler( candidates_func=candidates_func, n_startup_trials=1, consider_running_trials=True, ) def objective(trial: Trial) -> Sequence[float]: ret = [] for i in range(n_objectives): val = sum(trial.suggest_float(f"x{i}_{j}", 0, 1) for j in range(2)) ret.append(val) return ret study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize(objective, n_trials=2) assert len(study.trials) == 2 # fully suggested running trial running_trial_full = study.ask() _ = objective(running_trial_full) study.optimize(objective, n_trials=1) assert len(study.trials) == 4 assert sum(t.state == TrialState.RUNNING for t in study.trials) == 1 assert sum(t.state == TrialState.COMPLETE for t in study.trials) == 3 # partially suggested running trial running_trial_partial = study.ask() for i in range(n_objectives): running_trial_partial.suggest_float(f"x{i}_0", 0, 1) study.optimize(objective, n_trials=1) assert len(study.trials) == 6 assert sum(t.state == TrialState.RUNNING for t in study.trials) == 2 assert sum(t.state == TrialState.COMPLETE for t in study.trials) == 4 # not suggested running trial _ = study.ask() study.optimize(objective, n_trials=1) assert len(study.trials) == 8 assert sum(t.state == TrialState.RUNNING for t in study.trials) == 3 assert sum(t.state == TrialState.COMPLETE for t in study.trials) == 5 optuna-3.5.0/tests/integration_tests/test_catboost.py000066400000000000000000000110201453453102400231560ustar00rootroot00000000000000import types import numpy as np import pytest import optuna from optuna._imports import try_import from optuna.integration.catboost import CatBoostPruningCallback from optuna.testing.pruners import DeterministicPruner with try_import(): import catboost as cb pytestmark = pytest.mark.integration def test_catboost_pruning_callback_call() -> None: # The pruner is deactivated. study = optuna.create_study(pruner=DeterministicPruner(False)) trial = study.ask() pruning_callback = CatBoostPruningCallback(trial, "Logloss") info = types.SimpleNamespace( iteration=1, metrics={"learn": {"Logloss": [1.0]}, "validation": {"Logloss": [1.0]}} ) assert pruning_callback.after_iteration(info) # The pruner is activated. study = optuna.create_study(pruner=DeterministicPruner(True)) trial = study.ask() pruning_callback = CatBoostPruningCallback(trial, "Logloss") info = types.SimpleNamespace( iteration=1, metrics={"learn": {"Logloss": [1.0]}, "validation": {"Logloss": [1.0]}} ) assert not pruning_callback.after_iteration(info) METRICS = ["AUC", "Accuracy"] EVAL_SET_INDEXES = [None, 0, 1] @pytest.mark.parametrize("metric", METRICS) @pytest.mark.parametrize("eval_set_index", EVAL_SET_INDEXES) def test_catboost_pruning_callback_init_param(metric: str, eval_set_index: int) -> None: def objective(trial: optuna.trial.Trial) -> float: train_x = np.asarray([[1.0], [2.0]]) train_y = np.asarray([[1.0], [0.0]]) valid_x = np.asarray([[1.0], [2.0]]) valid_y = np.asarray([[1.0], [0.0]]) if eval_set_index is None: eval_set = [(valid_x, valid_y)] pruning_callback = CatBoostPruningCallback(trial, metric) else: eval_set = [(valid_x, valid_y), (valid_x, valid_y)] pruning_callback = CatBoostPruningCallback(trial, metric, eval_set_index) param = { "objective": "Logloss", "eval_metric": metric, } gbm = cb.CatBoostClassifier(**param) gbm.fit( train_x, train_y, eval_set=eval_set, verbose=0, callbacks=[pruning_callback], ) # Invoke pruning manually. pruning_callback.check_pruned() return 1.0 study = optuna.create_study(pruner=DeterministicPruner(True)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.PRUNED study = optuna.create_study(pruner=DeterministicPruner(False)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 # TODO(Hemmi): Remove the skip decorator after CatBoost's error handling is fixed. # See https://github.com/optuna/optuna/pull/4190 for more details. @pytest.mark.skip(reason="Temporally skip due to unknown CatBoost error.") @pytest.mark.parametrize( "metric, eval_set_index", [ ("foo_metric", None), ("AUC", 100), ], ) def test_catboost_pruning_callback_errors(metric: str, eval_set_index: int) -> None: # This test aims to cover the ValueError block in CatBoostPruningCallback.after_iteration(). # However, catboost currently terminates with a SystemError when python>=3.9 or pytest>=7.2.0, # otherwise terminates with RecursionError. This is because after_iteration() is called in a # Cython function in the catboost library, which is causing the unexpected error behavior. # Note that the difference in error type is mainly because the _Py_CheckRecursionLimit # variable used in limited C API was removed after python 3.9. def objective(trial: optuna.trial.Trial) -> float: train_x = np.asarray([[1.0], [2.0]]) train_y = np.asarray([[1.0], [0.0]]) valid_x = np.asarray([[1.0], [2.0]]) valid_y = np.asarray([[1.0], [0.0]]) pruning_callback = CatBoostPruningCallback(trial, metric, eval_set_index) param = { "objective": "Logloss", "eval_metric": "AUC", } gbm = cb.CatBoostClassifier(**param) gbm.fit( train_x, train_y, eval_set=[(valid_x, valid_y)], verbose=0, callbacks=[pruning_callback], ) # Invoke pruning manually. pruning_callback.check_pruned() return 1.0 # Unknown validation name or metric. study = optuna.create_study(pruner=DeterministicPruner(False)) with pytest.raises(ValueError): study.optimize(objective, n_trials=1) optuna-3.5.0/tests/integration_tests/test_cma.py000066400000000000000000000325561453453102400221210ustar00rootroot00000000000000import math from typing import Any from typing import Dict from unittest.mock import call from unittest.mock import patch import warnings import _pytest.capture import pytest import optuna from optuna._imports import try_import from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.integration.cma import _Optimizer from optuna.study._study_direction import StudyDirection from optuna.testing.distributions import UnsupportedDistribution from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial import TrialState with try_import(): import cma pytestmark = pytest.mark.integration def test_cmaes_deprecation_warning() -> None: with pytest.warns(FutureWarning): optuna.integration.CmaEsSampler() class TestPyCmaSampler: @staticmethod def test_init_cma_opts() -> None: sampler = optuna.integration.PyCmaSampler( x0={"x": 0, "y": 0}, sigma0=0.1, cma_stds={"x": 1, "y": 1}, seed=1, cma_opts={"popsize": 5}, ) study = optuna.create_study(sampler=sampler) with patch("optuna.integration.cma._Optimizer") as mock_obj: mock_obj.ask.return_value = {"x": -1, "y": -1} study.optimize( lambda t: t.suggest_int("x", -1, 1) + t.suggest_int("y", -1, 1), n_trials=2 ) assert mock_obj.mock_calls[0] == call( { "x": IntDistribution(low=-1, high=1), "y": IntDistribution(low=-1, high=1), }, {"x": 0, "y": 0}, 0.1, {"x": 1, "y": 1}, {"popsize": 5, "seed": 1, "verbose": -2}, ) @staticmethod def test_init_default_values() -> None: sampler = optuna.integration.PyCmaSampler() seed = sampler._cma_opts.get("seed") assert isinstance(seed, int) assert 0 < seed assert isinstance(sampler._independent_sampler, optuna.samplers.RandomSampler) @staticmethod def test_warn_independent_sampling(capsys: _pytest.capture.CaptureFixture) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", ["a", "b"]) if x == "a": return trial.suggest_float("y", 0, 1) else: return trial.suggest_float("z", 0, 1) # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = optuna.integration.PyCmaSampler( warn_independent_sampling=True, n_startup_trials=0 ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert err @staticmethod def test_infer_relative_search_space_1d() -> None: sampler = optuna.integration.PyCmaSampler() study = optuna.create_study(sampler=sampler) # The distribution has only one candidate. study.optimize(lambda t: t.suggest_int("x", 1, 1), n_trials=1) assert sampler.infer_relative_search_space(study, study.best_trial) == {} @staticmethod def test_sample_relative_1d() -> None: independent_sampler = optuna.samplers.RandomSampler() sampler = optuna.integration.PyCmaSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) # If search space is one dimensional, the independent sampler is always used. with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_object: study.optimize(lambda t: t.suggest_int("x", -1, 1), n_trials=2) assert mock_object.call_count == 2 @staticmethod def test_sample_relative_n_startup_trials() -> None: independent_sampler = optuna.samplers.RandomSampler() sampler = optuna.integration.PyCmaSampler( n_startup_trials=2, independent_sampler=independent_sampler ) study = optuna.create_study(sampler=sampler) # The independent sampler is used for Trial#0 and Trial#1. # The CMA-ES is used for Trial#2. with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_independent, patch.object( sampler, "sample_relative", wraps=sampler.sample_relative ) as mock_relative: study.optimize( lambda t: t.suggest_int("x", -1, 1) + t.suggest_int("y", -1, 1), n_trials=3 ) assert mock_independent.call_count == 4 # The objective function has two parameters. assert mock_relative.call_count == 3 @staticmethod def test_initialize_x0_with_unsupported_distribution() -> None: with pytest.raises(NotImplementedError): optuna.integration.PyCmaSampler._initialize_x0({"x": UnsupportedDistribution()}) @staticmethod def test_initialize_sigma0_with_unsupported_distribution() -> None: with pytest.raises(NotImplementedError): optuna.integration.PyCmaSampler._initialize_sigma0({"x": UnsupportedDistribution()}) @staticmethod def test_call_after_trial_of_independent_sampler() -> None: independent_sampler = optuna.samplers.RandomSampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = optuna.integration.PyCmaSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) with patch.object( independent_sampler, "after_trial", wraps=independent_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 class TestOptimizer: @staticmethod @pytest.fixture def search_space() -> Dict[str, BaseDistribution]: return { "c": CategoricalDistribution(("a", "b")), "d": FloatDistribution(-1, 9, step=2), "i": IntDistribution(-1, 1), "ii": IntDistribution(-1, 3, step=2), "il": IntDistribution(2, 16, log=True), "l": FloatDistribution(0.001, 0.1, log=True), "u": FloatDistribution(-2, 2), } @staticmethod @pytest.fixture def x0() -> Dict[str, Any]: return { "c": "a", "d": -1, "i": -1, "ii": -1, "il": 2, "l": 0.001, "u": -2, } @staticmethod def test_init(search_space: Dict[str, BaseDistribution], x0: Dict[str, Any]) -> None: # TODO(c-bata): Avoid exact assertion checks eps = 1e-10 with patch("cma.CMAEvolutionStrategy") as mock_obj: optuna.integration.cma._Optimizer( search_space, x0, 0.2, None, {"popsize": 5, "seed": 1} ) assert mock_obj.mock_calls[0] == call( [0, 0, -1, -1, math.log(2), math.log(0.001), -2], 0.2, { "BoundaryHandler": cma.BoundTransform, "bounds": [ [-0.5, -1.0, -1.5, -2.0, math.log(1.5), math.log(0.001), -2], [1.5, 11.0, 1.5, 4.0, math.log(16.5), math.log(0.1) - eps, 2 - eps], ], "popsize": 5, "seed": 1, }, ) @staticmethod def test_init_with_unsupported_distribution() -> None: with pytest.raises(NotImplementedError): optuna.integration.cma._Optimizer( {"x": UnsupportedDistribution()}, {"x": 0}, 0.2, None, {} ) @staticmethod @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_tell( search_space: Dict[str, BaseDistribution], x0: Dict[str, Any], direction: StudyDirection ) -> None: optimizer = optuna.integration.cma._Optimizer( search_space, x0, 0.2, None, {"popsize": 3, "seed": 1} ) trials = [_create_frozen_trial(x0, search_space)] assert -1 == optimizer.tell(trials, direction) trials = [_create_frozen_trial(x0, search_space, number=i) for i in range(3)] assert 2 == optimizer.tell(trials, direction) @staticmethod @pytest.mark.parametrize("state", [TrialState.FAIL, TrialState.RUNNING, TrialState.PRUNED]) def test_tell_filter_by_state( search_space: Dict[str, BaseDistribution], x0: Dict[str, Any], state: TrialState ) -> None: optimizer = optuna.integration.cma._Optimizer( search_space, x0, 0.2, None, {"popsize": 2, "seed": 1} ) trials = [_create_frozen_trial(x0, search_space)] trials.append(_create_frozen_trial(x0, search_space, state, len(trials))) assert -1 == optimizer.tell(trials, StudyDirection.MINIMIZE) @staticmethod def test_tell_filter_by_distribution( search_space: Dict[str, BaseDistribution], x0: Dict[str, Any] ) -> None: optimizer = optuna.integration.cma._Optimizer( search_space, x0, 0.2, None, {"popsize": 2, "seed": 1} ) trials = [_create_frozen_trial(x0, search_space)] distributions = trials[0].distributions.copy() distributions["additional"] = FloatDistribution(0, 100) trials.append(_create_frozen_trial(x0, distributions, number=1)) assert 1 == optimizer.tell(trials, StudyDirection.MINIMIZE) @staticmethod def test_ask(search_space: Dict[str, BaseDistribution], x0: Dict[str, Any]) -> None: trials = [_create_frozen_trial(x0, search_space, number=i) for i in range(3)] # Create 0-th individual. optimizer = _Optimizer(search_space, x0, 0.2, None, {"popsize": 3, "seed": 1}) last_told = optimizer.tell(trials, StudyDirection.MINIMIZE) params0 = optimizer.ask(trials, last_told) # Ignore parameters with incompatible distributions and create new individual. optimizer = _Optimizer(search_space, x0, 0.2, None, {"popsize": 3, "seed": 1}) last_told = optimizer.tell(trials, StudyDirection.MINIMIZE) distributions = trials[0].distributions.copy() distributions["additional"] = FloatDistribution(0, 100) trials.append(_create_frozen_trial(x0, distributions, number=len(trials))) params1 = optimizer.ask(trials, last_told) assert params0 != params1 assert "additional" not in params1 # Create first individual. optimizer = _Optimizer(search_space, x0, 0.2, None, {"popsize": 3, "seed": 1}) trials.append(_create_frozen_trial(x0, search_space, number=len(trials))) last_told = optimizer.tell(trials, StudyDirection.MINIMIZE) params2 = optimizer.ask(trials, last_told) assert params0 != params2 optimizer = _Optimizer(search_space, x0, 0.2, None, {"popsize": 3, "seed": 1}) last_told = optimizer.tell(trials, StudyDirection.MINIMIZE) # Other worker adds three trials. for _ in range(3): trials.append(_create_frozen_trial(x0, search_space, number=len(trials))) params3 = optimizer.ask(trials, last_told) assert params0 != params3 assert params2 != params3 @staticmethod def test_is_compatible(search_space: Dict[str, BaseDistribution], x0: Dict[str, Any]) -> None: optimizer = optuna.integration.cma._Optimizer(search_space, x0, 0.1, None, {}) # Compatible. trial = _create_frozen_trial(x0, search_space) assert optimizer._is_compatible(trial) # Compatible. trial = _create_frozen_trial(x0, dict(search_space, u=FloatDistribution(-10, 10))) assert optimizer._is_compatible(trial) # Compatible. trial = _create_frozen_trial( dict(x0, unknown=7), dict(search_space, unknown=FloatDistribution(0, 10)) ) assert optimizer._is_compatible(trial) # Incompatible ('u' doesn't exist). param = dict(x0) del param["u"] dist = dict(search_space) del dist["u"] trial = _create_frozen_trial(param, dist) assert not optimizer._is_compatible(trial) # Incompatible (the value of 'u' is out of range). trial = _create_frozen_trial( dict(x0, u=20), dict(search_space, u=FloatDistribution(-100, 100)) ) assert not optimizer._is_compatible(trial) # Error (different distribution class). trial = _create_frozen_trial(x0, dict(search_space, u=IntDistribution(-2, 2))) with pytest.raises(ValueError): optimizer._is_compatible(trial) def _create_frozen_trial( params: Dict[str, Any], param_distributions: Dict[str, BaseDistribution], state: TrialState = TrialState.COMPLETE, number: int = 0, ) -> FrozenTrial: return FrozenTrial( number=number, value=1.0, state=state, user_attrs={}, system_attrs={}, params=params, distributions=param_distributions, intermediate_values={}, datetime_start=None, datetime_complete=None, trial_id=number, ) optuna-3.5.0/tests/integration_tests/test_dask.py000066400000000000000000000112351453453102400222720ustar00rootroot00000000000000from contextlib import contextmanager import time from typing import Iterator import numpy as np import pytest import optuna from optuna._imports import try_import from optuna.integration.dask import _OptunaSchedulerExtension from optuna.integration.dask import DaskStorage from optuna.testing.tempfile_pool import NamedTemporaryFilePool from optuna.trial import Trial with try_import() as _imports: from distributed import Client from distributed import Scheduler from distributed import wait from distributed import Worker from distributed.utils_test import clean from distributed.utils_test import gen_cluster pytestmark = pytest.mark.integration STORAGE_MODES = ["inmemory", "sqlite"] @contextmanager def get_storage_url(specifier: str) -> Iterator: tmpfile = None try: if specifier == "inmemory": url = None elif specifier == "sqlite": tmpfile = NamedTemporaryFilePool().tempfile() url = "sqlite:///{}".format(tmpfile.name) else: raise ValueError( "Invalid specifier entered. Was expecting 'inmemory' or 'sqlite'" f"but got {specifier} instead" ) yield url finally: if tmpfile is not None: tmpfile.close() def objective(trial: Trial) -> float: x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 def objective_slow(trial: Trial) -> float: time.sleep(2) return objective(trial) @pytest.fixture def client() -> "Client": # type: ignore[misc] with clean(): with Client(dashboard_address=":0") as client: # type: ignore[no-untyped-call] yield client def test_experimental(client: "Client") -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): DaskStorage() def test_no_client_informative_error() -> None: with pytest.raises(ValueError, match="No global client found"): DaskStorage() def test_name_unique(client: "Client") -> None: s1 = DaskStorage() s2 = DaskStorage() assert s1.name != s2.name @pytest.mark.parametrize("storage_specifier", STORAGE_MODES) def test_study_optimize(client: "Client", storage_specifier: str) -> None: with get_storage_url(storage_specifier) as url: storage = DaskStorage(storage=url) study = optuna.create_study(storage=storage) assert not study.trials futures = [ client.submit( # type: ignore[no-untyped-call] study.optimize, objective, n_trials=1, pure=False ) for _ in range(10) ] wait(futures) # type: ignore[no-untyped-call] assert len(study.trials) == 10 @pytest.mark.parametrize("storage_specifier", STORAGE_MODES) def test_get_base_storage(client: "Client", storage_specifier: str) -> None: with get_storage_url(storage_specifier) as url: dask_storage = DaskStorage(url) storage = dask_storage.get_base_storage() expected_type = type(optuna.storages.get_storage(url)) assert type(storage) is expected_type @pytest.mark.parametrize("direction", ["maximize", "minimize"]) def test_study_direction_best_value(client: "Client", direction: str) -> None: # Regression test for https://github.com/jrbourbeau/dask-optuna/issues/15 pytest.importorskip("pandas") storage = DaskStorage() study = optuna.create_study(storage=storage, direction=direction) f = client.submit(study.optimize, objective, n_trials=10) # type: ignore[no-untyped-call] wait(f) # type: ignore[no-untyped-call] # Ensure that study.best_value matches up with the expected value from # the trials DataFrame trials_value = study.trials_dataframe()["value"] if direction == "maximize": expected = trials_value.max() else: expected = trials_value.min() np.testing.assert_allclose(expected, study.best_value) if _imports.is_successful(): @gen_cluster(client=True) async def test_daskstorage_registers_extension( c: "Client", s: "Scheduler", a: "Worker", b: "Worker" ) -> None: assert "optuna" not in s.extensions await DaskStorage() assert "optuna" in s.extensions assert type(s.extensions["optuna"]) is _OptunaSchedulerExtension @gen_cluster(client=True) async def test_name(c: "Client", s: "Scheduler", a: "Worker", b: "Worker") -> None: await DaskStorage(name="foo") ext = s.extensions["optuna"] assert len(ext.storages) == 1 assert type(ext.storages["foo"]) is optuna.storages.InMemoryStorage await DaskStorage(name="bar") assert len(ext.storages) == 2 assert type(ext.storages["bar"]) is optuna.storages.InMemoryStorage optuna-3.5.0/tests/integration_tests/test_integration.py000066400000000000000000000017201453453102400236710ustar00rootroot00000000000000import pytest pytestmark = pytest.mark.integration def test_import() -> None: from optuna.integration import dask # NOQA from optuna.integration import DaskStorage # NOQA from optuna.integration import lightgbm # NOQA from optuna.integration import LightGBMPruningCallback # NOQA from optuna.integration import xgboost # NOQA from optuna.integration import XGBoostPruningCallback # NOQA with pytest.raises(ImportError): from optuna.integration import unknown_module # type: ignore # NOQA def test_module_attributes() -> None: import optuna assert hasattr(optuna.integration, "dask") assert hasattr(optuna.integration, "lightgbm") assert hasattr(optuna.integration, "xgboost") assert hasattr(optuna.integration, "LightGBMPruningCallback") assert hasattr(optuna.integration, "XGBoostPruningCallback") with pytest.raises(AttributeError): optuna.integration.unknown_attribute # type: ignore optuna-3.5.0/tests/integration_tests/test_lightgbm.py000066400000000000000000000150171453453102400231470ustar00rootroot00000000000000from functools import partial from unittest.mock import patch import numpy as np import pytest import optuna from optuna._imports import try_import from optuna.integration.lightgbm import LightGBMPruningCallback from optuna.testing.pruners import DeterministicPruner with try_import(): import lightgbm as lgb pytestmark = pytest.mark.integration # If `True`, `lgb.cv(..)` will be used in the test, otherwise `lgb.train(..)` will be used. CV_FLAGS = [False, True] @pytest.mark.parametrize("cv", CV_FLAGS) def test_lightgbm_pruning_callback_call(cv: bool) -> None: callback_env = partial( lgb.callback.CallbackEnv, model="test", params={}, begin_iteration=0, end_iteration=1, iteration=1, ) if cv: env = callback_env(evaluation_result_list=[(("cv_agg", "binary_error", 1.0, False, 1.0))]) else: env = callback_env(evaluation_result_list=[("validation", "binary_error", 1.0, False)]) # The pruner is deactivated. study = optuna.create_study(pruner=DeterministicPruner(False)) trial = study.ask() pruning_callback = LightGBMPruningCallback(trial, "binary_error", valid_name="validation") pruning_callback(env) # The pruner is activated. study = optuna.create_study(pruner=DeterministicPruner(True)) trial = study.ask() pruning_callback = LightGBMPruningCallback(trial, "binary_error", valid_name="validation") with pytest.raises(optuna.TrialPruned): pruning_callback(env) @pytest.mark.parametrize("cv", CV_FLAGS) def test_lightgbm_pruning_callback(cv: bool) -> None: study = optuna.create_study(pruner=DeterministicPruner(True)) study.optimize(partial(objective, cv=cv), n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.PRUNED study = optuna.create_study(pruner=DeterministicPruner(False)) study.optimize(partial(objective, cv=cv), n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 # Use non default validation name. custom_valid_name = "my_validation" study = optuna.create_study(pruner=DeterministicPruner(False)) study.optimize(lambda trial: objective(trial, valid_name=custom_valid_name, cv=cv), n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 # Check "maximize" direction. study = optuna.create_study(pruner=DeterministicPruner(True), direction="maximize") study.optimize(lambda trial: objective(trial, metric="auc", cv=cv), n_trials=1, catch=()) assert study.trials[0].state == optuna.trial.TrialState.PRUNED study = optuna.create_study(pruner=DeterministicPruner(False), direction="maximize") study.optimize(lambda trial: objective(trial, metric="auc", cv=cv), n_trials=1, catch=()) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 @pytest.mark.parametrize( "cv, interval, num_boost_round", [ (True, 1, 1), (True, 2, 1), (True, 2, 2), (False, 1, 1), (False, 2, 1), (False, 2, 2), ], ) def test_lightgbm_pruning_callback_with_interval( cv: bool, interval: int, num_boost_round: int ) -> None: study = optuna.create_study(pruner=DeterministicPruner(False)) with patch("optuna.trial.Trial.report") as mock: study.optimize( partial(objective, cv=cv, interval=interval, num_boost_round=num_boost_round), n_trials=1, ) if interval <= num_boost_round: assert mock.call_count == 1 else: assert mock.call_count == 0 assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 study = optuna.create_study(pruner=DeterministicPruner(True)) study.optimize( partial(objective, cv=cv, interval=interval, num_boost_round=num_boost_round), n_trials=1 ) if interval > num_boost_round: assert study.trials[0].state == optuna.trial.TrialState.COMPLETE else: assert study.trials[0].state == optuna.trial.TrialState.PRUNED @pytest.mark.parametrize("cv", CV_FLAGS) def test_lightgbm_pruning_callback_errors(cv: bool) -> None: # Unknown metric. study = optuna.create_study(pruner=DeterministicPruner(False)) with pytest.raises(ValueError): study.optimize( lambda trial: objective(trial, metric="foo_metric", cv=cv), n_trials=1, catch=() ) if not cv: # Unknown validation name. study = optuna.create_study(pruner=DeterministicPruner(False)) with pytest.raises(ValueError): study.optimize( lambda trial: objective( trial, valid_name="valid_1", force_default_valid_names=True ), n_trials=1, catch=(), ) # Check consistency of study direction. study = optuna.create_study(pruner=DeterministicPruner(False)) with pytest.raises(ValueError): study.optimize(lambda trial: objective(trial, metric="auc", cv=cv), n_trials=1, catch=()) study = optuna.create_study(pruner=DeterministicPruner(False), direction="maximize") with pytest.raises(ValueError): study.optimize( lambda trial: objective(trial, metric="binary_error", cv=cv), n_trials=1, catch=() ) def objective( trial: optuna.trial.Trial, metric: str = "binary_error", valid_name: str = "valid_0", interval: int = 1, num_boost_round: int = 1, force_default_valid_names: bool = False, cv: bool = False, ) -> float: dtrain = lgb.Dataset(np.asarray([[1.0], [2.0], [3.0], [4.0]]), label=[1.0, 0.0, 1.0, 0.0]) dtest = lgb.Dataset(np.asarray([[1.0]]), label=[1.0]) if force_default_valid_names: valid_names = None else: valid_names = [valid_name] verbose_callback = lgb.log_evaluation() pruning_callback = LightGBMPruningCallback( trial, metric, valid_name=valid_name, report_interval=interval ) if cv: lgb.cv( {"objective": "binary", "metric": ["auc", "binary_error"]}, dtrain, num_boost_round, nfold=2, callbacks=[verbose_callback, pruning_callback], ) else: lgb.train( {"objective": "binary", "metric": ["auc", "binary_error"]}, dtrain, num_boost_round, valid_sets=[dtest], valid_names=valid_names, callbacks=[verbose_callback, pruning_callback], ) return 1.0 optuna-3.5.0/tests/integration_tests/test_mlflow.py000066400000000000000000000441171453453102400226550ustar00rootroot00000000000000from typing import Callable from typing import List from typing import Optional from typing import Tuple from typing import Union import numpy as np import py import pytest import optuna from optuna._imports import try_import from optuna.integration.mlflow import MLflowCallback with try_import(): import mlflow from mlflow.tracking import MlflowClient from mlflow.utils.mlflow_tags import MLFLOW_PARENT_RUN_ID pytestmark = pytest.mark.integration def _objective_func(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_float("y", 20, 30, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) trial.set_user_attr("my_user_attr", "my_user_attr_value") return (x - 2) ** 2 + (y - 25) ** 2 + z def _multiobjective_func(trial: optuna.trial.Trial) -> Tuple[float, float]: x = trial.suggest_float("x", low=-1.0, high=1.0) y = trial.suggest_float("y", low=20, high=30, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) first_objective = (x - 2) ** 2 + (y - 25) ** 2 + z second_objective = (x - 2) ** 3 + (y - 25) ** 3 - z return first_objective, second_objective # This is tool function for a temporary fix on Optuna side. It avoids an error with user # attributes that are too long. It should be fixed on MLflow side later. # When it is fixed on MLflow side this test can be removed. # see https://github.com/optuna/optuna/issues/1340 # see https://github.com/mlflow/mlflow/issues/2931 def _objective_func_long_user_attr(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_float("y", 20, 30, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) long_str = str(list(range(5000))) trial.set_user_attr("my_user_attr", long_str) return (x - 2) ** 2 + (y - 25) ** 2 + z @pytest.mark.parametrize("name,expected", [(None, "Default"), ("foo", "foo")]) def test_use_existing_or_default_experiment( tmpdir: py.path.local, name: Optional[str], expected: str ) -> None: if name is not None: tracking_uri = f"file:{tmpdir}" mlflow.set_tracking_uri(tracking_uri) mlflow.set_experiment(name) else: # Target directory can't exist when initializing first # run with default experiment at non-default uri. tracking_uri = f"file:{tmpdir}/foo" mlflow.set_tracking_uri(tracking_uri) mlflc = MLflowCallback(tracking_uri=tracking_uri, create_experiment=False) study = optuna.create_study() for _ in range(10): # Simulate multiple optimization runs under same experiment. study.optimize(_objective_func, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiment = mlfl_client.search_experiments()[0] runs = mlfl_client.search_runs(experiment.experiment_id) assert experiment.name == expected assert len(runs) == 10 def test_study_name(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" n_trials = 3 mlflc = MLflowCallback(tracking_uri=tracking_uri) study = optuna.create_study(study_name=study_name) study.optimize(_objective_func, n_trials=n_trials, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) assert len(mlfl_client.search_experiments()) == 1 experiment = mlfl_client.search_experiments()[0] runs = mlfl_client.search_runs(experiment.experiment_id) assert experiment.name == study_name assert len(runs) == n_trials def test_use_existing_experiment_by_id(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" mlflow.set_tracking_uri(tracking_uri) experiment_id = mlflow.create_experiment("foo") mlflow_kwargs = {"experiment_id": experiment_id} mlflc = MLflowCallback( tracking_uri=tracking_uri, create_experiment=False, mlflow_kwargs=mlflow_kwargs ) study = optuna.create_study() for _ in range(10): study.optimize(_objective_func, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiment_list = mlfl_client.search_experiments() assert len(experiment_list) == 1 experiment = experiment_list[0] assert experiment.experiment_id == experiment_id assert experiment.name == "foo" runs = mlfl_client.search_runs(experiment_id) assert len(runs) == 10 def test_metric_name(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" metric_name = "my_metric_name" mlflc = MLflowCallback(tracking_uri=tracking_uri, metric_name=metric_name) study = optuna.create_study(study_name="my_study") study.optimize(_objective_func, n_trials=3, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() experiment = experiments[0] experiment_id = experiment.experiment_id first_run = mlfl_client.search_runs(experiment_id)[0] first_run_dict = first_run.to_dictionary() assert metric_name in first_run_dict["data"]["metrics"] @pytest.mark.parametrize( "names,expected", [ ("foo", ["foo_0", "foo_1"]), (["foo", "bar"], ["foo", "bar"]), (("foo", "bar"), ["foo", "bar"]), ], ) def test_metric_name_multiobjective( tmpdir: py.path.local, names: Union[str, List[str]], expected: List[str] ) -> None: tracking_uri = f"file:{tmpdir}" mlflc = MLflowCallback(tracking_uri=tracking_uri, metric_name=names) study = optuna.create_study(study_name="my_study", directions=["minimize", "maximize"]) study.optimize(_multiobjective_func, n_trials=3, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() experiment = experiments[0] experiment_id = experiment.experiment_id first_run = mlfl_client.search_runs(experiment_id)[0] first_run_dict = first_run.to_dictionary() assert all([e in first_run_dict["data"]["metrics"] for e in expected]) @pytest.mark.parametrize("run_name,expected", [(None, "0"), ("foo", "foo")]) def test_run_name(tmpdir: py.path.local, run_name: Optional[str], expected: str) -> None: tracking_uri = f"file:{tmpdir}" mlflow_kwargs = {"run_name": run_name} mlflc = MLflowCallback(tracking_uri=tracking_uri, mlflow_kwargs=mlflow_kwargs) study = optuna.create_study() study.optimize(_objective_func, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiment = mlfl_client.search_experiments()[0] run = mlfl_client.search_runs(experiment.experiment_id)[0] tags = run.data.tags assert tags["mlflow.runName"] == expected # This is a test for a temporary fix on Optuna side. It avoids an error with user # attributes that are too long. It should be fixed on MLflow side later. # When it is fixed on MLflow side this test can be removed. # see https://github.com/optuna/optuna/issues/1340 # see https://github.com/mlflow/mlflow/issues/2931 def test_tag_truncation(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" n_trials = 3 mlflc = MLflowCallback(tracking_uri=tracking_uri) study = optuna.create_study(study_name=study_name) study.optimize(_objective_func_long_user_attr, n_trials=n_trials, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() assert len(experiments) == 1 experiment = experiments[0] assert experiment.name == study_name experiment_id = experiment.experiment_id runs = mlfl_client.search_runs(experiment_id) assert len(runs) == n_trials first_run = runs[0] first_run_dict = first_run.to_dictionary() my_user_attr = first_run_dict["data"]["tags"]["my_user_attr"] assert len(my_user_attr) <= 5000 def test_nest_trials(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" mlflow.set_tracking_uri(tracking_uri) mlflow.set_experiment(study_name) mlflc = MLflowCallback(tracking_uri=tracking_uri, mlflow_kwargs={"nested": True}) study = optuna.create_study(study_name=study_name) n_trials = 3 with mlflow.start_run() as parent_run: study.optimize(_objective_func, n_trials=n_trials, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() experiment_id = experiments[0].experiment_id all_runs = mlfl_client.search_runs([experiment_id]) child_runs = [r for r in all_runs if MLFLOW_PARENT_RUN_ID in r.data.tags] assert len(all_runs) == n_trials + 1 assert len(child_runs) == n_trials assert all(r.data.tags[MLFLOW_PARENT_RUN_ID] == parent_run.info.run_id for r in child_runs) assert all(set(r.data.params.keys()) == {"x", "y", "z"} for r in child_runs) assert all(set(r.data.metrics.keys()) == {"value"} for r in child_runs) @pytest.mark.parametrize("n_jobs", [2, 4]) def test_multiple_jobs(tmpdir: py.path.local, n_jobs: int) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" # The race-condition usually happens after first trial for each job. n_trials = n_jobs * 2 mlflc = MLflowCallback(tracking_uri=tracking_uri) study = optuna.create_study(study_name=study_name) study.optimize(_objective_func, n_trials=n_trials, callbacks=[mlflc], n_jobs=n_jobs) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() assert len(experiments) == 1 experiment_id = experiments[0].experiment_id runs = mlfl_client.search_runs([experiment_id]) assert len(runs) == n_trials def test_mlflow_callback_fails_when_nest_trials_is_false_and_active_run_exists( tmpdir: py.path.local, ) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" mlflow.set_tracking_uri(tracking_uri) mlflow.set_experiment(study_name) mlflc = MLflowCallback(tracking_uri=tracking_uri) study = optuna.create_study(study_name=study_name) with mlflow.start_run(): with pytest.raises(Exception, match=r"Run with UUID \w+ is already active."): study.optimize(_objective_func, n_trials=1, callbacks=[mlflc]) def test_tag_always_logged(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" n_trials = 3 mlflc = MLflowCallback(tracking_uri=tracking_uri) study = optuna.create_study(study_name=study_name) study.optimize(_objective_func, n_trials=n_trials, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiment = mlfl_client.search_experiments()[0] runs = mlfl_client.search_runs([experiment.experiment_id]) assert all((r.data.tags["direction"] == "MINIMIZE") for r in runs) assert all((r.data.tags["state"] == "COMPLETE") for r in runs) @pytest.mark.parametrize("tag_study_user_attrs", [True, False]) def test_tag_study_user_attrs(tmpdir: py.path.local, tag_study_user_attrs: bool) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" n_trials = 3 mlflc = MLflowCallback(tracking_uri=tracking_uri, tag_study_user_attrs=tag_study_user_attrs) study = optuna.create_study(study_name=study_name) study.set_user_attr("my_study_attr", "a") study.optimize(_objective_func_long_user_attr, n_trials=n_trials, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() assert len(experiments) == 1 experiment = experiments[0] assert experiment.name == study_name experiment_id = experiment.experiment_id runs = mlfl_client.search_runs([experiment_id]) assert len(runs) == n_trials if tag_study_user_attrs: assert all((r.data.tags["my_study_attr"] == "a") for r in runs) else: assert all(("my_study_attr" not in r.data.tags) for r in runs) @pytest.mark.parametrize("tag_trial_user_attrs", [True, False]) def test_tag_trial_user_attrs(tmpdir: py.path.local, tag_trial_user_attrs: bool) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" n_trials = 3 mlflc = MLflowCallback(tracking_uri=tracking_uri, tag_trial_user_attrs=tag_trial_user_attrs) study = optuna.create_study(study_name=study_name) study.optimize(_objective_func, n_trials=n_trials, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiment = mlfl_client.search_experiments()[0] runs = mlfl_client.search_runs([experiment.experiment_id]) if tag_trial_user_attrs: assert all((r.data.tags["my_user_attr"] == "my_user_attr_value") for r in runs) else: assert all(("my_user_attr" not in r.data.tags) for r in runs) def test_log_mlflow_tags(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" expected_tags = {"foo": 0, "bar": 1} mlflow_kwargs = {"tags": expected_tags} mlflc = MLflowCallback(tracking_uri=tracking_uri, mlflow_kwargs=mlflow_kwargs) study = optuna.create_study() study.optimize(_objective_func, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiment = mlfl_client.search_experiments()[0] run = mlfl_client.search_runs(experiment.experiment_id)[0] tags = run.data.tags assert all([k in tags.keys() for k in expected_tags.keys()]) assert all([tags[key] == str(value) for key, value in expected_tags.items()]) @pytest.mark.parametrize("n_jobs", [1, 2, 4]) def test_track_in_mlflow_decorator(tmpdir: py.path.local, n_jobs: int) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" n_trials = n_jobs * 2 metric_name = "additional_metric" metric = 3.14 mlflc = MLflowCallback(tracking_uri=tracking_uri) def _objective_func(trial: optuna.trial.Trial) -> float: """Objective function""" x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_float("y", 20, 30, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) trial.set_user_attr("my_user_attr", "my_user_attr_value") mlflow.log_metric(metric_name, metric) return (x - 2) ** 2 + (y - 25) ** 2 + z tracked_objective = mlflc.track_in_mlflow()(_objective_func) study = optuna.create_study(study_name=study_name) study.optimize(tracked_objective, n_trials=n_trials, callbacks=[mlflc], n_jobs=n_jobs) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() assert len(experiments) == 1 experiment = experiments[0] assert experiment.name == study_name experiment_id = experiment.experiment_id runs = mlfl_client.search_runs(experiment_id) assert len(runs) == n_trials first_run = runs[0] first_run_dict = first_run.to_dictionary() assert metric_name in first_run_dict["data"]["metrics"] assert first_run_dict["data"]["metrics"][metric_name] == metric assert tracked_objective.__name__ == _objective_func.__name__ assert tracked_objective.__doc__ == _objective_func.__doc__ @pytest.mark.parametrize( "func,names,values", [ (_objective_func, ["metric"], [27.0]), (_multiobjective_func, ["metric1", "metric2"], [27.0, -127.0]), ], ) def test_log_metric( tmpdir: py.path.local, func: Callable, names: List[str], values: List[float] ) -> None: tracking_uri = f"file:{tmpdir}" study_name = "my_study" mlflc = MLflowCallback(tracking_uri=tracking_uri, metric_name=names) study = optuna.create_study( study_name=study_name, directions=["minimize" for _ in range(len(values))] ) study.enqueue_trial({"x": 1.0, "y": 20.0, "z": 1.0}) study.optimize(func, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() experiment = experiments[0] experiment_id = experiment.experiment_id runs = mlfl_client.search_runs(experiment_id) assert len(runs) == 1 run = runs[0] run_dict = run.to_dictionary() assert all(name in run_dict["data"]["metrics"] for name in names) assert all([run_dict["data"]["metrics"][name] == val for name, val in zip(names, values)]) def test_log_metric_none(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" metric_name = "metric" study_name = "my_study" mlflc = MLflowCallback(tracking_uri=tracking_uri, metric_name=metric_name) study = optuna.create_study(study_name=study_name) study.optimize(lambda _: np.nan, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() experiment = experiments[0] experiment_id = experiment.experiment_id runs = mlfl_client.search_runs(experiment_id) assert len(runs) == 1 run = runs[0] run_dict = run.to_dictionary() # When `values` is `None`, do not save values with metric names. assert metric_name not in run_dict["data"]["metrics"] def test_log_params(tmpdir: py.path.local) -> None: tracking_uri = f"file:{tmpdir}" metric_name = "metric" study_name = "my_study" mlflc = MLflowCallback(tracking_uri=tracking_uri, metric_name=metric_name) study = optuna.create_study(study_name=study_name) study.enqueue_trial({"x": 1.0, "y": 20.0, "z": 1.0}) study.optimize(_objective_func, n_trials=1, callbacks=[mlflc]) mlfl_client = MlflowClient(tracking_uri) experiments = mlfl_client.search_experiments() experiment = experiments[0] experiment_id = experiment.experiment_id runs = mlfl_client.search_runs(experiment_id) assert len(runs) == 1 run = runs[0] run_dict = run.to_dictionary() for param_name, param_value in study.best_params.items(): assert param_name in run_dict["data"]["params"] assert run_dict["data"]["params"][param_name] == str(param_value) assert run_dict["data"]["tags"][f"{param_name}_distribution"] == str( study.best_trial.distributions[param_name] ) @pytest.mark.parametrize("metrics", [["foo"], ["foo", "bar", "baz"]]) def test_multiobjective_raises_on_name_mismatch(tmpdir: py.path.local, metrics: List[str]) -> None: tracking_uri = f"file:{tmpdir}" mlflc = MLflowCallback(tracking_uri=tracking_uri, metric_name=metrics) study = optuna.create_study(study_name="my_study", directions=["minimize", "maximize"]) with pytest.raises(ValueError): study.optimize(_multiobjective_func, n_trials=1, callbacks=[mlflc]) optuna-3.5.0/tests/integration_tests/test_pytorch_distributed.py000066400000000000000000000324111453453102400254410ustar00rootroot00000000000000import datetime import os from typing import Optional import pytest import optuna from optuna._imports import try_import from optuna.integration import TorchDistributedTrial from optuna.testing.pruners import DeterministicPruner from optuna.testing.storages import StorageSupplier with try_import(): import torch import torch.distributed as dist pytestmark = pytest.mark.integration STORAGE_MODES = [ "inmemory", "sqlite", "cached_sqlite", "journal", "journal_redis", ] @pytest.fixture(scope="session", autouse=True) def init_process_group() -> None: if "OMPI_COMM_WORLD_SIZE" not in os.environ: pytest.skip("This test is expected to be launch with mpirun.") # This function is automatically called at the beginning of the pytest session. os.environ["WORLD_SIZE"] = os.environ["OMPI_COMM_WORLD_SIZE"] os.environ["RANK"] = os.environ["OMPI_COMM_WORLD_RANK"] os.environ["MASTER_ADDR"] = "127.0.0.1" os.environ["MASTER_PORT"] = "20000" dist.init_process_group("gloo", timeout=datetime.timedelta(seconds=15)) def test_torch_distributed_trial_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): if dist.get_rank() == 0: study = optuna.create_study() TorchDistributedTrial(study.ask()) else: TorchDistributedTrial(None) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") def test_torch_distributed_trial_invalid_argument() -> None: with pytest.raises(ValueError): if dist.get_rank() == 0: TorchDistributedTrial(None) else: study = optuna.create_study() TorchDistributedTrial(study.ask()) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_float(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) x1 = trial.suggest_float("x", 0, 1) assert 0 <= x1 <= 1 x2 = trial.suggest_float("x", 0, 1) assert x1 == x2 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_uniform(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) x1 = trial.suggest_uniform("x", 0, 1) assert 0 <= x1 <= 1 x2 = trial.suggest_uniform("x", 0, 1) assert x1 == x2 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_loguniform(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) x1 = trial.suggest_loguniform("x", 1e-7, 1) assert 1e-7 <= x1 <= 1 x2 = trial.suggest_loguniform("x", 1e-7, 1) assert x1 == x2 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_discrete_uniform(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) x1 = trial.suggest_discrete_uniform("x", 0, 10, 2) assert 0 <= x1 <= 10 assert x1 % 2 == 0 x2 = trial.suggest_discrete_uniform("x", 0, 10, 2) assert x1 == x2 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) x1 = trial.suggest_int("x", 0, 10) assert 0 <= x1 <= 10 x2 = trial.suggest_int("x", 0, 10) assert x1 == x2 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_categorical(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) x1 = trial.suggest_categorical("x", ("a", "b", "c")) assert x1 in {"a", "b", "c"} x2 = trial.suggest_categorical("x", ("a", "b", "c")) assert x1 == x2 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_report(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study: Optional[optuna.study.Study] = None if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) trial.report(1, 0) if dist.get_rank() == 0: assert study is not None study.trials[0].intermediate_values[0] == 1 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_report_nan(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study: Optional[optuna.study.Study] = None if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) with pytest.raises(TypeError): trial.report("abc", 0) # type: ignore[arg-type] if dist.get_rank() == 0: assert study is not None assert len(study.trials[0].intermediate_values) == 0 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("is_pruning", [False, True]) def test_should_prune(storage_mode: str, is_pruning: bool) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage, pruner=DeterministicPruner(is_pruning)) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) trial.report(1, 0) assert trial.should_prune() == is_pruning @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) trial.set_user_attr("dataset", "mnist") trial.set_user_attr("batch_size", 128) assert trial.user_attrs["dataset"] == "mnist" assert trial.user_attrs["batch_size"] == 128 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") def test_user_attrs_with_exception() -> None: with StorageSupplier("sqlite") as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) with pytest.raises(TypeError): trial.set_user_attr("not serializable", torch.Tensor([1, 2])) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_number(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) assert trial.number == 0 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_datetime_start(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) assert isinstance(trial.datetime_start, datetime.datetime) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_params(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) trial.suggest_float("f", 0, 1) trial.suggest_int("i", 0, 1) trial.suggest_categorical("c", ("a", "b", "c")) params = trial.params assert 0 <= params["f"] <= 1 assert 0 <= params["i"] <= 1 assert params["c"] in {"a", "b", "c"} @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_distributions(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) trial.suggest_float("u", 0, 1) trial.suggest_float("lu", 1e-7, 1, log=True) trial.suggest_float("du", 0, 1, step=0.5) trial.suggest_int("i", 0, 1) trial.suggest_int("il", 1, 128, log=True) trial.suggest_categorical("c", ("a", "b", "c")) distributions = trial.distributions assert distributions["u"] == optuna.distributions.FloatDistribution(0, 1) assert distributions["lu"] == optuna.distributions.FloatDistribution(1e-7, 1, log=True) assert distributions["du"] == optuna.distributions.FloatDistribution(0, 1, step=0.5) assert distributions["i"] == optuna.distributions.IntDistribution(0, 1) assert distributions["il"] == optuna.distributions.IntDistribution(1, 128, log=True) assert distributions["c"] == optuna.distributions.CategoricalDistribution(("a", "b", "c")) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_updates_properties(storage_mode: str) -> None: """Check for any distributed deadlock following a property read.""" with StorageSupplier(storage_mode) as storage: if dist.get_rank() == 0: study = optuna.create_study(storage=storage) trial = TorchDistributedTrial(study.ask()) else: trial = TorchDistributedTrial(None) trial.suggest_float("f", 0, 1) trial.suggest_int("i", 0, 1) trial.suggest_categorical("c", ("a", "b", "c")) property_names = [ p for p in dir(TorchDistributedTrial) if isinstance(getattr(TorchDistributedTrial, p), property) ] # Rank 0 can read properties without deadlock. if dist.get_rank() == 0: [getattr(trial, p) for p in property_names] dist.barrier() # Same with rank 1. if dist.get_rank() == 1: [getattr(trial, p) for p in property_names] dist.barrier() @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_pass_frozen_trial_to_torch_distributed(storage_mode: str) -> None: # Regression test of #4697 def objective(trial: optuna.trial.BaseTrial) -> float: trial = optuna.integration.TorchDistributedTrial(trial if dist.get_rank() == 0 else None) x = trial.suggest_float("x", low=-100, high=100) return x * x with StorageSupplier(storage_mode) as storage: study = optuna.create_study(direction="minimize", storage=storage) study.optimize(objective, n_trials=1) best_trial = study.best_trial objective(best_trial) optuna-3.5.0/tests/integration_tests/test_pytorch_ignite.py000066400000000000000000000025511453453102400244000ustar00rootroot00000000000000from typing import Iterable from unittest.mock import patch import pytest import optuna from optuna._imports import try_import from optuna.testing.pruners import DeterministicPruner with try_import(): from ignite.engine import Engine pytestmark = pytest.mark.integration def test_pytorch_ignite_pruning_handler() -> None: def update(engine: Engine, batch: Iterable) -> None: pass trainer = Engine(update) evaluator = Engine(update) # The pruner is activated. study = optuna.create_study(pruner=DeterministicPruner(True)) trial = study.ask() handler = optuna.integration.PyTorchIgnitePruningHandler(trial, "accuracy", trainer) with patch.object(trainer, "state", epoch=3): with patch.object(evaluator, "state", metrics={"accuracy": 1}): with pytest.raises(optuna.TrialPruned): handler(evaluator) assert study.trials[0].intermediate_values == {3: 1} # The pruner is not activated. study = optuna.create_study(pruner=DeterministicPruner(False)) trial = study.ask() handler = optuna.integration.PyTorchIgnitePruningHandler(trial, "accuracy", trainer) with patch.object(trainer, "state", epoch=5): with patch.object(evaluator, "state", metrics={"accuracy": 2}): handler(evaluator) assert study.trials[0].intermediate_values == {5: 2} optuna-3.5.0/tests/integration_tests/test_pytorch_lightning.py000066400000000000000000000154561453453102400251140ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Sequence import numpy as np import pytest import optuna from optuna._imports import try_import from optuna.integration import PyTorchLightningPruningCallback from optuna.testing.pruners import DeterministicPruner from optuna.testing.storages import StorageSupplier with try_import() as _imports: import lightning.pytorch as pl from lightning.pytorch import LightningModule import torch from torch import nn from torch.multiprocessing.spawn import ProcessRaisedException import torch.nn.functional as F if not _imports.is_successful(): LightningModule = object # type: ignore[assignment, misc] # NOQA pytestmark = pytest.mark.integration class Model(LightningModule): def __init__(self) -> None: super().__init__() self._model = nn.Sequential(nn.Linear(4, 8)) self.validation_step_outputs: list["torch.Tensor"] = [] def forward(self, data: "torch.Tensor") -> "torch.Tensor": return self._model(data) def training_step( self, batch: Sequence["torch.Tensor"], batch_nb: int ) -> dict[str, "torch.Tensor"]: data, target = batch output = self.forward(data) loss = F.nll_loss(output, target) return {"loss": loss} def validation_step(self, batch: Sequence["torch.Tensor"], batch_nb: int) -> "torch.Tensor": data, target = batch output = self.forward(data) pred = output.argmax(dim=1, keepdim=True) accuracy = pred.eq(target.view_as(pred)).double().mean() self.validation_step_outputs.append(accuracy) return accuracy def on_validation_epoch_end( self, ) -> None: if not len(self.validation_step_outputs): return accuracy = sum(self.validation_step_outputs) / len(self.validation_step_outputs) self.log("accuracy", accuracy) def configure_optimizers(self) -> "torch.optim.Optimizer": return torch.optim.SGD(self._model.parameters(), lr=1e-2) def train_dataloader(self) -> "torch.utils.data.DataLoader": return self._generate_dummy_dataset() def val_dataloader(self) -> "torch.utils.data.DataLoader": return self._generate_dummy_dataset() def _generate_dummy_dataset(self) -> "torch.utils.data.DataLoader": data = torch.zeros(3, 4, dtype=torch.float32) target = torch.zeros(3, dtype=torch.int64) dataset = torch.utils.data.TensorDataset(data, target) return torch.utils.data.DataLoader(dataset, batch_size=1) class ModelDDP(Model): def __init__(self) -> None: super().__init__() def validation_step(self, batch: Sequence["torch.Tensor"], batch_nb: int) -> "torch.Tensor": data, target = batch output = self.forward(data) pred = output.argmax(dim=1, keepdim=True) accuracy = pred.eq(target.view_as(pred)).double().mean() if self.global_rank == 0: accuracy = torch.tensor(0.3) elif self.global_rank == 1: accuracy = torch.tensor(0.6) self.log("accuracy", accuracy, sync_dist=True) return accuracy def on_validation_epoch_end(self) -> None: return def test_pytorch_lightning_pruning_callback() -> None: def objective(trial: optuna.trial.Trial) -> float: callback = PyTorchLightningPruningCallback(trial, monitor="accuracy") trainer = pl.Trainer( max_epochs=2, accelerator="cpu", enable_checkpointing=False, callbacks=[callback], ) model = Model() trainer.fit(model) return 1.0 study = optuna.create_study(pruner=DeterministicPruner(True)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.PRUNED study = optuna.create_study(pruner=DeterministicPruner(False)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 def test_pytorch_lightning_pruning_callback_monitor_is_invalid() -> None: study = optuna.create_study(pruner=DeterministicPruner(True)) trial = study.ask() callback = PyTorchLightningPruningCallback(trial, "InvalidMonitor") trainer = pl.Trainer( max_epochs=1, enable_checkpointing=False, callbacks=[callback], ) model = Model() with pytest.warns(UserWarning): callback.on_validation_end(trainer, model) @pytest.mark.parametrize("storage_mode", ["sqlite", "cached_sqlite"]) def test_pytorch_lightning_pruning_callback_ddp_monitor( storage_mode: str, ) -> None: def objective(trial: optuna.trial.Trial) -> float: callback = PyTorchLightningPruningCallback(trial, monitor="accuracy") trainer = pl.Trainer( max_epochs=2, accelerator="cpu", devices=2, enable_checkpointing=False, callbacks=[callback], strategy="ddp_spawn", ) model = ModelDDP() trainer.fit(model) # Evoke pruning manually. callback.check_pruned() return 1.0 with StorageSupplier(storage_mode) as storage: study = optuna.create_study(storage=storage, pruner=DeterministicPruner(True)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.PRUNED assert list(study.trials[0].intermediate_values.keys()) == [0] np.testing.assert_almost_equal(study.trials[0].intermediate_values[0], 0.45) study = optuna.create_study(storage=storage, pruner=DeterministicPruner(False)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 assert list(study.trials[0].intermediate_values.keys()) == [0, 1] np.testing.assert_almost_equal(study.trials[0].intermediate_values[0], 0.45) np.testing.assert_almost_equal(study.trials[0].intermediate_values[1], 0.45) def test_pytorch_lightning_pruning_callback_ddp_unsupported_storage() -> None: storage_mode = "inmemory" def objective(trial: optuna.trial.Trial) -> float: callback = PyTorchLightningPruningCallback(trial, monitor="accuracy") trainer = pl.Trainer( max_epochs=1, accelerator="cpu", devices=2, enable_checkpointing=False, callbacks=[callback], strategy="ddp_spawn", ) model = ModelDDP() trainer.fit(model) # Evoke pruning manually. callback.check_pruned() return 1.0 with StorageSupplier(storage_mode) as storage: study = optuna.create_study(storage=storage, pruner=DeterministicPruner(True)) with pytest.raises(ProcessRaisedException): study.optimize(objective, n_trials=1) optuna-3.5.0/tests/integration_tests/test_sklearn.py000066400000000000000000000337261453453102400230200ustar00rootroot00000000000000from __future__ import annotations from unittest.mock import MagicMock from unittest.mock import patch import warnings import numpy as np import pytest import scipy as sp from sklearn.datasets import make_blobs from sklearn.datasets import make_regression from sklearn.decomposition import PCA from sklearn.exceptions import ConvergenceWarning from sklearn.exceptions import NotFittedError from sklearn.linear_model import LogisticRegression from sklearn.linear_model import SGDClassifier from sklearn.neighbors import KernelDensity from sklearn.tree import DecisionTreeRegressor from optuna import distributions from optuna import integration from optuna.samplers import BruteForceSampler from optuna.study import create_study from optuna.terminator.erroreval import _CROSS_VALIDATION_SCORES_KEY pytestmark = pytest.mark.integration def test_is_arraylike() -> None: assert integration.sklearn._is_arraylike([]) assert integration.sklearn._is_arraylike(np.zeros(5)) assert not integration.sklearn._is_arraylike(1) def test_num_samples() -> None: x1 = np.random.random((10, 10)) x2 = [1, 2, 3] assert integration.sklearn._num_samples(x1) == 10 assert integration.sklearn._num_samples(x2) == 3 def test_make_indexable() -> None: x1 = np.random.random((10, 10)) x2 = sp.sparse.coo_matrix(x1) x3 = [1, 2, 3] assert hasattr(integration.sklearn._make_indexable(x1), "__getitem__") assert hasattr(integration.sklearn._make_indexable(x2), "__getitem__") assert hasattr(integration.sklearn._make_indexable(x3), "__getitem__") assert integration.sklearn._make_indexable(None) is None @pytest.mark.parametrize("enable_pruning", [True, False]) @pytest.mark.parametrize("fit_params", ["", "coef_init"]) @pytest.mark.filterwarnings("ignore::UserWarning") def test_optuna_search(enable_pruning: bool, fit_params: str) -> None: X, y = make_blobs(n_samples=10) est = SGDClassifier(max_iter=5, tol=1e-03) param_dist = {"alpha": distributions.FloatDistribution(1e-04, 1e03, log=True)} optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, enable_pruning=enable_pruning, error_score="raise", max_iter=5, random_state=0, return_train_score=True, ) with pytest.raises(NotFittedError): optuna_search._check_is_fitted() if fit_params == "coef_init" and not enable_pruning: optuna_search.fit(X, y, coef_init=np.ones((3, 2), dtype=np.float64)) else: optuna_search.fit(X, y) optuna_search.trials_dataframe() optuna_search.decision_function(X) optuna_search.predict(X) optuna_search.score(X, y) @pytest.mark.filterwarnings("ignore::UserWarning") def test_optuna_search_properties() -> None: X, y = make_blobs(n_samples=10) est = LogisticRegression(tol=1e-03) param_dist = {"C": distributions.FloatDistribution(1e-04, 1e03, log=True)} optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, error_score="raise", random_state=0, return_train_score=True ) optuna_search.fit(X, y) optuna_search.set_user_attr("dataset", "blobs") assert optuna_search._estimator_type == "classifier" assert isinstance(optuna_search.best_index_, int) assert isinstance(optuna_search.best_params_, dict) assert isinstance(optuna_search.cv_results_, dict) for cv_result_list_ in optuna_search.cv_results_.values(): assert len(cv_result_list_) == optuna_search.n_trials_ assert optuna_search.best_score_ is not None assert optuna_search.best_trial_ is not None assert np.allclose(optuna_search.classes_, np.array([0, 1, 2])) assert optuna_search.n_trials_ == 10 assert optuna_search.user_attrs_ == {"dataset": "blobs"} assert type(optuna_search.predict_log_proba(X)) == np.ndarray assert type(optuna_search.predict_proba(X)) == np.ndarray @pytest.mark.filterwarnings("ignore::UserWarning") def test_optuna_search_score_samples() -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() optuna_search = integration.OptunaSearchCV( est, {}, cv=3, error_score="raise", random_state=0, return_train_score=True ) optuna_search.fit(X) assert optuna_search.score_samples(X) is not None @pytest.mark.filterwarnings("ignore::UserWarning") def test_optuna_search_transforms() -> None: X, y = make_blobs(n_samples=10) est = PCA() optuna_search = integration.OptunaSearchCV( est, {}, cv=3, error_score="raise", random_state=0, return_train_score=True ) optuna_search.fit(X) assert type(optuna_search.transform(X)) == np.ndarray assert type(optuna_search.inverse_transform(X)) == np.ndarray def test_optuna_search_invalid_estimator() -> None: X, y = make_blobs(n_samples=10) est = "not an estimator" optuna_search = integration.OptunaSearchCV( est, {}, cv=3, error_score="raise", random_state=0, return_train_score=True ) with pytest.raises(ValueError, match="estimator must be a scikit-learn estimator."): optuna_search.fit(X) def test_optuna_search_pruning_without_partial_fit() -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, enable_pruning=True, error_score="raise", random_state=0, return_train_score=True, ) with pytest.raises(ValueError, match="estimator must support partial_fit."): optuna_search.fit(X) def test_optuna_search_negative_max_iter() -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, max_iter=-1, error_score="raise", random_state=0, return_train_score=True, ) with pytest.raises(ValueError, match="max_iter must be > 0"): optuna_search.fit(X) def test_optuna_search_tuple_instead_of_distribution() -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() param_dist = {"kernel": ("gaussian", "linear")} optuna_search = integration.OptunaSearchCV( est, param_dist, # type: ignore cv=3, error_score="raise", random_state=0, return_train_score=True, ) with pytest.raises(ValueError, match="must be a optuna distribution."): optuna_search.fit(X) def test_optuna_search_study_with_minimize() -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() study = create_study(direction="minimize") optuna_search = integration.OptunaSearchCV( est, {}, cv=3, error_score="raise", random_state=0, return_train_score=True, study=study ) with pytest.raises(ValueError, match="direction of study must be 'maximize'."): optuna_search.fit(X) @pytest.mark.parametrize("verbose", [1, 2]) def test_optuna_search_verbosity(verbose: int) -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, error_score="raise", random_state=0, return_train_score=True, verbose=verbose, ) optuna_search.fit(X) def test_optuna_search_subsample() -> None: X, y = make_blobs(n_samples=10) est = KernelDensity() param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, error_score="raise", random_state=0, return_train_score=True, subsample=5, ) optuna_search.fit(X) @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_objective_y_None() -> None: X, y = make_blobs(n_samples=10) est = SGDClassifier(max_iter=5, tol=1e-03) param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, enable_pruning=True, error_score="raise", random_state=0, return_train_score=True, ) with pytest.raises(ValueError): optuna_search.fit(X) @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_objective_error_score_nan() -> None: X, y = make_blobs(n_samples=10) est = SGDClassifier(max_iter=5, tol=1e-03) param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, enable_pruning=True, max_iter=5, error_score=np.nan, random_state=0, return_train_score=True, ) with pytest.raises( ValueError, match="This SGDClassifier estimator requires y to be passed, but the target y is None.", ): optuna_search.fit(X) for trial in optuna_search.study_.get_trials(): assert np.all(np.isnan(list(trial.intermediate_values.values()))) # "_score" stores every score value for train and test validation holds. for name, value in trial.user_attrs.items(): if name.endswith("_score"): assert np.isnan(value) @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_objective_error_score_invalid() -> None: X, y = make_blobs(n_samples=10) est = SGDClassifier(max_iter=5, tol=1e-03) param_dist = {} # type: ignore optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, enable_pruning=True, max_iter=5, error_score="invalid error score", random_state=0, return_train_score=True, ) with pytest.raises(ValueError, match="error_score must be 'raise' or numeric."): optuna_search.fit(X) # This test checks whether OptunaSearchCV completes the study without halting, even if some trials # fails due to misconfiguration. @pytest.mark.parametrize( "param_dist,all_params", [ ({"max_depth": distributions.IntDistribution(0, 1)}, [0, 1]), ({"max_depth": distributions.IntDistribution(0, 0)}, [0]), ], ) @pytest.mark.filterwarnings("ignore::RuntimeWarning") @pytest.mark.filterwarnings("ignore::UserWarning") def test_no_halt_with_error( param_dist: dict[str, distributions.BaseDistribution], all_params: list[int] ) -> None: X, y = make_regression(n_samples=100, n_features=10) estimator = DecisionTreeRegressor() study = create_study(sampler=BruteForceSampler(), direction="maximize") # DecisionTreeRegressor raises ValueError when max_depth==0. optuna_search = integration.OptunaSearchCV( estimator, param_dist, study=study, ) optuna_search.fit(X, y) all_suggested_values = [t.params["max_depth"] for t in study.trials] assert len(all_suggested_values) == len(all_params) for a in all_params: assert a in all_suggested_values # TODO(himkt): Remove this method with the deletion of deprecated distributions. # https://github.com/optuna/optuna/issues/2941 @pytest.mark.filterwarnings("ignore::FutureWarning") def test_optuna_search_convert_deprecated_distribution() -> None: param_dist = { "ud": distributions.UniformDistribution(low=0, high=10), "dud": distributions.DiscreteUniformDistribution(low=0, high=10, q=2), "lud": distributions.LogUniformDistribution(low=1, high=10), "id": distributions.IntUniformDistribution(low=0, high=10), "idd": distributions.IntUniformDistribution(low=0, high=10, step=2), "ild": distributions.IntLogUniformDistribution(low=1, high=10), } expected_param_dist = { "ud": distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": distributions.IntDistribution(low=1, high=10, log=True, step=1), } with pytest.raises(ValueError): optuna_search = integration.OptunaSearchCV( KernelDensity(), param_dist, ) # It confirms that ask doesn't convert non-deprecated distributions. optuna_search = integration.OptunaSearchCV( KernelDensity(), expected_param_dist, ) assert optuna_search.param_distributions == expected_param_dist def test_callbacks() -> None: callbacks = [] for _ in range(2): callback = MagicMock() callback.__call__ = MagicMock(return_value=None) # type: ignore callbacks.append(callback) n_trials = 5 X, y = make_blobs(n_samples=10) est = SGDClassifier(max_iter=5, tol=1e-03) param_dist = {"alpha": distributions.FloatDistribution(1e-04, 1e03, log=True)} optuna_search = integration.OptunaSearchCV( est, param_dist, cv=3, enable_pruning=True, max_iter=5, n_trials=n_trials, error_score=np.nan, callbacks=callbacks, # type: ignore ) with warnings.catch_warnings(): warnings.simplefilter("ignore", category=ConvergenceWarning) optuna_search.fit(X, y) for callback in callbacks: for trial in optuna_search.trials_: callback.assert_any_call(optuna_search.study_, trial) assert callback.call_count == n_trials @pytest.mark.filterwarnings("ignore::UserWarning") @patch("optuna.integration.sklearn.cross_validate") def test_terminator_cv_score_reporting(mock: MagicMock) -> None: scores = { "fit_time": np.array([2.01, 1.78, 3.22]), "score_time": np.array([0.33, 0.35, 0.48]), "test_score": np.array([0.04, 0.80, 0.70]), } mock.return_value = scores X, _ = make_blobs(n_samples=10) est = PCA() optuna_search = integration.OptunaSearchCV(est, {}, cv=3, error_score="raise", random_state=0) optuna_search.fit(X) for trial in optuna_search.study_.trials: assert (trial.system_attrs[_CROSS_VALIDATION_SCORES_KEY] == scores["test_score"]).all() optuna-3.5.0/tests/integration_tests/test_tensorboard.py000066400000000000000000000071511453453102400236740ustar00rootroot00000000000000import os import shutil import tempfile import pytest import optuna from optuna._imports import try_import from optuna.integration.tensorboard import TensorBoardCallback with try_import(): from tensorboard.backend.event_processing.event_accumulator import EventAccumulator pytestmark = pytest.mark.integration def _objective_func(trial: optuna.trial.Trial) -> float: u = trial.suggest_int("u", 0, 10, step=2) v = trial.suggest_int("v", 1, 10, log=True) w = trial.suggest_float("w", -1.0, 1.0, step=0.1) x = trial.suggest_float("x", -1.0, 1.0) y = trial.suggest_float("y", 20.0, 30.0, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) trial.set_user_attr("my_user_attr", "my_user_attr_value") return u + v + w + (x - 2) ** 2 + (y - 25) ** 2 + z def test_study_name() -> None: dirname = tempfile.mkdtemp() metric_name = "target" study_name = "test_tensorboard_integration" tbcallback = TensorBoardCallback(dirname, metric_name) study = optuna.create_study(study_name=study_name) study.optimize(_objective_func, n_trials=1, callbacks=[tbcallback]) event_acc = EventAccumulator(os.path.join(dirname, "trial-0")) event_acc.Reload() try: assert len(event_acc.Tensors("target")) == 1 except Exception as e: raise e finally: shutil.rmtree(dirname) def test_cast_float() -> None: def objective(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", 1, 2) y = trial.suggest_float("y", 1, 2, log=True) assert isinstance(x, float) assert isinstance(y, float) return x + y dirname = tempfile.mkdtemp() metric_name = "target" study_name = "test_tensorboard_integration" tbcallback = TensorBoardCallback(dirname, metric_name) study = optuna.create_study(study_name=study_name) study.optimize(objective, n_trials=1, callbacks=[tbcallback]) def test_categorical() -> None: def objective(trial: optuna.trial.Trial) -> float: x = trial.suggest_categorical("x", [1, 2, 3]) assert isinstance(x, int) return x dirname = tempfile.mkdtemp() metric_name = "target" study_name = "test_tensorboard_integration" tbcallback = TensorBoardCallback(dirname, metric_name) study = optuna.create_study(study_name=study_name) study.optimize(objective, n_trials=1, callbacks=[tbcallback]) def test_categorical_mixed_types() -> None: def objective(trial: optuna.trial.Trial) -> float: x = trial.suggest_categorical("x", [None, 1, 2, 3.14, True, "foo"]) assert x is None or isinstance(x, (int, float, bool, str)) return len(str(x)) dirname = tempfile.mkdtemp() metric_name = "target" study_name = "test_tensorboard_integration" tbcallback = TensorBoardCallback(dirname, metric_name) study = optuna.create_study(study_name=study_name) study.optimize(objective, n_trials=10, callbacks=[tbcallback]) def test_categorical_unsupported_types() -> None: def objective(trial: optuna.trial.Trial) -> float: x = trial.suggest_categorical("x", [[1, 2], [3, 4, 5], [6]]) # type: ignore[list-item] assert isinstance(x, list) return len(x) dirname = tempfile.mkdtemp() metric_name = "target" study_name = "test_tensorboard_integration" tbcallback = TensorBoardCallback(dirname, metric_name) study = optuna.create_study(study_name=study_name) study.optimize(objective, n_trials=10, callbacks=[tbcallback]) def test_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): TensorBoardCallback(dirname="", metric_name="") optuna-3.5.0/tests/integration_tests/test_wandb.py000066400000000000000000000236621453453102400224520ustar00rootroot00000000000000from typing import Any from typing import Dict from typing import List from typing import Sequence from typing import Tuple from typing import Union from unittest import mock import pytest import optuna from optuna.integration import WeightsAndBiasesCallback pytestmark = pytest.mark.integration def _objective_func(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", low=-10, high=10) y = trial.suggest_float("y", low=1, high=10, log=True) return (x - 2) ** 2 + (y - 25) ** 2 def _multiobjective_func(trial: optuna.trial.Trial) -> Tuple[float, float]: x = trial.suggest_float("x", low=-10, high=10) y = trial.suggest_float("y", low=1, high=10, log=True) first_objective = (x - 2) ** 2 + (y - 25) ** 2 second_objective = (x - 2) ** 3 + (y - 25) ** 3 return first_objective, second_objective @mock.patch("optuna.integration.wandb.wandb") def test_run_initialized(wandb: mock.MagicMock) -> None: wandb.sdk.wandb_run.Run = mock.MagicMock n_trials = 10 wandb_kwargs = { "project": "optuna", "group": "summary", "job_type": "logging", "mode": "offline", "tags": ["test-tag"], } WeightsAndBiasesCallback(metric_name="mse", wandb_kwargs=wandb_kwargs, as_multirun=False) wandb.init.assert_called_once_with( project="optuna", group="summary", job_type="logging", mode="offline", tags=["test-tag"] ) wandbc = WeightsAndBiasesCallback( metric_name="mse", wandb_kwargs=wandb_kwargs, as_multirun=True ) wandb.run = None study = optuna.create_study(direction="minimize") _wrapped_func = wandbc.track_in_wandb()(lambda t: 1.0) wandb.init.reset_mock() trial = study.ask() _wrapped_func(trial) wandb.init.assert_called_once_with( project="optuna", group="summary", job_type="logging", mode="offline", tags=["test-tag"] ) wandb.init.reset_mock() study.optimize(_objective_func, n_trials=n_trials, callbacks=[wandbc]) wandb.init.assert_called_with( project="optuna", group="summary", job_type="logging", mode="offline", tags=["test-tag"] ) assert wandb.init.call_count == n_trials wandb.init().finish.assert_called() assert wandb.init().finish.call_count == n_trials @mock.patch("optuna.integration.wandb.wandb") @pytest.mark.parametrize("as_multirun", [True, False]) def test_attributes_set_on_epoch(wandb: mock.MagicMock, as_multirun: bool) -> None: wandb.sdk.wandb_run.Run = mock.MagicMock expected_config: Dict[str, Any] = {"direction": ["MINIMIZE"]} trial_params = {"x": 1.1, "y": 2.2} expected_config_with_params = {**expected_config, **trial_params} study = optuna.create_study(direction="minimize") wandbc = WeightsAndBiasesCallback(as_multirun=as_multirun) if as_multirun: wandb.run = None study.enqueue_trial(trial_params) study.optimize(_objective_func, n_trials=1, callbacks=[wandbc]) if as_multirun: wandb.init().config.update.assert_called_once_with(expected_config_with_params) else: wandb.run.config.update.assert_called_once_with(expected_config) @mock.patch("optuna.integration.wandb.wandb") @pytest.mark.parametrize("as_multirun", [True, False]) def test_multiobjective_attributes_set_on_epoch(wandb: mock.MagicMock, as_multirun: bool) -> None: wandb.sdk.wandb_run.Run = mock.MagicMock expected_config: Dict[str, Any] = {"direction": ["MINIMIZE", "MAXIMIZE"]} trial_params = {"x": 1.1, "y": 2.2} expected_config_with_params = {**expected_config, **trial_params} study = optuna.create_study(directions=["minimize", "maximize"]) wandbc = WeightsAndBiasesCallback(as_multirun=as_multirun) if as_multirun: wandb.run = None study.enqueue_trial(trial_params) study.optimize(_multiobjective_func, n_trials=1, callbacks=[wandbc]) if as_multirun: wandb.init().config.update.assert_called_once_with(expected_config_with_params) else: wandb.run.config.update.assert_called_once_with(expected_config) @mock.patch("optuna.integration.wandb.wandb") def test_log_api_call_count(wandb: mock.MagicMock) -> None: wandb.sdk.wandb_run.Run = mock.MagicMock study = optuna.create_study() wandbc = WeightsAndBiasesCallback() @wandbc.track_in_wandb() def _decorated_objective(trial: optuna.trial.Trial) -> float: result = _objective_func(trial) wandb.run.log({"result": result}) return result target_n_trials = 10 study.optimize(_objective_func, n_trials=target_n_trials, callbacks=[wandbc]) assert wandb.run.log.call_count == target_n_trials wandbc = WeightsAndBiasesCallback(as_multirun=True) wandb.run.reset_mock() study.optimize(_decorated_objective, n_trials=target_n_trials, callbacks=[wandbc]) assert wandb.run.log.call_count == 2 * target_n_trials wandb.run = None study.optimize(_objective_func, n_trials=target_n_trials, callbacks=[wandbc]) assert wandb.init().log.call_count == target_n_trials @pytest.mark.parametrize( "metric,as_multirun,expected", [("value", False, ["x", "y", "value"]), ("foo", True, ["x", "y", "foo", "trial_number"])], ) @mock.patch("optuna.integration.wandb.wandb") def test_values_registered_on_epoch( wandb: mock.MagicMock, metric: str, as_multirun: bool, expected: List[str] ) -> None: def assert_call_args(log_func: mock.MagicMock, as_multirun: bool) -> None: call_args = log_func.call_args assert list(call_args[0][0].keys()) == expected assert call_args[1] == {"step": None if as_multirun else 0} wandb.sdk.wandb_run.Run = mock.MagicMock if as_multirun: wandb.run = None log_func = wandb.init().log else: log_func = wandb.run.log study = optuna.create_study() wandbc = WeightsAndBiasesCallback(metric_name=metric, as_multirun=as_multirun) study.optimize(_objective_func, n_trials=1, callbacks=[wandbc]) assert_call_args(log_func, as_multirun) @pytest.mark.parametrize("metric,expected", [("foo", ["x", "y", "foo", "trial_number"])]) @mock.patch("optuna.integration.wandb.wandb") def test_values_registered_on_epoch_with_logging( wandb: mock.MagicMock, metric: str, expected: List[str] ) -> None: wandb.sdk.wandb_run.Run = mock.MagicMock study = optuna.create_study() wandbc = WeightsAndBiasesCallback(metric_name=metric, as_multirun=True) @wandbc.track_in_wandb() def _decorated_objective(trial: optuna.trial.Trial) -> float: result = _objective_func(trial) wandb.run.log({"result": result}) return result study.enqueue_trial({"x": 2, "y": 3}) study.optimize(_decorated_objective, n_trials=1, callbacks=[wandbc]) logged_in_decorator = wandb.run.log.mock_calls[0][1][0] logged_in_callback = wandb.run.log.mock_calls[1][1][0] assert len(wandb.run.log.mock_calls) == 2 assert list(logged_in_decorator) == ["result"] assert list(logged_in_callback) == expected call_args = wandb.run.log.call_args assert call_args[1] == {"step": 0} @pytest.mark.parametrize( "metrics,as_multirun,expected", [ ("value", False, ["x", "y", "value_0", "value_1"]), ("value", True, ["x", "y", "value_0", "value_1", "trial_number"]), (["foo", "bar"], False, ["x", "y", "foo", "bar"]), (("foo", "bar"), True, ["x", "y", "foo", "bar", "trial_number"]), ], ) @mock.patch("optuna.integration.wandb.wandb") def test_multiobjective_values_registered_on_epoch( wandb: mock.MagicMock, metrics: Union[str, Sequence[str]], as_multirun: bool, expected: List[str], ) -> None: def assert_call_args(log_func: mock.MagicMock, as_multirun: bool) -> None: call_args = log_func.call_args assert list(call_args[0][0].keys()) == expected assert call_args[1] == {"step": None if as_multirun else 0} wandb.sdk.wandb_run.Run = mock.MagicMock if as_multirun: wandb.run = None log_func = wandb.init().log else: log_func = wandb.run.log study = optuna.create_study(directions=["minimize", "maximize"]) wandbc = WeightsAndBiasesCallback(metric_name=metrics, as_multirun=as_multirun) study.optimize(_multiobjective_func, n_trials=1, callbacks=[wandbc]) assert_call_args(log_func, as_multirun) @pytest.mark.parametrize( "metrics,expected", [ ("value", ["x", "y", "value_0", "value_1", "trial_number"]), (("foo", "bar"), ["x", "y", "foo", "bar", "trial_number"]), ], ) @mock.patch("optuna.integration.wandb.wandb") def test_multiobjective_values_registered_on_epoch_with_logging( wandb: mock.MagicMock, metrics: Union[str, Sequence[str]], expected: List[str] ) -> None: wandbc = WeightsAndBiasesCallback(as_multirun=True, metric_name=metrics) @wandbc.track_in_wandb() def _decorated_objective(trial: optuna.trial.Trial) -> Tuple[float, float]: result0, result1 = _multiobjective_func(trial) wandb.run.log({"result0": result0, "result1": result1}) return result0, result1 study = optuna.create_study(directions=["minimize", "maximize"]) study.enqueue_trial({"x": 2, "y": 3}) study.optimize(_decorated_objective, n_trials=1, callbacks=[wandbc]) logged_in_decorator = wandb.run.log.mock_calls[0][1][0] logged_in_callback = wandb.run.log.mock_calls[1][1][0] assert len(wandb.run.log.mock_calls) == 2 assert list(logged_in_decorator) == ["result0", "result1"] assert list(logged_in_callback) == expected call_args = wandb.run.log.call_args assert call_args[1] == {"step": 0} @pytest.mark.parametrize("metrics", [["foo"], ["foo", "bar", "baz"]]) @mock.patch("optuna.integration.wandb.wandb") def test_multiobjective_raises_on_name_mismatch(wandb: mock.MagicMock, metrics: List[str]) -> None: wandb.sdk.wandb_run.Run = mock.MagicMock study = optuna.create_study(directions=["minimize", "maximize"]) wandbc = WeightsAndBiasesCallback(metric_name=metrics) with pytest.raises(ValueError): study.optimize(_multiobjective_func, n_trials=1, callbacks=[wandbc]) optuna-3.5.0/tests/integration_tests/test_xgboost.py000066400000000000000000000053541453453102400230420ustar00rootroot00000000000000import numpy as np import pytest import optuna from optuna._imports import try_import from optuna.integration.xgboost import XGBoostPruningCallback from optuna.testing.pruners import DeterministicPruner with try_import(): import xgboost as xgb pytestmark = pytest.mark.integration def test_xgboost_pruning_callback_call() -> None: # The pruner is deactivated. study = optuna.create_study(pruner=DeterministicPruner(False)) trial = study.ask() pruning_callback = XGBoostPruningCallback(trial, "validation-logloss") pruning_callback.after_iteration( model=None, epoch=1, evals_log={"validation": {"logloss": [1.0]}} ) # The pruner is activated. study = optuna.create_study(pruner=DeterministicPruner(True)) trial = study.ask() pruning_callback = XGBoostPruningCallback(trial, "validation-logloss") with pytest.raises(optuna.TrialPruned): pruning_callback.after_iteration( model=None, epoch=1, evals_log={"validation": {"logloss": [1.0]}} ) def test_xgboost_pruning_callback() -> None: def objective(trial: optuna.trial.Trial) -> float: dtrain = xgb.DMatrix(np.asarray([[1.0]]), label=[1.0]) dtest = xgb.DMatrix(np.asarray([[1.0]]), label=[1.0]) pruning_callback = XGBoostPruningCallback(trial, "validation-logloss") xgb.train( {"objective": "binary:logistic"}, dtrain, 1, evals=[(dtest, "validation")], verbose_eval=False, callbacks=[pruning_callback], ) return 1.0 study = optuna.create_study(pruner=DeterministicPruner(True)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.PRUNED study = optuna.create_study(pruner=DeterministicPruner(False)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 def test_xgboost_pruning_callback_cv() -> None: def objective(trial: optuna.trial.Trial) -> float: dtrain = xgb.DMatrix(np.ones((2, 1)), label=[1.0, 1.0]) params = { "objective": "binary:logistic", } pruning_callback = optuna.integration.XGBoostPruningCallback(trial, "test-logloss") xgb.cv(params, dtrain, callbacks=[pruning_callback], nfold=2) return 1.0 study = optuna.create_study(pruner=DeterministicPruner(True)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.PRUNED study = optuna.create_study(pruner=DeterministicPruner(False)) study.optimize(objective, n_trials=1) assert study.trials[0].state == optuna.trial.TrialState.COMPLETE assert study.trials[0].value == 1.0 optuna-3.5.0/tests/multi_objective_tests/000077500000000000000000000000001453453102400205765ustar00rootroot00000000000000optuna-3.5.0/tests/multi_objective_tests/__init__.py000066400000000000000000000000001453453102400226750ustar00rootroot00000000000000optuna-3.5.0/tests/multi_objective_tests/samplers_tests/000077500000000000000000000000001453453102400236465ustar00rootroot00000000000000optuna-3.5.0/tests/multi_objective_tests/samplers_tests/__init__.py000066400000000000000000000000001453453102400257450ustar00rootroot00000000000000optuna-3.5.0/tests/multi_objective_tests/samplers_tests/test_motpe.py000066400000000000000000000044421453453102400264070ustar00rootroot00000000000000from typing import Dict from typing import Tuple from unittest.mock import Mock from unittest.mock import patch import pytest import optuna from optuna import multi_objective from optuna.multi_objective.samplers import MOTPEMultiObjectiveSampler from optuna.samplers import MOTPESampler class MockSystemAttr: def __init__(self) -> None: self.value: Dict[str, dict] = {} def set_trial_system_attr(self, _: int, key: str, value: dict) -> None: self.value[key] = value @pytest.mark.filterwarnings("ignore::FutureWarning") def test_reseed_rng() -> None: sampler = MOTPEMultiObjectiveSampler() original_random_state = sampler._motpe_sampler._rng.rng.get_state() with patch.object( sampler._motpe_sampler, "reseed_rng", wraps=sampler._motpe_sampler.reseed_rng ) as mock_object: sampler.reseed_rng() assert mock_object.call_count == 1 assert str(original_random_state) != str(sampler._motpe_sampler._rng.rng.get_state()) @pytest.mark.filterwarnings("ignore::FutureWarning") def test_sample_relative() -> None: sampler = MOTPEMultiObjectiveSampler() # Study and frozen-trial are not supposed to be accessed. study = Mock(spec=[]) frozen_trial = Mock(spec=[]) assert sampler.sample_relative(study, frozen_trial, {}) == {} @pytest.mark.filterwarnings("ignore::FutureWarning") def test_infer_relative_search_space() -> None: sampler = MOTPEMultiObjectiveSampler() # Study and frozen-trial are not supposed to be accessed. study = Mock(spec=[]) frozen_trial = Mock(spec=[]) assert sampler.infer_relative_search_space(study, frozen_trial) == {} @pytest.mark.filterwarnings("ignore::FutureWarning") def test_sample_independent() -> None: sampler = MOTPEMultiObjectiveSampler() study = optuna.multi_objective.create_study( directions=["minimize", "maximize"], sampler=sampler ) def _objective(trial: multi_objective.trial.MultiObjectiveTrial) -> Tuple[float, float]: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x, y with patch.object( MOTPESampler, "sample_independent", wraps=sampler._motpe_sampler.sample_independent, ) as mock: study.optimize(_objective, n_trials=10) assert mock.call_count == 20 optuna-3.5.0/tests/multi_objective_tests/samplers_tests/test_nsga2.py000066400000000000000000000200641453453102400262730ustar00rootroot00000000000000from collections import Counter from typing import List from typing import Tuple from unittest.mock import patch import pytest import optuna from optuna import multi_objective from optuna.study._study_direction import StudyDirection pytestmark = pytest.mark.filterwarnings("ignore::FutureWarning") def test_population_size() -> None: # Set `population_size` to 10. sampler = multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size=10) study = multi_objective.create_study(["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[multi_objective.samplers._nsga2._GENERATION_KEY] for t in study.trials] ) assert generations == {0: 10, 1: 10, 2: 10, 3: 10} # Set `population_size` to 2. sampler = multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size=2) study = multi_objective.create_study(["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[multi_objective.samplers._nsga2._GENERATION_KEY] for t in study.trials] ) assert generations == {i: 2 for i in range(20)} # Invalid population size. with pytest.raises(ValueError): # Less than 2. multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size=1) with pytest.raises(TypeError): # Not an integer. multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size=2.5) # type: ignore def test_mutation_prob() -> None: multi_objective.samplers.NSGAIIMultiObjectiveSampler(mutation_prob=None) multi_objective.samplers.NSGAIIMultiObjectiveSampler(mutation_prob=0.0) multi_objective.samplers.NSGAIIMultiObjectiveSampler(mutation_prob=0.5) multi_objective.samplers.NSGAIIMultiObjectiveSampler(mutation_prob=1.0) with pytest.raises(ValueError): multi_objective.samplers.NSGAIIMultiObjectiveSampler(mutation_prob=-0.5) with pytest.raises(ValueError): multi_objective.samplers.NSGAIIMultiObjectiveSampler(mutation_prob=1.1) def test_crossover_prob() -> None: multi_objective.samplers.NSGAIIMultiObjectiveSampler(crossover_prob=0.0) multi_objective.samplers.NSGAIIMultiObjectiveSampler(crossover_prob=0.5) multi_objective.samplers.NSGAIIMultiObjectiveSampler(crossover_prob=1.0) with pytest.raises(ValueError): multi_objective.samplers.NSGAIIMultiObjectiveSampler(crossover_prob=-0.5) with pytest.raises(ValueError): multi_objective.samplers.NSGAIIMultiObjectiveSampler(crossover_prob=1.1) def test_swapping_prob() -> None: multi_objective.samplers.NSGAIIMultiObjectiveSampler(swapping_prob=0.0) multi_objective.samplers.NSGAIIMultiObjectiveSampler(swapping_prob=0.5) multi_objective.samplers.NSGAIIMultiObjectiveSampler(swapping_prob=1.0) with pytest.raises(ValueError): multi_objective.samplers.NSGAIIMultiObjectiveSampler(swapping_prob=-0.5) with pytest.raises(ValueError): multi_objective.samplers.NSGAIIMultiObjectiveSampler(swapping_prob=1.1) def test_fast_non_dominated_sort() -> None: # Single objective. directions = [StudyDirection.MINIMIZE] trials = [ _create_frozen_trial(0, [10]), _create_frozen_trial(1, [20]), _create_frozen_trial(2, [20]), _create_frozen_trial(3, [30]), ] population_per_rank = multi_objective.samplers._nsga2._fast_non_dominated_sort( trials, directions ) assert [{t.number for t in population} for population in population_per_rank] == [ {0}, {1, 2}, {3}, ] # Two objective. directions = [StudyDirection.MAXIMIZE, StudyDirection.MAXIMIZE] trials = [ _create_frozen_trial(0, [10, 30]), _create_frozen_trial(1, [10, 10]), _create_frozen_trial(2, [20, 20]), _create_frozen_trial(3, [30, 10]), _create_frozen_trial(4, [15, 15]), ] population_per_rank = multi_objective.samplers._nsga2._fast_non_dominated_sort( trials, directions ) assert [{t.number for t in population} for population in population_per_rank] == [ {0, 2, 3}, {4}, {1}, ] # Three objective. directions = [StudyDirection.MAXIMIZE, StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE] trials = [ _create_frozen_trial(0, [5, 5, 4]), _create_frozen_trial(1, [5, 5, 5]), _create_frozen_trial(2, [9, 9, 0]), _create_frozen_trial(3, [5, 7, 5]), _create_frozen_trial(4, [0, 0, 9]), _create_frozen_trial(5, [0, 9, 9]), ] population_per_rank = multi_objective.samplers._nsga2._fast_non_dominated_sort( trials, directions ) assert [{t.number for t in population} for population in population_per_rank] == [ {2}, {0, 3, 5}, {1}, {4}, ] def test_crowding_distance_sort() -> None: trials = [ _create_frozen_trial(0, [5]), _create_frozen_trial(1, [6]), _create_frozen_trial(2, [9]), _create_frozen_trial(3, [0]), ] multi_objective.samplers._nsga2._crowding_distance_sort(trials) assert [t.number for t in trials] == [2, 3, 0, 1] trials = [ _create_frozen_trial(0, [5, 0]), _create_frozen_trial(1, [6, 0]), _create_frozen_trial(2, [9, 0]), _create_frozen_trial(3, [0, 0]), ] multi_objective.samplers._nsga2._crowding_distance_sort(trials) assert [t.number for t in trials] == [2, 3, 0, 1] def test_study_system_attr_for_population_cache() -> None: sampler = multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size=10) study = multi_objective.create_study(["minimize"], sampler=sampler) def get_cached_entries( study: multi_objective.study.MultiObjectiveStudy, ) -> List[Tuple[int, List[int]]]: study_system_attrs = study._storage.get_study_system_attrs(study._study_id) return [ v for k, v in study_system_attrs.items() if k.startswith(multi_objective.samplers._nsga2._POPULATION_CACHE_KEY_PREFIX) ] study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 0 study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=1) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 0 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 1 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. def test_reseed_rng() -> None: sampler = multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size=10) original_random_state = sampler._rng.rng.get_state() original_random_sampler_random_state = sampler._random_sampler._sampler._rng.rng.get_state() with patch.object( sampler._random_sampler, "reseed_rng", wraps=sampler._random_sampler.reseed_rng ) as mock_object: sampler.reseed_rng() assert mock_object.call_count == 1 assert str(original_random_state) != str(sampler._rng.rng.get_state()) assert str(original_random_sampler_random_state) != str( sampler._random_sampler._sampler._rng.rng.get_state() ) # TODO(ohta): Consider to move this utility function to `optuna.testing` module. def _create_frozen_trial( number: int, values: List[float] ) -> multi_objective.trial.FrozenMultiObjectiveTrial: trial = optuna.trial.FrozenTrial( number=number, trial_id=number, state=optuna.trial.TrialState.COMPLETE, value=None, datetime_start=None, datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values=dict(enumerate(values)), ) return multi_objective.trial.FrozenMultiObjectiveTrial(len(values), trial) optuna-3.5.0/tests/multi_objective_tests/samplers_tests/test_samplers.py000066400000000000000000000073161453453102400271140ustar00rootroot00000000000000from typing import Callable from unittest.mock import patch import numpy as np import pytest import optuna from optuna import multi_objective from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.multi_objective.samplers import BaseMultiObjectiveSampler pytestmark = pytest.mark.filterwarnings("ignore::FutureWarning") parametrize_sampler = pytest.mark.parametrize( "sampler_class", [ optuna.multi_objective.samplers.RandomMultiObjectiveSampler, optuna.multi_objective.samplers.NSGAIIMultiObjectiveSampler, ], ) @parametrize_sampler @pytest.mark.parametrize( "distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(0.0, 1.0), FloatDistribution(-1.0, 0.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.1), FloatDistribution(-10.2, 10.2, step=0.1), IntDistribution(-10, 10), IntDistribution(0, 10), IntDistribution(-10, 0), IntDistribution(-10, 10, step=2), IntDistribution(0, 10, step=2), IntDistribution(-10, 0, step=2), CategoricalDistribution((1, 2, 3)), CategoricalDistribution(("a", "b", "c")), CategoricalDistribution((1, "a")), ], ) def test_sample_independent( sampler_class: Callable[[], BaseMultiObjectiveSampler], distribution: BaseDistribution ) -> None: study = optuna.multi_objective.study.create_study( ["minimize", "maximize"], sampler=sampler_class() ) for i in range(100): value = study.sampler.sample_independent( study, _create_new_trial(study), "x", distribution ) assert distribution._contains(distribution.to_internal_repr(value)) if not isinstance(distribution, CategoricalDistribution): # Please see https://github.com/optuna/optuna/pull/393 why this assertion is needed. assert not isinstance(value, np.floating) if isinstance(distribution, FloatDistribution): if distribution.step is not None: # Check the value is a multiple of `distribution.q` which is # the quantization interval of the distribution. value -= distribution.low value /= distribution.step round_value = np.round(value) np.testing.assert_almost_equal(round_value, value) def test_random_mo_sampler_reseed_rng() -> None: sampler = optuna.multi_objective.samplers.RandomMultiObjectiveSampler() original_random_state = sampler._sampler._rng.rng.get_state() with patch.object( sampler._sampler, "reseed_rng", wraps=sampler._sampler.reseed_rng ) as mock_object: sampler.reseed_rng() assert mock_object.call_count == 1 assert str(original_random_state) != str(sampler._sampler._rng.rng.get_state()) @pytest.mark.parametrize( "sampler_class", [ optuna.multi_objective.samplers.RandomMultiObjectiveSampler, optuna.multi_objective.samplers.NSGAIIMultiObjectiveSampler, optuna.multi_objective.samplers.MOTPEMultiObjectiveSampler, ], ) def test_deprecated_warning(sampler_class: Callable[[], BaseMultiObjectiveSampler]) -> None: with pytest.warns(FutureWarning): sampler_class() def _create_new_trial( study: multi_objective.study.MultiObjectiveStudy, ) -> multi_objective.trial.FrozenMultiObjectiveTrial: trial_id = study._study._storage.create_new_trial(study._study._study_id) trial = study._study._storage.get_trial(trial_id) return multi_objective.trial.FrozenMultiObjectiveTrial(study.n_objectives, trial) optuna-3.5.0/tests/multi_objective_tests/test_study.py000066400000000000000000000125151453453102400233630ustar00rootroot00000000000000from typing import List from typing import Tuple import uuid import _pytest.capture import pytest import optuna from optuna.study._study_direction import StudyDirection from optuna.testing.storages import StorageSupplier pytestmark = pytest.mark.filterwarnings("ignore::FutureWarning") def test_create_study() -> None: study = optuna.multi_objective.create_study(["maximize"]) assert study.n_objectives == 1 assert study.directions == [StudyDirection.MAXIMIZE] study = optuna.multi_objective.create_study(["maximize", "minimize"]) assert study.n_objectives == 2 assert study.directions == [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE] with pytest.raises(ValueError): # Empty `directions` isn't allowed. study = optuna.multi_objective.create_study([]) def test_load_study() -> None: with StorageSupplier("sqlite") as storage: study_name = str(uuid.uuid4()) with pytest.raises(KeyError): # Test loading an unexisting study. optuna.multi_objective.study.load_study(study_name=study_name, storage=storage) # Create a new study. created_study = optuna.multi_objective.study.create_study( ["minimize"], study_name=study_name, storage=storage ) # Test loading an existing study. loaded_study = optuna.multi_objective.study.load_study( study_name=study_name, storage=storage ) assert created_study._study._study_id == loaded_study._study._study_id @pytest.mark.parametrize("n_objectives", [1, 2, 3]) def test_optimize(n_objectives: int) -> None: directions = ["minimize" for _ in range(n_objectives)] study = optuna.multi_objective.create_study(directions) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> List[float]: return [trial.suggest_uniform("v{}".format(i), 0, 5) for i in range(n_objectives)] study.optimize(objective, n_trials=10) assert len(study.trials) == 10 for trial in study.trials: assert len(trial.values) == n_objectives def test_pareto_front() -> None: study = optuna.multi_objective.create_study(["minimize", "maximize"]) assert {tuple(t.values) for t in study.get_pareto_front_trials()} == set() study.optimize(lambda t: [2, 2], n_trials=1) assert {tuple(t.values) for t in study.get_pareto_front_trials()} == {(2, 2)} study.optimize(lambda t: [1, 1], n_trials=1) assert {tuple(t.values) for t in study.get_pareto_front_trials()} == {(1, 1), (2, 2)} study.optimize(lambda t: [3, 1], n_trials=1) assert {tuple(t.values) for t in study.get_pareto_front_trials()} == {(1, 1), (2, 2)} study.optimize(lambda t: [1, 3], n_trials=1) assert {tuple(t.values) for t in study.get_pareto_front_trials()} == {(1, 3)} assert len(study.get_pareto_front_trials()) == 1 study.optimize(lambda t: [1, 3], n_trials=1) # The trial result is the same as the above one. assert {tuple(t.values) for t in study.get_pareto_front_trials()} == {(1, 3)} assert len(study.get_pareto_front_trials()) == 2 def test_study_user_attrs() -> None: study = optuna.multi_objective.create_study(["minimize", "maximize"]) assert study.user_attrs == {} study.set_user_attr("foo", "bar") assert study.user_attrs == {"foo": "bar"} study.set_user_attr("baz", "qux") assert study.user_attrs == {"foo": "bar", "baz": "qux"} study.set_user_attr("foo", "quux") assert study.user_attrs == {"foo": "quux", "baz": "qux"} def test_enqueue_trial() -> None: study = optuna.multi_objective.create_study(["minimize", "maximize"]) study.enqueue_trial({"x": 2}) study.enqueue_trial({"x": 3}) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> List[float]: if trial.number == 0: assert trial.suggest_uniform("x", 0, 100) == 2 elif trial.number == 1: assert trial.suggest_uniform("x", 0, 100) == 3 return [0, 0] study.optimize(objective, n_trials=2) def test_callbacks() -> None: study = optuna.multi_objective.create_study(["minimize", "maximize"]) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> Tuple[float, float]: x = trial.suggest_float("x", 0, 10) y = trial.suggest_float("y", 0, 10) return x, y list0 = [] list1 = [] callbacks = [ lambda study, trial: list0.append(trial.number), lambda study, trial: list1.append(trial.number), ] study.optimize(objective, n_trials=2, callbacks=callbacks) assert list0 == [0, 1] assert list1 == [0, 1] def test_log_completed_trial(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.set_verbosity(optuna.logging.INFO) study = optuna.multi_objective.create_study(["minimize", "maximize"]) study.optimize(lambda t: (1.0, 1.0), n_trials=1) _, err = capsys.readouterr() assert "Trial 0" in err optuna.logging.set_verbosity(optuna.logging.WARNING) study.optimize(lambda t: (1.0, 1.0), n_trials=1) _, err = capsys.readouterr() assert "Trial 1" not in err optuna.logging.set_verbosity(optuna.logging.DEBUG) study.optimize(lambda t: (1.0, 1.0), n_trials=1) _, err = capsys.readouterr() assert "Trial 2" in err optuna-3.5.0/tests/multi_objective_tests/test_trial.py000066400000000000000000000166541453453102400233360ustar00rootroot00000000000000from __future__ import annotations import datetime from typing import List from typing import Tuple import pytest import optuna from optuna.multi_objective.trial import FrozenMultiObjectiveTrial from optuna.study._study_direction import StudyDirection from optuna.trial import TrialState pytestmark = pytest.mark.filterwarnings("ignore::FutureWarning") def test_suggest() -> None: study = optuna.multi_objective.create_study(["maximize", "maximize"]) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> Tuple[float, float]: p0 = trial.suggest_float("p0", -10, 10) p1 = trial.suggest_float("p1", 3, 5) p2 = trial.suggest_float("p2", 0.00001, 0.1, log=True) p3 = trial.suggest_float("p3", 100, 200, step=5) p4 = trial.suggest_int("p4", -20, -15) p5 = trial.suggest_categorical("p5", [7, 1, 100]) p6 = trial.suggest_float("p6", -10, 10, step=1.0) p7 = trial.suggest_int("p7", 1, 7, log=True) return ( p0 + p1 + p2, p3 + p4 + p5 + p6 + p7, ) study.optimize(objective, n_trials=10) def test_report() -> None: study = optuna.multi_objective.create_study(["maximize", "minimize", "maximize"]) def objective( trial: optuna.multi_objective.trial.MultiObjectiveTrial, ) -> Tuple[float, float, float]: if trial.number == 0: trial.report((1, 2, 3), 1) trial.report((10, 20, 30), 2) return 100, 200, 300 study.optimize(objective, n_trials=2) trial = study.trials[0] assert trial.intermediate_values == {1: (1, 2, 3), 2: (10, 20, 30)} assert trial.values == (100, 200, 300) assert trial.last_step == 2 trial = study.trials[1] assert trial.intermediate_values == {} assert trial.values == (100, 200, 300) assert trial.last_step is None def test_number() -> None: study = optuna.multi_objective.create_study(["maximize", "minimize", "maximize"]) def objective( trial: optuna.multi_objective.trial.MultiObjectiveTrial, number: int ) -> List[float]: assert trial.number == number return [0, 0, 0] for i in range(10): study.optimize(lambda t: objective(t, i), n_trials=1) for i, trial in enumerate(study.trials): assert trial.number == i def test_user_attrs() -> None: study = optuna.multi_objective.create_study(["maximize", "minimize", "maximize"]) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> List[float]: trial.set_user_attr("foo", "bar") assert trial.user_attrs == {"foo": "bar"} trial.set_user_attr("baz", "qux") assert trial.user_attrs == {"foo": "bar", "baz": "qux"} trial.set_user_attr("foo", "quux") assert trial.user_attrs == {"foo": "quux", "baz": "qux"} return [0, 0, 0] study.optimize(objective, n_trials=1) assert study.trials[0].user_attrs == {"foo": "quux", "baz": "qux"} def test_params_and_distributions() -> None: study = optuna.multi_objective.create_study(["maximize", "minimize", "maximize"]) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> List[float]: x = trial.suggest_uniform("x", 0, 10) assert set(trial.params.keys()) == {"x"} assert set(trial.distributions.keys()) == {"x"} assert isinstance(trial.distributions["x"], optuna.distributions.FloatDistribution) return [x, x, x] study.optimize(objective, n_trials=1) trial = study.trials[0] assert set(trial.params.keys()) == {"x"} assert set(trial.distributions.keys()) == {"x"} assert isinstance(trial.distributions["x"], optuna.distributions.FloatDistribution) def test_datetime() -> None: study = optuna.multi_objective.create_study(["maximize", "minimize", "maximize"]) def objective(trial: optuna.multi_objective.trial.MultiObjectiveTrial) -> List[float]: assert isinstance(trial.datetime_start, datetime.datetime) return [0, 0, 0] study.optimize(objective, n_trials=1) assert isinstance(study.trials[0].datetime_start, datetime.datetime) assert isinstance(study.trials[0].datetime_complete, datetime.datetime) def test_dominates() -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] def create_trial( values: List[float], state: TrialState = TrialState.COMPLETE ) -> FrozenMultiObjectiveTrial: n_objectives = len(values) trial = optuna.trial.FrozenTrial( state=state, intermediate_values=dict(enumerate(values)), # The following attributes aren't used in this test case. number=0, value=None, datetime_start=None, datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, trial_id=0, ) return FrozenMultiObjectiveTrial(n_objectives, trial) # The numbers of objectives for `t0` and `t1` don't match. with pytest.raises(ValueError): t0 = create_trial([1]) # One objective. t1 = create_trial([1, 2]) # Two objectives. t0._dominates(t1, directions) # The numbers of objectives and directions don't match. with pytest.raises(ValueError): t0 = create_trial([1]) # One objective. t1 = create_trial([1]) # One objective. t0._dominates(t1, directions) # `t0` dominates `t1`. t0 = create_trial([0, 2]) t1 = create_trial([1, 1]) assert t0._dominates(t1, directions) assert not t1._dominates(t0, directions) # `t0` dominates `t1`. t0 = create_trial([0, 1]) t1 = create_trial([1, 1]) assert t0._dominates(t1, directions) assert not t1._dominates(t0, directions) # `t0` and `t1` don't dominate each other. t0 = create_trial([1, 1]) t1 = create_trial([1, 1]) assert not t0._dominates(t1, directions) assert not t1._dominates(t0, directions) # `t0` and `t1` don't dominate each other. t0 = create_trial([0, 1]) t1 = create_trial([1, 2]) assert not t0._dominates(t1, directions) assert not t1._dominates(t0, directions) for t0_state in [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]: t0 = create_trial([1, 1], t0_state) for t1_state in [ TrialState.COMPLETE, TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED, ]: # If `t0` has not the COMPLETE state, it never dominates other trials. t1 = create_trial([0, 2], t1_state) assert not t0._dominates(t1, directions) if t1_state == TrialState.COMPLETE: # If `t0` isn't COMPLETE and `t1` is COMPLETE, `t1` dominates `t0`. assert t1._dominates(t0, directions) else: # If `t1` isn't COMPLETE, it doesn't dominate others. assert not t1._dominates(t0, directions) @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. study = optuna.multi_objective.create_study(["maximize"]) kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. study.optimize(lambda trial: [trial.suggest_int("x", -1, 1, *args)], n_trials=1) optuna-3.5.0/tests/multi_objective_tests/visualization_tests/000077500000000000000000000000001453453102400247215ustar00rootroot00000000000000optuna-3.5.0/tests/multi_objective_tests/visualization_tests/__init__.py000066400000000000000000000000001453453102400270200ustar00rootroot00000000000000optuna-3.5.0/tests/multi_objective_tests/visualization_tests/test_pareto_front.py000066400000000000000000000250731453453102400310430ustar00rootroot00000000000000import itertools from typing import List from typing import Optional from typing import Tuple import numpy as np import pytest import optuna from optuna.multi_objective.visualization import plot_pareto_front pytestmark = pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1], [1, 0]]) def test_plot_pareto_front_2d( include_dominated_trials: bool, axis_order: Optional[List[int]] ) -> None: # Test with no trial. study = optuna.multi_objective.create_study(["minimize", "minimize"]) figure = plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) assert len(figure.data) == 1 assert figure.data[0]["x"] == () assert figure.data[0]["y"] == () # Test with three trials. study.enqueue_trial({"x": 1, "y": 1}) study.enqueue_trial({"x": 1, "y": 0}) study.enqueue_trial({"x": 0, "y": 1}) study.optimize(lambda t: [t.suggest_int("x", 0, 1), t.suggest_int("y", 0, 1)], n_trials=3) figure = plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) assert len(figure.data) == 1 data: List[Tuple[int, ...]] if include_dominated_trials: # The last elements come from dominated trial that is enqueued firstly. data = [(1, 0, 1), (0, 1, 1)] if axis_order is None: assert figure.data[0]["x"] == data[0] assert figure.data[0]["y"] == data[1] else: assert figure.data[0]["x"] == data[axis_order[0]] assert figure.data[0]["y"] == data[axis_order[1]] else: data = [(1, 0), (0, 1)] if axis_order is None: assert figure.data[0]["x"] == data[0] assert figure.data[0]["y"] == data[1] else: assert figure.data[0]["x"] == data[axis_order[0]] assert figure.data[0]["y"] == data[axis_order[1]] titles = ["Objective {}".format(i) for i in range(2)] if axis_order is None: assert figure.layout.xaxis.title.text == titles[0] assert figure.layout.yaxis.title.text == titles[1] else: assert figure.layout.xaxis.title.text == titles[axis_order[0]] assert figure.layout.yaxis.title.text == titles[axis_order[1]] # Test with `names` argument. with pytest.raises(ValueError): plot_pareto_front(study, names=[], include_dominated_trials=include_dominated_trials) with pytest.raises(ValueError): plot_pareto_front(study, names=["Foo"], include_dominated_trials=include_dominated_trials) with pytest.raises(ValueError): plot_pareto_front( study, names=["Foo", "Bar", "Baz"], include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) names = ["Foo", "Bar"] figure = plot_pareto_front( study, names=names, include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) if axis_order is None: assert figure.layout.xaxis.title.text == names[0] assert figure.layout.yaxis.title.text == names[1] else: assert figure.layout.xaxis.title.text == names[axis_order[0]] assert figure.layout.yaxis.title.text == names[axis_order[1]] @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None] + list(itertools.permutations(range(3), 3))) def test_plot_pareto_front_3d( include_dominated_trials: bool, axis_order: Optional[List[int]] ) -> None: # Test with no trial. study = optuna.multi_objective.create_study(["minimize", "minimize", "minimize"]) figure = plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) assert len(figure.data) == 1 assert figure.data[0]["x"] == () assert figure.data[0]["y"] == () assert figure.data[0]["z"] == () # Test with three trials. study.enqueue_trial({"x": 1, "y": 1, "z": 1}) study.enqueue_trial({"x": 1, "y": 0, "z": 1}) study.enqueue_trial({"x": 1, "y": 1, "z": 0}) study.optimize( lambda t: [t.suggest_int("x", 0, 1), t.suggest_int("y", 0, 1), t.suggest_int("z", 0, 1)], n_trials=3, ) figure = plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) assert len(figure.data) == 1 data: List[Tuple[int, ...]] if include_dominated_trials: # The last elements come from dominated trial that is enqueued firstly. data = [(1, 1, 1), (0, 1, 1), (1, 0, 1)] if axis_order is None: assert figure.data[0]["x"] == data[0] assert figure.data[0]["y"] == data[1] assert figure.data[0]["z"] == data[2] else: assert figure.data[0]["x"] == data[axis_order[0]] assert figure.data[0]["y"] == data[axis_order[1]] assert figure.data[0]["z"] == data[axis_order[2]] else: data = [(1, 1), (0, 1), (1, 0)] if axis_order is None: assert figure.data[0]["x"] == data[0] assert figure.data[0]["y"] == data[1] assert figure.data[0]["z"] == data[2] else: assert figure.data[0]["x"] == data[axis_order[0]] assert figure.data[0]["y"] == data[axis_order[1]] assert figure.data[0]["z"] == data[axis_order[2]] titles = ["Objective {}".format(i) for i in range(3)] if axis_order is None: assert figure.layout.scene.xaxis.title.text == titles[0] assert figure.layout.scene.yaxis.title.text == titles[1] assert figure.layout.scene.zaxis.title.text == titles[2] else: assert figure.layout.scene.xaxis.title.text == titles[axis_order[0]] assert figure.layout.scene.yaxis.title.text == titles[axis_order[1]] assert figure.layout.scene.zaxis.title.text == titles[axis_order[2]] # Test with `names` argument. with pytest.raises(ValueError): plot_pareto_front( study, names=[], include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) with pytest.raises(ValueError): plot_pareto_front( study, names=["Foo"], include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) with pytest.raises(ValueError): plot_pareto_front( study, names=["Foo", "Bar"], include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) with pytest.raises(ValueError): plot_pareto_front( study, names=["Foo", "Bar", "Baz", "Qux"], include_dominated_trials=include_dominated_trials, axis_order=axis_order, ) names = ["Foo", "Bar", "Baz"] figure = plot_pareto_front(study, names=names, axis_order=axis_order) if axis_order is None: assert figure.layout.scene.xaxis.title.text == names[0] assert figure.layout.scene.yaxis.title.text == names[1] assert figure.layout.scene.zaxis.title.text == names[2] else: assert figure.layout.scene.xaxis.title.text == names[axis_order[0]] assert figure.layout.scene.yaxis.title.text == names[axis_order[1]] assert figure.layout.scene.zaxis.title.text == names[axis_order[2]] @pytest.mark.parametrize("include_dominated_trials", [False, True]) def test_plot_pareto_front_unsupported_dimensions(include_dominated_trials: bool) -> None: # Unsupported: n_objectives == 1. with pytest.raises(ValueError): study = optuna.multi_objective.create_study(["minimize"]) study.optimize(lambda t: [0], n_trials=1) plot_pareto_front(study, include_dominated_trials=include_dominated_trials) # Supported: n_objectives == 2. study = optuna.multi_objective.create_study(["minimize", "minimize"]) study.optimize(lambda t: [0, 0], n_trials=1) plot_pareto_front(study, include_dominated_trials=include_dominated_trials) # Supported: n_objectives == 3. study = optuna.multi_objective.create_study(["minimize", "minimize", "minimize"]) study.optimize(lambda t: [0, 0, 0], n_trials=1) plot_pareto_front(study, include_dominated_trials=include_dominated_trials) # Unsupported: n_objectives == 4. with pytest.raises(ValueError): study = optuna.multi_objective.create_study( ["minimize", "minimize", "minimize", "minimize"] ) study.optimize(lambda t: [0, 0, 0, 0], n_trials=1) plot_pareto_front(study, include_dominated_trials=include_dominated_trials) @pytest.mark.parametrize("dimension", [2, 3]) @pytest.mark.parametrize("include_dominated_trials", [False, True]) def test_plot_pareto_front_invalid_axis_order( dimension: int, include_dominated_trials: bool ) -> None: study = optuna.multi_objective.create_study(["minimize"] * dimension) # Invalid: len(axis_order) != dimension with pytest.raises(ValueError): invalid_axis_order = list(range(dimension + 1)) assert len(invalid_axis_order) != dimension plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=invalid_axis_order, ) # Invalid: np.unique(axis_order).size != dimension with pytest.raises(ValueError): invalid_axis_order = list(range(dimension)) invalid_axis_order[1] = invalid_axis_order[0] assert np.unique(invalid_axis_order).size != dimension plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=invalid_axis_order, ) # Invalid: max(axis_order) > (dimension - 1) with pytest.raises(ValueError): invalid_axis_order = list(range(dimension)) invalid_axis_order[-1] += 1 assert max(invalid_axis_order) > (dimension - 1) plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=invalid_axis_order, ) # Invalid: min(axis_order) < 0 with pytest.raises(ValueError): study = optuna.multi_objective.create_study(["minimize", "minimize"]) invalid_axis_order = list(range(dimension)) invalid_axis_order[0] -= 1 assert min(invalid_axis_order) < 0 plot_pareto_front( study, include_dominated_trials=include_dominated_trials, axis_order=invalid_axis_order, ) optuna-3.5.0/tests/pruners_tests/000077500000000000000000000000001453453102400171105ustar00rootroot00000000000000optuna-3.5.0/tests/pruners_tests/__init__.py000066400000000000000000000000001453453102400212070ustar00rootroot00000000000000optuna-3.5.0/tests/pruners_tests/test_hyperband.py000066400000000000000000000176571453453102400225150ustar00rootroot00000000000000from typing import Callable from unittest import mock import numpy import pytest import optuna MIN_RESOURCE = 1 MAX_RESOURCE = 16 REDUCTION_FACTOR = 2 N_BRACKETS = 4 EARLY_STOPPING_RATE_LOW = 0 EARLY_STOPPING_RATE_HIGH = 3 N_REPORTS = 10 EXPECTED_N_TRIALS_PER_BRACKET = 10 def test_hyperband_pruner_intermediate_values() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) def objective(trial: optuna.trial.Trial) -> float: for i in range(N_REPORTS): trial.report(i, step=i) return 1.0 study.optimize(objective, n_trials=N_BRACKETS * EXPECTED_N_TRIALS_PER_BRACKET) trials = study.trials assert len(trials) == N_BRACKETS * EXPECTED_N_TRIALS_PER_BRACKET def test_bracket_study() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) bracket_study = pruner._create_bracket_study(study, 0) with pytest.raises(AttributeError): bracket_study.optimize(lambda *args: 1.0) with pytest.raises(AttributeError): bracket_study.set_user_attr("abc", 100) for attr in ("user_attrs", "system_attrs"): with pytest.raises(AttributeError): getattr(bracket_study, attr) with pytest.raises(AttributeError): bracket_study.trials_dataframe() bracket_study.get_trials() bracket_study.direction bracket_study._storage bracket_study._study_id bracket_study.pruner bracket_study.study_name # As `_BracketStudy` is defined inside `HyperbandPruner`, # we cannot do `assert isinstance(bracket_study, _BracketStudy)`. # This is why the below line is ignored by mypy checks. bracket_study._bracket_id # type: ignore def test_hyperband_max_resource_is_auto() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) def objective(trial: optuna.trial.Trial) -> float: for i in range(N_REPORTS): trial.report(1.0, i) if trial.should_prune(): raise optuna.TrialPruned() return 1.0 study.optimize(objective, n_trials=N_BRACKETS * EXPECTED_N_TRIALS_PER_BRACKET) assert N_REPORTS == pruner._max_resource def test_hyperband_max_resource_value_error() -> None: with pytest.raises(ValueError): _ = optuna.pruners.HyperbandPruner(max_resource="not_appropriate") @pytest.mark.parametrize( "sampler_init_func", [ lambda: optuna.samplers.RandomSampler(), (lambda: optuna.samplers.TPESampler(n_startup_trials=1)), ( lambda: optuna.samplers.GridSampler( search_space={"value": numpy.linspace(0.0, 1.0, 8, endpoint=False).tolist()} ) ), (lambda: optuna.samplers.CmaEsSampler(n_startup_trials=1)), ], ) def test_hyperband_filter_study( sampler_init_func: Callable[[], optuna.samplers.BaseSampler] ) -> None: def objective(trial: optuna.trial.Trial) -> float: return trial.suggest_float("value", 0.0, 1.0) n_trials = 8 n_brackets = 4 expected_n_trials_per_bracket = n_trials // n_brackets with mock.patch( "optuna.pruners.HyperbandPruner._get_bracket_id", new=mock.Mock(side_effect=lambda study, trial: trial.number % n_brackets), ): for method_name in [ "infer_relative_search_space", "sample_relative", "sample_independent", ]: sampler = sampler_init_func() pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR, ) with mock.patch( "optuna.samplers.{}.{}".format(sampler.__class__.__name__, method_name), wraps=getattr(sampler, method_name), ) as method_mock: study = optuna.study.create_study(sampler=sampler, pruner=pruner) study.optimize(objective, n_trials=n_trials) args = method_mock.call_args[0] study = args[0] trials = study.get_trials() assert len(trials) == expected_n_trials_per_bracket @pytest.mark.parametrize( "pruner_init_func", [ lambda: optuna.pruners.NopPruner(), lambda: optuna.pruners.MedianPruner(), lambda: optuna.pruners.ThresholdPruner(lower=0.5), lambda: optuna.pruners.SuccessiveHalvingPruner(), ], ) def test_hyperband_no_filter_study( pruner_init_func: Callable[[], optuna.pruners.BasePruner] ) -> None: def objective(trial: optuna.trial.Trial) -> float: return trial.suggest_float("value", 0.0, 1.0) n_trials = 10 for method_name in [ "infer_relative_search_space", "sample_relative", "sample_independent", ]: sampler = optuna.samplers.RandomSampler() pruner = pruner_init_func() with mock.patch( "optuna.samplers.{}.{}".format(sampler.__class__.__name__, method_name), wraps=getattr(sampler, method_name), ) as method_mock: study = optuna.study.create_study(sampler=sampler, pruner=pruner) study.optimize(objective, n_trials=n_trials) args = method_mock.call_args[0] study = args[0] trials = study.get_trials() assert len(trials) == n_trials @pytest.mark.parametrize( "sampler_init_func", [ lambda: optuna.samplers.RandomSampler(), (lambda: optuna.samplers.TPESampler(n_startup_trials=1)), ( lambda: optuna.samplers.GridSampler( search_space={"value": numpy.linspace(0.0, 1.0, 10, endpoint=False).tolist()} ) ), (lambda: optuna.samplers.CmaEsSampler(n_startup_trials=1)), ], ) def test_hyperband_no_call_of_filter_study_in_should_prune( sampler_init_func: Callable[[], optuna.samplers.BaseSampler] ) -> None: def objective(trial: optuna.trial.Trial) -> float: with mock.patch("optuna.pruners._filter_study") as method_mock: for i in range(N_REPORTS): trial.report(i, step=i) if trial.should_prune(): method_mock.assert_not_called() raise optuna.TrialPruned() else: method_mock.assert_not_called() return 1.0 sampler = sampler_init_func() pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) study = optuna.study.create_study(sampler=sampler, pruner=pruner) study.optimize(objective, n_trials=10) def test_incompatibility_between_bootstrap_count_and_auto_max_resource() -> None: with pytest.raises(ValueError): optuna.pruners.HyperbandPruner(max_resource="auto", bootstrap_count=1) def test_hyperband_pruner_and_grid_sampler() -> None: pruner = optuna.pruners.HyperbandPruner( min_resource=MIN_RESOURCE, max_resource=MAX_RESOURCE, reduction_factor=REDUCTION_FACTOR ) search_space = {"x": [-50, 0, 50], "y": [-99, 0, 99]} sampler = optuna.samplers.GridSampler(search_space) study = optuna.study.create_study(sampler=sampler, pruner=pruner) def objective(trial: optuna.trial.Trial) -> float: for i in range(N_REPORTS): trial.report(i, step=i) x = trial.suggest_float("x", -100, 100) y = trial.suggest_int("y", -100, 100) return x**2 + y**2 study.optimize(objective, n_trials=10) trials = study.trials assert len(trials) == 9 optuna-3.5.0/tests/pruners_tests/test_median.py000066400000000000000000000112221453453102400217540ustar00rootroot00000000000000from typing import List from typing import Tuple import pytest import optuna def test_median_pruner_with_one_trial() -> None: pruner = optuna.pruners.MedianPruner(0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) # A pruner is not activated at a first trial. assert not trial.should_prune() @pytest.mark.parametrize("direction_value", [("minimize", 2), ("maximize", 0.5)]) def test_median_pruner_intermediate_values(direction_value: Tuple[str, float]) -> None: direction, intermediate_value = direction_value pruner = optuna.pruners.MedianPruner(0, 0) study = optuna.study.create_study(direction=direction, pruner=pruner) trial = study.ask() trial.report(1, 1) study.tell(trial, 1) trial = study.ask() # A pruner is not activated if a trial has no intermediate values. assert not trial.should_prune() trial.report(intermediate_value, 1) # A pruner is activated if a trial has an intermediate value. assert trial.should_prune() @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_median_pruner_intermediate_values_nan() -> None: pruner = optuna.pruners.MedianPruner(0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(float("nan"), 1) # A pruner is not activated if the study does not have any previous trials. assert not trial.should_prune() study.tell(trial, -1) # -1 is used because we can not tell with nan. trial = study.ask() trial.report(float("nan"), 1) # A pruner is activated if the best intermediate value of this trial is NaN. assert trial.should_prune() study.tell(trial, -1) # -1 is used because we can not tell with nan. trial = study.ask() trial.report(1, 1) # A pruner is not activated if the median intermediate value is NaN. assert not trial.should_prune() def test_median_pruner_n_startup_trials() -> None: pruner = optuna.pruners.MedianPruner(2, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) study.tell(trial, 1) trial = study.ask() trial.report(2, 1) # A pruner is not activated during startup trials. assert not trial.should_prune() study.tell(trial, 2) trial = study.ask() trial.report(3, 1) # A pruner is activated after startup trials. assert trial.should_prune() def test_median_pruner_n_warmup_steps() -> None: pruner = optuna.pruners.MedianPruner(0, 1) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 0) trial.report(1, 1) study.tell(trial, 1) trial = study.ask() trial.report(2, 0) # A pruner is not activated during warm-up steps. assert not trial.should_prune() trial.report(2, 1) # A pruner is activated after warm-up steps. assert trial.should_prune() @pytest.mark.parametrize( "n_warmup_steps,interval_steps,report_steps,expected_prune_steps", [ (0, 1, 1, [0, 1, 2, 3, 4, 5]), (1, 1, 1, [1, 2, 3, 4, 5]), (1, 2, 1, [1, 3, 5]), (0, 3, 10, list(range(29))), (2, 3, 10, list(range(10, 29))), (0, 10, 3, [0, 1, 2, 12, 13, 14, 21, 22, 23]), (2, 10, 3, [3, 4, 5, 12, 13, 14, 24, 25, 26]), ], ) def test_median_pruner_interval_steps( n_warmup_steps: int, interval_steps: int, report_steps: int, expected_prune_steps: List[int] ) -> None: pruner = optuna.pruners.MedianPruner(0, n_warmup_steps, interval_steps) study = optuna.study.create_study(pruner=pruner) trial = study.ask() last_step = max(expected_prune_steps) + 1 for i in range(last_step): trial.report(0, i) study.tell(trial, 0) trial = study.ask() pruned = [] for i in range(last_step): if i % report_steps == 0: trial.report(2, i) if trial.should_prune(): pruned.append(i) assert pruned == expected_prune_steps def test_median_pruner_n_min_trials() -> None: pruner = optuna.pruners.MedianPruner(2, 0, 1, n_min_trials=2) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(4, 1) trial.report(2, 2) study.tell(trial, 2) trial = study.ask() trial.report(3, 1) study.tell(trial, 3) trial = study.ask() trial.report(4, 1) trial.report(3, 2) # A pruner is not activated before the values at step 2 observed n_min_trials times. assert not trial.should_prune() study.tell(trial, 3) trial = study.ask() trial.report(4, 1) trial.report(3, 2) # A pruner is activated after the values at step 2 observed n_min_trials times. assert trial.should_prune() optuna-3.5.0/tests/pruners_tests/test_nop.py000066400000000000000000000004221453453102400213130ustar00rootroot00000000000000import optuna def test_nop_pruner() -> None: pruner = optuna.pruners.NopPruner() study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) # A NopPruner instance is always deactivated. assert not trial.should_prune() optuna-3.5.0/tests/pruners_tests/test_patient.py000066400000000000000000000053411453453102400221700ustar00rootroot00000000000000from typing import List import pytest import optuna def test_patient_pruner_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.pruners.PatientPruner(None, 0) def test_patient_pruner_patience() -> None: optuna.pruners.PatientPruner(None, 0) optuna.pruners.PatientPruner(None, 1) with pytest.raises(ValueError): optuna.pruners.PatientPruner(None, -1) def test_patient_pruner_min_delta() -> None: optuna.pruners.PatientPruner(None, 0, 0.0) optuna.pruners.PatientPruner(None, 0, 1.0) with pytest.raises(ValueError): optuna.pruners.PatientPruner(None, 0, -1) def test_patient_pruner_with_one_trial() -> None: pruner = optuna.pruners.PatientPruner(None, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 0) # The pruner is not activated at a first trial. assert not trial.should_prune() @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_patient_pruner_intermediate_values_nan() -> None: pruner = optuna.pruners.PatientPruner(None, 0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() # A pruner is not activated if a trial does not have any intermediate values. assert not trial.should_prune() trial.report(float("nan"), 0) # A pruner is not activated if a trial has only one intermediate value. assert not trial.should_prune() trial.report(1.0, 1) # A pruner is not activated if a trial has only nan in intermediate values. assert not trial.should_prune() trial.report(float("nan"), 2) # A pruner is not activated if a trial has only nan in intermediate values. assert not trial.should_prune() @pytest.mark.parametrize( "patience,min_delta,direction,intermediates,expected_prune_steps", [ (0, 0, "maximize", [1, 0], [1]), (1, 0, "maximize", [2, 1, 0], [2]), (0, 0, "minimize", [0, 1], [1]), (1, 0, "minimize", [0, 1, 2], [2]), (0, 1.0, "maximize", [1, 0], []), (1, 1.0, "maximize", [3, 2, 1, 0], [3]), (0, 1.0, "minimize", [0, 1], []), (1, 1.0, "minimize", [0, 1, 2, 3], [3]), ], ) def test_patient_pruner_intermediate_values( patience: int, min_delta: float, direction: str, intermediates: List[int], expected_prune_steps: List[int], ) -> None: pruner = optuna.pruners.PatientPruner(None, patience, min_delta) study = optuna.study.create_study(pruner=pruner, direction=direction) trial = study.ask() pruned = [] for step, value in enumerate(intermediates): trial.report(value, step) if trial.should_prune(): pruned.append(step) assert pruned == expected_prune_steps optuna-3.5.0/tests/pruners_tests/test_percentile.py000066400000000000000000000205711453453102400226600ustar00rootroot00000000000000import math from typing import List from typing import Tuple import warnings import pytest import optuna from optuna.pruners import _percentile from optuna.study import Study from optuna.study import StudyDirection from optuna.trial import TrialState def test_percentile_pruner_percentile() -> None: optuna.pruners.PercentilePruner(0.0) optuna.pruners.PercentilePruner(25.0) optuna.pruners.PercentilePruner(100.0) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(-0.1) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(100.1) def test_percentile_pruner_n_startup_trials() -> None: optuna.pruners.PercentilePruner(25.0, n_startup_trials=0) optuna.pruners.PercentilePruner(25.0, n_startup_trials=5) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, n_startup_trials=-1) def test_percentile_pruner_n_warmup_steps() -> None: optuna.pruners.PercentilePruner(25.0, n_warmup_steps=0) optuna.pruners.PercentilePruner(25.0, n_warmup_steps=5) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, n_warmup_steps=-1) def test_percentile_pruner_interval_steps() -> None: optuna.pruners.PercentilePruner(25.0, interval_steps=1) optuna.pruners.PercentilePruner(25.0, interval_steps=5) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, interval_steps=-1) with pytest.raises(ValueError): optuna.pruners.PercentilePruner(25.0, interval_steps=0) def test_percentile_pruner_with_one_trial() -> None: pruner = optuna.pruners.PercentilePruner(25.0, 0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, 1) # A pruner is not activated at a first trial. assert not trial.should_prune() @pytest.mark.parametrize( "direction_value", [("minimize", [1, 2, 3, 4, 5], 2.1), ("maximize", [1, 2, 3, 4, 5], 3.9)] ) def test_25_percentile_pruner_intermediate_values( direction_value: Tuple[str, List[float], float] ) -> None: direction, intermediate_values, latest_value = direction_value pruner = optuna.pruners.PercentilePruner(25.0, 0, 0) study = optuna.study.create_study(direction=direction, pruner=pruner) for v in intermediate_values: trial = study.ask() trial.report(v, 1) study.tell(trial, v) trial = study.ask() # A pruner is not activated if a trial has no intermediate values. assert not trial.should_prune() trial.report(latest_value, 1) # A pruner is activated if a trial has an intermediate value. assert trial.should_prune() @pytest.mark.filterwarnings("ignore::RuntimeWarning") def test_25_percentile_pruner_intermediate_values_nan() -> None: pruner = optuna.pruners.PercentilePruner(25.0, 0, 0) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(float("nan"), 1) # A pruner is not activated if the study does not have any previous trials. assert not trial.should_prune() study.tell(trial, -1) trial = study.ask() trial.report(float("nan"), 1) # A pruner is activated if the best intermediate value of this trial is NaN. assert trial.should_prune() study.tell(trial, -1) trial = study.ask() trial.report(1, 1) # A pruner is not activated if the 25 percentile intermediate value is NaN. assert not trial.should_prune() @pytest.mark.parametrize( "direction_expected", [(StudyDirection.MINIMIZE, 0.1), (StudyDirection.MAXIMIZE, 0.2)] ) def test_get_best_intermediate_result_over_steps( direction_expected: Tuple[StudyDirection, float] ) -> None: direction, expected = direction_expected if direction == StudyDirection.MINIMIZE: study = optuna.study.create_study(direction="minimize") else: study = optuna.study.create_study(direction="maximize") # FrozenTrial.intermediate_values has no elements. trial_id_empty = study._storage.create_new_trial(study._study_id) trial_empty = study._storage.get_trial(trial_id_empty) with pytest.raises(ValueError): _percentile._get_best_intermediate_result_over_steps(trial_empty, direction) # Input value has no NaNs but float values. trial_id_float = study._storage.create_new_trial(study._study_id) trial_float = optuna.trial.Trial(study, trial_id_float) trial_float.report(0.1, step=0) trial_float.report(0.2, step=1) frozen_trial_float = study._storage.get_trial(trial_id_float) assert expected == _percentile._get_best_intermediate_result_over_steps( frozen_trial_float, direction ) # Input value has a float value and a NaN. trial_id_float_nan = study._storage.create_new_trial(study._study_id) trial_float_nan = optuna.trial.Trial(study, trial_id_float_nan) trial_float_nan.report(0.3, step=0) trial_float_nan.report(float("nan"), step=1) frozen_trial_float_nan = study._storage.get_trial(trial_id_float_nan) assert 0.3 == _percentile._get_best_intermediate_result_over_steps( frozen_trial_float_nan, direction ) # Input value has a NaN only. trial_id_nan = study._storage.create_new_trial(study._study_id) trial_nan = optuna.trial.Trial(study, trial_id_nan) trial_nan.report(float("nan"), step=0) frozen_trial_nan = study._storage.get_trial(trial_id_nan) with warnings.catch_warnings(): warnings.simplefilter("ignore", category=RuntimeWarning) assert math.isnan( _percentile._get_best_intermediate_result_over_steps(frozen_trial_nan, direction) ) def test_get_percentile_intermediate_result_over_trials() -> None: def setup_study(trial_num: int, _intermediate_values: List[List[float]]) -> Study: _study = optuna.study.create_study(direction="minimize") trial_ids = [_study._storage.create_new_trial(_study._study_id) for _ in range(trial_num)] for step, values in enumerate(_intermediate_values): # Study does not have any complete trials. with pytest.raises(ValueError): completed_trials = _study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) _direction = _study.direction _percentile._get_percentile_intermediate_result_over_trials( completed_trials, _direction, step, 25, 1 ) for i in range(trial_num): trial_id = trial_ids[i] value = values[i] _study._storage.set_trial_intermediate_value(trial_id, step, value) # Set trial states complete because this method ignores incomplete trials. for trial_id in trial_ids: _study._storage.set_trial_state_values(trial_id, state=TrialState.COMPLETE) return _study # Input value has no NaNs but float values (step=0). intermediate_values = [[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]] study = setup_study(9, intermediate_values) all_trials = study.get_trials() direction = study.direction assert 0.3 == _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 0, 25.0, 1 ) # Input value has a float value and NaNs (step=1). intermediate_values.append( [0.1, 0.2, 0.3, 0.4, 0.5, float("nan"), float("nan"), float("nan"), float("nan")] ) study = setup_study(9, intermediate_values) all_trials = study.get_trials() direction = study.direction assert 0.2 == _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 1, 25.0, 1 ) # Input value has NaNs only (step=2). intermediate_values.append( [ float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), float("nan"), ] ) study = setup_study(9, intermediate_values) all_trials = study.get_trials() direction = study.direction with warnings.catch_warnings(): warnings.simplefilter("ignore", category=RuntimeWarning) assert math.isnan( _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 2, 75, 1 ) ) # n_min_trials = 2. assert math.isnan( _percentile._get_percentile_intermediate_result_over_trials( all_trials, direction, 2, 75, 2 ) ) optuna-3.5.0/tests/pruners_tests/test_successive_halving.py000066400000000000000000000242421453453102400244110ustar00rootroot00000000000000from typing import Tuple import pytest import optuna @pytest.mark.parametrize("direction_value", [("minimize", 2), ("maximize", 0.5)]) def test_successive_halving_pruner_intermediate_values(direction_value: Tuple[str, float]) -> None: direction, intermediate_value = direction_value pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(direction=direction, pruner=pruner) trial = study.ask() trial.report(1, 1) # A pruner is not activated at a first trial. assert not trial.should_prune() trial = study.ask() # A pruner is not activated if a trial has no intermediate values. assert not trial.should_prune() trial.report(intermediate_value, 1) # A pruner is activated if a trial has an intermediate value. assert trial.should_prune() def test_successive_halving_pruner_rung_check() -> None: pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) # Report 7 trials in advance. for i in range(7): trial = study.ask() trial.report(0.1 * (i + 1), step=7) pruner.prune(study=study, trial=study._storage.get_trial(trial._trial_id)) # Report a trial that has the 7-th value from bottom. trial = study.ask() trial.report(0.75, step=7) pruner.prune(study=study, trial=study._storage.get_trial(trial._trial_id)) trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs # Report a trial that has the third value from bottom. trial = study.ask() trial.report(0.25, step=7) trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_1" in trial_system_attrs assert "completed_rung_2" not in trial_system_attrs # Report a trial that has the lowest value. trial = study.ask() trial.report(0.05, step=7) trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_2" in trial_system_attrs assert "completed_rung_3" not in trial_system_attrs def test_successive_halving_pruner_first_trial_is_not_pruned() -> None: pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() for i in range(10): trial.report(1, step=i) # The first trial is not pruned. assert not trial.should_prune() # The trial completed until rung 3. trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" in trial_system_attrs assert "completed_rung_2" in trial_system_attrs assert "completed_rung_3" in trial_system_attrs assert "completed_rung_4" not in trial_system_attrs def test_successive_halving_pruner_with_nan() -> None: pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=2, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = optuna.trial.Trial(study, study._storage.create_new_trial(study._study_id)) # A pruner is not activated if the step is not a rung completion point. trial.report(float("nan"), step=1) assert not trial.should_prune() # A pruner is activated if the step is a rung completion point and # the intermediate value is NaN. trial.report(float("nan"), step=2) assert trial.should_prune() @pytest.mark.parametrize("n_reports", range(3)) @pytest.mark.parametrize("n_trials", [1, 2]) def test_successive_halving_pruner_with_auto_min_resource(n_reports: int, n_trials: int) -> None: pruner = optuna.pruners.SuccessiveHalvingPruner(min_resource="auto") study = optuna.study.create_study(sampler=optuna.samplers.RandomSampler(), pruner=pruner) assert pruner._min_resource is None def objective(trial: optuna.trial.Trial) -> float: for i in range(n_reports): trial.report(1.0 / (i + 1), i) if trial.should_prune(): raise optuna.TrialPruned() return 1.0 study.optimize(objective, n_trials=n_trials) if n_reports > 0 and n_trials > 1: assert pruner._min_resource is not None and pruner._min_resource > 0 else: assert pruner._min_resource is None def test_successive_halving_pruner_with_invalid_str_to_min_resource() -> None: with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner(min_resource="fixed") def test_successive_halving_pruner_min_resource_parameter() -> None: # min_resource=0: Error (must be `min_resource >= 1`). with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner( min_resource=0, reduction_factor=2, min_early_stopping_rate=0 ) # min_resource=1: The rung 0 ends at step 1. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs # min_resource=2: The rung 0 ends at step 2. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=2, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" not in trial_system_attrs trial.report(1, step=2) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs def test_successive_halving_pruner_reduction_factor_parameter() -> None: # reduction_factor=1: Error (must be `reduction_factor >= 2`). with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=1, min_early_stopping_rate=0 ) # reduction_factor=2: The rung 0 ends at step 1. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs # reduction_factor=3: The rung 1 ends at step 3. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=3, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs trial.report(1, step=2) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_1" not in trial_system_attrs trial.report(1, step=3) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_1" in trial_system_attrs assert "completed_rung_2" not in trial_system_attrs def test_successive_halving_pruner_min_early_stopping_rate_parameter() -> None: # min_early_stopping_rate=-1: Error (must be `min_early_stopping_rate >= 0`). with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=-1 ) # min_early_stopping_rate=0: The rung 0 ends at step 1. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=0 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs # min_early_stopping_rate=1: The rung 0 ends at step 2. pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, min_early_stopping_rate=1 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1, step=1) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" not in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs trial.report(1, step=2) assert not trial.should_prune() trial_system_attrs = trial.storage.get_trial_system_attrs(trial._trial_id) assert "completed_rung_0" in trial_system_attrs assert "completed_rung_1" not in trial_system_attrs def test_successive_halving_pruner_bootstrap_parameter() -> None: with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner(bootstrap_count=-1) with pytest.raises(ValueError): optuna.pruners.SuccessiveHalvingPruner(bootstrap_count=1, min_resource="auto") pruner = optuna.pruners.SuccessiveHalvingPruner( min_resource=1, reduction_factor=2, bootstrap_count=1 ) study = optuna.study.create_study(pruner=pruner) trial1 = study.ask() trial2 = study.ask() trial1.report(1, step=1) assert trial1.should_prune() trial2.report(1, step=1) assert not trial2.should_prune() optuna-3.5.0/tests/pruners_tests/test_threshold.py000066400000000000000000000056621453453102400225260ustar00rootroot00000000000000import pytest import optuna def test_threshold_pruner_with_ub() -> None: pruner = optuna.pruners.ThresholdPruner(upper=2.0, n_warmup_steps=0, interval_steps=1) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(1.0, 1) assert not trial.should_prune() trial.report(3.0, 2) assert trial.should_prune() def test_threshold_pruner_with_lt() -> None: pruner = optuna.pruners.ThresholdPruner(lower=2.0, n_warmup_steps=0, interval_steps=1) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(3.0, 1) assert not trial.should_prune() trial.report(1.0, 2) assert trial.should_prune() def test_threshold_pruner_with_two_side() -> None: pruner = optuna.pruners.ThresholdPruner( lower=0.0, upper=1.0, n_warmup_steps=0, interval_steps=1 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(-0.1, 1) assert trial.should_prune() trial.report(0.0, 2) assert not trial.should_prune() trial.report(0.4, 3) assert not trial.should_prune() trial.report(1.0, 4) assert not trial.should_prune() trial.report(1.1, 5) assert trial.should_prune() def test_threshold_pruner_with_invalid_inputs() -> None: with pytest.raises(TypeError): optuna.pruners.ThresholdPruner(lower="val", upper=1.0) # type: ignore with pytest.raises(TypeError): optuna.pruners.ThresholdPruner(lower=0.0, upper="val") # type: ignore with pytest.raises(TypeError): optuna.pruners.ThresholdPruner(lower=None, upper=None) def test_threshold_pruner_with_nan() -> None: pruner = optuna.pruners.ThresholdPruner( lower=0.0, upper=1.0, n_warmup_steps=0, interval_steps=1 ) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(float("nan"), 1) assert trial.should_prune() def test_threshold_pruner_n_warmup_steps() -> None: pruner = optuna.pruners.ThresholdPruner(lower=0.0, upper=1.0, n_warmup_steps=2) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(-10.0, 0) assert not trial.should_prune() trial.report(100.0, 1) assert not trial.should_prune() trial.report(-100.0, 3) assert trial.should_prune() trial.report(1.0, 4) assert not trial.should_prune() trial.report(1000.0, 5) assert trial.should_prune() def test_threshold_pruner_interval_steps() -> None: pruner = optuna.pruners.ThresholdPruner(lower=0.0, upper=1.0, interval_steps=2) study = optuna.study.create_study(pruner=pruner) trial = study.ask() trial.report(-10.0, 0) assert trial.should_prune() trial.report(100.0, 1) assert not trial.should_prune() trial.report(-100.0, 2) assert trial.should_prune() trial.report(10.0, 3) assert not trial.should_prune() trial.report(1000.0, 4) assert trial.should_prune() optuna-3.5.0/tests/samplers_tests/000077500000000000000000000000001453453102400172405ustar00rootroot00000000000000optuna-3.5.0/tests/samplers_tests/__init__.py000066400000000000000000000000001453453102400213370ustar00rootroot00000000000000optuna-3.5.0/tests/samplers_tests/test_brute_force.py000066400000000000000000000224461453453102400231600ustar00rootroot00000000000000import numpy as np import pytest import optuna from optuna import samplers from optuna.samplers._brute_force import _TreeNode from optuna.trial import Trial def test_tree_node_add_paths() -> None: tree = _TreeNode() leafs = [ tree.add_path([("a", [0, 1, 2], 0), ("b", [0.0, 1.0], 0.0)]), tree.add_path([("a", [0, 1, 2], 0), ("b", [0.0, 1.0], 1.0)]), tree.add_path([("a", [0, 1, 2], 0), ("b", [0.0, 1.0], 1.0)]), tree.add_path([("a", [0, 1, 2], 1), ("b", [0.0, 1.0], 0.0), ("c", [0, 1], 0)]), tree.add_path([("a", [0, 1, 2], 1), ("b", [0.0, 1.0], 0.0)]), ] for leaf in leafs: assert leaf is not None if leaf.children is None: leaf.set_leaf() assert tree == _TreeNode( param_name="a", children={ 0: _TreeNode( param_name="b", children={ 0.0: _TreeNode(param_name=None, children={}), 1.0: _TreeNode(param_name=None, children={}), }, ), 1: _TreeNode( param_name="b", children={ 0.0: _TreeNode( param_name="c", children={ 0: _TreeNode(param_name=None, children={}), 1: _TreeNode(), }, ), 1.0: _TreeNode(), }, ), 2: _TreeNode(), }, ) def test_tree_node_add_paths_error() -> None: with pytest.raises(ValueError): tree = _TreeNode() tree.add_path([("a", [0, 1, 2], 0)]) tree.add_path([("a", [0, 1], 0)]) with pytest.raises(ValueError): tree = _TreeNode() tree.add_path([("a", [0, 1, 2], 0)]) tree.add_path([("b", [0, 1, 2], 0)]) def test_tree_node_count_unexpanded() -> None: tree = _TreeNode( param_name="a", children={ 0: _TreeNode( param_name="b", children={ 0.0: _TreeNode(param_name=None, children={}), 1.0: _TreeNode(param_name=None, children={}), }, ), 1: _TreeNode( param_name="b", children={ 0.0: _TreeNode( param_name="c", children={ 0: _TreeNode(param_name=None, children={}), 1: _TreeNode(), }, ), 1.0: _TreeNode(), }, ), 2: _TreeNode(), }, ) assert tree.count_unexpanded() == 3 def test_study_optimize_with_single_search_space() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 2) if a == 0: b = trial.suggest_float("b", -1.0, 1.0, step=0.5) return a + b elif a == 1: c = trial.suggest_categorical("c", ["x", "y", None]) if c == "x": return a + 1 else: return a - 1 else: return a * 2 study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective) expected_suggested_values = [ {"a": 0, "b": -1.0}, {"a": 0, "b": -0.5}, {"a": 0, "b": 0.0}, {"a": 0, "b": 0.5}, {"a": 0, "b": 1.0}, {"a": 1, "c": "x"}, {"a": 1, "c": "y"}, {"a": 1, "c": None}, {"a": 2}, ] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in all_suggested_values: assert a in expected_suggested_values def test_study_optimize_with_pruned_trials() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 2) if a == 0: trial.suggest_float("b", -1.0, 1.0, step=0.5) raise optuna.TrialPruned elif a == 1: c = trial.suggest_categorical("c", ["x", "y", None]) if c == "x": return a + 1 else: return a - 1 else: return a * 2 study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective) expected_suggested_values = [ {"a": 0, "b": -1.0}, {"a": 0, "b": -0.5}, {"a": 0, "b": 0.0}, {"a": 0, "b": 0.5}, {"a": 0, "b": 1.0}, {"a": 1, "c": "x"}, {"a": 1, "c": "y"}, {"a": 1, "c": None}, {"a": 2}, ] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in all_suggested_values: assert a in expected_suggested_values def test_study_optimize_with_infinite_search_space() -> None: def objective(trial: Trial) -> float: return trial.suggest_float("a", 0, 2) study = optuna.create_study(sampler=samplers.BruteForceSampler()) with pytest.raises(ValueError): study.optimize(objective) def test_study_optimize_with_nan() -> None: def objective(trial: Trial) -> float: trial.suggest_categorical("a", [0.0, float("nan")]) return 1.0 study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective) all_suggested_values = [t.params["a"] for t in study.trials] assert len(all_suggested_values) == 2 assert 0.0 in all_suggested_values assert np.isnan(all_suggested_values[0]) or np.isnan(all_suggested_values[1]) def test_study_optimize_with_single_search_space_user_added() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 2) if a == 0: b = trial.suggest_float("b", -1.0, 1.0, step=0.5) return a + b elif a == 1: c = trial.suggest_categorical("c", ["x", "y", None]) if c == "x": return a + 1 else: return a - 1 else: return a * 2 study = optuna.create_study(sampler=samplers.BruteForceSampler()) # Manually add a trial. This should not be tried again. study.add_trial( optuna.create_trial( params={"a": 0, "b": -1.0}, value=0.0, distributions={ "a": optuna.distributions.IntDistribution(0, 2), "b": optuna.distributions.FloatDistribution(-1.0, 1.0, step=0.5), }, ) ) study.optimize(objective) expected_suggested_values = [ {"a": 0, "b": -1.0}, {"a": 0, "b": -0.5}, {"a": 0, "b": 0.0}, {"a": 0, "b": 0.5}, {"a": 0, "b": 1.0}, {"a": 1, "c": "x"}, {"a": 1, "c": "y"}, {"a": 1, "c": None}, {"a": 2}, ] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in all_suggested_values: assert a in expected_suggested_values def test_study_optimize_with_nonconstant_search_space() -> None: def objective_nonconstant_range(trial: Trial) -> float: x = trial.suggest_int("x", -1, trial.number) return x study = optuna.create_study(sampler=samplers.BruteForceSampler()) with pytest.raises(ValueError): study.optimize(objective_nonconstant_range, n_trials=10) def objective_increasing_variable(trial: Trial) -> float: return sum(trial.suggest_int(f"x{i}", 0, 0) for i in range(2)) study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.add_trial( optuna.create_trial( params={"x0": 0}, value=0.0, distributions={"x0": optuna.distributions.IntDistribution(0, 0)}, ) ) with pytest.raises(ValueError): study.optimize(objective_increasing_variable, n_trials=10) def objective_decreasing_variable(trial: Trial) -> float: return trial.suggest_int("x0", 0, 0) study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.add_trial( optuna.create_trial( params={"x0": 0, "x1": 0}, value=0.0, distributions={ "x0": optuna.distributions.IntDistribution(0, 0), "x1": optuna.distributions.IntDistribution(0, 0), }, ) ) with pytest.raises(ValueError): study.optimize(objective_decreasing_variable, n_trials=10) def test_study_optimize_with_failed_trials() -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", 0, 99) # NOQA[F811] return np.nan study = optuna.create_study(sampler=samplers.BruteForceSampler()) study.optimize(objective, n_trials=100) expected_suggested_values = [{"x": i} for i in range(100)] all_suggested_values = [t.params for t in study.trials] assert len(all_suggested_values) == len(expected_suggested_values) for a in expected_suggested_values: assert a in all_suggested_values def test_parallel_optimize() -> None: study = optuna.create_study(sampler=samplers.BruteForceSampler()) trial1 = study.ask() trial2 = study.ask() x1 = trial1.suggest_categorical("x", ["a", "b"]) x2 = trial2.suggest_categorical("x", ["a", "b"]) assert {x1, x2} == {"a", "b"} optuna-3.5.0/tests/samplers_tests/test_cmaes.py000066400000000000000000000701531453453102400217470ustar00rootroot00000000000000import math from typing import Any from typing import Dict from typing import List from typing import Optional from unittest.mock import MagicMock from unittest.mock import Mock from unittest.mock import patch import warnings import _pytest.capture from cmaes import CMA from cmaes import CMAwM from cmaes import SepCMA import numpy as np import pytest import optuna from optuna import create_trial from optuna._transform import _SearchSpaceTransform from optuna.testing.storages import StorageSupplier from optuna.trial import FrozenTrial from optuna.trial import TrialState def test_consider_pruned_trials_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.CmaEsSampler(consider_pruned_trials=True) def test_with_margin_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.CmaEsSampler(with_margin=True) def test_lr_adapt_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.CmaEsSampler(lr_adapt=True) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize( "use_separable_cma, cma_class_str", [(False, "optuna.samplers._cmaes.cmaes.CMA"), (True, "optuna.samplers._cmaes.cmaes.SepCMA")], ) @pytest.mark.parametrize("popsize", [None, 8]) def test_init_cmaes_opts( use_separable_cma: bool, cma_class_str: str, popsize: Optional[int] ) -> None: sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, use_separable_cma=use_separable_cma, popsize=popsize, ) study = optuna.create_study(sampler=sampler) with patch(cma_class_str) as cma_class: cma_obj = MagicMock() cma_obj.ask.return_value = np.array((-1, -1)) cma_obj.generation = 0 cma_class.return_value = cma_obj study.optimize( lambda t: t.suggest_float("x", -1, 1) + t.suggest_float("y", -1, 1), n_trials=2 ) assert cma_class.call_count == 1 _, actual_kwargs = cma_class.call_args assert np.array_equal(actual_kwargs["mean"], np.array([0.5, 0.5])) assert actual_kwargs["sigma"] == 0.1 assert np.allclose(actual_kwargs["bounds"], np.array([(0, 1), (0, 1)])) assert actual_kwargs["seed"] == np.random.RandomState(1).randint(1, np.iinfo(np.int32).max) assert actual_kwargs["n_max_resampling"] == 10 * 2 expected_popsize = 4 + math.floor(3 * math.log(2)) if popsize is None else popsize assert actual_kwargs["population_size"] == expected_popsize @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("popsize", [None, 8]) def test_init_cmaes_opts_with_margin(popsize: Optional[int]) -> None: sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, popsize=popsize, with_margin=True, ) study = optuna.create_study(sampler=sampler) with patch("optuna.samplers._cmaes.cmaes.CMAwM") as cma_class: cma_obj = MagicMock() cma_obj.ask.return_value = np.array((-1, -1)) cma_obj.generation = 0 cma_class.return_value = cma_obj study.optimize( lambda t: t.suggest_float("x", -1, 1) + t.suggest_int("y", -1, 1), n_trials=2 ) assert cma_class.call_count == 1 _, actual_kwargs = cma_class.call_args assert np.array_equal(actual_kwargs["mean"], np.array([0.5, 0.5])) assert actual_kwargs["sigma"] == 0.1 assert np.allclose(actual_kwargs["bounds"], np.array([(0, 1), (0, 1)])) assert np.allclose(actual_kwargs["steps"], np.array([0.0, 0.5])) assert actual_kwargs["seed"] == np.random.RandomState(1).randint(1, np.iinfo(np.int32).max) assert actual_kwargs["n_max_resampling"] == 10 * 2 expected_popsize = 4 + math.floor(3 * math.log(2)) if popsize is None else popsize assert actual_kwargs["population_size"] == expected_popsize @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("popsize", [None, 8]) def test_init_cmaes_opts_lr_adapt(popsize: Optional[int]) -> None: sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, popsize=popsize, lr_adapt=True, ) study = optuna.create_study(sampler=sampler) with patch("optuna.samplers._cmaes.cmaes.CMA") as cma_class: cma_obj = MagicMock() cma_obj.ask.return_value = np.array((-1, -1)) cma_obj.generation = 0 cma_class.return_value = cma_obj study.optimize( lambda t: t.suggest_float("x", -1, 1) + t.suggest_float("y", -1, 1), n_trials=2 ) assert cma_class.call_count == 1 _, actual_kwargs = cma_class.call_args assert actual_kwargs["lr_adapt"] is True @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) def test_warm_starting_cmaes(with_margin: bool) -> None: def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y source_study = optuna.create_study() source_study.optimize(objective, 20) source_trials = source_study.get_trials(deepcopy=False) with patch("optuna.samplers._cmaes.cmaes.get_warm_start_mgd") as mock_func_ws: mock_func_ws.return_value = (np.zeros(2), 0.0, np.zeros((2, 2))) sampler = optuna.samplers.CmaEsSampler( seed=1, n_startup_trials=1, with_margin=with_margin, source_trials=source_trials ) study = optuna.create_study(sampler=sampler) study.optimize(objective, 2) assert mock_func_ws.call_count == 1 @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) def test_warm_starting_cmaes_maximize(with_margin: bool) -> None: def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) # Objective values are negative. return -(x**2) - (y - 5) ** 2 source_study = optuna.create_study(direction="maximize") source_study.optimize(objective, 20) source_trials = source_study.get_trials(deepcopy=False) with patch("optuna.samplers._cmaes.cmaes.get_warm_start_mgd") as mock_func_ws: mock_func_ws.return_value = (np.zeros(2), 0.0, np.zeros((2, 2))) sampler = optuna.samplers.CmaEsSampler( seed=1, n_startup_trials=1, with_margin=with_margin, source_trials=source_trials ) study = optuna.create_study(sampler=sampler, direction="maximize") study.optimize(objective, 2) assert mock_func_ws.call_count == 1 solutions_arg = mock_func_ws.call_args[0][0] is_positive = [x[1] >= 0 for x in solutions_arg] assert all(is_positive) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") def test_should_raise_exception() -> None: dummy_source_trials = [create_trial(value=i, state=TrialState.COMPLETE) for i in range(10)] with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( x0={"x": 0.1, "y": 0.1}, source_trials=dummy_source_trials, ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( sigma0=0.1, source_trials=dummy_source_trials, ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( use_separable_cma=True, source_trials=dummy_source_trials, ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler( restart_strategy="invalid-restart-strategy", ) with pytest.raises(ValueError): optuna.samplers.CmaEsSampler(use_separable_cma=True, with_margin=True) @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) def test_incompatible_search_space(with_margin: bool) -> None: def objective1(trial: optuna.Trial) -> float: x0 = trial.suggest_float("x0", 2, 3) x1 = trial.suggest_float("x1", 1e-2, 1e2, log=True) return x0 + x1 source_study = optuna.create_study() source_study.optimize(objective1, 20) # Should not raise an exception. sampler = optuna.samplers.CmaEsSampler( with_margin=with_margin, source_trials=source_study.trials ) target_study1 = optuna.create_study(sampler=sampler) target_study1.optimize(objective1, 20) def objective2(trial: optuna.Trial) -> float: x0 = trial.suggest_float("x0", 2, 3) x1 = trial.suggest_float("x1", 1e-2, 1e2, log=True) x2 = trial.suggest_float("x2", 1e-2, 1e2, log=True) return x0 + x1 + x2 # Should raise an exception. sampler = optuna.samplers.CmaEsSampler( with_margin=with_margin, source_trials=source_study.trials ) target_study2 = optuna.create_study(sampler=sampler) with pytest.raises(ValueError): target_study2.optimize(objective2, 20) def test_infer_relative_search_space_1d() -> None: sampler = optuna.samplers.CmaEsSampler() study = optuna.create_study(sampler=sampler) # The distribution has only one candidate. study.optimize(lambda t: t.suggest_int("x", 1, 1), n_trials=1) assert sampler.infer_relative_search_space(study, study.best_trial) == {} def test_sample_relative_1d() -> None: independent_sampler = optuna.samplers.RandomSampler() sampler = optuna.samplers.CmaEsSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) # If search space is one dimensional, the independent sampler is always used. with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_object: study.optimize(lambda t: t.suggest_int("x", -1, 1), n_trials=2) assert mock_object.call_count == 2 def test_sample_relative_n_startup_trials() -> None: independent_sampler = optuna.samplers.RandomSampler() sampler = optuna.samplers.CmaEsSampler( n_startup_trials=2, independent_sampler=independent_sampler ) study = optuna.create_study(sampler=sampler) def objective(t: optuna.Trial) -> float: value = t.suggest_int("x", -1, 1) + t.suggest_int("y", -1, 1) if t.number == 0: raise Exception("first trial is failed") return float(value) # The independent sampler is used for Trial#0 (FAILED), Trial#1 (COMPLETE) # and Trial#2 (COMPLETE). The CMA-ES is used for Trial#3 (COMPLETE). with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_independent, patch.object( sampler, "sample_relative", wraps=sampler.sample_relative ) as mock_relative: study.optimize(objective, n_trials=4, catch=(Exception,)) assert mock_independent.call_count == 6 # The objective function has two parameters. assert mock_relative.call_count == 4 def test_get_trials() -> None: with patch( "optuna.Study._get_trials", new=Mock(side_effect=lambda deepcopy, use_cache: _create_trials()), ): sampler = optuna.samplers.CmaEsSampler(consider_pruned_trials=False) study = optuna.create_study(sampler=sampler) trials = sampler._get_trials(study) assert len(trials) == 1 sampler = optuna.samplers.CmaEsSampler(consider_pruned_trials=True) study = optuna.create_study(sampler=sampler) trials = sampler._get_trials(study) assert len(trials) == 2 assert trials[0].value == 1.0 assert trials[1].value == 2.0 def _create_trials() -> List[FrozenTrial]: trials = [] trials.append( FrozenTrial( number=0, value=1.0, state=optuna.trial.TrialState.COMPLETE, user_attrs={}, system_attrs={}, params={}, distributions={}, intermediate_values={}, datetime_start=None, datetime_complete=None, trial_id=0, ) ) trials.append( FrozenTrial( number=1, value=None, state=optuna.trial.TrialState.PRUNED, user_attrs={}, system_attrs={}, params={}, distributions={}, intermediate_values={0: 2.0}, datetime_start=None, datetime_complete=None, trial_id=0, ) ) return trials @pytest.mark.parametrize( "options, key", [ ({"with_margin": False, "use_separable_cma": False}, "cma:"), ({"with_margin": True, "use_separable_cma": False}, "cmawm:"), ({"with_margin": False, "use_separable_cma": True}, "sepcma:"), ], ) def test_sampler_attr_key(options: Dict[str, bool], key: str) -> None: # Test sampler attr_key property. sampler = optuna.samplers.CmaEsSampler( with_margin=options["with_margin"], use_separable_cma=options["use_separable_cma"] ) assert sampler._attr_keys.optimizer(0).startswith(key) assert sampler._attr_keys.popsize().startswith(key) assert sampler._attr_keys.n_restarts().startswith(key) assert sampler._attr_keys.n_restarts_with_large.startswith(key) assert sampler._attr_keys.poptype.startswith(key) assert sampler._attr_keys.small_n_eval.startswith(key) assert sampler._attr_keys.large_n_eval.startswith(key) assert sampler._attr_keys.generation(0).startswith(key) for restart_strategy in ["ipop", "bipop"]: sampler._restart_strategy = restart_strategy for i in range(3): assert sampler._attr_keys.generation(i).startswith( (key + "{}:restart_{}:".format(restart_strategy, i) + "generation") ) @pytest.mark.parametrize("popsize", [None, 16]) def test_population_size_is_multiplied_when_enable_ipop(popsize: Optional[int]) -> None: inc_popsize = 2 sampler = optuna.samplers.CmaEsSampler( x0={"x": 0, "y": 0}, sigma0=0.1, seed=1, n_startup_trials=1, restart_strategy="ipop", popsize=popsize, inc_popsize=inc_popsize, ) study = optuna.create_study(sampler=sampler) def objective(trial: optuna.Trial) -> float: _ = trial.suggest_float("x", -1, 1) _ = trial.suggest_float("y", -1, 1) return 1.0 with patch("optuna.samplers._cmaes.cmaes.CMA") as cma_class_mock, patch( "optuna.samplers._cmaes.pickle" ) as pickle_mock: pickle_mock.dump.return_value = b"serialized object" should_stop_mock = MagicMock() should_stop_mock.return_value = True cma_obj = CMA( mean=np.array([-1, -1], dtype=float), sigma=1.3, bounds=np.array([[-1, 1], [-1, 1]], dtype=float), population_size=popsize, # Already tested by test_init_cmaes_opts(). ) cma_obj.should_stop = should_stop_mock cma_class_mock.return_value = cma_obj initial_popsize = cma_obj.population_size study.optimize(objective, n_trials=2 + initial_popsize) assert cma_obj.should_stop.call_count == 1 _, actual_kwargs = cma_class_mock.call_args assert actual_kwargs["population_size"] == inc_popsize * initial_popsize @pytest.mark.parametrize("sampler_opts", [{}, {"use_separable_cma": True}, {"with_margin": True}]) def test_restore_optimizer_from_substrings(sampler_opts: Dict[str, Any]) -> None: popsize = 8 sampler = optuna.samplers.CmaEsSampler(popsize=popsize, **sampler_opts) optimizer = sampler._restore_optimizer([]) assert optimizer is None def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=popsize + 2) optimizer = sampler._restore_optimizer(study.trials) assert optimizer is not None assert optimizer.generation == 1 if sampler._with_margin: assert isinstance(optimizer, CMAwM) elif sampler._use_separable_cma: assert isinstance(optimizer, SepCMA) else: assert isinstance(optimizer, CMA) @pytest.mark.parametrize( "sampler_opts", [ {"restart_strategy": "ipop"}, {"restart_strategy": "bipop"}, {"restart_strategy": "ipop", "use_separable_cma": True}, {"restart_strategy": "bipop", "use_separable_cma": True}, {"restart_strategy": "ipop", "with_margin": True}, {"restart_strategy": "bipop", "with_margin": True}, ], ) def test_restore_optimizer_after_restart(sampler_opts: Dict[str, Any]) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 if sampler_opts.get("with_margin"): cma_class = CMAwM elif sampler_opts.get("use_separable_cma"): cma_class = SepCMA else: cma_class = CMA with patch.object(cma_class, "should_stop") as mock_method: mock_method.return_value = True sampler = optuna.samplers.CmaEsSampler(popsize=5, **sampler_opts) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=5 + 2) optimizer = sampler._restore_optimizer(study.trials, 1) assert optimizer is not None assert optimizer.generation == 0 @pytest.mark.parametrize( "sampler_opts, restart_strategy", [ ({"use_separable_cma": True}, "ipop"), ({"use_separable_cma": True}, "bipop"), ({"with_margin": True}, "ipop"), ({"with_margin": True}, "bipop"), ], ) def test_restore_optimizer_with_other_option( sampler_opts: Dict[str, Any], restart_strategy: str ) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 with patch.object(CMA, "should_stop") as mock_method: mock_method.return_value = True sampler = optuna.samplers.CmaEsSampler(popsize=5, restart_strategy=restart_strategy) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=5 + 2) # Restore optimizer via SepCMA or CMAwM samplers. sampler = optuna.samplers.CmaEsSampler(**sampler_opts) optimizer = sampler._restore_optimizer(study.trials) assert optimizer is None @pytest.mark.parametrize( "sampler_opts", [ {"restart_strategy": "ipop"}, {"restart_strategy": "bipop"}, {"restart_strategy": "ipop", "use_separable_cma": True}, {"restart_strategy": "bipop", "use_separable_cma": True}, {"restart_strategy": "ipop", "with_margin": True}, {"restart_strategy": "bipop", "with_margin": True}, ], ) def test_get_solution_trials(sampler_opts: Dict[str, Any]) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 popsize = 5 sampler = optuna.samplers.CmaEsSampler(popsize=popsize, **sampler_opts) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=popsize + 2) # The number of solutions for generation 0 equals population size. assert len(sampler._get_solution_trials(study.trials, 0, 0)) == popsize # The number of solutions for generation 1 is 1. assert len(sampler._get_solution_trials(study.trials, 1, 0)) == 1 @pytest.mark.parametrize( "sampler_opts, restart_strategy", [ ({"use_separable_cma": True}, "ipop"), ({"use_separable_cma": True}, "bipop"), ({"with_margin": True}, "ipop"), ({"with_margin": True}, "bipop"), ], ) def test_get_solution_trials_with_other_options( sampler_opts: Dict[str, Any], restart_strategy: str ) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 sampler = optuna.samplers.CmaEsSampler(popsize=5, restart_strategy=restart_strategy) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=5 + 2) # The number of solutions is 0 after changed samplers sampler = optuna.samplers.CmaEsSampler(**sampler_opts) assert len(sampler._get_solution_trials(study.trials, 0, 0)) == 0 @pytest.mark.parametrize( "sampler_opts", [ {"restart_strategy": "ipop"}, {"restart_strategy": "bipop"}, {"restart_strategy": "ipop", "use_separable_cma": True}, {"restart_strategy": "bipop", "use_separable_cma": True}, {"restart_strategy": "ipop", "with_margin": True}, {"restart_strategy": "bipop", "with_margin": True}, ], ) def test_get_solution_trials_after_restart(sampler_opts: Dict[str, Any]) -> None: def objective(trial: optuna.Trial) -> float: x1 = trial.suggest_float("x1", -10, 10, step=1) x2 = trial.suggest_float("x2", -10, 10) return x1**2 + x2**2 if sampler_opts.get("with_margin"): cma_class = CMAwM elif sampler_opts.get("use_separable_cma"): cma_class = SepCMA else: cma_class = CMA popsize = 5 with patch.object(cma_class, "should_stop") as mock_method: mock_method.return_value = True sampler = optuna.samplers.CmaEsSampler(popsize=popsize, **sampler_opts) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=popsize + 2) # The number of solutions for generation=0 and n_restarts=0 equals population size. assert len(sampler._get_solution_trials(study.trials, 0, 0)) == popsize # The number of solutions for generation=1 and n_restarts=0 is 0. assert len(sampler._get_solution_trials(study.trials, 1, 0)) == 0 # The number of solutions for generation=0 and n_restarts=1 is 1 since it was restarted. assert len(sampler._get_solution_trials(study.trials, 0, 1)) == 1 @pytest.mark.parametrize( "dummy_optimizer_str,attr_len", [ ("012", 1), ("01234", 1), ("012345", 2), ], ) def test_split_and_concat_optimizer_string(dummy_optimizer_str: str, attr_len: int) -> None: sampler = optuna.samplers.CmaEsSampler() with patch("optuna.samplers._cmaes._SYSTEM_ATTR_MAX_LENGTH", 5): attrs = sampler._split_optimizer_str(dummy_optimizer_str) assert len(attrs) == attr_len actual = sampler._concat_optimizer_attrs(attrs) assert dummy_optimizer_str == actual def test_call_after_trial_of_base_sampler() -> None: independent_sampler = optuna.samplers.RandomSampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = optuna.samplers.CmaEsSampler(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) with patch.object( independent_sampler, "after_trial", wraps=independent_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_is_compatible_search_space() -> None: transform = _SearchSpaceTransform( { "x0": optuna.distributions.FloatDistribution(2, 3), "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), } ) assert optuna.samplers._cmaes._is_compatible_search_space( transform, { "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), "x0": optuna.distributions.FloatDistribution(2, 3), }, ) # Same search space size, but different param names. assert not optuna.samplers._cmaes._is_compatible_search_space( transform, { "x0": optuna.distributions.FloatDistribution(2, 3), "foo": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), }, ) # x2 is added. assert not optuna.samplers._cmaes._is_compatible_search_space( transform, { "x0": optuna.distributions.FloatDistribution(2, 3), "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), "x2": optuna.distributions.FloatDistribution(2, 3, step=0.1), }, ) # x0 is not found. assert not optuna.samplers._cmaes._is_compatible_search_space( transform, { "x1": optuna.distributions.CategoricalDistribution(["foo", "bar", "baz", "qux"]), }, ) def test_internal_optimizer_with_margin() -> None: def objective_discrete(trial: optuna.Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y def objective_mixed(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y def objective_continuous(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -10, 10) return x**2 + y objectives = [objective_discrete, objective_mixed, objective_continuous] for objective in objectives: with patch("optuna.samplers._cmaes.cmaes.CMAwM") as cmawm_class_mock: sampler = optuna.samplers.CmaEsSampler(with_margin=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert cmawm_class_mock.call_count == 1 @pytest.mark.parametrize("warn_independent_sampling", [True, False]) def test_warn_independent_sampling( capsys: _pytest.capture.CaptureFixture, warn_independent_sampling: bool ) -> None: def objective_single(trial: optuna.trial.Trial) -> float: return trial.suggest_float("x", 0, 1) def objective_shrink(trial: optuna.trial.Trial) -> float: if trial.number != 5: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) z = trial.suggest_float("z", 0, 1) return x + y + z else: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x + y def objective_expand(trial: optuna.trial.Trial) -> float: if trial.number != 5: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) return x + y else: x = trial.suggest_float("x", 0, 1) y = trial.suggest_float("y", 0, 1) z = trial.suggest_float("z", 0, 1) return x + y + z for objective in [objective_single, objective_shrink, objective_expand]: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = optuna.samplers.CmaEsSampler(warn_independent_sampling=warn_independent_sampling) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert (err != "") == warn_independent_sampling @pytest.mark.filterwarnings("ignore::optuna.exceptions.ExperimentalWarning") @pytest.mark.parametrize("with_margin", [False, True]) @pytest.mark.parametrize("storage_name", ["sqlite", "journal"]) def test_rdb_storage(with_margin: bool, storage_name: str) -> None: # Confirm `study._storage.set_trial_system_attr` does not fail in several storages. def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y with StorageSupplier(storage_name) as storage: study = optuna.create_study( sampler=optuna.samplers.CmaEsSampler(with_margin=with_margin), storage=storage, ) study.optimize(objective, n_trials=3) optuna-3.5.0/tests/samplers_tests/test_grid.py000066400000000000000000000221511453453102400215770ustar00rootroot00000000000000import itertools from typing import Dict from typing import List from typing import Mapping from typing import Sequence from typing import Union from typing import ValuesView import numpy as np import pytest import optuna from optuna import samplers from optuna.samplers._grid import GridValueType from optuna.storages import RetryFailedTrialCallback from optuna.testing.objectives import fail_objective from optuna.testing.objectives import pruned_objective from optuna.testing.storages import StorageSupplier from optuna.trial import Trial def test_study_optimize_with_single_search_space() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 100) b = trial.suggest_float("b", -0.1, 0.1) c = trial.suggest_categorical("c", ("x", "y", None, 1, 2.0)) d = trial.suggest_float("d", -5, 5, step=1) e = trial.suggest_float("e", 0.0001, 1, log=True) if c == "x": return a * d else: return b * e # Test that all combinations of the grid is sampled. search_space = { "b": np.arange(-0.1, 0.1, 0.05), "c": ("x", "y", None, 1, 2.0), "d": [-5.0, 5.0], "e": [0.1], "a": list(range(0, 100, 20)), } study = optuna.create_study(sampler=samplers.GridSampler(search_space)) # type: ignore study.optimize(objective) def sorted_values( d: Mapping[str, Sequence[GridValueType]] ) -> ValuesView[Sequence[GridValueType]]: return dict(sorted(d.items())).values() all_grids = itertools.product(*sorted_values(search_space)) # type: ignore all_suggested_values = [tuple([p for p in sorted_values(t.params)]) for t in study.trials] assert set(all_grids) == set(all_suggested_values) # Test a non-existing parameter name in the grid. search_space = {"a": list(range(0, 100, 20))} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) # type: ignore with pytest.raises(ValueError): study.optimize(objective) # Test a value with out of range. search_space = { "a": [110], # 110 is out of range specified by the suggest method. "b": [0], "c": ["x"], "d": [0], "e": [0.1], } study = optuna.create_study(sampler=samplers.GridSampler(search_space)) # type: ignore with pytest.warns(UserWarning): study.optimize(objective) def test_study_optimize_with_exceeding_number_of_trials() -> None: def objective(trial: Trial) -> float: return trial.suggest_int("a", 0, 100) # When `n_trials` is `None`, the optimization stops just after all grids are evaluated. search_space: Dict[str, List[GridValueType]] = {"a": [0, 50]} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) study.optimize(objective, n_trials=None) assert len(study.trials) == 2 # If the optimization is triggered after all grids are evaluated, an additional trial runs. study.optimize(objective, n_trials=None) assert len(study.trials) == 3 def test_study_optimize_with_pruning() -> None: # Pruned trials should count towards grid consumption. search_space: Dict[str, List[GridValueType]] = {"a": [0, 50]} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) study.optimize(pruned_objective, n_trials=None) assert len(study.trials) == 2 def test_study_optimize_with_fail() -> None: def objective(trial: Trial) -> float: return trial.suggest_int("a", 0, 100) # Failed trials should count towards grid consumption. search_space: Dict[str, List[GridValueType]] = {"a": [0, 50]} study = optuna.create_study(sampler=samplers.GridSampler(search_space)) study.optimize(fail_objective, n_trials=1, catch=ValueError) study.optimize(objective, n_trials=None) assert len(study.trials) == 2 def test_study_optimize_with_numpy_related_search_space() -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 0, 10) b = trial.suggest_float("b", -0.1, 0.1) return a + b # Test that all combinations of the grid is sampled. search_space = { "a": np.linspace(0, 10, 11), "b": np.arange(-0.1, 0.1, 0.05), } with StorageSupplier("sqlite") as storage: study = optuna.create_study( sampler=samplers.GridSampler(search_space), # type: ignore storage=storage, ) study.optimize(objective, n_trials=None) def test_study_optimize_with_multiple_search_spaces() -> None: def objective(trial: Trial) -> float: a = trial.suggest_int("a", 0, 100) b = trial.suggest_float("b", -100, 100) return a * b # Run 3 trials with a search space. search_space_0 = {"a": [0, 50], "b": [-50, 0, 50]} sampler_0 = samplers.GridSampler(search_space_0) study = optuna.create_study(sampler=sampler_0) study.optimize(objective, n_trials=3) assert len(study.trials) == 3 for t in study.trials: assert sampler_0._same_search_space(t.system_attrs["search_space"]) # Run 2 trials with another space. search_space_1 = {"a": [0, 25], "b": [-50]} sampler_1 = samplers.GridSampler(search_space_1) study.sampler = sampler_1 study.optimize(objective, n_trials=2) assert not sampler_0._same_search_space(sampler_1._search_space) assert len(study.trials) == 5 for t in study.trials[:3]: assert sampler_0._same_search_space(t.system_attrs["search_space"]) for t in study.trials[3:5]: assert sampler_1._same_search_space(t.system_attrs["search_space"]) # Run 3 trials with the first search space again. study.sampler = sampler_0 study.optimize(objective, n_trials=3) assert len(study.trials) == 8 for t in study.trials[:3]: assert sampler_0._same_search_space(t.system_attrs["search_space"]) for t in study.trials[3:5]: assert sampler_1._same_search_space(t.system_attrs["search_space"]) for t in study.trials[5:]: assert sampler_0._same_search_space(t.system_attrs["search_space"]) def test_cast_value() -> None: samplers.GridSampler._check_value("x", None) samplers.GridSampler._check_value("x", True) samplers.GridSampler._check_value("x", False) samplers.GridSampler._check_value("x", -1) samplers.GridSampler._check_value("x", -1.5) samplers.GridSampler._check_value("x", float("nan")) samplers.GridSampler._check_value("x", "foo") samplers.GridSampler._check_value("x", "") with pytest.warns(UserWarning): samplers.GridSampler._check_value("x", [1]) def test_has_same_search_space() -> None: search_space: Dict[str, List[Union[int, str]]] = {"x": [3, 2, 1], "y": ["a", "b", "c"]} sampler = samplers.GridSampler(search_space) assert sampler._same_search_space(search_space) assert sampler._same_search_space({"x": [3, 2, 1], "y": ["a", "b", "c"]}) assert not sampler._same_search_space({"y": ["c", "a", "b"], "x": [1, 2, 3]}) assert not sampler._same_search_space({"x": [3, 2, 1, 0], "y": ["a", "b", "c"]}) assert not sampler._same_search_space({"x": [3, 2], "y": ["a", "b", "c"]}) def test_retried_trial() -> None: sampler = samplers.GridSampler({"a": [0, 50]}) study = optuna.create_study(sampler=sampler) trial = study.ask() trial.suggest_int("a", 0, 100) callback = RetryFailedTrialCallback() callback(study, study.trials[0]) study.optimize(lambda trial: trial.suggest_int("a", 0, 100)) assert len(study.trials) == 3 assert study.trials[0].params["a"] == study.trials[1].params["a"] assert study.trials[0].system_attrs["grid_id"] == study.trials[1].system_attrs["grid_id"] def test_enqueued_trial() -> None: sampler = samplers.GridSampler({"a": [0, 50]}) study = optuna.create_study(sampler=sampler) study.enqueue_trial({"a": 100}) study.optimize(lambda trial: trial.suggest_int("a", 0, 100)) assert len(study.trials) == 3 assert study.trials[0].params["a"] == 100 assert sorted([study.trials[1].params["a"], study.trials[2].params["a"]]) == [0, 50] def test_same_seed_trials() -> None: grid_values = [0, 20, 40, 60, 80, 100] seed = 0 sampler1 = samplers.GridSampler({"a": grid_values}, seed) study1 = optuna.create_study(sampler=sampler1) study1.optimize(lambda trial: trial.suggest_int("a", 0, 100)) sampler2 = samplers.GridSampler({"a": grid_values}, seed) study2 = optuna.create_study(sampler=sampler2) study2.optimize(lambda trial: trial.suggest_int("a", 0, 100)) for i in range(len(grid_values)): assert study1.trials[i].params["a"] == study2.trials[i].params["a"] def test_enqueued_insufficient_trial() -> None: sampler = samplers.GridSampler({"a": [0, 50]}) study = optuna.create_study(sampler=sampler) study.enqueue_trial({}) with pytest.raises(ValueError): study.optimize(lambda trial: trial.suggest_int("a", 0, 100)) def test_nan() -> None: sampler = optuna.samplers.GridSampler({"x": [0, float("nan")]}) study = optuna.create_study(sampler=sampler) study.optimize( lambda trial: 1 if np.isnan(trial.suggest_categorical("x", [0, float("nan")])) else 0 ) assert len(study.get_trials()) == 2 optuna-3.5.0/tests/samplers_tests/test_lazy_random_state.py000066400000000000000000000003241453453102400243670ustar00rootroot00000000000000from optuna.samplers._lazy_random_state import LazyRandomState def test_lazy_state() -> None: state = LazyRandomState() assert state._rng is None state.rng.seed(1) assert state._rng is not None optuna-3.5.0/tests/samplers_tests/test_nsgaii.py000066400000000000000000001130271453453102400221270ustar00rootroot00000000000000from __future__ import annotations from collections import Counter from collections.abc import Callable from collections.abc import Sequence import copy import itertools from typing import Any from unittest.mock import MagicMock from unittest.mock import Mock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna._transform import _SearchSpaceTransform from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.samplers import BaseSampler from optuna.samplers import NSGAIISampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._lazy_random_state import LazyRandomState from optuna.samplers.nsgaii import BaseCrossover from optuna.samplers.nsgaii import BLXAlphaCrossover from optuna.samplers.nsgaii import SBXCrossover from optuna.samplers.nsgaii import SPXCrossover from optuna.samplers.nsgaii import UNDXCrossover from optuna.samplers.nsgaii import UniformCrossover from optuna.samplers.nsgaii import VSBXCrossover from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.samplers.nsgaii._child_generation_strategy import NSGAIIChildGenerationStrategy from optuna.samplers.nsgaii._crossover import _inlined_categorical_uniform_crossover from optuna.samplers.nsgaii._dominates import _constrained_dominates from optuna.samplers.nsgaii._dominates import _validate_constraints from optuna.samplers.nsgaii._elite_population_selection_strategy import ( NSGAIIElitePopulationSelectionStrategy, ) from optuna.samplers.nsgaii._elite_population_selection_strategy import _calc_crowding_distance from optuna.samplers.nsgaii._elite_population_selection_strategy import _crowding_distance_sort from optuna.samplers.nsgaii._elite_population_selection_strategy import _fast_non_dominated_sort from optuna.samplers.nsgaii._sampler import _GENERATION_KEY from optuna.study._multi_objective import _dominates from optuna.study._study_direction import StudyDirection from optuna.trial import FrozenTrial def _nan_equal(a: Any, b: Any) -> bool: if isinstance(a, float) and isinstance(b, float) and np.isnan(a) and np.isnan(b): return True return a == b def test_population_size() -> None: # Set `population_size` to 10. sampler = NSGAIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers.nsgaii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {0: 10, 1: 10, 2: 10, 3: 10} # Set `population_size` to 2. sampler = NSGAIISampler(population_size=2) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers.nsgaii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {i: 2 for i in range(20)} # Invalid population size. with pytest.raises(ValueError): # Less than 2. NSGAIISampler(population_size=1) with pytest.raises(ValueError): mock_crossover = MagicMock(spec=BaseCrossover) mock_crossover.configure_mock(n_parents=3) NSGAIISampler(population_size=2, crossover=mock_crossover) def test_mutation_prob() -> None: NSGAIISampler(mutation_prob=None) NSGAIISampler(mutation_prob=0.0) NSGAIISampler(mutation_prob=0.5) NSGAIISampler(mutation_prob=1.0) with pytest.raises(ValueError): NSGAIISampler(mutation_prob=-0.5) with pytest.raises(ValueError): NSGAIISampler(mutation_prob=1.1) def test_crossover_prob() -> None: NSGAIISampler(crossover_prob=0.0) NSGAIISampler(crossover_prob=0.5) NSGAIISampler(crossover_prob=1.0) with pytest.raises(ValueError): NSGAIISampler(crossover_prob=-0.5) with pytest.raises(ValueError): NSGAIISampler(crossover_prob=1.1) def test_swapping_prob() -> None: NSGAIISampler(swapping_prob=0.0) NSGAIISampler(swapping_prob=0.5) NSGAIISampler(swapping_prob=1.0) with pytest.raises(ValueError): NSGAIISampler(swapping_prob=-0.5) with pytest.raises(ValueError): NSGAIISampler(swapping_prob=1.1) with pytest.raises(ValueError): UniformCrossover(swapping_prob=-0.5) with pytest.raises(ValueError): UniformCrossover(swapping_prob=1.1) @pytest.mark.parametrize("choices", [[-1, 0, 1], [True, False]]) def test_crossover_casting(choices: list[Any]) -> None: str_choices = list(map(str, choices)) def objective(trial: optuna.Trial) -> Sequence[float]: cat_1 = trial.suggest_categorical("cat_1", choices) cat_2 = trial.suggest_categorical("cat_2", str_choices) assert isinstance(cat_1, type(choices[0])) assert isinstance(cat_2, type(str_choices[0])) return 1.0, 2.0 population_size = 10 sampler = NSGAIISampler(population_size=population_size) study = optuna.create_study(directions=["minimize"] * 2, sampler=sampler) study.optimize(objective, n_trials=population_size * 2) def test_constraints_func_none() -> None: n_trials = 4 n_objectives = 2 sampler = NSGAIISampler(population_size=2) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials for trial in study.trials: assert _CONSTRAINTS_KEY not in trial.system_attrs @pytest.mark.parametrize("constraint_value", [-1.0, 0.0, 1.0, -float("inf"), float("inf")]) def test_constraints_func(constraint_value: float) -> None: n_trials = 4 n_objectives = 2 constraints_func_call_count = 0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: nonlocal constraints_func_call_count constraints_func_call_count += 1 return (constraint_value + trial.number,) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials assert constraints_func_call_count == n_trials for trial in study.trials: for x, y in zip(trial.system_attrs[_CONSTRAINTS_KEY], (constraint_value + trial.number,)): assert x == y def test_constraints_func_nan() -> None: n_trials = 4 n_objectives = 2 def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) with pytest.raises(ValueError): study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) trials = study.get_trials() assert len(trials) == 1 # The error stops optimization, but completed trials are recorded. assert all(0 <= x <= 1 for x in trials[0].params.values()) # The params are normal. assert trials[0].values == list(trials[0].params.values()) # The values are normal. assert trials[0].system_attrs[_CONSTRAINTS_KEY] is None # None is set for constraints. @pytest.mark.parametrize("direction1", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) @pytest.mark.parametrize("direction2", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) @pytest.mark.parametrize( "constraints_list", [ [[]], # empty constraint [[-float("inf")], [-1], [0]], # single constraint [ [c1, c2] for c1 in [-float("inf"), -1, 0] for c2 in [-float("inf"), -1, 0] ], # multiple constraints ], ) def test_constrained_dominates_feasible_vs_feasible( direction1: StudyDirection, direction2: StudyDirection, constraints_list: list[list[float]] ) -> None: directions = [direction1, direction2] # Check all pairs of trials consisting of these values, i.e., # [-inf, -inf], [-inf, -1], [-inf, 1], [-inf, inf], [-1, -inf], ... values_list = [ [x, y] for x in [-float("inf"), -1, 1, float("inf")] for y in [-float("inf"), -1, 1, float("inf")] ] values_constraints_list = [(vs, cs) for vs in values_list for cs in constraints_list] # The results of _constrained_dominates match _dominates in all feasible cases. for values1, constraints1 in values_constraints_list: for values2, constraints2 in values_constraints_list: t1 = _create_frozen_trial(0, values1, constraints1) t2 = _create_frozen_trial(1, values2, constraints2) assert _constrained_dominates(t1, t2, directions) == _dominates(t1, t2, directions) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_constrained_dominates_feasible_vs_infeasible( direction: StudyDirection, ) -> None: # Check all pairs of trials consisting of these constraint values. constraints_1d_feasible = [-float("inf"), -1, 0] constraints_1d_infeasible = [2, float("inf")] directions = [direction] # Feasible constraints. constraints_list1 = [ [c1, c2] for c1 in constraints_1d_feasible for c2 in constraints_1d_feasible ] # Infeasible constraints. constraints_list2 = [ [c1, c2] for c1 in constraints_1d_feasible + constraints_1d_infeasible for c2 in constraints_1d_infeasible ] # In the following code, we test that the feasible trials always dominate # the infeasible trials. for constraints1 in constraints_list1: for constraints2 in constraints_list2: t1 = _create_frozen_trial(0, [0], constraints1) t2 = _create_frozen_trial(1, [1], constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) t1 = _create_frozen_trial(0, [1], constraints1) t2 = _create_frozen_trial(1, [0], constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_constrained_dominates_infeasible_vs_infeasible(direction: StudyDirection) -> None: inf = float("inf") directions = [direction] # The following table illustrates the violations of some constraint values. # When both trials are infeasible, the trial with smaller violation dominates # the one with larger violation. # # c2 # â•”â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•— # â•‘ │ -1 │ 0 │ 1 │ 2 │ ∞ â•‘ # ╟─────┼─────┼─────┼─────┼─────┼─────╢ # â•‘ -1 │ │ 1 │ 2 │ ∞ â•‘ # ╟─────┼─feasible ─┼─────┼─────┼─────╢ # â•‘ 0 │ │ 1 │ 2 │ ∞ â•‘ # c1 ╟─────┼─────┼─────┼─────┼─────┼─────╢ # â•‘ 1 │ 1 │ 1 │ 2 │ 3 │ ∞ â•‘ # ╟─────┼─────┼─────┼─────┼─────┼─────╢ # â•‘ 2 │ 2 │ 2 │ 3 │ 4 │ ∞ â•‘ # ╟─────┼─────┼─────┼─────┼─────┼─────╢ # â•‘ ∞ │ ∞ │ ∞ │ ∞ │ ∞ │ ∞ â•‘ # â•šâ•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â• # # Check all pairs of these constraints. constraints_infeasible_sorted: list[list[list[float]]] constraints_infeasible_sorted = [ # These constraints have violation 1. [[1, -inf], [1, -1], [1, 0], [0, 1], [-1, 1], [-inf, 1]], # These constraints have violation 2. [[2, -inf], [2, -1], [2, 0], [1, 1], [0, 2], [-1, 2], [-inf, 2]], # These constraints have violation 3. [[3, -inf], [3, -1], [3, 0], [2, 1], [1, 2], [0, 3], [-1, 3], [-inf, 3]], # These constraints have violation inf. [ [-inf, inf], [-1, inf], [0, inf], [1, inf], [inf, inf], [inf, 1], [inf, 0], [inf, -1], [inf, -inf], ], ] # Check that constraints with smaller violations dominate constraints with larger violation. for i in range(len(constraints_infeasible_sorted)): for j in range(i + 1, len(constraints_infeasible_sorted)): # Every constraint in constraints_infeasible_sorted[i] dominates # every constraint in constraints_infeasible_sorted[j]. for constraints1 in constraints_infeasible_sorted[i]: for constraints2 in constraints_infeasible_sorted[j]: t1 = _create_frozen_trial(0, [0], constraints1) t2 = _create_frozen_trial(1, [1], constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) t1 = _create_frozen_trial(0, [1], constraints1) t2 = _create_frozen_trial(1, [0], constraints2) assert _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) # Check that constraints with same violations are incomparable. for constraints_with_same_violations in constraints_infeasible_sorted: for constraints1 in constraints_with_same_violations: for constraints2 in constraints_with_same_violations: t1 = _create_frozen_trial(0, [0], constraints1) t2 = _create_frozen_trial(1, [1], constraints2) assert not _constrained_dominates(t1, t2, directions) assert not _constrained_dominates(t2, t1, directions) def _assert_population_per_rank( trials: list[FrozenTrial], direction: list[StudyDirection], population_per_rank: list[list[FrozenTrial]], ) -> None: # Check that the number of trials do not change. flattened = [trial for rank in population_per_rank for trial in rank] assert len(flattened) == len(trials) with warnings.catch_warnings(): warnings.simplefilter("ignore", UserWarning) # Check that the trials in the same rank do not dominate each other. for i in range(len(population_per_rank)): for trial1 in population_per_rank[i]: for trial2 in population_per_rank[i]: assert not _constrained_dominates(trial1, trial2, direction) # Check that each trial is dominated by some trial in the rank above. for i in range(len(population_per_rank) - 1): for trial2 in population_per_rank[i + 1]: assert any( _constrained_dominates(trial1, trial2, direction) for trial1 in population_per_rank[i] ) @pytest.mark.parametrize("direction1", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) @pytest.mark.parametrize("direction2", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_fast_non_dominated_sort_no_constraints( direction1: StudyDirection, direction2: StudyDirection ) -> None: directions = [direction1, direction2] value_list = [10, 20, 20, 30, float("inf"), float("inf"), -float("inf")] values = [[v1, v2] for v1 in value_list for v2 in value_list] trials = [_create_frozen_trial(i, v) for i, v in enumerate(values)] population_per_rank = _fast_non_dominated_sort(copy.copy(trials), directions, _dominates) _assert_population_per_rank(trials, directions, population_per_rank) def test_fast_non_dominated_sort_with_constraints() -> None: value_list = [10, 20, 20, 30, float("inf"), float("inf"), -float("inf")] values = [[v1, v2] for v1 in value_list for v2 in value_list] constraint_list = [-float("inf"), -2, 0, 1, 2, 3, float("inf")] constraints = [[c1, c2] for c1 in constraint_list for c2 in constraint_list] trials = [ _create_frozen_trial(i, v, c) for i, (v, c) in enumerate(itertools.product(values, constraints)) ] directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] population_per_rank = _fast_non_dominated_sort( copy.copy(trials), directions, _constrained_dominates ) _assert_population_per_rank(trials, directions, population_per_rank) def test_validate_constraints() -> None: with pytest.raises(ValueError): _validate_constraints( [_create_frozen_trial(0, [1], [0, float("nan")])], constraints_func=lambda _: [0], ) @pytest.mark.parametrize( "values_and_constraints", [ [([10], None), ([20], None), ([20], [0]), ([20], [1]), ([30], [-1])], [ ([50, 30], None), ([30, 50], None), ([20, 20], [3, 3]), ([30, 10], [0, -1]), ([15, 15], [4, 4]), ], ], ) def test_fast_non_dominated_sort_missing_constraint_values( values_and_constraints: list[tuple[list[float], list[float]]] ) -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) values_dim = len(values_and_constraints[0][0]) for directions in itertools.product( [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE], repeat=values_dim ): trials = [_create_frozen_trial(i, v, c) for i, (v, c) in enumerate(values_and_constraints)] with pytest.warns(UserWarning): population_per_rank = _fast_non_dominated_sort( copy.copy(trials), list(directions), _constrained_dominates ) _assert_population_per_rank(trials, list(directions), population_per_rank) @pytest.mark.parametrize("n_dims", [1, 2, 3]) def test_fast_non_dominated_sort_empty(n_dims: int) -> None: for directions in itertools.product( [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE], repeat=n_dims ): trials: list[FrozenTrial] = [] population_per_rank = _fast_non_dominated_sort(trials, list(directions), _dominates) assert population_per_rank == [] @pytest.mark.parametrize( "values, expected_dist", [ ([[5], [6], [9], [0]], [6 / 9, 4 / 9, float("inf"), float("inf")]), ([[5, 0], [6, 0], [9, 0], [0, 0]], [6 / 9, 4 / 9, float("inf"), float("inf")]), ( [[5, -1], [6, 0], [9, 1], [0, 2]], [float("inf"), 4 / 9 + 2 / 3, float("inf"), float("inf")], ), ([[5]], [0]), ([[5], [5]], [0, 0]), ( [[1], [2], [float("inf")]], [float("inf"), float("inf"), float("inf")], ), ( [[float("-inf")], [1], [2]], [float("inf"), float("inf"), float("inf")], ), ([[float("inf")], [float("inf")], [float("inf")]], [0, 0, 0]), ([[-float("inf")], [-float("inf")], [-float("inf")]], [0, 0, 0]), ([[-float("inf")], [float("inf")]], [float("inf"), float("inf")]), ( [[-float("inf")], [-float("inf")], [-float("inf")], [0], [1], [2], [float("inf")]], [0, 0, float("inf"), float("inf"), 1, float("inf"), float("inf")], ), ], ) def test_calc_crowding_distance(values: list[list[float]], expected_dist: list[float]) -> None: trials = [_create_frozen_trial(i, value) for i, value in enumerate(values)] crowding_dist = _calc_crowding_distance(trials) for i in range(len(trials)): assert _nan_equal(crowding_dist[i], expected_dist[i]), i @pytest.mark.parametrize( "values", [ [[5], [6], [9], [0]], [[5, 0], [6, 0], [9, 0], [0, 0]], [[5, -1], [6, 0], [9, 1], [0, 2]], [[1], [2], [float("inf")]], [[float("-inf")], [1], [2]], ], ) def test_crowding_distance_sort(values: list[list[float]]) -> None: """Checks that trials are sorted by the values of `_calc_crowding_distance`.""" trials = [_create_frozen_trial(i, value) for i, value in enumerate(values)] crowding_dist = _calc_crowding_distance(trials) _crowding_distance_sort(trials) sorted_dist = [crowding_dist[t.number] for t in trials] assert sorted_dist == sorted(sorted_dist, reverse=True) def test_study_system_attr_for_population_cache() -> None: sampler = NSGAIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) def get_cached_entries( study: optuna.study.Study, ) -> list[tuple[int, list[int]]]: study_system_attrs = study._storage.get_study_system_attrs(study._study_id) return [ v for k, v in study_system_attrs.items() if k.startswith(optuna.samplers.nsgaii._sampler._POPULATION_CACHE_KEY_PREFIX) ] study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 0 study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=1) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 0 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 1 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. def test_constraints_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(constraints_func=lambda _: [0]) def test_elite_population_selection_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(elite_population_selection_strategy=lambda study, population: []) def test_child_generation_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(child_generation_strategy=lambda study, search_space, parent_population: {}) def test_after_trial_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIISampler(after_trial_strategy=lambda study, trial, state, value: None) # TODO(ohta): Consider to move this utility function to `optuna.testing` module. def _create_frozen_trial( number: int, values: Sequence[float], constraints: Sequence[float] | None = None ) -> optuna.trial.FrozenTrial: trial = optuna.trial.create_trial( state=optuna.trial.TrialState.COMPLETE, values=list(values), system_attrs={} if constraints is None else {_CONSTRAINTS_KEY: list(constraints)}, ) trial.number = number trial._trial_id = number return trial def test_elite_population_selection_strategy_invalid_value() -> None: with pytest.raises(ValueError): NSGAIIElitePopulationSelectionStrategy(population_size=1) @pytest.mark.parametrize( "objectives, expected_elite_population", [ ( [[1.0, 4.0], [2.0, 3.0], [3.0, 2.0], [4.0, 1.0]], [[1.0, 4.0], [2.0, 3.0], [3.0, 2.0], [4.0, 1.0]], ), ( [[1.0, 2.0], [2.0, 1.0], [3.0, 3.0], [4.0, 4.0]], [[1.0, 2.0], [2.0, 1.0], [3.0, 3.0], [4.0, 4.0]], ), ( [[1.0, 2.0], [2.0, 1.0], [5.0, 3.0], [3.0, 5.0], [4.0, 4.0]], [[1.0, 2.0], [2.0, 1.0], [5.0, 3.0], [3.0, 5.0]], ), ], ) def test_elite_population_selection_strategy_result( objectives: list[list[float]], expected_elite_population: list[list[float]], ) -> None: population_size = 4 elite_population_selection_strategy = NSGAIIElitePopulationSelectionStrategy( population_size=population_size ) study = optuna.create_study(directions=["minimize", "minimize"]) study.add_trials([optuna.create_trial(values=values) for values in objectives]) elite_population_values = [ trial.values for trial in elite_population_selection_strategy(study, study.get_trials()) ] assert len(elite_population_values) == population_size for values in elite_population_values: assert values in expected_elite_population @pytest.mark.parametrize( "mutation_prob,crossover,crossover_prob,swapping_prob", [ (1.2, UniformCrossover(), 0.9, 0.5), (-0.2, UniformCrossover(), 0.9, 0.5), (None, UniformCrossover(), 1.2, 0.5), (None, UniformCrossover(), -0.2, 0.5), (None, UniformCrossover(), 0.9, 1.2), (None, UniformCrossover(), 0.9, -0.2), (None, 3, 0.9, 0.5), ], ) def test_child_generation_strategy_invalid_value( mutation_prob: float, crossover: BaseCrossover | int, crossover_prob: float, swapping_prob: float, ) -> None: with pytest.raises(ValueError): NSGAIIChildGenerationStrategy( mutation_prob=mutation_prob, crossover=crossover, # type: ignore[arg-type] crossover_prob=crossover_prob, swapping_prob=swapping_prob, rng=LazyRandomState(), ) @pytest.mark.parametrize( "mutation_prob,child_params", [(0.0, {"x": 1.0, "y": 0.0}), (1.0, {})], ) def test_child_generation_strategy_mutation_prob( mutation_prob: int, child_params: dict[str, float] ) -> None: child_generation_strategy = NSGAIIChildGenerationStrategy( crossover_prob=0.0, crossover=UniformCrossover(), mutation_prob=mutation_prob, swapping_prob=0.5, rng=LazyRandomState(seed=1), ) study = MagicMock(spec=optuna.study.Study) search_space = MagicMock(spec=dict) search_space.keys.return_value = ["x", "y"] parent_population = [ optuna.trial.create_trial( params={"x": 1.0, "y": 0}, distributions={ "x": FloatDistribution(0, 10), "y": CategoricalDistribution([-1, 0, 1]), }, value=5.0, ) ] assert child_generation_strategy(study, search_space, parent_population) == child_params def test_child_generation_strategy_generation_key() -> None: n_params = 2 def objective(trial: optuna.Trial) -> list[float]: xs = [trial.suggest_float(f"x{dim}", -10, 10) for dim in range(n_params)] return xs mock_func = MagicMock(spec=Callable, return_value={"x0": 0.0, "x1": 1.1}) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study = optuna.create_study( sampler=NSGAIISampler(population_size=2, child_generation_strategy=mock_func), directions=["minimize", "minimize"], ) study.optimize(objective, n_trials=3) assert mock_func.call_count == 1 for i, trial in enumerate(study.get_trials()): if i < 2: assert trial.system_attrs[_GENERATION_KEY] == 0 elif i == 2: assert trial.system_attrs[_GENERATION_KEY] == 1 @patch( "optuna.samplers.nsgaii._child_generation_strategy.perform_crossover", return_value={"x": 3.0, "y": 2.0}, ) def test_child_generation_strategy_crossover_prob(mock_func: MagicMock) -> None: study = MagicMock(spec=optuna.study.Study) search_space = MagicMock(spec=dict) search_space.keys.return_value = ["x", "y"] parent_population = [ optuna.trial.create_trial( params={"x": 1.0, "y": 0}, distributions={ "x": FloatDistribution(0, 10), "y": CategoricalDistribution([-1, 0, 1]), }, value=5.0, ) ] child_generation_strategy_always_not_crossover = NSGAIIChildGenerationStrategy( crossover_prob=0.0, crossover=UniformCrossover(), mutation_prob=None, swapping_prob=0.5, rng=LazyRandomState(seed=1), ) assert child_generation_strategy_always_not_crossover( study, search_space, parent_population ) == {"x": 1.0} assert mock_func.call_count == 0 child_generation_strategy_always_crossover = NSGAIIChildGenerationStrategy( crossover_prob=1.0, crossover=UniformCrossover(), mutation_prob=0.0, swapping_prob=0.5, rng=LazyRandomState(), ) assert child_generation_strategy_always_crossover(study, search_space, parent_population) == { "x": 3.0, "y": 2.0, } assert mock_func.call_count == 1 def test_call_after_trial_of_random_sampler() -> None: sampler = NSGAIISampler() study = optuna.create_study(sampler=sampler) with patch.object( sampler._random_sampler, "after_trial", wraps=sampler._random_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_call_after_trial_of_after_trial_strategy() -> None: sampler = NSGAIISampler() study = optuna.create_study(sampler=sampler) with patch.object(sampler, "_after_trial_strategy") as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @patch("optuna.samplers.nsgaii._after_trial_strategy._process_constraints_after_trial") def test_nsgaii_after_trial_strategy(mock_func: MagicMock) -> None: def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) state = optuna.trial.TrialState.FAIL study = optuna.create_study() trial = optuna.trial.create_trial(state=state) after_trial_strategy_without_constrains = NSGAIIAfterTrialStrategy() after_trial_strategy_without_constrains(study, trial, state) assert mock_func.call_count == 0 after_trial_strategy_with_constrains = NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) after_trial_strategy_with_constrains(study, trial, state) assert mock_func.call_count == 1 parametrize_nsga2_sampler = pytest.mark.parametrize( "sampler_class", [ lambda: NSGAIISampler(population_size=2, crossover=UniformCrossover()), lambda: NSGAIISampler(population_size=2, crossover=BLXAlphaCrossover()), lambda: NSGAIISampler(population_size=2, crossover=SBXCrossover()), lambda: NSGAIISampler(population_size=2, crossover=VSBXCrossover()), lambda: NSGAIISampler(population_size=3, crossover=UNDXCrossover()), lambda: NSGAIISampler(population_size=3, crossover=UNDXCrossover()), ], ) @parametrize_nsga2_sampler @pytest.mark.parametrize("n_objectives", [1, 2, 3]) def test_crossover_objectives(n_objectives: int, sampler_class: Callable[[], BaseSampler]) -> None: n_trials = 8 study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler_class()) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials ) assert len(study.trials) == n_trials @parametrize_nsga2_sampler @pytest.mark.parametrize("n_params", [1, 2, 3]) def test_crossover_dims(n_params: int, sampler_class: Callable[[], BaseSampler]) -> None: def objective(trial: optuna.Trial) -> float: xs = [trial.suggest_float(f"x{dim}", -10, 10) for dim in range(n_params)] return sum(xs) n_objectives = 1 n_trials = 8 study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler_class()) study.optimize(objective, n_trials=n_trials) assert len(study.trials) == n_trials @pytest.mark.parametrize( "crossover,population_size", [ (UniformCrossover(), 1), (BLXAlphaCrossover(), 1), (SBXCrossover(), 1), (VSBXCrossover(), 1), (UNDXCrossover(), 2), (SPXCrossover(), 2), ], ) def test_crossover_invalid_population(crossover: BaseCrossover, population_size: int) -> None: with pytest.raises(ValueError): NSGAIISampler(population_size=population_size, crossover=crossover) @pytest.mark.parametrize( "crossover", [ UniformCrossover(), BLXAlphaCrossover(), SPXCrossover(), SBXCrossover(), VSBXCrossover(), UNDXCrossover(), ], ) def test_crossover_numerical_distribution(crossover: BaseCrossover) -> None: study = optuna.study.create_study() rng = np.random.RandomState() search_space = {"x": FloatDistribution(1, 10), "y": IntDistribution(1, 10)} numerical_transform = _SearchSpaceTransform(search_space) parent_params = np.array([[1.0, 2], [3.0, 4]]) if crossover.n_parents == 3: parent_params = np.append(parent_params, [[5.0, 6]], axis=0) child_params = crossover.crossover(parent_params, rng, study, numerical_transform.bounds) assert child_params.ndim == 1 assert len(child_params) == len(search_space) assert not any(np.isnan(child_params)) assert not any(np.isinf(child_params)) def test_crossover_inlined_categorical_distribution() -> None: search_space: dict[str, BaseDistribution] = { "x": CategoricalDistribution(choices=["a", "c"]), "y": CategoricalDistribution(choices=["b", "d"]), } parent_params = np.array([["a", "b"], ["c", "d"]]) rng = np.random.RandomState() child_params = _inlined_categorical_uniform_crossover(parent_params, rng, 0.5, search_space) assert child_params.ndim == 1 assert len(child_params) == len(search_space) assert all([isinstance(param, str) for param in child_params]) # Is child param from correct distribution? search_space["x"].to_internal_repr(child_params[0]) search_space["y"].to_internal_repr(child_params[1]) @pytest.mark.parametrize( "crossover", [ UniformCrossover(), BLXAlphaCrossover(), SPXCrossover(), SBXCrossover(), VSBXCrossover(), UNDXCrossover(), ], ) def test_crossover_duplicated_param_values(crossover: BaseCrossover) -> None: param_values = [1.0, 2.0] study = optuna.study.create_study() rng = np.random.RandomState() search_space = {"x": FloatDistribution(1, 10), "y": IntDistribution(1, 10)} numerical_transform = _SearchSpaceTransform(search_space) parent_params = np.array([param_values, param_values]) if crossover.n_parents == 3: parent_params = np.append(parent_params, [param_values], axis=0) child_params = crossover.crossover(parent_params, rng, study, numerical_transform.bounds) assert child_params.ndim == 1 np.testing.assert_almost_equal(child_params, param_values) @pytest.mark.parametrize( "crossover,rand_value,expected_params", [ (UniformCrossover(), 0.0, np.array([1.0, 2.0])), # p1. (UniformCrossover(), 0.5, np.array([3.0, 4.0])), # p2. (UniformCrossover(), 1.0, np.array([3.0, 4.0])), # p2. (BLXAlphaCrossover(), 0.0, np.array([0.0, 1.0])), # p1 - [1, 1]. (BLXAlphaCrossover(), 0.5, np.array([2.0, 3.0])), # (p1 + p2) / 2. (BLXAlphaCrossover(), 1.0, np.array([4.0, 5.0])), # p2 + [1, 1]. # G = [3, 4], xks=[[-1, 0], [3, 4]. [7, 8]]. (SPXCrossover(), 0.0, np.array([7, 8])), # rs = [0, 0], xks[-1]. (SPXCrossover(), 0.5, np.array([2.75735931, 3.75735931])), # rs = [0.5, 0.25]. (SPXCrossover(), 1.0, np.array([-1.0, 0.0])), # rs = [1, 1], xks[0]. (SBXCrossover(), 0.0, np.array([2.0, 3.0])), # c1 = (p1 + p2) / 2. (SBXCrossover(), 0.5, np.array([3.0, 4.0])), # p2. (SBXCrossover(), 1.0, np.array([3.0, 4.0])), # p2. (VSBXCrossover(), 0.0, np.array([2.0, 3.0])), # c1 = (p1 + p2) / 2. (VSBXCrossover(), 0.5, np.array([3.0, 4.0])), # p2. (VSBXCrossover(), 1.0, np.array([3.0, 4.0])), # p2. # p1, p2 and p3 are on x + 1, and distance from child to PSL is 0. (UNDXCrossover(), -0.5, np.array([3.0, 4.0])), # [2, 3] + [-1, -1] + [0, 0]. (UNDXCrossover(), 0.0, np.array([2.0, 3.0])), # [2, 3] + [0, 0] + [0, 0]. (UNDXCrossover(), 0.5, np.array([1.0, 2.0])), # [2, 3] + [-1, -1] + [0, 0]. ], ) def test_crossover_deterministic( crossover: BaseCrossover, rand_value: float, expected_params: np.ndarray ) -> None: study = optuna.study.create_study() search_space: dict[str, BaseDistribution] = { "x": FloatDistribution(1, 10), "y": FloatDistribution(1, 10), } numerical_transform = _SearchSpaceTransform(search_space) parent_params = np.array([[1.0, 2.0], [3.0, 4.0]]) if crossover.n_parents == 3: parent_params = np.append(parent_params, [[5.0, 6.0]], axis=0) def _rand(*args: Any, **kwargs: Any) -> Any: if len(args) == 0: return rand_value return np.full(args[0], rand_value) def _normal(*args: Any, **kwargs: Any) -> Any: if kwargs.get("size") is None: return rand_value return np.full(kwargs.get("size"), rand_value) # type: ignore[arg-type] rng = Mock() rng.rand = Mock(side_effect=_rand) rng.normal = Mock(side_effect=_normal) child_params = crossover.crossover(parent_params, rng, study, numerical_transform.bounds) np.testing.assert_almost_equal(child_params, expected_params) optuna-3.5.0/tests/samplers_tests/test_nsgaiii.py000066400000000000000000000476361453453102400223140ustar00rootroot00000000000000from __future__ import annotations from collections import Counter from collections.abc import Sequence from unittest.mock import MagicMock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna.samplers import BaseSampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _associate_individuals_with_reference_points, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _generate_default_reference_point, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _normalize_objective_values, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import ( _preserve_niche_individuals, ) from optuna.samplers._nsgaiii._elite_population_selection_strategy import _COEF from optuna.samplers._nsgaiii._elite_population_selection_strategy import _filter_inf from optuna.samplers._nsgaiii._sampler import _POPULATION_CACHE_KEY_PREFIX from optuna.samplers._nsgaiii._sampler import NSGAIIISampler from optuna.samplers.nsgaii import BaseCrossover from optuna.samplers.nsgaii import BLXAlphaCrossover from optuna.samplers.nsgaii import SBXCrossover from optuna.samplers.nsgaii import SPXCrossover from optuna.samplers.nsgaii import UNDXCrossover from optuna.samplers.nsgaii import UniformCrossover from optuna.samplers.nsgaii import VSBXCrossover from optuna.samplers.nsgaii._after_trial_strategy import NSGAIIAfterTrialStrategy from optuna.trial import create_trial from optuna.trial import FrozenTrial def test_population_size() -> None: # Set `population_size` to 10. sampler = NSGAIIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers._nsgaiii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {0: 10, 1: 10, 2: 10, 3: 10} # Set `population_size` to 2. sampler = NSGAIIISampler(population_size=2) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=40) generations = Counter( [t.system_attrs[optuna.samplers._nsgaiii._sampler._GENERATION_KEY] for t in study.trials] ) assert generations == {i: 2 for i in range(20)} # Invalid population size. with pytest.raises(ValueError): # Less than 2. NSGAIIISampler(population_size=1) with pytest.raises(ValueError): mock_crossover = MagicMock(spec=BaseCrossover) mock_crossover.configure_mock(n_parents=3) NSGAIIISampler(population_size=2, crossover=mock_crossover) def test_mutation_prob() -> None: NSGAIIISampler(mutation_prob=None) NSGAIIISampler(mutation_prob=0.0) NSGAIIISampler(mutation_prob=0.5) NSGAIIISampler(mutation_prob=1.0) with pytest.raises(ValueError): NSGAIIISampler(mutation_prob=-0.5) with pytest.raises(ValueError): NSGAIIISampler(mutation_prob=1.1) def test_crossover_prob() -> None: NSGAIIISampler(crossover_prob=0.0) NSGAIIISampler(crossover_prob=0.5) NSGAIIISampler(crossover_prob=1.0) with pytest.raises(ValueError): NSGAIIISampler(crossover_prob=-0.5) with pytest.raises(ValueError): NSGAIIISampler(crossover_prob=1.1) def test_swapping_prob() -> None: NSGAIIISampler(swapping_prob=0.0) NSGAIIISampler(swapping_prob=0.5) NSGAIIISampler(swapping_prob=1.0) with pytest.raises(ValueError): NSGAIIISampler(swapping_prob=-0.5) with pytest.raises(ValueError): NSGAIIISampler(swapping_prob=1.1) def test_constraints_func_none() -> None: n_trials = 4 n_objectives = 2 sampler = NSGAIIISampler(population_size=2) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) assert len(study.trials) == n_trials for trial in study.trials: assert _CONSTRAINTS_KEY not in trial.system_attrs @pytest.mark.parametrize("constraint_value", [-1.0, 0.0, 1.0, -float("inf"), float("inf")]) def test_constraints_func(constraint_value: float) -> None: n_trials = 4 n_objectives = 2 constraints_func_call_count = 0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: nonlocal constraints_func_call_count constraints_func_call_count += 1 return (constraint_value + trial.number,) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) assert len(study.trials) == n_trials assert constraints_func_call_count == n_trials for trial in study.trials: for x, y in zip(trial.system_attrs[_CONSTRAINTS_KEY], (constraint_value + trial.number,)): assert x == y def test_constraints_func_nan() -> None: n_trials = 4 n_objectives = 2 def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = NSGAIIISampler(population_size=2, constraints_func=constraints_func) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) with pytest.raises(ValueError): study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) trials = study.get_trials() assert len(trials) == 1 # The error stops optimization, but completed trials are recorded. assert all(0 <= x <= 1 for x in trials[0].params.values()) # The params are normal. assert trials[0].values == list(trials[0].params.values()) # The values are normal. assert trials[0].system_attrs[_CONSTRAINTS_KEY] is None # None is set for constraints. def test_study_system_attr_for_population_cache() -> None: sampler = NSGAIIISampler(population_size=10) study = optuna.create_study(directions=["minimize"], sampler=sampler) def get_cached_entries( study: optuna.study.Study, ) -> list[tuple[int, list[int]]]: study_system_attrs = study._storage.get_study_system_attrs(study._study_id) return [ v for k, v in study_system_attrs.items() if k.startswith(_POPULATION_CACHE_KEY_PREFIX) ] study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 0 study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=1) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 0 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. study.optimize(lambda t: [t.suggest_float("x", 0, 9)], n_trials=10) cached_entries = get_cached_entries(study) assert len(cached_entries) == 1 assert cached_entries[0][0] == 1 # Cached generation. assert len(cached_entries[0][1]) == 10 # Population size. def test_constraints_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIIISampler(constraints_func=lambda _: [0]) def test_child_generation_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIIISampler(child_generation_strategy=lambda study, search_space, parent_population: {}) def test_after_trial_strategy_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): NSGAIIISampler(after_trial_strategy=lambda study, trial, state, value: None) def test_call_after_trial_of_random_sampler() -> None: sampler = NSGAIIISampler() study = optuna.create_study(sampler=sampler) with patch.object( sampler._random_sampler, "after_trial", wraps=sampler._random_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_call_after_trial_of_after_trial_strategy() -> None: sampler = NSGAIIISampler() study = optuna.create_study(sampler=sampler) with patch.object(sampler, "_after_trial_strategy") as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @patch("optuna.samplers.nsgaii._after_trial_strategy._process_constraints_after_trial") def test_nsgaii_after_trial_strategy(mock_func: MagicMock) -> None: def constraints_func(_: FrozenTrial) -> Sequence[float]: return (float("nan"),) state = optuna.trial.TrialState.FAIL study = optuna.create_study() trial = optuna.trial.create_trial(state=state) after_trial_strategy_without_constrains = NSGAIIAfterTrialStrategy() after_trial_strategy_without_constrains(study, trial, state) assert mock_func.call_count == 0 after_trial_strategy_with_constrains = NSGAIIAfterTrialStrategy( constraints_func=constraints_func ) after_trial_strategy_with_constrains(study, trial, state) assert mock_func.call_count == 1 parametrize_crossover_population = pytest.mark.parametrize( "crossover,population_size", [ (UniformCrossover(), 2), (BLXAlphaCrossover(), 2), (SBXCrossover(), 2), (VSBXCrossover(), 2), (UNDXCrossover(), 2), (UNDXCrossover(), 3), ], ) @parametrize_crossover_population @pytest.mark.parametrize("n_objectives", [1, 2, 3]) def test_crossover_objectives( n_objectives: int, crossover: BaseSampler, population_size: int ) -> None: n_trials = 8 sampler = NSGAIIISampler(population_size=population_size) study = optuna.create_study(directions=["minimize"] * n_objectives, sampler=sampler) study.optimize( lambda t: [t.suggest_float(f"x{i}", 0, 1) for i in range(n_objectives)], n_trials=n_trials, ) assert len(study.trials) == n_trials @parametrize_crossover_population @pytest.mark.parametrize("n_params", [1, 2, 3]) def test_crossover_dims(n_params: int, crossover: BaseSampler, population_size: int) -> None: def objective(trial: optuna.Trial) -> float: xs = [trial.suggest_float(f"x{dim}", -10, 10) for dim in range(n_params)] return sum(xs) n_trials = 8 sampler = NSGAIIISampler(population_size=population_size) study = optuna.create_study(directions=["minimize"], sampler=sampler) study.optimize(objective, n_trials=n_trials) assert len(study.trials) == n_trials @pytest.mark.parametrize( "crossover,population_size", [ (UniformCrossover(), 1), (BLXAlphaCrossover(), 1), (SBXCrossover(), 1), (VSBXCrossover(), 1), (UNDXCrossover(), 2), (SPXCrossover(), 2), ], ) def test_crossover_invalid_population(crossover: BaseCrossover, population_size: int) -> None: with pytest.raises(ValueError): NSGAIIISampler(population_size=population_size, crossover=crossover) @pytest.mark.parametrize( "n_objectives,dividing_parameter,expected_reference_points", [ (1, 3, [[3.0]]), (2, 2, [[0.0, 2.0], [1.0, 1.0], [2.0, 0.0]]), (2, 3, [[0.0, 3.0], [1.0, 2.0], [2.0, 1.0], [3.0, 0.0]]), ( 3, 2, [ [0.0, 0.0, 2.0], [0.0, 1.0, 1.0], [0.0, 2.0, 0.0], [1.0, 0.0, 1.0], [1.0, 1.0, 0.0], [2.0, 0.0, 0.0], ], ), ], ) def test_generate_reference_point( n_objectives: int, dividing_parameter: int, expected_reference_points: Sequence[Sequence[int]] ) -> None: actual_reference_points = sorted( _generate_default_reference_point(n_objectives, dividing_parameter).tolist() ) assert actual_reference_points == expected_reference_points @pytest.mark.parametrize( "objective_values, expected_normalized_value", [ ( [ [1.0, 2.0], [float("inf"), 0.5], ], [ [1.0, 2.0], [1.0, 0.5], ], ), ( [ [1.0, float("inf")], [float("inf"), 0.5], ], [ [1.0, 0.5], [1.0, 0.5], ], ), ( [ [1.0, float("inf")], [3.0, 1.0], [2.0, 3.0], [float("inf"), 0.5], ], [ [1.0, 3.0 + _COEF * 2.5], [3.0, 1.0], [2.0, 3.0], [3.0 + _COEF * 2.0, 0.5], ], ), ( [ [2.0, 3.0], [-float("inf"), 3.5], ], [ [2.0, 3.0], [2.0, 3.5], ], ), ( [ [2.0, -float("inf")], [-float("inf"), 3.5], ], [ [2.0, 3.5], [2.0, 3.5], ], ), ( [ [4.0, -float("inf")], [3.0, 1.0], [2.0, 3.0], [-float("inf"), 3.5], ], [ [4.0, 1.0 - _COEF * 2.5], [3.0, 1.0], [2.0, 3.0], [2.0 - _COEF * 2.0, 3.5], ], ), ( [ [1.0, float("inf")], [3.0, -float("inf")], [float("inf"), 2.0], [-float("inf"), 3.5], ], [ [1.0, 3.5 + _COEF * 1.5], [3.0, 2.0 - _COEF * 1.5], [3.0 + _COEF * 2.0, 2.0], [1.0 - _COEF * 2.0, 3.5], ], ), ], ) def test_filter_inf( objective_values: Sequence[Sequence[int]], expected_normalized_value: Sequence[Sequence[int]] ) -> None: population = [create_trial(values=values) for values in objective_values] np.testing.assert_almost_equal(_filter_inf(population), np.array(expected_normalized_value)) @pytest.mark.parametrize( "objective_values, expected_normalized_value", [ ( [ [2.71], [1.41], [3.14], ], [ [(2.71 - 1.41) / (3.14 - 1.41)], [0], [1.0], ], ), ( [ [1.0, 2.0, 3.0], [3.0, 1.0, 2.0], [2.0, 3.0, 1.0], [2.0, 2.0, 2.0], [4.0, 5.0, 6.0], [6.0, 4.0, 5.0], [5.0, 6.0, 4.0], [4.0, 4.0, 4.0], ], [ [0.0, 1.0 / 3.0, 2.0 / 3.0], [2.0 / 3.0, 0.0, 1.0 / 3.0], [1.0 / 3.0, 2.0 / 3.0, 0.0], [1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0], [1.0, 4.0 / 3.0, 5.0 / 3.0], [5.0 / 3.0, 1.0, 4.0 / 3.0], [4.0 / 3.0, 5.0 / 3.0, 1.0], [1.0, 1.0, 1.0], ], ), ( [ [1.0, 2.0, 3.0], [3.0, 1.0, 2.0], ], [ [0.0, 1.0, 1.0], [1.0, 0.0, 0.0], ], ), ], ) def test_normalize( objective_values: Sequence[Sequence[int]], expected_normalized_value: Sequence[Sequence[int]] ) -> None: np.testing.assert_almost_equal( _normalize_objective_values(np.array(objective_values)), np.array(expected_normalized_value), ) @pytest.mark.parametrize( "objective_values, expected_indices, expected_distances", [ ([[1.0], [2.0], [0.0], [3.0]], [0, 0, 0, 0], [0.0, 0.0, 0.0, 0.0]), ( [ [1.0, 2.0, 3.0], [3.0, 1.0, 2.0], [2.0, 3.0, 1.0], [2.0, 2.0, 2.0], [4.0, 5.0, 6.0], [6.0, 4.0, 5.0], [5.0, 6.0, 4.0], [4.0, 4.0, 4.0], [0.0, 1.0, 10.0], [10.0, 0.0, 1.0], [1.0, 10.0, 0.0], ], [4, 2, 1, 1, 4, 2, 1, 1, 5, 0, 3], [ 1.22474487, 1.22474487, 1.22474487, 2.0, 4.0620192, 4.0620192, 4.0620192, 4.0, 1.0, 1.0, 1.0, ], ), ], ) def test_associate( objective_values: Sequence[Sequence[float]], expected_indices: Sequence[int], expected_distances: Sequence[float], ) -> None: population = np.array(objective_values) n_objectives = population.shape[1] reference_points = _generate_default_reference_point( n_objectives=n_objectives, dividing_parameter=2 ) ( closest_reference_points, distance_reference_points, ) = _associate_individuals_with_reference_points(population, reference_points) assert np.all(closest_reference_points == expected_indices) np.testing.assert_almost_equal(distance_reference_points, expected_distances) @pytest.mark.parametrize( "population_value,closest_reference_points, distance_reference_points, " "expected_population_indices", [ ( [[1.0], [2.0], [0.0], [3.0], [3.5], [5.5], [1.2], [3.3], [4.8]], [0, 0, 0, 0, 0, 0, 0, 0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [3, 0, 2, 4, 1], ), ( [ [4.0, 5.0, 6.0], [6.0, 4.0, 5.0], [5.0, 6.0, 4.0], [4.0, 4.0, 4.0], [0.0, 1.0, 10.0], [10.0, 0.0, 1.0], [1.0, 10.0, 0.0], ], [4, 2, 1, 1, 4, 2, 1, 1, 5, 0, 3], [ 1.22474487, 1.22474487, 1.22474487, 2.0, 4.0620192, 4.0620192, 4.0620192, 4.0, 1.0, 1.0, 1.0, ], [6, 5, 4, 0, 1], ), ], ) def test_niching( population_value: Sequence[Sequence[float]], closest_reference_points: Sequence[int], distance_reference_points: Sequence[float], expected_population_indices: Sequence[int], ) -> None: sampler = NSGAIIISampler(seed=42) target_population_size = 5 elite_population_num = 4 population = [create_trial(values=value) for value in population_value] actual_additional_elite_population = [ trial.values for trial in _preserve_niche_individuals( target_population_size, elite_population_num, population, np.array(closest_reference_points), np.array(distance_reference_points), sampler._rng.rng, ) ] expected_additional_elite_population = [ population[idx].values for idx in expected_population_indices ] assert np.all(actual_additional_elite_population == expected_additional_elite_population) def test_niching_unexpected_target_population_size() -> None: sampler = NSGAIIISampler(seed=42) target_population_size = 2 elite_population_num = 1 population = [create_trial(values=[1.0])] with pytest.raises(ValueError): _preserve_niche_individuals( target_population_size, elite_population_num, population, np.array([0]), np.array([0.0]), sampler._rng.rng, ) optuna-3.5.0/tests/samplers_tests/test_partial_fixed.py000066400000000000000000000113201453453102400234610ustar00rootroot00000000000000from unittest.mock import patch import warnings import pytest import optuna from optuna.samplers import PartialFixedSampler from optuna.samplers import RandomSampler from optuna.trial import Trial def test_fixed_sampling() -> None: def objective(trial: Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -10, 10) return x**2 + y**2 study0 = optuna.create_study() study0.sampler = RandomSampler(seed=42) study0.optimize(objective, n_trials=1) x_sampled0 = study0.trials[0].params["x"] # Fix parameter ``y`` as 0. study1 = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study1.sampler = PartialFixedSampler( fixed_params={"y": 0}, base_sampler=RandomSampler(seed=42) ) study1.optimize(objective, n_trials=1) x_sampled1 = study1.trials[0].params["x"] y_sampled1 = study1.trials[0].params["y"] assert x_sampled1 == x_sampled0 assert y_sampled1 == 0 def test_float_to_int() -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 fixed_y = 0.5 # Parameters of Int-type-distribution are rounded to int-type, # even if they are defined as float-type. study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = PartialFixedSampler( fixed_params={"y": fixed_y}, base_sampler=study.sampler ) # Since `fixed_y` is out-of-the-range value in the corresponding suggest_int, # `UserWarning` will occur. with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) assert study.trials[0].params["y"] == int(fixed_y) @pytest.mark.parametrize("fixed_y", [-2, 2]) def test_out_of_the_range_numerical(fixed_y: int) -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -1, 1) y = trial.suggest_int("y", -1, 1) return x**2 + y**2 # It is possible to fix numerical parameters as out-of-the-range value. # `UserWarning` will occur. study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = PartialFixedSampler( fixed_params={"y": fixed_y}, base_sampler=study.sampler ) with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) assert study.trials[0].params["y"] == fixed_y def test_out_of_the_range_categorical() -> None: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -1, 1) y = trial.suggest_categorical("y", [-1, 0, 1]) return x**2 + y**2 fixed_y = 2 # It isn't possible to fix categorical parameters as out-of-the-range value. # `ValueError` will occur. study = optuna.create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = PartialFixedSampler( fixed_params={"y": fixed_y}, base_sampler=study.sampler ) with pytest.raises(ValueError): study.optimize(objective, n_trials=1) def test_partial_fixed_experimental_warning() -> None: study = optuna.create_study() with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.PartialFixedSampler(fixed_params={"x": 0}, base_sampler=study.sampler) def test_call_after_trial_of_base_sampler() -> None: base_sampler = RandomSampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = PartialFixedSampler(fixed_params={}, base_sampler=base_sampler) study = optuna.create_study(sampler=sampler) with patch.object(base_sampler, "after_trial", wraps=base_sampler.after_trial) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_fixed_none_value_sampling() -> None: def objective(trial: Trial) -> float: trial.suggest_categorical("x", (None, 0)) return 0.0 tpe = optuna.samplers.TPESampler() with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) # In this following case , "x" should sample only `None` sampler = optuna.samplers.PartialFixedSampler(fixed_params={"x": None}, base_sampler=tpe) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) for trial in study.trials: assert trial.params["x"] is None optuna-3.5.0/tests/samplers_tests/test_qmc.py000066400000000000000000000321521453453102400214340ustar00rootroot00000000000000from typing import Any from typing import Callable from typing import Dict from unittest.mock import Mock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna.distributions import BaseDistribution from optuna.trial import Trial from optuna.trial import TrialState _SEARCH_SPACE = { "x1": optuna.distributions.IntDistribution(0, 10), "x2": optuna.distributions.IntDistribution(1, 10, log=True), "x3": optuna.distributions.FloatDistribution(0, 10), "x4": optuna.distributions.FloatDistribution(1, 10, log=True), "x5": optuna.distributions.FloatDistribution(1, 10, step=3), "x6": optuna.distributions.CategoricalDistribution([1, 4, 7, 10]), } # TODO(kstoneriv3): `QMCSampler` can be initialized without this wrapper # Remove this after the experimental warning is removed. def _init_QMCSampler_without_exp_warning(**kwargs: Any) -> optuna.samplers.QMCSampler: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = optuna.samplers.QMCSampler(**kwargs) return sampler def test_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.QMCSampler() @pytest.mark.parametrize("qmc_type", ["sobol", "halton", "non-qmc"]) def test_invalid_qmc_type(qmc_type: str) -> None: if qmc_type == "non-qmc": with pytest.raises(ValueError): _init_QMCSampler_without_exp_warning(qmc_type=qmc_type) else: _init_QMCSampler_without_exp_warning(qmc_type=qmc_type) def test_initial_seeding() -> None: with patch.object(optuna.samplers.QMCSampler, "_log_asynchronous_seeding") as mock_log_async: sampler = _init_QMCSampler_without_exp_warning(scramble=True) mock_log_async.assert_called_once() assert isinstance(sampler._seed, int) def test_infer_relative_search_space() -> None: def objective(trial: Trial) -> float: ret: float = trial.suggest_int("x1", 0, 10) ret += trial.suggest_int("x2", 1, 10, log=True) ret += trial.suggest_float("x3", 0, 10) ret += trial.suggest_float("x4", 1, 10, log=True) ret += trial.suggest_float("x5", 1, 10, step=3) _ = trial.suggest_categorical("x6", [1, 4, 7, 10]) return ret sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) trial = Mock() # In case no past trials. assert sampler.infer_relative_search_space(study, trial) == {} # In case there is a past trial. study.optimize(objective, n_trials=1) relative_search_space = sampler.infer_relative_search_space(study, trial) assert len(relative_search_space.keys()) == 5 assert set(relative_search_space.keys()) == {"x1", "x2", "x3", "x4", "x5"} # In case self._initial_trial already exists. new_search_space: Dict[str, BaseDistribution] = {"x": Mock()} sampler._initial_search_space = new_search_space assert sampler.infer_relative_search_space(study, trial) == new_search_space def test_infer_initial_search_space() -> None: trial = Mock() sampler = _init_QMCSampler_without_exp_warning() # Can it handle empty search space? trial.distributions = {} initial_search_space = sampler._infer_initial_search_space(trial) assert initial_search_space == {} # Does it exclude only categorical distribution? search_space = _SEARCH_SPACE.copy() trial.distributions = search_space initial_search_space = sampler._infer_initial_search_space(trial) search_space.pop("x6") assert initial_search_space == search_space def test_sample_independent() -> None: objective: Callable[[Trial], float] = lambda t: t.suggest_categorical("x", [1.0, 2.0]) independent_sampler = optuna.samplers.RandomSampler() with patch.object( independent_sampler, "sample_independent", wraps=independent_sampler.sample_independent ) as mock_sample_indep: sampler = _init_QMCSampler_without_exp_warning(independent_sampler=independent_sampler) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=1) assert mock_sample_indep.call_count == 1 # Relative sampling of `QMCSampler` does not support categorical distribution. # Thus, `independent_sampler.sample_independent` is called twice. study.optimize(objective, n_trials=1) assert mock_sample_indep.call_count == 2 # Unseen parameter is sampled by independent sampler. new_objective: Callable[[Trial], int] = lambda t: t.suggest_int("y", 0, 10) study.optimize(new_objective, n_trials=1) assert mock_sample_indep.call_count == 3 def test_warn_asynchronous_seeding() -> None: # Relative sampling of `QMCSampler` does not support categorical distribution. # Thus, `independent_sampler.sample_independent` is called twice. # '_log_independent_sampling is not called in the first trial so called once in total. objective: Callable[[Trial], float] = lambda t: t.suggest_categorical("x", [1.0, 2.0]) with patch.object(optuna.samplers.QMCSampler, "_log_asynchronous_seeding") as mock_log_async: sampler = _init_QMCSampler_without_exp_warning( scramble=True, warn_asynchronous_seeding=False ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_async.call_count == 0 sampler = _init_QMCSampler_without_exp_warning(scramble=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_async.call_count == 1 def test_warn_independent_sampling() -> None: # Relative sampling of `QMCSampler` does not support categorical distribution. # Thus, `independent_sampler.sample_independent` is called twice. # '_log_independent_sampling is not called in the first trial so called once in total. objective: Callable[[Trial], float] = lambda t: t.suggest_categorical("x", [1.0, 2.0]) with patch.object(optuna.samplers.QMCSampler, "_log_independent_sampling") as mock_log_indep: sampler = _init_QMCSampler_without_exp_warning(warn_independent_sampling=False) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_indep.call_count == 0 sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=2) assert mock_log_indep.call_count == 1 def test_sample_relative() -> None: search_space = _SEARCH_SPACE.copy() search_space.pop("x6") sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) trial = Mock() # Make sure that sample type, shape is OK. for _ in range(3): sample = sampler.sample_relative(study, trial, search_space) assert 0 <= sample["x1"] <= 10 assert 1 <= sample["x2"] <= 10 assert 0 <= sample["x3"] <= 10 assert 1 <= sample["x4"] <= 10 assert 1 <= sample["x5"] <= 10 assert isinstance(sample["x1"], int) assert isinstance(sample["x2"], int) assert sample["x5"] in (1, 4, 7, 10) # If empty search_space, return {}. assert sampler.sample_relative(study, trial, {}) == {} def test_sample_relative_halton() -> None: n, d = 8, 5 search_space: Dict[str, BaseDistribution] = { f"x{i}": optuna.distributions.FloatDistribution(0, 1) for i in range(d) } sampler = _init_QMCSampler_without_exp_warning(scramble=False, qmc_type="halton") study = optuna.create_study(sampler=sampler) trial = Mock() # Make sure that sample type, shape is OK. samples = np.zeros((n, d)) for i in range(n): sample = sampler.sample_relative(study, trial, search_space) for j in range(d): samples[i, j] = sample[f"x{j}"] ref_samples = np.array( [ [0.0, 0.0, 0.0, 0.0, 0.0], [0.5, 0.33333333, 0.2, 0.14285714, 0.09090909], [0.25, 0.66666667, 0.4, 0.28571429, 0.18181818], [0.75, 0.11111111, 0.6, 0.42857143, 0.27272727], [0.125, 0.44444444, 0.8, 0.57142857, 0.36363636], [0.625, 0.77777778, 0.04, 0.71428571, 0.45454545], [0.375, 0.22222222, 0.24, 0.85714286, 0.54545455], [0.875, 0.55555556, 0.44, 0.02040816, 0.63636364], ] ) # If empty search_space, return {}. np.testing.assert_allclose(samples, ref_samples, rtol=1e-6) def test_sample_relative_sobol() -> None: n, d = 8, 5 search_space: Dict[str, BaseDistribution] = { f"x{i}": optuna.distributions.FloatDistribution(0, 1) for i in range(d) } sampler = _init_QMCSampler_without_exp_warning(scramble=False, qmc_type="sobol") study = optuna.create_study(sampler=sampler) trial = Mock() # Make sure that sample type, shape is OK. samples = np.zeros((n, d)) for i in range(n): sample = sampler.sample_relative(study, trial, search_space) for j in range(d): samples[i, j] = sample[f"x{j}"] ref_samples = np.array( [ [0.0, 0.0, 0.0, 0.0, 0.0], [0.5, 0.5, 0.5, 0.5, 0.5], [0.75, 0.25, 0.25, 0.25, 0.75], [0.25, 0.75, 0.75, 0.75, 0.25], [0.375, 0.375, 0.625, 0.875, 0.375], [0.875, 0.875, 0.125, 0.375, 0.875], [0.625, 0.125, 0.875, 0.625, 0.625], [0.125, 0.625, 0.375, 0.125, 0.125], ] ) # If empty search_space, return {}. np.testing.assert_allclose(samples, ref_samples, rtol=1e-6) @pytest.mark.parametrize("scramble", [True, False]) @pytest.mark.parametrize("qmc_type", ["sobol", "halton"]) @pytest.mark.parametrize("seed", [0, 12345]) def test_sample_relative_seeding(scramble: bool, qmc_type: str, seed: int) -> None: objective: Callable[[Trial], float] = lambda t: t.suggest_float("x", 0, 1) # Base case. sampler = _init_QMCSampler_without_exp_warning(scramble=scramble, qmc_type=qmc_type, seed=seed) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10, n_jobs=1) past_trials = study._storage.get_all_trials(study._study_id, states=(TrialState.COMPLETE,)) past_trials = [t for t in past_trials if t.number > 0] values = [t.params["x"] for t in past_trials] # Sequential case. sampler = _init_QMCSampler_without_exp_warning(scramble=scramble, qmc_type=qmc_type, seed=seed) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10, n_jobs=1) past_trials_sequential = study._storage.get_all_trials( study._study_id, states=(TrialState.COMPLETE,) ) past_trials_sequential = [t for t in past_trials_sequential if t.number > 0] values_sequential = [t.params["x"] for t in past_trials_sequential] np.testing.assert_allclose(values, values_sequential, rtol=1e-6) # Parallel case (n_jobs=3): # Same parameters might be evaluated multiple times. sampler = _init_QMCSampler_without_exp_warning(scramble=scramble, qmc_type=qmc_type, seed=seed) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=30, n_jobs=3) past_trials_parallel = study._storage.get_all_trials( study._study_id, states=(TrialState.COMPLETE,) ) past_trials_parallel = [t for t in past_trials_parallel if t.number > 0] values_parallel = [t.params["x"] for t in past_trials_parallel] for v in values: assert np.any( np.isclose(v, values_parallel, rtol=1e-6) ), f"v: {v} of values: {values} is not included in values_parallel: {values_parallel}." def test_call_after_trial() -> None: sampler = _init_QMCSampler_without_exp_warning() study = optuna.create_study(sampler=sampler) with patch.object( sampler._independent_sampler, "after_trial", wraps=sampler._independent_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 @pytest.mark.parametrize("qmc_type", ["sobol", "halton"]) def test_sample_qmc(qmc_type: str) -> None: sampler = _init_QMCSampler_without_exp_warning(qmc_type=qmc_type) study = Mock() search_space = _SEARCH_SPACE.copy() search_space.pop("x6") with patch.object(sampler, "_find_sample_id", side_effect=[0, 1, 2, 4, 9]) as _: # Make sure that the shape of sample is correct. sample = sampler._sample_qmc(study, search_space) assert sample.shape == (1, 5) def test_find_sample_id() -> None: sampler = _init_QMCSampler_without_exp_warning(qmc_type="halton", seed=0) study = optuna.create_study() for i in range(5): assert sampler._find_sample_id(study) == i # Change seed but without scramble. The hash should remain the same. with patch.object(sampler, "_seed", 1) as _: assert sampler._find_sample_id(study) == 5 # Seed is considered only when scrambling is enabled. with patch.object(sampler, "_scramble", True) as _: assert sampler._find_sample_id(study) == 0 # Change qmc_type. with patch.object(sampler, "_qmc_type", "sobol") as _: assert sampler._find_sample_id(study) == 0 optuna-3.5.0/tests/samplers_tests/test_samplers.py000066400000000000000000001161371453453102400225100ustar00rootroot00000000000000from __future__ import annotations from collections.abc import Callable from collections.abc import Sequence import multiprocessing from multiprocessing.managers import DictProxy import os import pickle from typing import Any from unittest.mock import patch import warnings from _pytest.fixtures import SubRequest from _pytest.mark.structures import MarkDecorator import numpy as np import pytest import optuna from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalChoiceType from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.integration.botorch import logei_candidates_func from optuna.integration.botorch import qei_candidates_func from optuna.samplers import BaseSampler from optuna.samplers._lazy_random_state import LazyRandomState from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.objectives import pruned_objective from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.trial import TrialState parametrize_sampler = pytest.mark.parametrize( "sampler_class", [ optuna.samplers.RandomSampler, lambda: optuna.samplers.TPESampler(n_startup_trials=0), lambda: optuna.samplers.TPESampler(n_startup_trials=0, multivariate=True), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0, use_separable_cma=True), pytest.param( lambda: optuna.integration.PyCmaSampler(n_startup_trials=0), marks=pytest.mark.integration, ), optuna.samplers.NSGAIISampler, optuna.samplers.NSGAIIISampler, optuna.samplers.QMCSampler, pytest.param( lambda: optuna.integration.BoTorchSampler( n_startup_trials=0, candidates_func=logei_candidates_func, ), marks=pytest.mark.integration, ), pytest.param( lambda: optuna.integration.BoTorchSampler( n_startup_trials=0, candidates_func=qei_candidates_func, ), marks=pytest.mark.integration, ), ], ) parametrize_relative_sampler = pytest.mark.parametrize( "relative_sampler_class", [ lambda: optuna.samplers.TPESampler(n_startup_trials=0, multivariate=True), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0, use_separable_cma=True), pytest.param( lambda: optuna.integration.PyCmaSampler(n_startup_trials=0), marks=pytest.mark.integration, ), ], ) parametrize_multi_objective_sampler = pytest.mark.parametrize( "multi_objective_sampler_class", [ optuna.samplers.NSGAIISampler, optuna.samplers.NSGAIIISampler, lambda: optuna.samplers.TPESampler(n_startup_trials=0), pytest.param( lambda: optuna.integration.BoTorchSampler(n_startup_trials=0), marks=pytest.mark.integration, ), ], ) sampler_class_with_seed: dict[str, tuple[Callable[[int], BaseSampler], bool]] = { "RandomSampler": (lambda seed: optuna.samplers.RandomSampler(seed=seed), False), "TPESampler": (lambda seed: optuna.samplers.TPESampler(seed=seed), False), "multivariate TPESampler": ( lambda seed: optuna.samplers.TPESampler(multivariate=True, seed=seed), False, ), "CmaEsSampler": (lambda seed: optuna.samplers.CmaEsSampler(seed=seed), False), "separable CmaEsSampler": ( lambda seed: optuna.samplers.CmaEsSampler(seed=seed, use_separable_cma=True), False, ), "PyCmaSampler": (lambda seed: optuna.integration.PyCmaSampler(seed=seed), True), "NSGAIISampler": (lambda seed: optuna.samplers.NSGAIISampler(seed=seed), False), "NSGAIIISampler": (lambda seed: optuna.samplers.NSGAIIISampler(seed=seed), False), "QMCSampler": (lambda seed: optuna.samplers.QMCSampler(seed=seed), False), "BoTorchSampler": (lambda seed: optuna.integration.BoTorchSampler(seed=seed), True), } param_sampler_with_seed = [] param_sampler_name_with_seed = [] for sampler_name, (sampler_class, integration_flag) in sampler_class_with_seed.items(): if integration_flag: param_sampler_with_seed.append( pytest.param(sampler_class, id=sampler_name, marks=pytest.mark.integration) ) param_sampler_name_with_seed.append( pytest.param(sampler_name, marks=pytest.mark.integration) ) else: param_sampler_with_seed.append(pytest.param(sampler_class, id=sampler_name)) param_sampler_name_with_seed.append(pytest.param(sampler_name)) parametrize_sampler_with_seed = pytest.mark.parametrize("sampler_class", param_sampler_with_seed) parametrize_sampler_name_with_seed = pytest.mark.parametrize( "sampler_name", param_sampler_name_with_seed ) @pytest.mark.parametrize( "sampler_class,expected_has_rng,expected_has_another_sampler", [ (optuna.samplers.RandomSampler, True, False), (lambda: optuna.samplers.TPESampler(n_startup_trials=0), True, True), (lambda: optuna.samplers.TPESampler(n_startup_trials=0, multivariate=True), True, True), (lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), True, True), pytest.param( lambda: optuna.integration.PyCmaSampler(n_startup_trials=0), False, True, marks=pytest.mark.integration, ), (optuna.samplers.NSGAIISampler, True, True), (optuna.samplers.NSGAIIISampler, True, True), ( lambda: optuna.samplers.PartialFixedSampler( fixed_params={"x": 0}, base_sampler=optuna.samplers.RandomSampler() ), False, True, ), (lambda: optuna.samplers.GridSampler(search_space={"x": [0]}), True, False), (lambda: optuna.samplers.QMCSampler(), False, True), pytest.param( lambda: optuna.integration.BoTorchSampler(n_startup_trials=0), False, True, marks=pytest.mark.integration, ), ], ) def test_sampler_reseed_rng( sampler_class: Callable[[], BaseSampler], expected_has_rng: bool, expected_has_another_sampler: bool, ) -> None: def _extract_attr_name_from_sampler_by_cls(sampler: BaseSampler, cls: Any) -> str | None: for name, attr in sampler.__dict__.items(): if isinstance(attr, cls): return name return None sampler = sampler_class() rng_name = _extract_attr_name_from_sampler_by_cls(sampler, LazyRandomState) has_rng = rng_name is not None assert expected_has_rng == has_rng if has_rng: rng_name = str(rng_name) original_random_state = sampler.__dict__[rng_name].rng.get_state() sampler.reseed_rng() random_state = sampler.__dict__[rng_name].rng.get_state() if not isinstance(sampler, optuna.samplers.CmaEsSampler): assert str(original_random_state) != str(random_state) else: # CmaEsSampler has a RandomState that is not reseed by its reseed_rng method. assert str(original_random_state) == str(random_state) had_sampler_name = _extract_attr_name_from_sampler_by_cls(sampler, BaseSampler) has_another_sampler = had_sampler_name is not None assert expected_has_another_sampler == has_another_sampler if has_another_sampler: had_sampler_name = str(had_sampler_name) had_sampler = sampler.__dict__[had_sampler_name] had_sampler_rng_name = _extract_attr_name_from_sampler_by_cls(had_sampler, LazyRandomState) original_had_sampler_random_state = had_sampler.__dict__[ had_sampler_rng_name ].rng.get_state() with patch.object( had_sampler, "reseed_rng", wraps=had_sampler.reseed_rng, ) as mock_object: sampler.reseed_rng() assert mock_object.call_count == 1 had_sampler = sampler.__dict__[had_sampler_name] had_sampler_random_state = had_sampler.__dict__[had_sampler_rng_name].rng.get_state() assert str(original_had_sampler_random_state) != str(had_sampler_random_state) def parametrize_suggest_method(name: str) -> MarkDecorator: return pytest.mark.parametrize( f"suggest_method_{name}", [ lambda t: t.suggest_float(name, 0, 10), lambda t: t.suggest_int(name, 0, 10), lambda t: t.suggest_categorical(name, [0, 1, 2]), lambda t: t.suggest_float(name, 0, 10, step=0.5), lambda t: t.suggest_float(name, 1e-7, 10, log=True), lambda t: t.suggest_int(name, 1, 10, log=True), ], ) @pytest.mark.parametrize( "sampler_class", [ lambda: optuna.samplers.CmaEsSampler(n_startup_trials=0), pytest.param( lambda: optuna.integration.PyCmaSampler(n_startup_trials=0), marks=pytest.mark.integration, ), ], ) def test_raise_error_for_samplers_during_multi_objectives( sampler_class: Callable[[], BaseSampler] ) -> None: study = optuna.study.create_study(directions=["maximize", "maximize"], sampler=sampler_class()) distribution = FloatDistribution(0.0, 1.0) with pytest.raises(ValueError): study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution) with pytest.raises(ValueError): trial = _create_new_trial(study) study.sampler.sample_relative( study, trial, study.sampler.infer_relative_search_space(study, trial) ) @pytest.mark.parametrize("seed", [0, 169208]) def test_pickle_random_sampler(seed: int) -> None: sampler = optuna.samplers.RandomSampler(seed) restored_sampler = pickle.loads(pickle.dumps(sampler)) assert sampler._rng.rng.bytes(10) == restored_sampler._rng.rng.bytes(10) @parametrize_sampler @pytest.mark.parametrize( "distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(0.0, 1.0), FloatDistribution(-1.0, 0.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.1), FloatDistribution(-10.2, 10.2, step=0.1), ], ) def test_float( sampler_class: Callable[[], BaseSampler], distribution: FloatDistribution, ) -> None: study = optuna.study.create_study(sampler=sampler_class()) points = np.array( [ study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution) for _ in range(100) ] ) assert np.all(points >= distribution.low) assert np.all(points <= distribution.high) assert not isinstance( study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution), np.floating, ) if distribution.step is not None: # Check all points are multiples of distribution.step. points -= distribution.low points /= distribution.step round_points = np.round(points) np.testing.assert_almost_equal(round_points, points) @parametrize_sampler @pytest.mark.parametrize( "distribution", [ IntDistribution(-10, 10), IntDistribution(0, 10), IntDistribution(-10, 0), IntDistribution(-10, 10, step=2), IntDistribution(0, 10, step=2), IntDistribution(-10, 0, step=2), IntDistribution(1, 100, log=True), ], ) def test_int(sampler_class: Callable[[], BaseSampler], distribution: IntDistribution) -> None: study = optuna.study.create_study(sampler=sampler_class()) points = np.array( [ study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution) for _ in range(100) ] ) assert np.all(points >= distribution.low) assert np.all(points <= distribution.high) assert not isinstance( study.sampler.sample_independent(study, _create_new_trial(study), "x", distribution), np.integer, ) @parametrize_sampler @pytest.mark.parametrize("choices", [(1, 2, 3), ("a", "b", "c"), (1, "a")]) def test_categorical( sampler_class: Callable[[], BaseSampler], choices: Sequence[CategoricalChoiceType] ) -> None: distribution = CategoricalDistribution(choices) study = optuna.study.create_study(sampler=sampler_class()) def sample() -> float: trial = _create_new_trial(study) param_value = study.sampler.sample_independent(study, trial, "x", distribution) return float(distribution.to_internal_repr(param_value)) points = np.asarray([sample() for i in range(100)]) # 'x' value is corresponding to an index of distribution.choices. assert np.all(points >= 0) assert np.all(points <= len(distribution.choices) - 1) round_points = np.round(points) np.testing.assert_almost_equal(round_points, points) @parametrize_relative_sampler @pytest.mark.parametrize( "x_distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.5), IntDistribution(3, 10), IntDistribution(1, 100, log=True), IntDistribution(3, 9, step=2), ], ) @pytest.mark.parametrize( "y_distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.5), IntDistribution(3, 10), IntDistribution(1, 100, log=True), IntDistribution(3, 9, step=2), ], ) def test_sample_relative_numerical( relative_sampler_class: Callable[[], BaseSampler], x_distribution: BaseDistribution, y_distribution: BaseDistribution, ) -> None: search_space: dict[str, BaseDistribution] = dict(x=x_distribution, y=y_distribution) study = optuna.study.create_study(sampler=relative_sampler_class()) trial = study.ask(search_space) study.tell(trial, sum(trial.params.values())) def sample() -> list[int | float]: params = study.sampler.sample_relative(study, _create_new_trial(study), search_space) return [params[name] for name in search_space] points = np.array([sample() for _ in range(10)]) for i, distribution in enumerate(search_space.values()): assert isinstance( distribution, ( FloatDistribution, IntDistribution, ), ) assert np.all(points[:, i] >= distribution.low) assert np.all(points[:, i] <= distribution.high) for param_value, distribution in zip(sample(), search_space.values()): assert not isinstance(param_value, np.floating) assert not isinstance(param_value, np.integer) if isinstance(distribution, IntDistribution): assert isinstance(param_value, int) else: assert isinstance(param_value, float) @parametrize_relative_sampler def test_sample_relative_categorical(relative_sampler_class: Callable[[], BaseSampler]) -> None: search_space: dict[str, BaseDistribution] = dict( x=CategoricalDistribution([1, 10, 100]), y=CategoricalDistribution([-1, -10, -100]) ) study = optuna.study.create_study(sampler=relative_sampler_class()) trial = study.ask(search_space) study.tell(trial, sum(trial.params.values())) def sample() -> list[float]: params = study.sampler.sample_relative(study, _create_new_trial(study), search_space) return [params[name] for name in search_space] points = np.array([sample() for _ in range(10)]) for i, distribution in enumerate(search_space.values()): assert isinstance(distribution, CategoricalDistribution) assert np.all([v in distribution.choices for v in points[:, i]]) for param_value in sample(): assert not isinstance(param_value, np.floating) assert not isinstance(param_value, np.integer) assert isinstance(param_value, int) @parametrize_relative_sampler @pytest.mark.parametrize( "x_distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.5), IntDistribution(1, 10), IntDistribution(1, 100, log=True), ], ) def test_sample_relative_mixed( relative_sampler_class: Callable[[], BaseSampler], x_distribution: BaseDistribution ) -> None: search_space: dict[str, BaseDistribution] = dict( x=x_distribution, y=CategoricalDistribution([-1, -10, -100]) ) study = optuna.study.create_study(sampler=relative_sampler_class()) trial = study.ask(search_space) study.tell(trial, sum(trial.params.values())) def sample() -> list[float]: params = study.sampler.sample_relative(study, _create_new_trial(study), search_space) return [params[name] for name in search_space] points = np.array([sample() for _ in range(10)]) assert isinstance( search_space["x"], ( FloatDistribution, IntDistribution, ), ) assert np.all(points[:, 0] >= search_space["x"].low) assert np.all(points[:, 0] <= search_space["x"].high) assert isinstance(search_space["y"], CategoricalDistribution) assert np.all([v in search_space["y"].choices for v in points[:, 1]]) for param_value, distribution in zip(sample(), search_space.values()): assert not isinstance(param_value, np.floating) assert not isinstance(param_value, np.integer) if isinstance( distribution, ( IntDistribution, CategoricalDistribution, ), ): assert isinstance(param_value, int) else: assert isinstance(param_value, float) @parametrize_sampler def test_conditional_sample_independent(sampler_class: Callable[[], BaseSampler]) -> None: # This test case reproduces the error reported in #2734. # See https://github.com/optuna/optuna/pull/2734#issuecomment-857649769. study = optuna.study.create_study(sampler=sampler_class()) categorical_distribution = CategoricalDistribution(choices=["x", "y"]) dependent_distribution = CategoricalDistribution(choices=["a", "b"]) study.add_trial( optuna.create_trial( params={"category": "x", "x": "a"}, distributions={"category": categorical_distribution, "x": dependent_distribution}, value=0.1, ) ) study.add_trial( optuna.create_trial( params={"category": "y", "y": "b"}, distributions={"category": categorical_distribution, "y": dependent_distribution}, value=0.1, ) ) _trial = _create_new_trial(study) category = study.sampler.sample_independent( study, _trial, "category", categorical_distribution ) assert category in ["x", "y"] value = study.sampler.sample_independent(study, _trial, category, dependent_distribution) assert value in ["a", "b"] def _create_new_trial(study: Study) -> FrozenTrial: trial_id = study._storage.create_new_trial(study._study_id) return study._storage.get_trial(trial_id) class FixedSampler(BaseSampler): def __init__( self, relative_search_space: dict[str, BaseDistribution], relative_params: dict[str, Any], unknown_param_value: Any, ) -> None: self.relative_search_space = relative_search_space self.relative_params = relative_params self.unknown_param_value = unknown_param_value def infer_relative_search_space( self, study: Study, trial: FrozenTrial ) -> dict[str, BaseDistribution]: return self.relative_search_space def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] ) -> dict[str, Any]: return self.relative_params def sample_independent( self, study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution, ) -> Any: return self.unknown_param_value def test_sample_relative() -> None: relative_search_space: dict[str, BaseDistribution] = { "a": FloatDistribution(low=0, high=5), "b": CategoricalDistribution(choices=("foo", "bar", "baz")), "c": IntDistribution(low=20, high=50), # Not exist in `relative_params`. } relative_params = { "a": 3.2, "b": "baz", } unknown_param_value = 30 sampler = FixedSampler(relative_search_space, relative_params, unknown_param_value) study = optuna.study.create_study(sampler=sampler) def objective(trial: Trial) -> float: # Predefined parameters are sampled by `sample_relative()` method. assert trial.suggest_float("a", 0, 5) == 3.2 assert trial.suggest_categorical("b", ["foo", "bar", "baz"]) == "baz" # Other parameters are sampled by `sample_independent()` method. assert trial.suggest_int("c", 20, 50) == unknown_param_value assert trial.suggest_float("d", 1, 100, log=True) == unknown_param_value assert trial.suggest_float("e", 20, 40) == unknown_param_value return 0.0 study.optimize(objective, n_trials=10, catch=()) for trial in study.trials: assert trial.params == {"a": 3.2, "b": "baz", "c": 30, "d": 30, "e": 30} @parametrize_sampler def test_nan_objective_value(sampler_class: Callable[[], BaseSampler]) -> None: study = optuna.create_study(sampler=sampler_class()) def objective(trial: Trial, base_value: float) -> float: return trial.suggest_float("x", 0.1, 0.2) + base_value # Non NaN objective values. for i in range(10, 1, -1): study.optimize(lambda t: objective(t, i), n_trials=1, catch=()) assert int(study.best_value) == 2 # NaN objective values. study.optimize(lambda t: objective(t, float("nan")), n_trials=1, catch=()) assert int(study.best_value) == 2 # Non NaN objective value. study.optimize(lambda t: objective(t, 1), n_trials=1, catch=()) assert int(study.best_value) == 1 @parametrize_sampler def test_partial_fixed_sampling(sampler_class: Callable[[], BaseSampler]) -> None: study = optuna.create_study(sampler=sampler_class()) def objective(trial: Trial) -> float: x = trial.suggest_float("x", -1, 1) y = trial.suggest_int("y", -1, 1) z = trial.suggest_float("z", -1, 1) return x + y + z # First trial. study.optimize(objective, n_trials=1) # Second trial. Here, the parameter ``y`` is fixed as 0. fixed_params = {"y": 0} with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) study.sampler = optuna.samplers.PartialFixedSampler(fixed_params, study.sampler) study.optimize(objective, n_trials=1) trial_params = study.trials[-1].params assert trial_params["y"] == fixed_params["y"] @parametrize_multi_objective_sampler @pytest.mark.parametrize( "distribution", [ FloatDistribution(-1.0, 1.0), FloatDistribution(0.0, 1.0), FloatDistribution(-1.0, 0.0), FloatDistribution(1e-7, 1.0, log=True), FloatDistribution(-10, 10, step=0.1), FloatDistribution(-10.2, 10.2, step=0.1), IntDistribution(-10, 10), IntDistribution(0, 10), IntDistribution(-10, 0), IntDistribution(-10, 10, step=2), IntDistribution(0, 10, step=2), IntDistribution(-10, 0, step=2), IntDistribution(1, 100, log=True), CategoricalDistribution((1, 2, 3)), CategoricalDistribution(("a", "b", "c")), CategoricalDistribution((1, "a")), ], ) def test_multi_objective_sample_independent( multi_objective_sampler_class: Callable[[], BaseSampler], distribution: BaseDistribution ) -> None: study = optuna.study.create_study( directions=["minimize", "maximize"], sampler=multi_objective_sampler_class() ) for i in range(100): value = study.sampler.sample_independent( study, _create_new_trial(study), "x", distribution ) assert distribution._contains(distribution.to_internal_repr(value)) if not isinstance(distribution, CategoricalDistribution): # Please see https://github.com/optuna/optuna/pull/393 why this assertion is needed. assert not isinstance(value, np.floating) if isinstance(distribution, FloatDistribution): if distribution.step is not None: # Check the value is a multiple of `distribution.step` which is # the quantization interval of the distribution. value -= distribution.low value /= distribution.step round_value = np.round(value) np.testing.assert_almost_equal(round_value, value) def test_before_trial() -> None: n_calls = 0 n_trials = 3 class SamplerBeforeTrial(optuna.samplers.RandomSampler): def before_trial(self, study: Study, trial: FrozenTrial) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None nonlocal n_calls n_calls += 1 sampler = SamplerBeforeTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.optimize( lambda t: [t.suggest_float("y", -3, 3), t.suggest_int("x", 0, 10)], n_trials=n_trials ) assert n_calls == n_trials def test_after_trial() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None assert state == TrialState.COMPLETE assert values is not None assert len(values) == 2 nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.optimize(lambda t: [t.suggest_float("y", -3, 3), t.suggest_int("x", 0, 10)], n_trials=3) assert n_calls == n_trials def test_after_trial_pruning() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None assert state == TrialState.PRUNED assert values is None nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) study.optimize(pruned_objective, n_trials=n_trials) assert n_calls == n_trials def test_after_trial_failing() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: assert len(study.trials) - 1 == trial.number assert trial.state == TrialState.RUNNING assert trial.values is None assert state == TrialState.FAIL assert values is None nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler) with pytest.raises(ValueError): study.optimize(fail_objective, n_trials=n_trials) # Called once after the first failing trial before returning from optimize. assert n_calls == 1 def test_after_trial_failing_in_after_trial() -> None: n_calls = 0 n_trials = 3 class SamplerAfterTrialAlwaysFail(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: nonlocal n_calls n_calls += 1 raise NotImplementedError # Arbitrary error for testing purpose. sampler = SamplerAfterTrialAlwaysFail() study = optuna.create_study(sampler=sampler) with pytest.raises(NotImplementedError): study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=n_trials) assert len(study.trials) == 1 assert n_calls == 1 sampler = SamplerAfterTrialAlwaysFail() study = optuna.create_study(sampler=sampler) # Not affected by `catch`. with pytest.raises(NotImplementedError): study.optimize( lambda t: t.suggest_int("x", 0, 10), n_trials=n_trials, catch=(NotImplementedError,) ) assert len(study.trials) == 1 assert n_calls == 2 def test_after_trial_with_study_tell() -> None: n_calls = 0 class SamplerAfterTrial(optuna.samplers.RandomSampler): def after_trial( self, study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None, ) -> None: nonlocal n_calls n_calls += 1 sampler = SamplerAfterTrial() study = optuna.create_study(sampler=sampler) assert n_calls == 0 study.tell(study.ask(), 1.0) assert n_calls == 1 @parametrize_sampler def test_sample_single_distribution(sampler_class: Callable[[], BaseSampler]) -> None: relative_search_space = { "a": CategoricalDistribution([1]), "b": IntDistribution(low=1, high=1), "c": IntDistribution(low=1, high=1, log=True), "d": FloatDistribution(low=1.0, high=1.0), "e": FloatDistribution(low=1.0, high=1.0, log=True), "f": FloatDistribution(low=1.0, high=1.0, step=1.0), } with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) # We need to test the construction of the model, so we should set `n_trials >= 2`. for _ in range(2): trial = study.ask(fixed_distributions=relative_search_space) study.tell(trial, 1.0) for param_name in relative_search_space.keys(): assert trial.params[param_name] == 1 @parametrize_sampler @parametrize_suggest_method("x") def test_single_parameter_objective( sampler_class: Callable[[], BaseSampler], suggest_method_x: Callable[[Trial], float] ) -> None: def objective(trial: Trial) -> float: return suggest_method_x(trial) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(objective, n_trials=10) assert len(study.trials) == 10 assert all(t.state == TrialState.COMPLETE for t in study.trials) @parametrize_sampler def test_conditional_parameter_objective(sampler_class: Callable[[], BaseSampler]) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", [True, False]) if x: return trial.suggest_float("y", 0, 1) return trial.suggest_float("z", 0, 1) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(objective, n_trials=10) assert len(study.trials) == 10 assert all(t.state == TrialState.COMPLETE for t in study.trials) @parametrize_sampler @parametrize_suggest_method("x") @parametrize_suggest_method("y") def test_combination_of_different_distributions_objective( sampler_class: Callable[[], BaseSampler], suggest_method_x: Callable[[Trial], float], suggest_method_y: Callable[[Trial], float], ) -> None: def objective(trial: Trial) -> float: return suggest_method_x(trial) + suggest_method_y(trial) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(objective, n_trials=3) assert len(study.trials) == 3 assert all(t.state == TrialState.COMPLETE for t in study.trials) @parametrize_sampler @pytest.mark.parametrize( "second_low,second_high", [ (0, 5), # Narrow range. (0, 20), # Expand range. (20, 30), # Set non-overlapping range. ], ) def test_dynamic_range_objective( sampler_class: Callable[[], BaseSampler], second_low: int, second_high: int ) -> None: def objective(trial: Trial, low: int, high: int) -> float: v = trial.suggest_float("x", low, high) v += trial.suggest_int("y", low, high) return v with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = sampler_class() study = optuna.study.create_study(sampler=sampler) study.optimize(lambda t: objective(t, 0, 10), n_trials=10) study.optimize(lambda t: objective(t, second_low, second_high), n_trials=10) assert len(study.trials) == 20 assert all(t.state == TrialState.COMPLETE for t in study.trials) # We add tests for constant objective functions to ensure the reproducibility of sorting. @parametrize_sampler_with_seed @pytest.mark.slow @pytest.mark.parametrize("objective_func", [lambda *args: sum(args), lambda *args: 0.0]) def test_reproducible(sampler_class: Callable[[int], BaseSampler], objective_func: Any) -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 1, 9) b = trial.suggest_float("b", 1, 9, log=True) c = trial.suggest_float("c", 1, 9, step=1) d = trial.suggest_int("d", 1, 9) e = trial.suggest_int("e", 1, 9, log=True) f = trial.suggest_int("f", 1, 9, step=2) g = trial.suggest_categorical("g", range(1, 10)) return objective_func(a, b, c, d, e, f, g) study = optuna.create_study(sampler=sampler_class(1)) study.optimize(objective, n_trials=15) study_same_seed = optuna.create_study(sampler=sampler_class(1)) study_same_seed.optimize(objective, n_trials=15) for i in range(15): assert study.trials[i].params == study_same_seed.trials[i].params study_different_seed = optuna.create_study(sampler=sampler_class(2)) study_different_seed.optimize(objective, n_trials=15) assert any( [study.trials[i].params != study_different_seed.trials[i].params for i in range(15)] ) @pytest.mark.slow @parametrize_sampler_with_seed def test_reseed_rng_change_sampling(sampler_class: Callable[[int], BaseSampler]) -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 1, 9) b = trial.suggest_float("b", 1, 9, log=True) c = trial.suggest_float("c", 1, 9, step=1) d = trial.suggest_int("d", 1, 9) e = trial.suggest_int("e", 1, 9, log=True) f = trial.suggest_int("f", 1, 9, step=2) g = trial.suggest_categorical("g", range(1, 10)) return a + b + c + d + e + f + g sampler = sampler_class(1) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=15) sampler_different_seed = sampler_class(1) sampler_different_seed.reseed_rng() study_different_seed = optuna.create_study(sampler=sampler_different_seed) study_different_seed.optimize(objective, n_trials=15) assert any( [study.trials[i].params != study_different_seed.trials[i].params for i in range(15)] ) # This function is used only in test_reproducible_in_other_process, but declared at top-level # because local function cannot be pickled, which occurs within multiprocessing. def run_optimize( k: int, sampler_name: str, sequence_dict: DictProxy, hash_dict: DictProxy, ) -> None: def objective(trial: Trial) -> float: a = trial.suggest_float("a", 1, 9) b = trial.suggest_float("b", 1, 9, log=True) c = trial.suggest_float("c", 1, 9, step=1) d = trial.suggest_int("d", 1, 9) e = trial.suggest_int("e", 1, 9, log=True) f = trial.suggest_int("f", 1, 9, step=2) g = trial.suggest_categorical("g", range(1, 10)) return a + b + c + d + e + f + g hash_dict[k] = hash("nondeterministic hash") sampler = sampler_class_with_seed[sampler_name][0](1) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=15) sequence_dict[k] = list(study.trials[-1].params.values()) @pytest.fixture def unset_seed_in_test(request: SubRequest) -> None: # Unset the hashseed at beginning and restore it at end regardless of an exception in the test. # See https://docs.pytest.org/en/stable/how-to/fixtures.html#adding-finalizers-directly # for details. hash_seed = os.getenv("PYTHONHASHSEED") if hash_seed is not None: del os.environ["PYTHONHASHSEED"] def restore_seed() -> None: if hash_seed is not None: os.environ["PYTHONHASHSEED"] = hash_seed request.addfinalizer(restore_seed) @pytest.mark.slow @parametrize_sampler_name_with_seed def test_reproducible_in_other_process(sampler_name: str, unset_seed_in_test: None) -> None: # This test should be tested without `PYTHONHASHSEED`. However, some tool such as tox # set the environmental variable "PYTHONHASHSEED" by default. # To do so, this test calls a finalizer: `unset_seed_in_test`. # Multiprocessing supports three way to start a process. # We use `spawn` option to create a child process as a fresh python process. # For more detail, see https://github.com/optuna/optuna/pull/3187#issuecomment-997673037. multiprocessing.set_start_method("spawn", force=True) manager = multiprocessing.Manager() sequence_dict: DictProxy = manager.dict() hash_dict: DictProxy = manager.dict() for i in range(3): p = multiprocessing.Process( target=run_optimize, args=(i, sampler_name, sequence_dict, hash_dict) ) p.start() p.join() # Hashes are expected to be different because string hashing is nondeterministic per process. assert not (hash_dict[0] == hash_dict[1] == hash_dict[2]) # But the sequences are expected to be the same. assert sequence_dict[0] == sequence_dict[1] == sequence_dict[2] @pytest.mark.parametrize("n_jobs", [1, 2]) @parametrize_relative_sampler def test_cache_is_invalidated( n_jobs: int, relative_sampler_class: Callable[[], BaseSampler] ) -> None: sampler = relative_sampler_class() study = optuna.study.create_study(sampler=sampler) def objective(trial: Trial) -> float: assert trial._relative_params is None assert study._thread_local.cached_all_trials is None trial.suggest_float("x", -10, 10) trial.suggest_float("y", -10, 10) assert trial._relative_params is not None assert study._thread_local.cached_all_trials is not None return -1 study.optimize(objective, n_trials=10, n_jobs=n_jobs) optuna-3.5.0/tests/samplers_tests/tpe_tests/000077500000000000000000000000001453453102400212525ustar00rootroot00000000000000optuna-3.5.0/tests/samplers_tests/tpe_tests/__init__.py000066400000000000000000000000001453453102400233510ustar00rootroot00000000000000optuna-3.5.0/tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py000066400000000000000000000470721453453102400276040ustar00rootroot00000000000000import random from typing import Callable from typing import Dict from typing import List from typing import Optional from typing import Union from unittest.mock import patch from unittest.mock import PropertyMock import numpy as np import pytest import optuna from optuna.samplers import _tpe from optuna.samplers import TPESampler class MockSystemAttr: def __init__(self) -> None: self.value: Dict[str, dict] = {} def set_trial_system_attr(self, _: int, key: str, value: dict) -> None: self.value[key] = value def suggest( sampler: optuna.samplers.BaseSampler, study: optuna.Study, trial: optuna.trial.FrozenTrial, distribution: optuna.distributions.BaseDistribution, past_trials: List[optuna.trial.FrozenTrial], ) -> float: attrs = MockSystemAttr() with patch.object(study._storage, "get_all_trials", return_value=past_trials), patch.object( study._storage, "set_trial_system_attr", side_effect=attrs.set_trial_system_attr ), patch.object(study._storage, "get_trial", return_value=trial), patch( "optuna.trial.Trial.system_attrs", new_callable=PropertyMock ) as mock1, patch( "optuna.trial.FrozenTrial.system_attrs", new_callable=PropertyMock, ) as mock2: mock1.return_value = attrs.value mock2.return_value = attrs.value suggestion = sampler.sample_independent(study, trial, "param-a", distribution) return suggestion def test_multi_objective_sample_independent_seed_fix() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) sampler = TPESampler(seed=0) assert suggest(sampler, study, trial, dist, past_trials) == suggestion sampler = TPESampler(seed=1) assert suggest(sampler, study, trial, dist, past_trials) != suggestion def test_multi_objective_sample_independent_prior() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) sampler = TPESampler(consider_prior=False, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion sampler = TPESampler(prior_weight=0.5, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion def test_multi_objective_sample_independent_n_startup_trial() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] trial = frozen_trial_factory(16, [0, 0]) def _suggest_and_return_call_count( sampler: optuna.samplers.BaseSampler, past_trials: List[optuna.trial.FrozenTrial], ) -> int: attrs = MockSystemAttr() with patch.object( study._storage, "get_all_trials", return_value=past_trials ), patch.object( study._storage, "set_trial_system_attr", side_effect=attrs.set_trial_system_attr ), patch.object( study._storage, "get_trial", return_value=trial ), patch( "optuna.trial.Trial.system_attrs", new_callable=PropertyMock ) as mock1, patch( "optuna.trial.FrozenTrial.system_attrs", new_callable=PropertyMock, ) as mock2, patch.object( optuna.samplers.RandomSampler, "sample_independent", return_value=1.0, ) as sample_method: mock1.return_value = attrs.value mock2.return_value = attrs.value sampler.sample_independent(study, trial, "param-a", dist) study._thread_local.cached_all_trials = None return sample_method.call_count sampler = TPESampler(n_startup_trials=16, seed=0) assert _suggest_and_return_call_count(sampler, past_trials[:-1]) == 1 sampler = TPESampler(n_startup_trials=16, seed=0) assert _suggest_and_return_call_count(sampler, past_trials) == 0 def test_multi_objective_sample_independent_misc_arguments() -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(32)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) # Test misc. parameters. sampler = TPESampler(n_ei_candidates=13, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion sampler = TPESampler(gamma=lambda _: 1, seed=0) assert suggest(sampler, study, trial, dist, past_trials) != suggestion @pytest.mark.parametrize("log, step", [(False, None), (True, None), (False, 0.1)]) def test_multi_objective_sample_independent_float_distributions( log: bool, step: Optional[float] ) -> None: # Prepare sample from float distribution for checking other distributions. study = optuna.create_study(directions=["minimize", "maximize"]) random.seed(128) float_dist = optuna.distributions.FloatDistribution(1.0, 100.0, log=log, step=step) if float_dist.step: value_fn: Optional[Callable[[int], float]] = ( lambda number: int(random.random() * 1000) * 0.1 ) else: value_fn = None past_trials = [ frozen_trial_factory( i, [random.random(), random.random()], dist=float_dist, value_fn=value_fn ) for i in range(16) ] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) float_suggestion = suggest(sampler, study, trial, float_dist, past_trials) assert 1.0 <= float_suggestion < 100.0 if float_dist.step == 0.1: assert abs(int(float_suggestion * 10) - float_suggestion * 10) < 1e-3 # Test sample is different when `float_dist.log` is True or float_dist.step != 1.0. random.seed(128) dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(16)] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) suggestion = suggest(sampler, study, trial, dist, past_trials) if float_dist.log or float_dist.step == 0.1: assert float_suggestion != suggestion else: assert float_suggestion == suggestion def test_multi_objective_sample_independent_categorical_distributions() -> None: """Test samples are drawn from the specified category.""" study = optuna.create_study(directions=["minimize", "maximize"]) random.seed(128) categories = [i * 0.3 + 1.0 for i in range(330)] def cat_value_fn(idx: int) -> float: return categories[random.randint(0, len(categories) - 1)] cat_dist = optuna.distributions.CategoricalDistribution(categories) past_trials = [ frozen_trial_factory( i, [random.random(), random.random()], dist=cat_dist, value_fn=cat_value_fn ) for i in range(16) ] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) categorical_suggestion = suggest(sampler, study, trial, cat_dist, past_trials) assert categorical_suggestion in categories @pytest.mark.parametrize( "log, step", [ (False, 1), (True, 1), (False, 2), ], ) def test_multi_objective_sample_int_distributions(log: bool, step: int) -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study(directions=["minimize", "maximize"]) random.seed(128) def int_value_fn(idx: int) -> float: return random.randint(1, 99) int_dist = optuna.distributions.IntDistribution(1, 99, log, step) past_trials = [ frozen_trial_factory( i, [random.random(), random.random()], dist=int_dist, value_fn=int_value_fn ) for i in range(16) ] trial = frozen_trial_factory(16, [0, 0]) sampler = TPESampler(seed=0) int_suggestion = suggest(sampler, study, trial, int_dist, past_trials) assert 1 <= int_suggestion <= 99 assert isinstance(int_suggestion, int) @pytest.mark.parametrize( "state", [ (optuna.trial.TrialState.FAIL,), (optuna.trial.TrialState.PRUNED,), (optuna.trial.TrialState.RUNNING,), (optuna.trial.TrialState.WAITING,), ], ) def test_multi_objective_sample_independent_handle_unsuccessful_states( state: optuna.trial.TrialState, ) -> None: study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) random.seed(128) # Prepare sampling result for later tests. past_trials = [frozen_trial_factory(i, [random.random(), random.random()]) for i in range(32)] trial = frozen_trial_factory(32, [0, 0]) sampler = TPESampler(seed=0) all_success_suggestion = suggest(sampler, study, trial, dist, past_trials) study._thread_local.cached_all_trials = None # Test unsuccessful trials are handled differently. state_fn = build_state_fn(state) past_trials = [ frozen_trial_factory(i, [random.random(), random.random()], state_fn=state_fn) for i in range(32) ] trial = frozen_trial_factory(32, [0, 0]) sampler = TPESampler(seed=0) partial_unsuccessful_suggestion = suggest(sampler, study, trial, dist, past_trials) assert partial_unsuccessful_suggestion != all_success_suggestion def test_multi_objective_sample_independent_ignored_states() -> None: """Tests FAIL, RUNNING, and WAITING states are equally.""" study = optuna.create_study(directions=["minimize", "maximize"]) dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ]: random.seed(128) state_fn = build_state_fn(state) past_trials = [ frozen_trial_factory(i, [random.random(), random.random()], state_fn=state_fn) for i in range(32) ] trial = frozen_trial_factory(32, [0, 0]) sampler = TPESampler(seed=0) suggestions.append(suggest(sampler, study, trial, dist, past_trials)) assert len(set(suggestions)) == 1 @pytest.mark.parametrize("direction0", ["minimize", "maximize"]) @pytest.mark.parametrize("direction1", ["minimize", "maximize"]) def test_split_complete_trials_multi_objective(direction0: str, direction1: str) -> None: study = optuna.create_study(directions=(direction0, direction1)) for values in ([-2.0, -1.0], [3.0, 3.0], [0.0, 1.0], [-1.0, 0.0]): value0, value1 = values if direction0 == "maximize": value0 = -value0 if direction1 == "maximize": value1 = -value1 study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, values=(value0, value1), params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) below_trials, above_trials = _tpe.sampler._split_complete_trials_multi_objective( study.trials, study, 2, ) assert [trial.number for trial in below_trials] == [0, 3] assert [trial.number for trial in above_trials] == [1, 2] def test_split_complete_trials_multi_objective_empty() -> None: study = optuna.create_study(directions=("minimize", "minimize")) assert _tpe.sampler._split_complete_trials_multi_objective([], study, 0) == ([], []) def test_calculate_nondomination_rank() -> None: # Single objective test_case = np.asarray([[10], [20], [20], [30]]) ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case, len(test_case))) assert ranks == [0, 1, 1, 2] # Two objectives test_case = np.asarray([[10, 30], [10, 10], [20, 20], [30, 10], [15, 15]]) ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case, len(test_case))) assert ranks == [1, 0, 2, 1, 1] # Three objectives test_case = np.asarray([[5, 5, 4], [5, 5, 5], [9, 9, 0], [5, 7, 5], [0, 0, 9], [0, 9, 9]]) ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case, len(test_case))) assert ranks == [0, 1, 0, 2, 0, 1] # The negative values are included. test_case = np.asarray( [[-5, -5, -4], [-5, -5, 5], [-9, -9, 0], [5, 7, 5], [0, 0, -9], [0, -9, 9]] ) ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case, len(test_case))) assert ranks == [0, 1, 0, 2, 0, 1] # The +inf is included. test_case = np.asarray( [[1, 1], [1, float("inf")], [float("inf"), 1], [float("inf"), float("inf")]] ) ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case, len(test_case))) assert ranks == [0, 1, 1, 2] # The -inf is included. test_case = np.asarray( [[1, 1], [1, -float("inf")], [-float("inf"), 1], [-float("inf"), -float("inf")]] ) ranks = list(_tpe.sampler._calculate_nondomination_rank(test_case, len(test_case))) assert ranks == [2, 1, 1, 0] def test_calculate_weights_below_for_multi_objective() -> None: # No sample. study = optuna.create_study(directions=["minimize", "minimize"]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective(study, [], None) assert len(weights_below) == 0 # One sample. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.5]) study.add_trials([trial0]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0], None ) assert len(weights_below) == 1 assert sum(weights_below) > 0 # Two samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.5]) trial1 = optuna.create_trial(values=[0.9, 0.4]) study.add_trials([trial0, trial1]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1], None, ) assert len(weights_below) == 2 assert weights_below[0] > weights_below[1] assert sum(weights_below) > 0 # Two equally contributed samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.8]) trial1 = optuna.create_trial(values=[0.8, 0.2]) study.add_trials([trial0, trial1]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1], None, ) assert len(weights_below) == 2 assert weights_below[0] == weights_below[1] assert sum(weights_below) > 0 # Duplicated samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.2, 0.8]) trial1 = optuna.create_trial(values=[0.2, 0.8]) study.add_trials([trial0, trial1]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1], None, ) assert len(weights_below) == 2 assert weights_below[0] == weights_below[1] assert sum(weights_below) > 0 # Three samples. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.3, 0.3]) trial1 = optuna.create_trial(values=[0.2, 0.8]) trial2 = optuna.create_trial(values=[0.8, 0.2]) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], None, ) assert len(weights_below) == 3 assert weights_below[0] > weights_below[1] assert weights_below[0] > weights_below[2] assert weights_below[1] == weights_below[2] assert sum(weights_below) > 0 # Zero/negative objective values. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[-0.3, -0.3]) trial1 = optuna.create_trial(values=[0.0, -0.8]) trial2 = optuna.create_trial(values=[-0.8, 0.0]) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], None, ) assert len(weights_below) == 3 assert weights_below[0] > weights_below[1] assert weights_below[0] > weights_below[2] assert np.isclose(weights_below[1], weights_below[2]) assert sum(weights_below) > 0 # +/-inf objective values. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[-float("inf"), -float("inf")]) trial1 = optuna.create_trial(values=[0.0, -float("inf")]) trial2 = optuna.create_trial(values=[-float("inf"), 0.0]) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], None, ) assert len(weights_below) == 3 assert all([np.isnan(w) for w in weights_below]) # Three samples with two infeasible trials. study = optuna.create_study(directions=["minimize", "minimize"]) trial0 = optuna.create_trial(values=[0.3, 0.3], system_attrs={"constraints": 2}) trial1 = optuna.create_trial(values=[0.2, 0.8], system_attrs={"constraints": 8}) trial2 = optuna.create_trial(values=[0.8, 0.2], system_attrs={"constraints": 0}) study.add_trials([trial0, trial1, trial2]) weights_below = _tpe.sampler._calculate_weights_below_for_multi_objective( study, [trial0, trial1, trial2], lambda trial: [trial.system_attrs["constraints"]], ) assert len(weights_below) == 3 assert weights_below[0] == _tpe.sampler.EPS assert weights_below[1] == _tpe.sampler.EPS assert weights_below[2] > 0 def frozen_trial_factory( number: int, values: List[float], dist: optuna.distributions.BaseDistribution = optuna.distributions.FloatDistribution( 1.0, 100.0 ), value_fn: Optional[Callable[[int], Union[int, float]]] = None, state_fn: Callable[ [int], optuna.trial.TrialState ] = lambda _: optuna.trial.TrialState.COMPLETE, ) -> optuna.trial.FrozenTrial: if value_fn is None: value = random.random() * 99.0 + 1.0 else: value = value_fn(number) trial = optuna.trial.FrozenTrial( number=number, trial_id=number, state=optuna.trial.TrialState.COMPLETE, value=None, datetime_start=None, datetime_complete=None, params={"param-a": value}, distributions={"param-a": dist}, user_attrs={}, system_attrs={}, intermediate_values={}, values=values, ) return trial def build_state_fn(state: optuna.trial.TrialState) -> Callable[[int], optuna.trial.TrialState]: def state_fn(idx: int) -> optuna.trial.TrialState: return [optuna.trial.TrialState.COMPLETE, state][idx % 2] return state_fn optuna-3.5.0/tests/samplers_tests/tpe_tests/test_parzen_estimator.py000066400000000000000000000341021453453102400262510ustar00rootroot00000000000000from typing import Callable from typing import Dict from typing import List import numpy as np import pytest from optuna import distributions from optuna.distributions import CategoricalChoiceType from optuna.samplers._tpe.parzen_estimator import _ParzenEstimator from optuna.samplers._tpe.parzen_estimator import _ParzenEstimatorParameters from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution from optuna.samplers._tpe.sampler import default_weights def assert_distribution_almost_equal( d1: _MixtureOfProductDistribution, d2: _MixtureOfProductDistribution ) -> None: np.testing.assert_almost_equal(d1.weights, d2.weights) for d1_, d2_ in zip(d1.distributions, d2.distributions): assert type(d1_) is type(d2_) for field1, field2 in zip(d1_, d2_): np.testing.assert_almost_equal(np.array(field1), np.array(field2)) SEARCH_SPACE = { "a": distributions.FloatDistribution(1.0, 100.0), "b": distributions.FloatDistribution(1.0, 100.0, log=True), "c": distributions.FloatDistribution(1.0, 100.0, step=3.0), "d": distributions.IntDistribution(1, 100), "e": distributions.IntDistribution(1, 100, log=True), "f": distributions.CategoricalDistribution(["x", "y", "z"]), "g": distributions.CategoricalDistribution([0.0, float("inf"), float("nan"), None]), } MULTIVARIATE_SAMPLES = { "a": np.array([1.0]), "b": np.array([1.0]), "c": np.array([1.0]), "d": np.array([1]), "e": np.array([1]), "f": np.array([1]), "g": np.array([1]), } @pytest.mark.parametrize("consider_prior", [True, False]) @pytest.mark.parametrize("multivariate", [True, False]) def test_init_parzen_estimator(consider_prior: bool, multivariate: bool) -> None: parameters = _ParzenEstimatorParameters( consider_prior=consider_prior, prior_weight=1.0, consider_magic_clip=False, consider_endpoints=False, weights=lambda x: np.arange(x) + 1.0, multivariate=multivariate, categorical_distance_func={}, ) mpe = _ParzenEstimator(MULTIVARIATE_SAMPLES, SEARCH_SPACE, parameters) weights = np.array([1] + consider_prior * [1], dtype=float) weights /= weights.sum() expected_univariate = _MixtureOfProductDistribution( weights=weights, distributions=[ _BatchedTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([49.5 if consider_prior else 99.0] + consider_prior * [99.0]), low=1.0, high=100.0, ), _BatchedTruncNormDistributions( mu=np.array([np.log(1.0)] + consider_prior * [np.log(100) / 2.0]), sigma=np.array( [np.log(100) / 2 if consider_prior else np.log(100.0)] + consider_prior * [np.log(100)] ), low=np.log(1.0), high=np.log(100.0), ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([49.5 if consider_prior else 100.5] + consider_prior * [102.0]), low=1.0, high=100.0, step=3.0, ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([49.5 if consider_prior else 99.5] + consider_prior * [100.0]), low=1, high=100, step=1, ), _BatchedTruncNormDistributions( mu=np.array( [np.log(1.0)] + consider_prior * [(np.log(100.5) + np.log(0.5)) / 2.0] ), sigma=np.array( [(np.log(100.5) + np.log(0.5)) / 2 if consider_prior else np.log(100.5)] + consider_prior * [np.log(100.5) - np.log(0.5)] ), low=np.log(0.5), high=np.log(100.5), ), _BatchedCategoricalDistributions( np.array([[0.2, 0.6, 0.2], [1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0]]) if consider_prior else np.array([[0.25, 0.5, 0.25]]) ), _BatchedCategoricalDistributions( np.array( [ [1.0 / 6.0, 0.5, 1.0 / 6.0, 1.0 / 6.0], [1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0], ] ) if consider_prior else np.array([[0.2, 0.4, 0.2, 0.2]]) ), ], ) SIGMA0 = 0.2 expected_multivarite = _MixtureOfProductDistribution( weights=weights, distributions=[ _BatchedTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([SIGMA0 * 99.0] + consider_prior * [99.0]), low=1.0, high=100.0, ), _BatchedTruncNormDistributions( mu=np.array([np.log(1.0)] + consider_prior * [np.log(100) / 2.0]), sigma=np.array([SIGMA0 * np.log(100)] + consider_prior * [np.log(100)]), low=np.log(1.0), high=np.log(100.0), ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([SIGMA0 * 102.0] + consider_prior * [102.0]), low=1.0, high=100.0, step=3.0, ), _BatchedDiscreteTruncNormDistributions( mu=np.array([1.0] + consider_prior * [50.5]), sigma=np.array([SIGMA0 * 100.0] + consider_prior * [100.0]), low=1, high=100, step=1, ), _BatchedTruncNormDistributions( mu=np.array( [np.log(1.0)] + consider_prior * [(np.log(100.5) + np.log(0.5)) / 2.0] ), sigma=np.array( [SIGMA0 * (np.log(100.5) - np.log(0.5))] + consider_prior * [np.log(100.5) - np.log(0.5)] ), low=np.log(0.5), high=np.log(100.5), ), _BatchedCategoricalDistributions( np.array([[0.2, 0.6, 0.2], [1.0 / 3.0, 1.0 / 3.0, 1.0 / 3.0]]) if consider_prior else np.array([[0.25, 0.5, 0.25]]) ), _BatchedCategoricalDistributions( np.array( [ [1.0 / 6.0, 0.5, 1.0 / 6.0, 1.0 / 6.0], [1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0, 1.0 / 4.0], ] if consider_prior else np.array([[0.2, 0.4, 0.2, 0.2]]) ) ), ], ) expected = expected_multivarite if multivariate else expected_univariate # Test that the distribution is correct. assert_distribution_almost_equal(mpe._mixture_distribution, expected) # Test that the sampled values are valid. samples = mpe.sample(np.random.RandomState(0), 10) for param, values in samples.items(): for value in values: assert SEARCH_SPACE[param]._contains(value) @pytest.mark.parametrize("mus", (np.asarray([]), np.asarray([0.4]), np.asarray([-0.4, 0.4]))) @pytest.mark.parametrize("prior_weight", [1.0, 0.01, 100.0]) @pytest.mark.parametrize("prior", (True, False)) @pytest.mark.parametrize("magic_clip", (True, False)) @pytest.mark.parametrize("endpoints", (True, False)) @pytest.mark.parametrize("multivariate", (True, False)) def test_calculate_shape_check( mus: np.ndarray, prior_weight: float, prior: bool, magic_clip: bool, endpoints: bool, multivariate: bool, ) -> None: parameters = _ParzenEstimatorParameters( prior_weight=prior_weight, consider_prior=prior, consider_magic_clip=magic_clip, consider_endpoints=endpoints, weights=default_weights, multivariate=multivariate, categorical_distance_func={}, ) mpe = _ParzenEstimator( {"a": mus}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters ) assert len(mpe._mixture_distribution.weights) == max(len(mus) + int(prior), 1) @pytest.mark.parametrize("mus", (np.asarray([]), np.asarray([0.4]), np.asarray([-0.4, 0.4]))) @pytest.mark.parametrize("prior_weight", [1.0, 0.01, 100.0]) @pytest.mark.parametrize("prior", (True, False)) @pytest.mark.parametrize("categorical_distance_func", ({}, {"c": lambda x, y: abs(x - y)})) def test_calculate_shape_check_categorical( mus: np.ndarray, prior_weight: float, prior: bool, categorical_distance_func: Dict[ str, Callable[[CategoricalChoiceType, CategoricalChoiceType], float], ], ) -> None: parameters = _ParzenEstimatorParameters( prior_weight=prior_weight, consider_prior=prior, consider_magic_clip=True, consider_endpoints=False, weights=default_weights, multivariate=False, categorical_distance_func=categorical_distance_func, ) mpe = _ParzenEstimator( {"c": mus}, {"c": distributions.CategoricalDistribution([0.0, 1.0, 2.0])}, parameters ) assert len(mpe._mixture_distribution.weights) == max(len(mus) + int(prior), 1) @pytest.mark.parametrize("prior_weight", [None, -1.0, 0.0]) @pytest.mark.parametrize("mus", (np.asarray([]), np.asarray([0.4]), np.asarray([-0.4, 0.4]))) def test_invalid_prior_weight(prior_weight: float, mus: np.ndarray) -> None: parameters = _ParzenEstimatorParameters( prior_weight=prior_weight, consider_prior=True, consider_magic_clip=False, consider_endpoints=False, weights=default_weights, multivariate=False, categorical_distance_func={}, ) with pytest.raises(ValueError): _ParzenEstimator({"a": mus}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters) # TODO(ytsmiling): Improve test coverage for weights. @pytest.mark.parametrize( "mus, flags, expected", [ [ np.asarray([]), {"prior": False, "magic_clip": False, "endpoints": True}, {"weights": [1.0], "mus": [0.0], "sigmas": [2.0]}, ], [ np.asarray([]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [1.0], "mus": [0.0], "sigmas": [2.0]}, ], [ np.asarray([0.4]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [0.5, 0.5], "mus": [0.4, 0.0], "sigmas": [0.6, 2.0]}, ], [ np.asarray([-0.4]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [0.5, 0.5], "mus": [-0.4, 0.0], "sigmas": [0.6, 2.0]}, ], [ np.asarray([-0.4, 0.4]), {"prior": True, "magic_clip": False, "endpoints": True}, {"weights": [1.0 / 3] * 3, "mus": [-0.4, 0.4, 0.0], "sigmas": [0.6, 0.6, 2.0]}, ], [ np.asarray([-0.4, 0.4]), {"prior": True, "magic_clip": False, "endpoints": False}, {"weights": [1.0 / 3] * 3, "mus": [-0.4, 0.4, 0.0], "sigmas": [0.4, 0.4, 2.0]}, ], [ np.asarray([-0.4, 0.4]), {"prior": False, "magic_clip": False, "endpoints": True}, {"weights": [0.5, 0.5], "mus": [-0.4, 0.4], "sigmas": [0.8, 0.8]}, ], [ np.asarray([-0.4, 0.4, 0.41, 0.42]), {"prior": False, "magic_clip": False, "endpoints": True}, { "weights": [0.25, 0.25, 0.25, 0.25], "mus": [-0.4, 0.4, 0.41, 0.42], "sigmas": [0.8, 0.8, 0.01, 0.58], }, ], [ np.asarray([-0.4, 0.4, 0.41, 0.42]), {"prior": False, "magic_clip": True, "endpoints": True}, { "weights": [0.25, 0.25, 0.25, 0.25], "mus": [-0.4, 0.4, 0.41, 0.42], "sigmas": [0.8, 0.8, 0.4, 0.58], }, ], ], ) def test_calculate( mus: np.ndarray, flags: Dict[str, bool], expected: Dict[str, List[float]] ) -> None: parameters = _ParzenEstimatorParameters( prior_weight=1.0, consider_prior=flags["prior"], consider_magic_clip=flags["magic_clip"], consider_endpoints=flags["endpoints"], weights=default_weights, multivariate=False, categorical_distance_func={}, ) mpe = _ParzenEstimator( {"a": mus}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters ) expected_distribution = _MixtureOfProductDistribution( weights=np.asarray(expected["weights"]), distributions=[ _BatchedTruncNormDistributions( mu=np.asarray(expected["mus"]), sigma=np.asarray(expected["sigmas"]), low=-1.0, high=1.0, ) ], ) assert_distribution_almost_equal(mpe._mixture_distribution, expected_distribution) @pytest.mark.parametrize( "weights", [ lambda x: np.zeros(x), lambda x: -np.ones(x), lambda x: float("inf") * np.ones(x), lambda x: -float("inf") * np.ones(x), lambda x: np.asarray([float("nan") for _ in range(x)]), ], ) def test_invalid_weights(weights: Callable[[int], np.ndarray]) -> None: parameters = _ParzenEstimatorParameters( prior_weight=1.0, consider_prior=False, consider_magic_clip=False, consider_endpoints=False, weights=weights, multivariate=False, categorical_distance_func={}, ) with pytest.raises(ValueError): _ParzenEstimator( {"a": np.asarray([0.0])}, {"a": distributions.FloatDistribution(-1.0, 1.0)}, parameters ) optuna-3.5.0/tests/samplers_tests/tpe_tests/test_probability_distributions.py000066400000000000000000000063131453453102400301700ustar00rootroot00000000000000import warnings import numpy as np from optuna.samplers._tpe.probability_distributions import _BatchedCategoricalDistributions from optuna.samplers._tpe.probability_distributions import _BatchedDiscreteTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _BatchedTruncNormDistributions from optuna.samplers._tpe.probability_distributions import _MixtureOfProductDistribution def test_mixture_of_product_distribution() -> None: dist0 = _BatchedTruncNormDistributions( mu=np.array([0.2, 3.0]), sigma=np.array([0.8, 1.0]), low=-1.0, high=1.0, ) dist1 = _BatchedDiscreteTruncNormDistributions( mu=np.array([0.0, 1.0]), sigma=np.array([1.0, 1.0]), low=-1.0, high=0.5, step=0.5, ) dist2 = _BatchedCategoricalDistributions(weights=np.array([[0.4, 0.6], [0.2, 0.8]])) mixture_distribution = _MixtureOfProductDistribution( weights=np.array([0.5, 0.5]), distributions=[dist0, dist1, dist2], ) samples = mixture_distribution.sample(np.random.RandomState(0), 5) # Test that the shapes are correct. assert samples.shape == (5, 3) # Test that the samples are in the valid range. assert np.all(dist0.low <= samples[:, 0]) assert np.all(samples[:, 0] <= dist0.high) assert np.all(dist1.low <= samples[:, 1]) assert np.all(samples[:, 1] <= dist1.high) np.testing.assert_almost_equal( np.fmod( samples[:, 1] - dist1.low, dist1.step, ), 0.0, ) assert np.all(0 <= samples[:, 2]) assert np.all(samples[:, 2] <= 1) assert np.all(np.fmod(samples[:, 2], 1.0) == 0.0) # Test reproducibility. assert np.all(samples == mixture_distribution.sample(np.random.RandomState(0), 5)) assert not np.all(samples == mixture_distribution.sample(np.random.RandomState(1), 5)) log_pdf = mixture_distribution.log_pdf(samples) assert log_pdf.shape == (5,) def test_mixture_of_product_distribution_extreme_case() -> None: rng = np.random.RandomState(0) mixture_distribution = _MixtureOfProductDistribution( weights=np.array([1.0, 0.0]), distributions=[ _BatchedTruncNormDistributions( mu=np.array([0.5, 0.3]), sigma=np.array([1e-10, 1.0]), low=-1.0, high=1.0, ), _BatchedDiscreteTruncNormDistributions( mu=np.array([-0.5, 1.0]), sigma=np.array([1e-10, 1.0]), low=-1.0, high=0.5, step=0.5, ), _BatchedCategoricalDistributions(weights=np.array([[0, 1], [0.2, 0.8]])), ], ) samples = mixture_distribution.sample(rng, 2) np.testing.assert_almost_equal(samples, np.array([[0.5, -0.5, 1.0]] * 2)) # The first point has the highest probability, # and all other points have probability almost zero. x = np.array([[0.5, 0.5, 1.0], [0.1, 0.5, 1.0], [0.5, 0.0, 1.0], [0.5, 0.5, 0.0]]) with warnings.catch_warnings(): warnings.simplefilter("ignore", RuntimeWarning) # Ignore log(0) warnings. log_pdf = mixture_distribution.log_pdf(x) assert np.all(log_pdf[1:] < -100) optuna-3.5.0/tests/samplers_tests/tpe_tests/test_sampler.py000066400000000000000000001343431453453102400243360ustar00rootroot00000000000000from __future__ import annotations import random from typing import Callable from typing import Dict from typing import Optional from typing import Union from unittest.mock import Mock from unittest.mock import patch import warnings import _pytest.capture import numpy as np import pytest import optuna from optuna import distributions from optuna.samplers import _tpe from optuna.samplers import TPESampler from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.trial import Trial @pytest.mark.parametrize("use_hyperband", [False, True]) def test_hyperopt_parameters(use_hyperband: bool) -> None: sampler = TPESampler(**TPESampler.hyperopt_parameters()) study = optuna.create_study( sampler=sampler, pruner=optuna.pruners.HyperbandPruner() if use_hyperband else None ) study.optimize(lambda t: t.suggest_float("x", 10, 20), n_trials=50) def test_multivariate_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.TPESampler(multivariate=True) def test_constraints_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): optuna.samplers.TPESampler(constraints_func=lambda _: (0,)) def test_warn_independent_sampling(capsys: _pytest.capture.CaptureFixture) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", ["a", "b"]) if x == "a": return trial.suggest_float("y", 0, 1) else: return trial.suggest_float("z", 0, 1) # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = TPESampler(multivariate=True, warn_independent_sampling=True, n_startup_trials=0) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert err def test_warn_independent_sampling_group(capsys: _pytest.capture.CaptureFixture) -> None: def objective(trial: Trial) -> float: x = trial.suggest_categorical("x", ["a", "b"]) if x == "a": return trial.suggest_float("y", 0, 1) else: return trial.suggest_float("z", 0, 1) # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) sampler = TPESampler( multivariate=True, warn_independent_sampling=True, group=True, n_startup_trials=0 ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10) _, err = capsys.readouterr() assert err == "" def test_infer_relative_search_space() -> None: sampler = TPESampler() search_space = { "a": distributions.FloatDistribution(1.0, 100.0), "b": distributions.FloatDistribution(1.0, 100.0, log=True), "c": distributions.FloatDistribution(1.0, 100.0, step=3.0), "d": distributions.IntDistribution(1, 100), "e": distributions.IntDistribution(0, 100, step=2), "f": distributions.IntDistribution(1, 100, log=True), "g": distributions.CategoricalDistribution(["x", "y", "z"]), } def obj(t: Trial) -> float: t.suggest_float("a", 1.0, 100.0) t.suggest_float("b", 1.0, 100.0, log=True) t.suggest_float("c", 1.0, 100.0, step=3.0) t.suggest_int("d", 1, 100) t.suggest_int("e", 0, 100, step=2) t.suggest_int("f", 1, 100, log=True) t.suggest_categorical("g", ["x", "y", "z"]) return 0.0 # Study and frozen-trial are not supposed to be accessed. study1 = Mock(spec=[]) frozen_trial = Mock(spec=[]) assert sampler.infer_relative_search_space(study1, frozen_trial) == {} study2 = optuna.create_study(sampler=sampler) study2.optimize(obj, n_trials=1) assert sampler.infer_relative_search_space(study2, study2.best_trial) == {} with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=True) study3 = optuna.create_study(sampler=sampler) study3.optimize(obj, n_trials=1) assert sampler.infer_relative_search_space(study3, study3.best_trial) == search_space @pytest.mark.parametrize("multivariate", [False, True]) def test_sample_relative_empty_input(multivariate: bool) -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=multivariate) # A frozen-trial is not supposed to be accessed. study = optuna.create_study() frozen_trial = Mock(spec=[]) assert sampler.sample_relative(study, frozen_trial, {}) == {} def test_sample_relative_prior() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(consider_prior=False, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(prior_weight=0.2, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion def test_sample_relative_n_startup_trial() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] trial = frozen_trial_factory(8) # sample_relative returns {} for only 4 observations. with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials[:4]): assert sampler.sample_relative(study, trial, {"param-a": dist}) == {} # sample_relative returns some value for only 7 observations. study._thread_local.cached_all_trials = None with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert "param-a" in sampler.sample_relative(study, trial, {"param-a": dist}).keys() def test_sample_relative_misc_arguments() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 40)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(40) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) # Test misc. parameters. with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_ei_candidates=13, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(gamma=lambda _: 5, n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler( weights=lambda n: np.asarray([i**2 + 1 for i in range(n)]), n_startup_trials=5, seed=0, multivariate=True, ) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_relative(study, trial, {"param-a": dist}) != suggestion def test_sample_relative_uniform_distributions() -> None: study = optuna.create_study() # Prepare sample from uniform distribution for cheking other distributions. uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_relative(study, trial, {"param-a": uni_dist}) assert 1.0 <= uniform_suggestion["param-a"] < 100.0 def test_sample_relative_log_uniform_distributions() -> None: """Prepare sample from uniform distribution for cheking other distributions.""" study = optuna.create_study() uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_relative(study, trial, {"param-a": uni_dist}) # Test sample from log-uniform is different from uniform. log_dist = optuna.distributions.FloatDistribution(1.0, 100.0, log=True) past_trials = [frozen_trial_factory(i, dist=log_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): loguniform_suggestion = sampler.sample_relative(study, trial, {"param-a": log_dist}) assert 1.0 <= loguniform_suggestion["param-a"] < 100.0 assert uniform_suggestion["param-a"] != loguniform_suggestion["param-a"] def test_sample_relative_disrete_uniform_distributions() -> None: """Test samples from discrete have expected intervals.""" study = optuna.create_study() disc_dist = optuna.distributions.FloatDistribution(1.0, 100.0, step=0.1) def value_fn(idx: int) -> float: random.seed(idx) return int(random.random() * 1000) * 0.1 past_trials = [frozen_trial_factory(i, dist=disc_dist, value_fn=value_fn) for i in range(1, 8)] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): discrete_uniform_suggestion = sampler.sample_relative(study, trial, {"param-a": disc_dist}) assert 1.0 <= discrete_uniform_suggestion["param-a"] <= 100.0 np.testing.assert_almost_equal( int(discrete_uniform_suggestion["param-a"] * 10), discrete_uniform_suggestion["param-a"] * 10, ) def test_sample_relative_categorical_distributions() -> None: """Test samples are drawn from the specified category.""" study = optuna.create_study() categories = [i * 0.3 + 1.0 for i in range(330)] def cat_value_fn(idx: int) -> float: random.seed(idx) return categories[random.randint(0, len(categories) - 1)] cat_dist = optuna.distributions.CategoricalDistribution(categories) past_trials = [ frozen_trial_factory(i, dist=cat_dist, value_fn=cat_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): categorical_suggestion = sampler.sample_relative(study, trial, {"param-a": cat_dist}) assert categorical_suggestion["param-a"] in categories @pytest.mark.parametrize("step", [1, 2]) def test_sample_relative_int_uniform_distributions(step: int) -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return step * random.randint(0, 100 // step) int_dist = optuna.distributions.IntDistribution(0, 100, step=step) past_trials = [ frozen_trial_factory(i, dist=int_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): int_suggestion = sampler.sample_relative(study, trial, {"param-a": int_dist}) assert 1 <= int_suggestion["param-a"] <= 100 assert isinstance(int_suggestion["param-a"], int) assert int_suggestion["param-a"] % step == 0 def test_sample_relative_int_loguniform_distributions() -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return random.randint(0, 100) intlog_dist = optuna.distributions.IntDistribution(1, 100, log=True) past_trials = [ frozen_trial_factory(i, dist=intlog_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) with patch.object(study._storage, "get_all_trials", return_value=past_trials): intlog_suggestion = sampler.sample_relative(study, trial, {"param-a": intlog_dist}) assert 1 <= intlog_suggestion["param-a"] <= 100 assert isinstance(intlog_suggestion["param-a"], int) @pytest.mark.parametrize( "state", [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ], ) def test_sample_relative_handle_unsuccessful_states( state: optuna.trial.TrialState, ) -> None: dist = optuna.distributions.FloatDistribution(1.0, 100.0) # Prepare sampling result for later tests. study = optuna.create_study() for i in range(1, 100): trial = frozen_trial_factory(i, dist=dist) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(100) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) all_success_suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) # Test unsuccessful trials are handled differently. study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 100): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(100) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) partial_unsuccessful_suggestion = sampler.sample_relative(study, trial, {"param-a": dist}) assert partial_unsuccessful_suggestion != all_success_suggestion def test_sample_relative_ignored_states() -> None: """Tests FAIL, RUNNING, and WAITING states are equally.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) suggestions.append(sampler.sample_relative(study, trial, {"param-a": dist})["param-a"]) assert len(set(suggestions)) == 1 def test_sample_relative_pruned_state() -> None: """Tests PRUNED state is treated differently from both FAIL and COMPLETE.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 40): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(40) with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(n_startup_trials=5, seed=0, multivariate=True) suggestions.append(sampler.sample_relative(study, trial, {"param-a": dist})["param-a"]) assert len(set(suggestions)) == 3 def test_sample_independent_prior() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_independent(study, trial, "param-a", dist) sampler = TPESampler(consider_prior=False, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion sampler = TPESampler(prior_weight=0.1, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion def test_sample_independent_n_startup_trial() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials[:4]): with patch.object( optuna.samplers.RandomSampler, "sample_independent", return_value=1.0 ) as sample_method: sampler.sample_independent(study, trial, "param-a", dist) assert sample_method.call_count == 1 sampler = TPESampler(n_startup_trials=5, seed=0) study._thread_local.cached_all_trials = None with patch.object(study._storage, "get_all_trials", return_value=past_trials): with patch.object( optuna.samplers.RandomSampler, "sample_independent", return_value=1.0 ) as sample_method: sampler.sample_independent(study, trial, "param-a", dist) assert sample_method.call_count == 0 def test_sample_independent_misc_arguments() -> None: study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=dist) for i in range(1, 8)] # Prepare a trial and a sample for later checks. trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): suggestion = sampler.sample_independent(study, trial, "param-a", dist) # Test misc. parameters. sampler = TPESampler(n_ei_candidates=13, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion sampler = TPESampler(gamma=lambda _: 5, n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion sampler = TPESampler( weights=lambda i: np.asarray([10 - j for j in range(i)]), n_startup_trials=5, seed=0 ) with patch("optuna.Study._get_trials", return_value=past_trials): assert sampler.sample_independent(study, trial, "param-a", dist) != suggestion def test_sample_independent_uniform_distributions() -> None: study = optuna.create_study() # Prepare sample from uniform distribution for cheking other distributions. uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_independent(study, trial, "param-a", uni_dist) assert 1.0 <= uniform_suggestion < 100.0 def test_sample_independent_log_uniform_distributions() -> None: """Prepare sample from uniform distribution for cheking other distributions.""" study = optuna.create_study() uni_dist = optuna.distributions.FloatDistribution(1.0, 100.0) past_trials = [frozen_trial_factory(i, dist=uni_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): uniform_suggestion = sampler.sample_independent(study, trial, "param-a", uni_dist) # Test sample from log-uniform is different from uniform. log_dist = optuna.distributions.FloatDistribution(1.0, 100.0, log=True) past_trials = [frozen_trial_factory(i, dist=log_dist) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): loguniform_suggestion = sampler.sample_independent(study, trial, "param-a", log_dist) assert 1.0 <= loguniform_suggestion < 100.0 assert uniform_suggestion != loguniform_suggestion def test_sample_independent_discrete_uniform_distributions() -> None: """Test samples from discrete have expected intervals.""" study = optuna.create_study() disc_dist = optuna.distributions.FloatDistribution(1.0, 100.0, step=0.1) def value_fn(idx: int) -> float: random.seed(idx) return int(random.random() * 1000) * 0.1 past_trials = [frozen_trial_factory(i, dist=disc_dist, value_fn=value_fn) for i in range(1, 8)] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch("optuna.Study.get_trials", return_value=past_trials): discrete_uniform_suggestion = sampler.sample_independent( study, trial, "param-a", disc_dist ) assert 1.0 <= discrete_uniform_suggestion <= 100.0 assert abs(int(discrete_uniform_suggestion * 10) - discrete_uniform_suggestion * 10) < 1e-3 def test_sample_independent_categorical_distributions() -> None: """Test samples are drawn from the specified category.""" study = optuna.create_study() categories = [i * 0.3 + 1.0 for i in range(330)] def cat_value_fn(idx: int) -> float: random.seed(idx) return categories[random.randint(0, len(categories) - 1)] cat_dist = optuna.distributions.CategoricalDistribution(categories) past_trials = [ frozen_trial_factory(i, dist=cat_dist, value_fn=cat_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): categorical_suggestion = sampler.sample_independent(study, trial, "param-a", cat_dist) assert categorical_suggestion in categories def test_sample_independent_int_uniform_distributions() -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return random.randint(0, 100) int_dist = optuna.distributions.IntDistribution(1, 100) past_trials = [ frozen_trial_factory(i, dist=int_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): int_suggestion = sampler.sample_independent(study, trial, "param-a", int_dist) assert 1 <= int_suggestion <= 100 assert isinstance(int_suggestion, int) def test_sample_independent_int_loguniform_distributions() -> None: """Test sampling from int distribution returns integer.""" study = optuna.create_study() def int_value_fn(idx: int) -> float: random.seed(idx) return random.randint(0, 100) intlog_dist = optuna.distributions.IntDistribution(1, 100, log=True) past_trials = [ frozen_trial_factory(i, dist=intlog_dist, value_fn=int_value_fn) for i in range(1, 8) ] trial = frozen_trial_factory(8) sampler = TPESampler(n_startup_trials=5, seed=0) with patch.object(study._storage, "get_all_trials", return_value=past_trials): intlog_suggestion = sampler.sample_independent(study, trial, "param-a", intlog_dist) assert 1 <= intlog_suggestion <= 100 assert isinstance(intlog_suggestion, int) @pytest.mark.parametrize( "state", [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ], ) def test_sample_independent_handle_unsuccessful_states(state: optuna.trial.TrialState) -> None: dist = optuna.distributions.FloatDistribution(1.0, 100.0) # Prepare sampling result for later tests. study = optuna.create_study() for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=2) all_success_suggestion = sampler.sample_independent(study, trial, "param-a", dist) # Test unsuccessful trials are handled differently. state_fn = build_state_fn(state) study = optuna.create_study() for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=2) partial_unsuccessful_suggestion = sampler.sample_independent(study, trial, "param-a", dist) assert partial_unsuccessful_suggestion != all_success_suggestion def test_sample_independent_ignored_states() -> None: """Tests FAIL, RUNNING, and WAITING states are equally.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.FAIL, optuna.trial.TrialState.RUNNING, optuna.trial.TrialState.WAITING, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=0) suggestions.append(sampler.sample_independent(study, trial, "param-a", dist)) assert len(set(suggestions)) == 1 def test_sample_independent_pruned_state() -> None: """Tests PRUNED state is treated differently from both FAIL and COMPLETE.""" dist = optuna.distributions.FloatDistribution(1.0, 100.0) suggestions = [] for state in [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.FAIL, optuna.trial.TrialState.PRUNED, ]: study = optuna.create_study() state_fn = build_state_fn(state) for i in range(1, 30): trial = frozen_trial_factory(i, dist=dist, state_fn=state_fn) study._storage.create_new_trial(study._study_id, template_trial=trial) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=5, seed=2) suggestions.append(sampler.sample_independent(study, trial, "param-a", dist)) assert len(set(suggestions)) == 3 def test_constrained_sample_independent_zero_startup() -> None: """Tests TPESampler with constrained option works when n_startup_trials=0.""" study = optuna.create_study() dist = optuna.distributions.FloatDistribution(1.0, 100.0) trial = frozen_trial_factory(30) sampler = TPESampler(n_startup_trials=0, seed=2, constraints_func=lambda _: (0,)) sampler.sample_independent(study, trial, "param-a", dist) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("constant_liar", [True, False]) @pytest.mark.parametrize("constraints", [True, False]) def test_split_trials(direction: str, constant_liar: bool, constraints: bool) -> None: study = optuna.create_study(direction=direction) for value in [-float("inf"), 0, 1, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=(value if direction == "minimize" else -value), params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, ) ) for step in [2, 1]: for value in [-float("inf"), 0, 1, float("inf"), float("nan")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, intermediate_values={step: (value if direction == "minimize" else -value)}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, ) ) if constraints: for value in [1, 2, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=0, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [value]}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.RUNNING, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [-1]}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.FAIL, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.WAITING, ) ) if constant_liar: states = [ optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED, optuna.trial.TrialState.RUNNING, ] else: states = [optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED] trials = study.get_trials(states=states) finished_trials = study.get_trials( states=(optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.PRUNED) ) for n_below in range(len(finished_trials) + 1): below_trials, above_trials = _tpe.sampler._split_trials( study, trials, n_below, constraints, ) below_trial_numbers = [trial.number for trial in below_trials] assert below_trial_numbers == list(range(n_below)) above_trial_numbers = [trial.number for trial in above_trials] assert above_trial_numbers == list(range(n_below, len(trials))) @pytest.mark.parametrize( "directions", [["minimize", "minimize"], ["maximize", "maximize"], ["minimize", "maximize"]] ) def test_split_trials_for_multiobjective_constant_liar(directions: list[str]) -> None: study = optuna.create_study(directions=directions) for obj1 in [-float("inf"), 0, 1, float("inf")]: val1 = obj1 if directions[0] == "minimize" else -obj1 for obj2 in [-float("inf"), 0, 1, float("inf")]: val2 = obj2 if directions[1] == "minimize" else -obj2 study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, values=[val1, val2], params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) for _ in range(5): study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.RUNNING, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.FAIL, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.WAITING, ) ) states = [optuna.trial.TrialState.COMPLETE, optuna.trial.TrialState.RUNNING] trials = study.get_trials(states=states) finished_trials = study.get_trials(states=(optuna.trial.TrialState.COMPLETE,)) ground_truth = [0, 1, 4, 2, 8, 5, 3, 6, 9, 12, 7, 10, 13, 11, 14, 15, 16, 17, 18, 19, 20] for n_below in range(1, len(finished_trials) + 1): below_trials, above_trials = _tpe.sampler._split_trials( study, trials, n_below, constraints_enabled=False, ) below_trial_numbers = [trial.number for trial in below_trials] assert below_trial_numbers == np.sort(ground_truth[:n_below]).tolist() above_trial_numbers = [trial.number for trial in above_trials] assert above_trial_numbers == np.sort(ground_truth[n_below:]).tolist() @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_split_complete_trials_single_objective(direction: str) -> None: study = optuna.create_study(direction=direction) for value in [-float("inf"), 0, 1, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=(value if direction == "minimize" else -value), params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) for n_below in range(len(study.trials) + 1): below_trials, above_trials = _tpe.sampler._split_complete_trials_single_objective( study.trials, study, n_below, ) assert [trial.number for trial in below_trials] == list(range(n_below)) assert [trial.number for trial in above_trials] == list(range(n_below, len(study.trials))) def test_split_complete_trials_single_objective_empty() -> None: study = optuna.create_study() assert _tpe.sampler._split_complete_trials_single_objective([], study, 0) == ([], []) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_split_pruned_trials(direction: str) -> None: study = optuna.create_study(direction=direction) for step in [2, 1]: for value in [-float("inf"), 0, 1, float("inf"), float("nan")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, intermediate_values={step: (value if direction == "minimize" else -value)}, ) ) study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.PRUNED, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, ) ) for n_below in range(len(study.trials) + 1): below_trials, above_trials = _tpe.sampler._split_pruned_trials( study.trials, study, n_below, ) assert [trial.number for trial in below_trials] == list(range(n_below)) assert [trial.number for trial in above_trials] == list(range(n_below, len(study.trials))) def test_split_pruned_trials_empty() -> None: study = optuna.create_study() assert _tpe.sampler._split_pruned_trials([], study, 0) == ([], []) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_split_infeasible_trials(direction: str) -> None: study = optuna.create_study(direction=direction) for value in [1, 2, float("inf")]: study.add_trial( optuna.create_trial( state=optuna.trial.TrialState.COMPLETE, value=0, params={"x": 0}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, 1.0)}, system_attrs={_CONSTRAINTS_KEY: [value]}, ) ) for n_below in range(len(study.trials) + 1): below_trials, above_trials = _tpe.sampler._split_infeasible_trials(study.trials, n_below) assert [trial.number for trial in below_trials] == list(range(n_below)) assert [trial.number for trial in above_trials] == list(range(n_below, len(study.trials))) def test_split_infeasible_trials_empty() -> None: assert _tpe.sampler._split_infeasible_trials([], 0) == ([], []) def frozen_trial_factory( idx: int, dist: optuna.distributions.BaseDistribution = optuna.distributions.FloatDistribution( 1.0, 100.0 ), state_fn: Callable[ [int], optuna.trial.TrialState ] = lambda _: optuna.trial.TrialState.COMPLETE, value_fn: Optional[Callable[[int], Union[int, float]]] = None, target_fn: Callable[[float], float] = lambda val: (val - 20.0) ** 2, interm_val_fn: Callable[[int], Dict[int, float]] = lambda _: {}, ) -> optuna.trial.FrozenTrial: if value_fn is None: random.seed(idx) value = random.random() * 99.0 + 1.0 else: value = value_fn(idx) return optuna.trial.FrozenTrial( number=idx, state=state_fn(idx), value=target_fn(value), datetime_start=None, datetime_complete=None, params={"param-a": value}, distributions={"param-a": dist}, user_attrs={}, system_attrs={}, intermediate_values=interm_val_fn(idx), trial_id=idx, ) def build_state_fn(state: optuna.trial.TrialState) -> Callable[[int], optuna.trial.TrialState]: def state_fn(idx: int) -> optuna.trial.TrialState: return [optuna.trial.TrialState.COMPLETE, state][idx % 2] return state_fn def test_call_after_trial_of_random_sampler() -> None: sampler = TPESampler() study = optuna.create_study(sampler=sampler) with patch.object( sampler._random_sampler, "after_trial", wraps=sampler._random_sampler.after_trial ) as mock_object: study.optimize(lambda _: 1.0, n_trials=1) assert mock_object.call_count == 1 def test_mixed_relative_search_space_pruned_and_completed_trials() -> None: def objective(trial: Trial) -> float: if trial.number == 0: trial.suggest_float("param1", 0, 1) raise optuna.exceptions.TrialPruned() if trial.number == 1: trial.suggest_float("param2", 0, 1) return 0 return 0 sampler = TPESampler(n_startup_trials=1, multivariate=True) study = optuna.create_study(sampler=sampler) study.optimize(objective, 3) def test_group() -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=True, group=True) study = optuna.create_study(sampler=sampler) with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=2) assert mock.call_count == 1 assert study.trials[-1].distributions == {"x": distributions.IntDistribution(low=0, high=10)} with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3), n_trials=1 ) assert mock.call_count == 1 assert study.trials[-1].distributions == { "y": distributions.IntDistribution(low=0, high=10), "z": distributions.FloatDistribution(low=-3, high=3), } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3) + t.suggest_float("u", 1e-2, 1e2, log=True) + bool(t.suggest_categorical("v", ["A", "B", "C"])), n_trials=1, ) assert mock.call_count == 2 assert study.trials[-1].distributions == { "u": distributions.FloatDistribution(low=1e-2, high=1e2, log=True), "v": distributions.CategoricalDistribution(choices=["A", "B", "C"]), "y": distributions.IntDistribution(low=0, high=10), "z": distributions.FloatDistribution(low=-3, high=3), } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize(lambda t: t.suggest_float("u", 1e-2, 1e2, log=True), n_trials=1) assert mock.call_count == 3 assert study.trials[-1].distributions == { "u": distributions.FloatDistribution(low=1e-2, high=1e2, log=True) } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_int("w", 2, 8, log=True), n_trials=1 ) assert mock.call_count == 4 assert study.trials[-1].distributions == { "y": distributions.IntDistribution(low=0, high=10), "w": distributions.IntDistribution(low=2, high=8, log=True), } with patch.object(sampler, "_sample_relative", wraps=sampler._sample_relative) as mock: study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=1) assert mock.call_count == 6 assert study.trials[-1].distributions == {"x": distributions.IntDistribution(low=0, high=10)} def test_invalid_multivariate_and_group() -> None: with pytest.raises(ValueError): _ = TPESampler(multivariate=False, group=True) def test_group_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): _ = TPESampler(multivariate=True, group=True) def test_constant_liar_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): _ = TPESampler(constant_liar=True) @pytest.mark.parametrize("multivariate", [True, False]) @pytest.mark.parametrize("multiobjective", [True, False]) def test_constant_liar_with_running_trial(multivariate: bool, multiobjective: bool) -> None: with warnings.catch_warnings(): warnings.simplefilter("ignore", optuna.exceptions.ExperimentalWarning) sampler = TPESampler(multivariate=multivariate, constant_liar=True, n_startup_trials=0) study = optuna.create_study(sampler=sampler) # Add a complete trial. trial0 = study.ask() trial0.suggest_int("x", 0, 10) trial0.suggest_float("y", 0, 10) trial0.suggest_categorical("z", [0, 1, 2]) study.tell(trial0, [0, 0] if multiobjective else 0) # Add running trials. trial1 = study.ask() trial1.suggest_int("x", 0, 10) trial2 = study.ask() trial2.suggest_float("y", 0, 10) trial3 = study.ask() trial3.suggest_categorical("z", [0, 1, 2]) # Test suggestion with running trials. trial = study.ask() trial.suggest_int("x", 0, 10) trial.suggest_float("y", 0, 10) trial.suggest_categorical("z", [0, 1, 2]) study.tell(trial, [0, 0] if multiobjective else 0) def test_categorical_distance_func_experimental_warning() -> None: with pytest.warns(optuna.exceptions.ExperimentalWarning): _ = TPESampler(categorical_distance_func={"c": lambda x, y: 0.0}) optuna-3.5.0/tests/samplers_tests/tpe_tests/test_truncnorm.py000066400000000000000000000046001453453102400247120ustar00rootroot00000000000000import sys import numpy as np import pytest from scipy.stats import truncnorm as truncnorm_scipy from optuna._imports import try_import import optuna.samplers._tpe._truncnorm as truncnorm_ours with try_import() as _imports: from scipy.stats._continuous_distns import _log_gauss_mass as _log_gauss_mass_scipy @pytest.mark.skipif( sys.version_info < (3, 8, 0), reason="SciPy 1.9.2 is not supported in Python 3.7" ) @pytest.mark.parametrize( "a,b", [(-np.inf, np.inf), (-10, +10), (-1, +1), (-1e-3, +1e-3), (10, 100), (-100, -10), (0, 0)], ) def test_ppf(a: float, b: float) -> None: for x in np.concatenate( [np.linspace(0, 1, num=100), np.array([sys.float_info.min, 1 - sys.float_info.epsilon])] ): assert truncnorm_ours.ppf(x, a, b) == pytest.approx( truncnorm_scipy.ppf(x, a, b), nan_ok=True ), f"ppf(x={x}, a={a}, b={b})" @pytest.mark.skipif( sys.version_info < (3, 8, 0), reason="SciPy 1.9.2 is not supported in Python 3.7" ) @pytest.mark.parametrize( "a,b", [(-np.inf, np.inf), (-10, +10), (-1, +1), (-1e-3, +1e-3), (10, 100), (-100, -10), (0, 0)], ) @pytest.mark.parametrize("loc", [-10, 0, 10]) @pytest.mark.parametrize("scale", [0.1, 1, 10]) def test_logpdf(a: float, b: float, loc: float, scale: float) -> None: for x in np.concatenate( [np.linspace(np.max([a, -100]), np.min([b, 100]), num=1000), np.array([-2000.0, +2000.0])] ): assert truncnorm_ours.logpdf(x, a, b, loc, scale) == pytest.approx( truncnorm_scipy.logpdf(x, a, b, loc, scale), nan_ok=True ), f"logpdf(x={x}, a={a}, b={b})" @pytest.mark.skipif( sys.version_info < (3, 8, 0), reason="SciPy 1.9.2 is not supported in Python 3.7" ) @pytest.mark.skipif( not _imports.is_successful(), reason="Failed to import SciPy's internal function." ) @pytest.mark.parametrize( "a,b", # we don't test (0, 0) as SciPy returns the incorrect value. [(-np.inf, np.inf), (-10, +10), (-1, +1), (-1e-3, +1e-3), (10, 100), (-100, -10)], ) def test_log_gass_mass(a: float, b: float) -> None: for x in np.concatenate( [np.linspace(0, 1, num=100), np.array([sys.float_info.min, 1 - sys.float_info.epsilon])] ): assert truncnorm_ours._log_gauss_mass(np.array([a]), np.array([b])) == pytest.approx( np.atleast_1d(_log_gauss_mass_scipy(a, b)), nan_ok=True ), f"_log_gauss_mass(x={x}, a={a}, b={b})" optuna-3.5.0/tests/search_space_tests/000077500000000000000000000000001453453102400200325ustar00rootroot00000000000000optuna-3.5.0/tests/search_space_tests/__init__.py000066400000000000000000000000001453453102400221310ustar00rootroot00000000000000optuna-3.5.0/tests/search_space_tests/test_group_decomposed.py000066400000000000000000000200421453453102400247770ustar00rootroot00000000000000import pytest from optuna import create_study from optuna import TrialPruned from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.search_space import _GroupDecomposedSearchSpace from optuna.search_space import _SearchSpaceGroup from optuna.testing.storages import StorageSupplier from optuna.trial import Trial def test_search_space_group() -> None: search_space_group = _SearchSpaceGroup() # No search space. assert search_space_group.search_spaces == [] # No distributions. search_space_group.add_distributions({}) assert search_space_group.search_spaces == [] # Add a single distribution. search_space_group.add_distributions({"x": IntDistribution(low=0, high=10)}) assert search_space_group.search_spaces == [{"x": IntDistribution(low=0, high=10)}] # Add a same single distribution. search_space_group.add_distributions({"x": IntDistribution(low=0, high=10)}) assert search_space_group.search_spaces == [{"x": IntDistribution(low=0, high=10)}] # Add disjoint distributions. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, ] # Add distributions, which include one of search spaces in the group. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), "u": FloatDistribution(low=1e-2, high=1e2, log=True), "v": CategoricalDistribution(choices=["A", "B", "C"]), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, { "u": FloatDistribution(low=1e-2, high=1e2, log=True), "v": CategoricalDistribution(choices=["A", "B", "C"]), }, ] # Add a distribution, which is included by one of search spaces in the group. search_space_group.add_distributions({"u": FloatDistribution(low=1e-2, high=1e2, log=True)}) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, ] # Add distributions whose intersection with one of search spaces in the group is not empty. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "w": IntDistribution(low=2, high=8, log=True), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, {"y": IntDistribution(low=0, high=10)}, {"z": FloatDistribution(low=-3, high=3)}, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, {"w": IntDistribution(low=2, high=8, log=True)}, ] # Add distributions which include some of search spaces in the group. search_space_group.add_distributions( { "y": IntDistribution(low=0, high=10), "w": IntDistribution(low=2, high=8, log=True), "t": FloatDistribution(low=10, high=100), } ) assert search_space_group.search_spaces == [ {"x": IntDistribution(low=0, high=10)}, {"y": IntDistribution(low=0, high=10)}, {"z": FloatDistribution(low=-3, high=3)}, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, {"w": IntDistribution(low=2, high=8, log=True)}, {"t": FloatDistribution(low=10, high=100)}, ] def test_group_decomposed_search_space() -> None: search_space = _GroupDecomposedSearchSpace() study = create_study() # No trial. assert search_space.calculate(study).search_spaces == [] # A single parameter. study.optimize(lambda t: t.suggest_int("x", 0, 10), n_trials=1) assert search_space.calculate(study).search_spaces == [{"x": IntDistribution(low=0, high=10)}] # Disjoint parameters. study.optimize(lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3), n_trials=1) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, ] # Parameters which include one of search spaces in the group. study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_float("z", -3, 3) + t.suggest_float("u", 1e-2, 1e2, log=True) + bool(t.suggest_categorical("v", ["A", "B", "C"])), n_trials=1, ) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "z": FloatDistribution(low=-3, high=3), "y": IntDistribution(low=0, high=10), }, { "u": FloatDistribution(low=1e-2, high=1e2, log=True), "v": CategoricalDistribution(choices=["A", "B", "C"]), }, ] # A parameter which is included by one of search spaces in thew group. study.optimize(lambda t: t.suggest_float("u", 1e-2, 1e2, log=True), n_trials=1) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, { "y": IntDistribution(low=0, high=10), "z": FloatDistribution(low=-3, high=3), }, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, ] # Parameters whose intersection with one of search spaces in the group is not empty. study.optimize( lambda t: t.suggest_int("y", 0, 10) + t.suggest_int("w", 2, 8, log=True), n_trials=1 ) assert search_space.calculate(study).search_spaces == [ {"x": IntDistribution(low=0, high=10)}, {"y": IntDistribution(low=0, high=10)}, {"z": FloatDistribution(low=-3, high=3)}, {"u": FloatDistribution(low=1e-2, high=1e2, log=True)}, {"v": CategoricalDistribution(choices=["A", "B", "C"])}, {"w": IntDistribution(low=2, high=8, log=True)}, ] search_space = _GroupDecomposedSearchSpace() study = create_study() # Failed or pruned trials are not considered in the calculation of # an intersection search space. def objective(trial: Trial, exception: Exception) -> float: trial.suggest_float("a", 0, 1) raise exception study.optimize(lambda t: objective(t, RuntimeError()), n_trials=1, catch=(RuntimeError,)) study.optimize(lambda t: objective(t, TrialPruned()), n_trials=1) assert search_space.calculate(study).search_spaces == [] # If two parameters have the same name but different distributions, # the first one takes priority. study.optimize(lambda t: t.suggest_float("a", -1, 1), n_trials=1) study.optimize(lambda t: t.suggest_float("a", 0, 1), n_trials=1) assert search_space.calculate(study).search_spaces == [ {"a": FloatDistribution(low=-1, high=1)} ] def test_group_decomposed_search_space_with_different_studies() -> None: search_space = _GroupDecomposedSearchSpace() with StorageSupplier("sqlite") as storage: study0 = create_study(storage=storage) study1 = create_study(storage=storage) search_space.calculate(study0) with pytest.raises(ValueError): # `_GroupDecomposedSearchSpace` isn't supposed to be used for multiple studies. search_space.calculate(study1) optuna-3.5.0/tests/search_space_tests/test_intersection.py000066400000000000000000000070101453453102400241470ustar00rootroot00000000000000import pytest from optuna import create_study from optuna import TrialPruned from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.search_space import intersection_search_space from optuna.search_space import IntersectionSearchSpace from optuna.testing.storages import StorageSupplier from optuna.trial import Trial def test_intersection_search_space() -> None: search_space = IntersectionSearchSpace() study = create_study() # No trial. assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # Waiting trial. study.enqueue_trial( {"y": 0, "x": 5}, {"y": FloatDistribution(-3, 3), "x": IntDistribution(0, 10)} ) assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # First trial. study.optimize(lambda t: t.suggest_float("y", -3, 3) + t.suggest_int("x", 0, 10), n_trials=1) assert search_space.calculate(study) == { "x": IntDistribution(low=0, high=10), "y": FloatDistribution(low=-3, high=3), } assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # Returned dict is sorted by parameter names. assert list(search_space.calculate(study).keys()) == ["x", "y"] # Second trial (only 'y' parameter is suggested in this trial). study.optimize(lambda t: t.suggest_float("y", -3, 3), n_trials=1) assert search_space.calculate(study) == {"y": FloatDistribution(low=-3, high=3)} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # Failed or pruned trials are not considered in the calculation of # an intersection search space. def objective(trial: Trial, exception: Exception) -> float: trial.suggest_float("z", 0, 1) raise exception study.optimize(lambda t: objective(t, RuntimeError()), n_trials=1, catch=(RuntimeError,)) study.optimize(lambda t: objective(t, TrialPruned()), n_trials=1) assert search_space.calculate(study) == {"y": FloatDistribution(low=-3, high=3)} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # If two parameters have the same name but different distributions, # those are regarded as different parameters. study.optimize(lambda t: t.suggest_float("y", -1, 1), n_trials=1) assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) # The search space remains empty once it is empty. study.optimize(lambda t: t.suggest_float("y", -3, 3) + t.suggest_int("x", 0, 10), n_trials=1) assert search_space.calculate(study) == {} assert search_space.calculate(study) == intersection_search_space( study.get_trials(deepcopy=False) ) def test_intersection_search_space_class_with_different_studies() -> None: search_space = IntersectionSearchSpace() with StorageSupplier("sqlite") as storage: study0 = create_study(storage=storage) study1 = create_study(storage=storage) search_space.calculate(study0) with pytest.raises(ValueError): # An `IntersectionSearchSpace` instance isn't supposed to be used for multiple studies. search_space.calculate(study1) optuna-3.5.0/tests/storages_tests/000077500000000000000000000000001453453102400172415ustar00rootroot00000000000000optuna-3.5.0/tests/storages_tests/__init__.py000066400000000000000000000000001453453102400213400ustar00rootroot00000000000000optuna-3.5.0/tests/storages_tests/rdb_tests/000077500000000000000000000000001453453102400212325ustar00rootroot00000000000000optuna-3.5.0/tests/storages_tests/rdb_tests/__init__.py000066400000000000000000000000001453453102400233310ustar00rootroot00000000000000optuna-3.5.0/tests/storages_tests/rdb_tests/create_db.py000066400000000000000000000075241453453102400235240ustar00rootroot00000000000000"""This script generates assets for testing schema migration. 1. Prepare Optuna If you want to generate a DB file for the latest version of Optuna, you have to edit `optuna/version.py` since we add a suffix to a version in the master branch. > cat optuna/version.py __version__ = "3.0.0b0.dev" Please temporarily remove the suffix when running this script. After generating an asset, the change should be reverted. If you want to generate a DB file for older versions of Optuna, you have to install it. I recommend you to create isolated environment using `venv` for this purpose. ```sh > deactivate # if you already use `venv` for development > python3 -m venv venv_gen > . venv_gen/bin/activate > pip install optuna==2.6.0 # install Optuna v2.6.0 ``` 2. Generate database ```sh > python3 create_db.py [I 2022-02-05 15:39:32,488] A new study created in RDB with name: single_empty ... > ``` 3. Switch Optuna version to the latest one If you use `venv`, simply `deactivate` and re-activate your development environment. """ from argparse import ArgumentParser from typing import Tuple from packaging import version import optuna def objective_test_upgrade(trial: optuna.trial.Trial) -> float: x = trial.suggest_float("x", -5, 5) # optuna==0.9.0 does not have suggest_float. y = trial.suggest_int("y", 0, 10) z = trial.suggest_categorical("z", [-5, 0, 5]) trial.storage.set_trial_system_attr(trial._trial_id, "a", 0) trial.set_user_attr("b", 1) trial.report(0.5, step=0) return x**2 + y**2 + z**2 def mo_objective_test_upgrade(trial: optuna.trial.Trial) -> Tuple[float, float]: x = trial.suggest_float("x", -5, 5) y = trial.suggest_int("y", 0, 10) z = trial.suggest_categorical("z", [-5, 0, 5]) trial.storage.set_trial_system_attr(trial._trial_id, "a", 0) trial.set_user_attr("b", 1) return x, x**2 + y**2 + z**2 def objective_test_upgrade_distributions(trial: optuna.trial.Trial) -> float: x1 = trial.suggest_float("x1", -5, 5) x2 = trial.suggest_float("x2", 1e-5, 1e-3, log=True) x3 = trial.suggest_float("x3", -6, 6, step=2) y1 = trial.suggest_int("y1", 0, 10) y2 = trial.suggest_int("y2", 1, 20, log=True) y3 = trial.suggest_int("y3", 5, 15, step=3) z = trial.suggest_categorical("z", [-5, 0, 5]) return x1**2 + x2**2 + x3**2 + y1**2 + y2**2 + y3**2 + z**2 if __name__ == "__main__": parser = ArgumentParser(description="Create SQLite database for schema upgrade tests.") parser.add_argument( "--storage-url", default=f"sqlite:///test_upgrade_assets/{optuna.__version__}.db" ) args = parser.parse_args() # Create an empty study. optuna.create_study(storage=args.storage_url, study_name="single_empty") # Create a study for single-objective optimization. study = optuna.create_study(storage=args.storage_url, study_name="single") study.set_user_attr("d", 3) study.optimize(objective_test_upgrade, n_trials=1) # Create a study for multi-objective optimization. try: optuna.create_study( storage=args.storage_url, study_name="multi_empty", directions=["minimize", "minimize"] ) study = optuna.create_study( storage=args.storage_url, study_name="multi", directions=["minimize", "minimize"] ) study.set_user_attr("d", 3) study.optimize(mo_objective_test_upgrade, n_trials=1) except TypeError: print(f"optuna=={optuna.__version__} does not support multi-objective optimization.") # Create a study for distributions upgrade. if version.parse(optuna.__version__) >= version.parse("2.4.0"): study = optuna.create_study(storage=args.storage_url, study_name="schema migration") study.optimize(objective_test_upgrade_distributions, n_trials=1) for s in optuna.get_all_study_summaries(args.storage_url): print(f"{s.study_name}, {s.n_trials}") optuna-3.5.0/tests/storages_tests/rdb_tests/test_models.py000066400000000000000000000407241453453102400241350ustar00rootroot00000000000000from datetime import datetime import pytest from sqlalchemy import create_engine from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import Session from optuna.storages._rdb.models import BaseModel from optuna.storages._rdb.models import StudyDirectionModel from optuna.storages._rdb.models import StudyModel from optuna.storages._rdb.models import StudySystemAttributeModel from optuna.storages._rdb.models import TrialHeartbeatModel from optuna.storages._rdb.models import TrialIntermediateValueModel from optuna.storages._rdb.models import TrialModel from optuna.storages._rdb.models import TrialSystemAttributeModel from optuna.storages._rdb.models import TrialUserAttributeModel from optuna.storages._rdb.models import TrialValueModel from optuna.storages._rdb.models import VersionInfoModel from optuna.study._study_direction import StudyDirection from optuna.trial import TrialState @pytest.fixture def session() -> Session: engine = create_engine("sqlite:///:memory:") BaseModel.metadata.create_all(engine) return Session(bind=engine) class TestStudyDirectionModel: @staticmethod def _create_model(session: Session) -> StudyModel: study = StudyModel(study_id=1, study_name="test-study") dummy_study = StudyModel(study_id=2, study_name="dummy-study") session.add( StudyDirectionModel( study_id=study.study_id, direction=StudyDirection.MINIMIZE, objective=0 ) ) session.add( StudyDirectionModel( study_id=dummy_study.study_id, direction=StudyDirection.MINIMIZE, objective=0 ) ) session.commit() return study @staticmethod def test_where_study_id(session: Session) -> None: study = TestStudyDirectionModel._create_model(session) assert 1 == len(StudyDirectionModel.where_study_id(study.study_id, session)) assert 0 == len(StudyDirectionModel.where_study_id(-1, session)) @staticmethod def test_cascade_delete_on_study(session: Session) -> None: directions = [ StudyDirectionModel(study_id=1, direction=StudyDirection.MINIMIZE, objective=0), StudyDirectionModel(study_id=1, direction=StudyDirection.MAXIMIZE, objective=1), ] study = StudyModel(study_id=1, study_name="test-study", directions=directions) session.add(study) session.commit() assert 2 == len(StudyDirectionModel.where_study_id(study.study_id, session)) session.delete(study) session.commit() assert 0 == len(StudyDirectionModel.where_study_id(study.study_id, session)) class TestStudySystemAttributeModel: @staticmethod def test_find_by_study_and_key(session: Session) -> None: study = StudyModel(study_id=1, study_name="test-study") session.add( StudySystemAttributeModel(study_id=study.study_id, key="sample-key", value_json="1") ) session.commit() attr = StudySystemAttributeModel.find_by_study_and_key(study, "sample-key", session) assert attr is not None and "1" == attr.value_json assert StudySystemAttributeModel.find_by_study_and_key(study, "not-found", session) is None @staticmethod def test_where_study_id(session: Session) -> None: sample_study = StudyModel(study_id=1, study_name="test-study") empty_study = StudyModel(study_id=2, study_name="test-study") session.add( StudySystemAttributeModel( study_id=sample_study.study_id, key="sample-key", value_json="1" ) ) assert 1 == len(StudySystemAttributeModel.where_study_id(sample_study.study_id, session)) assert 0 == len(StudySystemAttributeModel.where_study_id(empty_study.study_id, session)) # Check the case of unknown study_id. assert 0 == len(StudySystemAttributeModel.where_study_id(-1, session)) @staticmethod def test_cascade_delete_on_study(session: Session) -> None: study_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=study_id, study_name="test-study", directions=[direction]) study.system_attributes.append( StudySystemAttributeModel(study_id=study_id, key="sample-key1", value_json="1") ) study.system_attributes.append( StudySystemAttributeModel(study_id=study_id, key="sample-key2", value_json="2") ) session.add(study) session.commit() assert 2 == len(StudySystemAttributeModel.where_study_id(study_id, session)) session.delete(study) session.commit() assert 0 == len(StudySystemAttributeModel.where_study_id(study_id, session)) class TestTrialModel: @staticmethod def test_default_datetime(session: Session) -> None: # Regardless of the initial state the trial created here should have null datetime_start session.add(TrialModel(state=TrialState.WAITING)) session.commit() trial_model = session.query(TrialModel).first() assert trial_model is not None assert trial_model.datetime_start is None assert trial_model.datetime_complete is None @staticmethod def test_count(session: Session) -> None: study_1 = StudyModel(study_id=1, study_name="test-study-1") study_2 = StudyModel(study_id=2, study_name="test-study-2") session.add(TrialModel(study_id=study_1.study_id, state=TrialState.COMPLETE)) session.add(TrialModel(study_id=study_1.study_id, state=TrialState.RUNNING)) session.add(TrialModel(study_id=study_2.study_id, state=TrialState.RUNNING)) session.commit() assert 3 == TrialModel.count(session) assert 2 == TrialModel.count(session, study=study_1) assert 1 == TrialModel.count(session, state=TrialState.COMPLETE) @staticmethod def test_count_past_trials(session: Session) -> None: study_1 = StudyModel(study_id=1, study_name="test-study-1") study_2 = StudyModel(study_id=2, study_name="test-study-2") trial_1_1 = TrialModel(study_id=study_1.study_id, state=TrialState.COMPLETE) session.add(trial_1_1) session.commit() assert 0 == trial_1_1.count_past_trials(session) trial_1_2 = TrialModel(study_id=study_1.study_id, state=TrialState.RUNNING) session.add(trial_1_2) session.commit() assert 1 == trial_1_2.count_past_trials(session) trial_2_1 = TrialModel(study_id=study_2.study_id, state=TrialState.RUNNING) session.add(trial_2_1) session.commit() assert 0 == trial_2_1.count_past_trials(session) @staticmethod def test_cascade_delete_on_study(session: Session) -> None: study_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=study_id, study_name="test-study", directions=[direction]) study.trials.append(TrialModel(study_id=study.study_id, state=TrialState.COMPLETE)) study.trials.append(TrialModel(study_id=study.study_id, state=TrialState.RUNNING)) session.add(study) session.commit() assert 2 == TrialModel.count(session, study) session.delete(study) session.commit() assert 0 == TrialModel.count(session, study) class TestTrialUserAttributeModel: @staticmethod def test_find_by_trial_and_key(session: Session) -> None: study = StudyModel(study_id=1, study_name="test-study") trial = TrialModel(study_id=study.study_id) session.add( TrialUserAttributeModel(trial_id=trial.trial_id, key="sample-key", value_json="1") ) session.commit() attr = TrialUserAttributeModel.find_by_trial_and_key(trial, "sample-key", session) assert attr is not None assert "1" == attr.value_json assert TrialUserAttributeModel.find_by_trial_and_key(trial, "not-found", session) is None @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=trial_id, study_id=study.study_id, state=TrialState.COMPLETE) trial.user_attributes.append( TrialUserAttributeModel(trial_id=trial_id, key="sample-key1", value_json="1") ) trial.user_attributes.append( TrialUserAttributeModel(trial_id=trial_id, key="sample-key2", value_json="2") ) study.trials.append(trial) session.add(study) session.commit() assert 2 == len(TrialUserAttributeModel.where_trial_id(trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialUserAttributeModel.where_trial_id(trial_id, session)) class TestTrialSystemAttributeModel: @staticmethod def test_find_by_trial_and_key(session: Session) -> None: study = StudyModel(study_id=1, study_name="test-study") trial = TrialModel(study_id=study.study_id) session.add( TrialSystemAttributeModel(trial_id=trial.trial_id, key="sample-key", value_json="1") ) session.commit() attr = TrialSystemAttributeModel.find_by_trial_and_key(trial, "sample-key", session) assert attr is not None assert "1" == attr.value_json assert TrialSystemAttributeModel.find_by_trial_and_key(trial, "not-found", session) is None @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial_id = 1 direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=trial_id, study_id=study.study_id, state=TrialState.COMPLETE) trial.system_attributes.append( TrialSystemAttributeModel(trial_id=trial_id, key="sample-key1", value_json="1") ) trial.system_attributes.append( TrialSystemAttributeModel(trial_id=trial_id, key="sample-key2", value_json="2") ) study.trials.append(trial) session.add(study) session.commit() assert 2 == len(TrialSystemAttributeModel.where_trial_id(trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialSystemAttributeModel.where_trial_id(trial_id, session)) class TestTrialValueModel: @staticmethod def _create_model(session: Session) -> TrialModel: direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=1, study_id=study.study_id, state=TrialState.COMPLETE) session.add(study) session.add(trial) session.add( TrialValueModel( trial_id=trial.trial_id, objective=0, value=10, value_type=TrialValueModel.TrialValueType.FINITE, ) ) session.commit() return trial @staticmethod def test_find_by_trial_and_objective(session: Session) -> None: trial = TestTrialValueModel._create_model(session) trial_value = TrialValueModel.find_by_trial_and_objective(trial, 0, session) assert trial_value is not None assert 10 == trial_value.value assert TrialValueModel.find_by_trial_and_objective(trial, 1, session) is None @staticmethod def test_where_trial_id(session: Session) -> None: trial = TestTrialValueModel._create_model(session) trial_values = TrialValueModel.where_trial_id(trial.trial_id, session) assert 1 == len(trial_values) assert 0 == trial_values[0].objective assert 10 == trial_values[0].value @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial = TestTrialValueModel._create_model(session) trial.values.append( TrialValueModel( trial_id=1, objective=1, value=20, value_type=TrialValueModel.TrialValueType.FINITE ) ) session.commit() assert 2 == len(TrialValueModel.where_trial_id(trial.trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialValueModel.where_trial_id(trial.trial_id, session)) class TestTrialIntermediateValueModel: @staticmethod def _create_model(session: Session) -> TrialModel: direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=1, study_id=study.study_id, state=TrialState.COMPLETE) session.add(study) session.add(trial) session.add( TrialIntermediateValueModel( trial_id=trial.trial_id, step=0, intermediate_value=10, intermediate_value_type=TrialIntermediateValueModel.TrialIntermediateValueType.FINITE, # noqa: E501 ) ) session.commit() return trial @staticmethod def test_find_by_trial_and_step(session: Session) -> None: trial = TestTrialIntermediateValueModel._create_model(session) trial_intermediate_value = TrialIntermediateValueModel.find_by_trial_and_step( trial, 0, session ) assert trial_intermediate_value is not None assert 10 == trial_intermediate_value.intermediate_value assert TrialIntermediateValueModel.find_by_trial_and_step(trial, 1, session) is None @staticmethod def test_where_trial_id(session: Session) -> None: trial = TestTrialIntermediateValueModel._create_model(session) trial_intermediate_values = TrialIntermediateValueModel.where_trial_id( trial.trial_id, session ) assert 1 == len(trial_intermediate_values) assert 0 == trial_intermediate_values[0].step assert 10 == trial_intermediate_values[0].intermediate_value @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial = TestTrialIntermediateValueModel._create_model(session) trial.intermediate_values.append( TrialIntermediateValueModel( trial_id=1, step=1, intermediate_value=20, intermediate_value_type=TrialIntermediateValueModel.TrialIntermediateValueType.FINITE, # noqa: E501 ) ) session.commit() assert 2 == len(TrialIntermediateValueModel.where_trial_id(trial.trial_id, session)) session.delete(trial) session.commit() assert 0 == len(TrialIntermediateValueModel.where_trial_id(trial.trial_id, session)) class TestTrialHeartbeatModel: @staticmethod def _create_model(session: Session) -> TrialModel: direction = StudyDirectionModel(direction=StudyDirection.MINIMIZE, objective=0) study = StudyModel(study_id=1, study_name="test-study", directions=[direction]) trial = TrialModel(trial_id=1, study_id=study.study_id, state=TrialState.COMPLETE) session.add(study) session.add(trial) session.add(TrialHeartbeatModel(trial_id=trial.trial_id)) session.commit() return trial @staticmethod def test_where_trial_id(session: Session) -> None: trial = TestTrialHeartbeatModel._create_model(session) trial_heartbeat = TrialHeartbeatModel.where_trial_id(trial.trial_id, session) assert trial_heartbeat is not None assert isinstance(trial_heartbeat.heartbeat, datetime) @staticmethod def test_cascade_delete_on_trial(session: Session) -> None: trial = TestTrialHeartbeatModel._create_model(session) session.commit() assert TrialHeartbeatModel.where_trial_id(trial.trial_id, session) is not None session.delete(trial) session.commit() assert TrialHeartbeatModel.where_trial_id(trial.trial_id, session) is None class TestVersionInfoModel: @staticmethod def test_version_info_id_constraint(session: Session) -> None: session.add(VersionInfoModel(schema_version=1, library_version="0.0.1")) session.commit() # Test check constraint of version_info_id. session.add(VersionInfoModel(version_info_id=2, schema_version=2, library_version="0.0.2")) pytest.raises(IntegrityError, lambda: session.commit()) optuna-3.5.0/tests/storages_tests/rdb_tests/test_storage.py000066400000000000000000000272131453453102400243140ustar00rootroot00000000000000import os import platform import shutil import sys import tempfile from typing import Any from typing import Dict from typing import Optional from unittest.mock import patch import warnings import pytest from sqlalchemy.exc import IntegrityError import optuna from optuna import create_study from optuna import load_study from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.storages import RDBStorage from optuna.storages._rdb.models import SCHEMA_VERSION from optuna.storages._rdb.models import VersionInfoModel from optuna.storages._rdb.storage import _create_scoped_session from optuna.testing.tempfile_pool import NamedTemporaryFilePool from .create_db import mo_objective_test_upgrade from .create_db import objective_test_upgrade from .create_db import objective_test_upgrade_distributions def test_init() -> None: storage = create_test_storage() session = storage.scoped_session() version_info = session.query(VersionInfoModel).first() assert version_info is not None assert version_info.schema_version == SCHEMA_VERSION assert version_info.library_version == optuna.version.__version__ assert storage.get_current_version() == storage.get_head_version() assert storage.get_all_versions() == [ "v3.2.0.a", "v3.0.0.d", "v3.0.0.c", "v3.0.0.b", "v3.0.0.a", "v2.6.0.a", "v2.4.0.a", "v1.3.0.a", "v1.2.0.a", "v0.9.0.a", ] def test_init_url_template() -> None: with NamedTemporaryFilePool(suffix="{SCHEMA_VERSION}") as tf: storage = RDBStorage("sqlite:///" + tf.name) assert storage.engine.url.database is not None assert storage.engine.url.database.endswith(str(SCHEMA_VERSION)) def test_init_url_that_contains_percent_character() -> None: # Alembic's ini file regards '%' as the special character for variable expansion. # We checks `RDBStorage` does not raise an error even if a storage url contains the character. with NamedTemporaryFilePool(suffix="%") as tf: RDBStorage("sqlite:///" + tf.name) with NamedTemporaryFilePool(suffix="%foo") as tf: RDBStorage("sqlite:///" + tf.name) with NamedTemporaryFilePool(suffix="%foo%%bar") as tf: RDBStorage("sqlite:///" + tf.name) def test_init_db_module_import_error() -> None: expected_msg = ( "Failed to import DB access module for the specified storage URL. " "Please install appropriate one." ) with patch.dict(sys.modules, {"psycopg2": None}): with pytest.raises(ImportError, match=expected_msg): RDBStorage("postgresql://user:password@host/database") def test_engine_kwargs() -> None: create_test_storage(engine_kwargs={"pool_size": 5}) @pytest.mark.parametrize( "url,engine_kwargs,expected", [ ("mysql://localhost", {"pool_pre_ping": False}, False), ("mysql://localhost", {"pool_pre_ping": True}, True), ("mysql://localhost", {}, True), ("mysql+pymysql://localhost", {}, True), ("mysql://localhost", {"pool_size": 5}, True), ], ) def test_set_default_engine_kwargs_for_mysql( url: str, engine_kwargs: Dict[str, Any], expected: bool ) -> None: RDBStorage._set_default_engine_kwargs_for_mysql(url, engine_kwargs) assert engine_kwargs["pool_pre_ping"] is expected def test_set_default_engine_kwargs_for_mysql_with_other_rdb() -> None: # Do not change engine_kwargs if database is not MySQL. engine_kwargs: Dict[str, Any] = {} RDBStorage._set_default_engine_kwargs_for_mysql("sqlite:///example.db", engine_kwargs) assert "pool_pre_ping" not in engine_kwargs RDBStorage._set_default_engine_kwargs_for_mysql("postgres:///example.db", engine_kwargs) assert "pool_pre_ping" not in engine_kwargs def test_check_table_schema_compatibility() -> None: storage = create_test_storage() session = storage.scoped_session() # The schema version of a newly created storage is always up-to-date. storage._version_manager.check_table_schema_compatibility() # `SCHEMA_VERSION` has not been used for compatibility check since alembic was introduced. version_info = session.query(VersionInfoModel).one() version_info.schema_version = SCHEMA_VERSION - 1 session.commit() storage._version_manager.check_table_schema_compatibility() with pytest.raises(RuntimeError): storage._version_manager._set_alembic_revision( storage._version_manager._get_base_version() ) storage._version_manager.check_table_schema_compatibility() def create_test_storage(engine_kwargs: Optional[Dict[str, Any]] = None) -> RDBStorage: storage = RDBStorage("sqlite:///:memory:", engine_kwargs=engine_kwargs) return storage def test_create_scoped_session() -> None: storage = create_test_storage() # This object violates the unique constraint of version_info_id. v = VersionInfoModel(version_info_id=1, schema_version=1, library_version="0.0.1") with pytest.raises(IntegrityError): with _create_scoped_session(storage.scoped_session) as session: session.add(v) def test_upgrade_identity() -> None: storage = create_test_storage() # `upgrade()` has no effect because the storage version is already up-to-date. old_version = storage.get_current_version() storage.upgrade() new_version = storage.get_current_version() assert old_version == new_version @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @pytest.mark.parametrize( "optuna_version", [ "0.9.0.a", "1.2.0.a", "1.3.0.a", "2.4.0.a", "2.6.0.a", "3.0.0.a", "3.0.0.b", "3.0.0.c", "3.0.0.d", "3.2.0.a", ], ) def test_upgrade_single_objective_optimization(optuna_version: str) -> None: src_db_file = os.path.join( os.path.dirname(__file__), "test_upgrade_assets", f"{optuna_version}.db" ) with tempfile.TemporaryDirectory() as workdir: shutil.copyfile(src_db_file, f"{workdir}/sqlite.db") storage_url = f"sqlite:///{workdir}/sqlite.db" storage = RDBStorage( storage_url, skip_compatibility_check=True, skip_table_creation=True, ) assert storage.get_current_version() == f"v{optuna_version}" head_version = storage.get_head_version() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) storage.upgrade() assert head_version == storage.get_current_version() # Create a new study. study = create_study(storage=storage) assert len(study.trials) == 0 study.optimize(objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Check empty study. study = load_study(storage=storage, study_name="single_empty") assert len(study.trials) == 0 study.optimize(objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Resume single objective optimization. study = load_study(storage=storage, study_name="single") assert len(study.trials) == 1 study.optimize(objective_test_upgrade, n_trials=1) assert len(study.trials) == 2 for trial in study.trials: assert trial.user_attrs["b"] == 1 assert trial.intermediate_values[0] == 0.5 assert -5 <= trial.params["x"] <= 5 assert 0 <= trial.params["y"] <= 10 assert trial.params["z"] in (-5, 0, 5) assert trial.value is not None and 0 <= trial.value <= 150 assert study.user_attrs["d"] == 3 storage.engine.dispose() # Be sure to disconnect db @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @pytest.mark.parametrize( "optuna_version", ["2.4.0.a", "2.6.0.a", "3.0.0.a", "3.0.0.b", "3.0.0.c", "3.0.0.d", "3.2.0.a"] ) def test_upgrade_multi_objective_optimization(optuna_version: str) -> None: src_db_file = os.path.join( os.path.dirname(__file__), "test_upgrade_assets", f"{optuna_version}.db" ) with tempfile.TemporaryDirectory() as workdir: shutil.copyfile(src_db_file, f"{workdir}/sqlite.db") storage_url = f"sqlite:///{workdir}/sqlite.db" storage = RDBStorage(storage_url, skip_compatibility_check=True, skip_table_creation=True) assert storage.get_current_version() == f"v{optuna_version}" head_version = storage.get_head_version() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) storage.upgrade() assert head_version == storage.get_current_version() # Create a new study. study = create_study(storage=storage, directions=["minimize", "minimize"]) assert len(study.trials) == 0 study.optimize(mo_objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Check empty study. study = load_study(storage=storage, study_name="multi_empty") assert len(study.trials) == 0 study.optimize(mo_objective_test_upgrade, n_trials=1) assert len(study.trials) == 1 # Resume multi-objective optimization. study = load_study(storage=storage, study_name="multi") assert len(study.trials) == 1 study.optimize(mo_objective_test_upgrade, n_trials=1) assert len(study.trials) == 2 for trial in study.trials: assert trial.user_attrs["b"] == 1 assert -5 <= trial.params["x"] <= 5 assert 0 <= trial.params["y"] <= 10 assert trial.params["z"] in (-5, 0, 5) assert -5 <= trial.values[0] < 5 assert 0 <= trial.values[1] <= 150 assert study.user_attrs["d"] == 3 storage.engine.dispose() # Be sure to disconnect db @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @pytest.mark.parametrize( "optuna_version", ["2.4.0.a", "2.6.0.a", "3.0.0.a", "3.0.0.b", "3.0.0.c", "3.0.0.d", "3.2.0.a"] ) def test_upgrade_distributions(optuna_version: str) -> None: src_db_file = os.path.join( os.path.dirname(__file__), "test_upgrade_assets", f"{optuna_version}.db" ) with tempfile.TemporaryDirectory() as workdir: shutil.copyfile(src_db_file, f"{workdir}/sqlite.db") storage_url = f"sqlite:///{workdir}/sqlite.db" storage = RDBStorage(storage_url, skip_compatibility_check=True, skip_table_creation=True) assert storage.get_current_version() == f"v{optuna_version}" head_version = storage.get_head_version() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=FutureWarning) storage.upgrade() assert head_version == storage.get_current_version() new_study = load_study(storage=storage, study_name="schema migration") new_distribution_dict = new_study.trials[0]._distributions assert isinstance(new_distribution_dict["x1"], FloatDistribution) assert isinstance(new_distribution_dict["x2"], FloatDistribution) assert isinstance(new_distribution_dict["x3"], FloatDistribution) assert isinstance(new_distribution_dict["y1"], IntDistribution) assert isinstance(new_distribution_dict["y2"], IntDistribution) assert isinstance(new_distribution_dict["z"], CategoricalDistribution) # Check if Study.optimize can run on new storage. with warnings.catch_warnings(): warnings.simplefilter("ignore", category=UserWarning) new_study.optimize(objective_test_upgrade_distributions, n_trials=1) storage.engine.dispose() # Be sure to disconnect db optuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/000077500000000000000000000000001453453102400253025ustar00rootroot00000000000000optuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/0.9.0.a.db000066400000000000000000002100001453453102400264650ustar00rootroot00000000000000SQLite format 3@  .0: ý»Pä f ² Á  © ^oÄþ6»Å‚%%ƒQtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER, step INTEGER, value FLOAT, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚5 %%„-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚R ;;„;tabletrial_system_attributestrial_system_attributes CREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )M a;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes ‚H 77„/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚„tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, study_id INTEGER, state VARCHAR(8) NOT NULL, value FLOAT, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚S;;„=tablestudy_system_attributesstudy_system_attributesCREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )Ma;indexsqlite_autoindex_study_system_attributes_1study_system_attributes‚I77„1tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name)-‚1tablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, direction VARCHAR(8) NOT NULL, PRIMARY KEY (study_id) ) ÒæÒsingleMINIMIZE%single_emptyMINIMIZE ååð single% single_empty ôô  0.9.0 ööd3 ùù d ööc2 ùù c ²²LAACOMPLETE@Rƒ¯ò± À2021-09-01 12:23:35.9490902021-09-01 12:23:36.010581 ÷÷ b1 úú  b èñè a0  _number0 íôí a   _number ©VS !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}Q y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10}}U x@ž’k¤B{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}} ìúóì z y  x ññ  ?à ûû  optuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/1.2.0.a.db000066400000000000000000002300001453453102400264610ustar00rootroot00000000000000SQLite format 3@  .0: ýÐPä f ² Á  © ^oÄþ6»ÅÐ|)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version‚%%ƒQtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER, step INTEGER, value FLOAT, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚5 %%„-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚R ;;„;tabletrial_system_attributestrial_system_attributes CREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )M a;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes ‚H 77„/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚„tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, study_id INTEGER, state VARCHAR(8) NOT NULL, value FLOAT, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚S;;„=tablestudy_system_attributesstudy_system_attributesCREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )Ma;indexsqlite_autoindex_study_system_attributes_1study_system_attributes‚I77„1tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name)-‚1tablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, direction VARCHAR(8) NOT NULL, PRIMARY KEY (study_id) ) ÒæÒsingleMINIMIZE%single_emptyMINIMIZE ååð single% single_empty ôô  1.2.0 ööd3 ùù d ööc2 ùù c ²²LAACOMPLETE@ ›¦ºçê2021-09-01 12:23:53.7428542021-09-01 12:23:53.835863 ÷÷ b1 úú  b èñè a0  _number0 íôí a   _number ©VR  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}Q y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10}}U x¿ð)OÁ {"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}} ìúóì z y  x ññ  ?à ûû  ôô v1.2.0.a ôô  v1.2.0.aoptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/1.3.0.a.db000066400000000000000000002300001453453102400264620ustar00rootroot00000000000000SQLite format 3@  .0: ý¾Pä f ² Á  — L]²ì$©³¾j)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version‚%%ƒQtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER, step INTEGER, value FLOAT, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚5 %%„-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚R ;;„;tabletrial_system_attributestrial_system_attributes CREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )M a;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes ‚H 77„/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚'„)tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, value FLOAT, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚S;;„=tablestudy_system_attributesstudy_system_attributesCREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )Ma;indexsqlite_autoindex_study_system_attributes_1study_system_attributes‚I77„1tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name)-‚1tablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, direction VARCHAR(8) NOT NULL, PRIMARY KEY (study_id) ) ÒæÒsingleMINIMIZE%single_emptyMINIMIZE ååð single% single_empty ôô  1.3.0 ööd3 ùù d ööc2 ùù c ±±MAACOMPLETE@C_Þ0TÏø2021-09-01 12:24:03.4611682021-09-01 12:24:03.548125 ÷÷ b1 úú  b ÷÷ a0 úú  a ÷©K÷R !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}U xÀúŠ ‚t{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}} ìúóì z y  x ññ  ?à ûû  ôô v1.3.0.a ôô  v1.3.0.aoptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/2.4.0.a.db000066400000000000000000003100001453453102400264630ustar00rootroot00000000000000SQLite format 3@ 9 9.WJûû Ý ù E T ª9£îÿTŽÆ&UÓÓQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚5%%„-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚R ;;„;tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚H 77„/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚S;;„=tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚I77„1tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty ôô  2.4.0 ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d ìöìc2c2 ñùñc c .ºt.DAACOMPLETE2022-02-16 15:46:59.3167642022-02-16 15:46:59.348160DAACOMPLETE2022-02-16 15:46:59.2792552022-02-16 15:46:59.295442DAACOMPLETE2022-02-16 15:46:59.2113912022-02-16 15:46:59.232018 ã÷íãb1b1 b1 êúòêbb  b Ê÷ÞÔÊa0a0-nsga2:generation0 a0 ÓúÛãÓaa-nsga2:generation  a U©K÷Ÿ@ ê ‘ . Í m ª US !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}^ 3y3{"name": "IntUniformDistribution", "attributes": {"low": 5, "high": 14, "step": 3}}a 9y2{"name": "IntLogUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}^ 3y1 {"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}_ 7x3{"name": "DiscreteUniformDistribution", "attributes": {"low": -6, "high": 6, "q": 2}}a+x2?(OFr0–ÿ{"name": "LogUniformDistribution", "attributes": {"low": 1e-05, "high": 0.001}}Wx1¿‘剶|{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}T!z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}]3y {"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}Vx@ ø;$ù {"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}}R !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y {"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}U x¿õ;\™íÌ{"name": "UniformDistribution", "attributes": {"low": -5, "high": 5}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ÁñáÑÁ@sà@S=r @\qt@ ø;$ù  @_°²½DUM æûôíæ   áñá?à  ?à ôûô  î £q& Ý ù E T ª9£îÿTŽÆ&U`Ó&s??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )îÐe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesî}%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_valuesî%%„-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_paramsî¤;;„;tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚H 77„/tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚S;;„=tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚I77„1tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) ) ôô v2.4.0.a ôô  v2.4.0.a j µ` Ù   q 8 Å r +)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_versionQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚5%%„-tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json VARCHAR(2048), PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚R ;;„;tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json VARCHAR(2048), PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes optuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/2.6.0.a.db000066400000000000000000003300001453453102400264670ustar00rootroot00000000000000SQLite format 3@ ::.WJûû ˜ å M n »SÆ+wÃò[Š•‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty ôô  2.6.0 ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d ìöìc2c2 ñùñc c .ºt.DAACOMPLETE2022-02-16 15:46:42.9495672022-02-16 15:46:42.979450DAACOMPLETE2022-02-16 15:46:42.9136282022-02-16 15:46:42.929011DAACOMPLETE2022-02-16 15:46:42.8481302022-02-16 15:46:42.869193 ã÷íãb1b1 b1 êúòêbb  b Ê÷ÞÔÊa0a0-nsga2:generation0 a0 ÓúÛãÓaa-nsga2:generation  a D¥Gó—9 ä ‡ $ ¼ \ ù ™ DS !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}^ 3y3 {"name": "IntUniformDistribution", "attributes": {"low": 5, "high": 14, "step": 3}}a 9y2 {"name": "IntLogUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}^ 3y1{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}f Cx3{"name": "DiscreteUniformDistribution", "attributes": {"low": -6.0, "high": 6.0, "q": 2.0}}a+x2?B¢l–½šë{"name": "LogUniformDistribution", "attributes": {"low": 1e-05, "high": 0.001}}[x1@~Hcnæè{"name": "UniformDistribution", "attributes": {"low": -5.0, "high": 5.0}}S!z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}Zx@\Vi[|¬{"name": "UniformDistribution", "attributes": {"low": -5.0, "high": 5.0}}R !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}\ 3y{"name": "IntUniformDistribution", "attributes": {"low": 0, "high": 10, "step": 1}}Y x@'ñ.]F{"name": "UniformDistribution", "attributes": {"low": -5.0, "high": 5.0}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ÁñáÑÁ@tõú÷Õ˜¸ @Fk&A¸Z*@\Vi[|¬  @TS)•©‹ æûôíæ   áñá?à  ?à ôûô    Æq& ˜ å M n »SÆ+wÃò[Š•LL??„otabletrial_intermzS-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats9‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params›;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   /µi ë ² ƒ J × „c"/Û)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats‚--ƒotabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ôô v2.6.0.a ôô  v2.6.0.aoptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.a.db000066400000000000000000003300001453453102400264620ustar00rootroot00000000000000SQLite format 3@ ==.WJûû ˜ å M n »SÆ+wÃò[Š•‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty ôô  3.0.0 ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d ìöìc2c2 ñùñc c .ºt.DAACOMPLETE2022-02-23 20:30:55.7491452022-02-23 20:30:55.787565DAACOMPLETE2022-02-23 20:30:55.6933652022-02-23 20:30:55.719393DAACOMPLETE2022-02-23 20:30:55.5916912022-02-23 20:30:55.620748 ã÷íãb1b1 b1 êúòêbb  b Ê÷ÞÔÊa0a0-nsga2:generation0 a0 ÓúÛãÓaa-nsga2:generation  a ¸‹&Ò\ ö ¡ * ± B Û u  ¸T !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3 {"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?Ùþî`v{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1@Þúí¾Î{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSx?™q'TûÎ{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}R  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s SxÀüió©µ{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ÁñáÑÁ@gÇñèê]4 @P .Éa?™q'TûÎ  @%ÃJY"„7 æûôíæ   áñá?à  ?à ôûô    Æq& ˜ å M n »SÆ+wÃò[Š•LL??„otabletrial_intermzS-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats9‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params›;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   /µi ë ² ƒ J × „c"/Û)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats‚--ƒotabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚p??„otabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ôô v3.0.0.a ôô  v3.0.0.aoptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.b.db000066400000000000000000003300001453453102400264630ustar00rootroot00000000000000SQLite format 3@ 77.[2ûû Ý  E f ³K¾#o»êS‚–‚g??„]tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty îî# 3.0.0b1.dev ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d ìöìc2c2 ñùñc c .ºt.DAACOMPLETE2022-04-27 16:44:16.7912732022-04-27 16:44:16.815170DAACOMPLETE2022-04-27 16:44:16.7529482022-04-27 16:44:16.770557DAACOMPLETE2022-04-27 16:44:16.6854652022-04-27 16:44:16.705778 í÷íb1 b1 òúòb  b Ô÷ÞÔa0-nsga2:generation0 a0 ÛúÛãa-nsga2:generation  a ¹‹&Ò\ ö ¡ * ± B Û u  ¹S  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?9èɆÑ\ö{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1¿Ôù®=ŽµÀ{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSxÀh¼‡__{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}R  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sx@W)‘GB{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ÁñáÑÁ@b#oíÏú© @?À’ÈFp}Àh¼‡__  @4°-÷h® æûôíæ   ññ  ?à ûû     ¾q& Ý  E f ³K¾#o»êS‚–UU??„]tabletrial_intermediate_viS-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats(??„]tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params›;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) ) # 8µi ë ² ƒ J à l+8ä)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeats‚--ƒotabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚g??„]tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ôô v3.0.0.b ôô  v3.0.0.boptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.c.db000066400000000000000000003300001453453102400264640ustar00rootroot00000000000000SQLite format 3@ 77.[5ûû Ý  E f ³K¾#o»êS‚Qe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty óó  3.0.0c ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d ìöìc2c2 ñùñc c .ºt.DAACOMPLETE2022-05-31 18:18:17.6635512022-05-31 18:18:17.683868DAACOMPLETE2022-05-31 18:18:17.6268432022-05-31 18:18:17.642115DAACOMPLETE2022-05-31 18:18:17.5582862022-05-31 18:18:17.578216 í÷íb1 b1 òúòb  b Ô÷ÞÔa0-nsga2:generation0 a0 ÛúÛãa-nsga2:generation  a ¸‹&Ñ[ õ   ) ° A Ú t ¸S  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1 {"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3ü{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?¸åâÐÉ${"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1?ûóÜ% S0{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S!z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy {"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSx?üÇ` °ÿø{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sx@¤vב.{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ÁñáÑÁ@ðÕ‹» @[Oohí?üÇ` °ÿø  @S”øƒm€á æûôíæ   êê ?àFINITE ûû   ¾q& Ý  E f ³K¾#o»êS‚gS™??…;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )¾e?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesk%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params›;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   V  µi ë ² ƒ J ± ^ô  µ)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version‚--ƒotabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeatsQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesƒ??…;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ôô v3.0.0.c ôô  v3.0.0.coptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.0.0.d.db000066400000000000000000003300001453453102400264650ustar00rootroot00000000000000SQLite format 3@ 77.[5ûû Ý  E f ³K¾#o»ê:‚ççQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values‚E%%„Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_ µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty îî# 3.0.0b1.dev ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d ìöìc2c2 ñùñc c .ºt.DAACOMPLETE2022-06-02 21:07:53.1555082022-06-02 21:07:53.176120DAACOMPLETE2022-06-02 21:07:53.1188402022-06-02 21:07:53.133590DAACOMPLETE2022-06-02 21:07:53.0516352022-06-02 21:07:53.069650 í÷íb1 b1 òúòb  b Ô÷ÞÔa0-nsga2:generation0 a0 ÛúÛãa-nsga2:generation  a ¸‹&Ñ[ õ   ) ° A Ú t ¸S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2??B9⪊{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1¿õ‰ÍG•Ú({"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}dAy{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSxÀ†püÆ.{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s Sx?þì¬SÛzl{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ¥êÓ¼¥@{üþ]î®±FINITE @Df*ù4`¦FINITEÀ†püÆ.FINITE @P/ÌuƒŠFINITE æûôíæ   êê ?àFINITE ûû   ¾q& Ý  E f ³K¾#o»ê:‚Nç:™??…;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )×e?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_values„%%„Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params›;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )K]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) )   = ðµi ë ² j 1 ˜ EÛüðœ)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version‚--ƒotabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeatsQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesƒ??…;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚E%%„Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚I ;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes ôô v3.0.0.d ôô  v3.0.0.doptuna-3.5.0/tests/storages_tests/rdb_tests/test_upgrade_assets/3.2.0.a.db000066400000000000000000003400001453453102400264650ustar00rootroot00000000000000SQLite format 3@ 11.WHûû µïäÔʵ-schema migrationmulti#multi_empty single%single_empty ¶ËÕ¶åð-schema migration multi#multi_empty single% single_empty òò  3.2.0.a ‘ñáÑÁ±¡‘MINIMIZE MINIMIZEMINIMIZE MINIMIZEMINIMIZEMINIMIZE  MINIMIZE ÑûôíæßØÑ    ìöìd3d3 ñùñd d   .ºt.DAACOMPLETE2023-03-13 13:23:06.5814252023-03-13 13:23:06.604608DAACOMPLETE2023-03-13 13:23:06.5440522023-03-13 13:23:06.559461DAACOMPLETE2023-03-13 13:23:06.4814672023-03-13 13:23:06.500233 ïûõï  í÷íb1 b1 òúòb  b Ô÷ÞÔa0-nsga2:generation0 a0 ÛúÛãa-nsga2:generation  a ¹‹&Ñ[ ö ¡ * ± B Û u  ¹S  !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}e Ay3{"name": "IntDistribution", "attributes": {"log": false, "step": 3, "low": 5, "high": 14}}d ?y2{"name": "IntDistribution", "attributes": {"log": true, "step": 1, "low": 1, "high": 20}}e Ay1 {"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}m Qx3ú{"name": "FloatDistribution", "attributes": {"step": 2.0, "low": -6.0, "high": 6.0, "log": false}}wWx2?JôG?˜{"name": "FloatDistribution", "attributes": {"step": null, "low": 1e-05, "high": 0.001, "log": true}}uSx1¿ò1´>™-D{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}tSx@*¾iøú{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}}S !z{"name": "CategoricalDistribution", "attributes": {"choices": [-5, 0, 5]}}c Ay{"name": "IntDistribution", "attributes": {"log": false, "step": 1, "low": 0, "high": 10}}s SxÀ®£ëh»{"name": "FloatDistribution", "attributes": {"step": null, "low": -5.0, "high": 5.0, "log": false}} –úóìäÜÔ˹°§ž–z y3 y2 y1 x3 x2x1zyx z y  x ¥êÓ¼¥@t¤°oõE·FINITE @5 žÆFINITE@*¾iøúFINITE @G†ÆðEŽÐFINITE æûôíæ   êê ?àFINITE ûû   ñq& Ý  E f ³Kñd¦Éaà(dX 1uindexix_trials_study_idtrials CREATE INDEX ix_trials_study_id ON trials (study_id)‚ „ tabletrialstrials CREATE TABLE trials ( trial_id INTEGER NOT NULL, number INTEGER, study_id INTEGER, state VARCHAR(8) NOT NULL, datetime_start DATETIME, datetime_complete DATETIME, PRIMARY KEY (trial_id), FOREIGN KEY(study_id) REFERENCES studies (study_id) )‚J;;„+tablestudy_system_attributesstudy_system_attributes CREATE TABLE study_system_attributes ( study_system_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_system_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )M a;indexsqlite_autoindex_study_system_attributes_1study_system_attributes ‚@77„tablestudy_user_attributesstudy_user_attributesCREATE TABLE study_user_attributes ( study_user_attribute_id INTEGER NOT NULL, study_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (study_user_attribute_id), UNIQUE (study_id, "key"), FOREIGN KEY(study_id) REFERENCES studies (study_id) )I]7indexsqlite_autoindex_study_user_attributes_1study_user_attributes‚J--„Gtablestudy_directionsstudy_directionsCREATE TABLE study_directions ( study_direction_id INTEGER NOT NULL, direction VARCHAR(8) NOT NULL, study_id INTEGER NOT NULL, objective INTEGER NOT NULL, PRIMARY KEY (study_direction_id), UNIQUE (study_id, objective), FOREIGN KEY(study_id) REFERENCES studies (study_id) )?S-indexsqlite_autoindex_study_directions_1study_directions\%%‚{tableversion_infoversion_infoCREATE TABLE version_info ( version_info_id INTEGER NOT NULL, schema_version INTEGER, library_version VARCHAR(256), PRIMARY KEY (version_info_id), CHECK (version_info_id=1) )j7indexix_studies_study_namestudiesCREATE UNIQUE INDEX ix_studies_study_name ON studies (study_name) otablestudiesstudiesCREATE TABLE studies ( study_id INTEGER NOT NULL, study_name VARCHAR(512) NOT NULL, PRIMARY KEY (study_id) ) û®¾s ' Ø © p ( ïV™º®Z)++‚ tablealembic_versionalembic_versionCREATE TABLE alembic_version ( version_num VARCHAR(32) NOT NULL, CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num) )=Q+indexsqlite_autoindex_alembic_version_1alembic_version‚--ƒotabletrial_heartbeatstrial_heartbeatsCREATE TABLE trial_heartbeats ( trial_heartbeat_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, heartbeat DATETIME NOT NULL, PRIMARY KEY (trial_heartbeat_id), UNIQUE (trial_id), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )?S-indexsqlite_autoindex_trial_heartbeats_1trial_heartbeatsQe?indexsqlite_autoindex_trial_intermediate_values_1trial_intermediate_valuesƒ??…;tabletrial_intermediate_valuestrial_intermediate_valuesCREATE TABLE trial_intermediate_values ( trial_intermediate_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, step INTEGER NOT NULL, intermediate_value FLOAT, intermediate_value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_intermediate_value_id), UNIQUE (trial_id, step), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_values_1trial_values‚E%%„Mtabletrial_valuestrial_valuesCREATE TABLE trial_values ( trial_value_id INTEGER NOT NULL, trial_id INTEGER NOT NULL, objective INTEGER NOT NULL, value FLOAT, value_type VARCHAR(7) NOT NULL, PRIMARY KEY (trial_value_id), UNIQUE (trial_id, objective), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )7K%indexsqlite_autoindex_trial_params_1trial_params‚,%%„tabletrial_paramstrial_paramsCREATE TABLE trial_params ( param_id INTEGER NOT NULL, trial_id INTEGER, param_name VARCHAR(512), param_value FLOAT, distribution_json TEXT, PRIMARY KEY (param_id), UNIQUE (trial_id, param_name), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )Ma;indexsqlite_autoindex_trial_system_attributes_1trial_system_attributes‚I;;„)tabletrial_system_attributestrial_system_attributesCREATE TABLE trial_system_attributes ( trial_system_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_system_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )I ]7indexsqlite_autoindex_trial_user_attributes_1trial_user_attributes‚? 77„tabletrial_user_attributestrial_user_attributes CREATE TABLE trial_user_attributes ( trial_user_attribute_id INTEGER NOT NULL, trial_id INTEGER, "key" VARCHAR(512), value_json TEXT, PRIMARY KEY (trial_user_attribute_id), UNIQUE (trial_id, "key"), FOREIGN KEY(trial_id) REFERENCES trials (trial_id) )   ôô v3.2.0.a ôô  v3.2.0.aoptuna-3.5.0/tests/storages_tests/test_cached_storage.py000066400000000000000000000124321453453102400236070ustar00rootroot00000000000000from unittest.mock import patch import pytest import optuna from optuna.storages._cached_storage import _CachedStorage from optuna.storages._rdb.storage import RDBStorage from optuna.study import StudyDirection from optuna.trial import TrialState def test_create_trial() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) frozen_trial = optuna.trial.FrozenTrial( number=1, state=TrialState.RUNNING, value=None, datetime_start=None, datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, ) with patch.object(base_storage, "_create_new_trial", return_value=frozen_trial): storage.create_new_trial(study_id) storage.create_new_trial(study_id) def test_set_trial_state_values() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) trial_id = storage.create_new_trial(study_id) storage.set_trial_state_values(trial_id, state=TrialState.COMPLETE) cached_trial = storage.get_trial(trial_id) base_trial = base_storage.get_trial(trial_id) assert cached_trial == base_trial def test_uncached_set() -> None: """Test CachedStorage does flush to persistent storages. The CachedStorage flushes any modification of trials to a persistent storage immediately. """ base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) trial_id = storage.create_new_trial(study_id) trial = storage.get_trial(trial_id) with patch.object(base_storage, "set_trial_state_values", return_value=True) as set_mock: storage.set_trial_state_values(trial_id, state=trial.state, values=(0.3,)) assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_param", return_value=True) as set_mock: storage.set_trial_param( trial_id, "paramA", 1.2, optuna.distributions.FloatDistribution(-0.2, 2.3) ) assert set_mock.call_count == 1 for state in [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL, TrialState.WAITING]: trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_state_values", return_value=True) as set_mock: storage.set_trial_state_values(trial_id, state=state) assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_intermediate_value", return_value=None) as set_mock: storage.set_trial_intermediate_value(trial_id, 3, 0.3) assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_system_attr", return_value=None) as set_mock: storage.set_trial_system_attr(trial_id, "attrA", "foo") assert set_mock.call_count == 1 trial_id = storage.create_new_trial(study_id) with patch.object(base_storage, "set_trial_user_attr", return_value=None) as set_mock: storage.set_trial_user_attr(trial_id, "attrB", "bar") assert set_mock.call_count == 1 def test_read_trials_from_remote_storage() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name="test-study" ) storage._read_trials_from_remote_storage(study_id) # Non-existent study. with pytest.raises(KeyError): storage._read_trials_from_remote_storage(study_id + 1) # Create a trial via CachedStorage and update it via backend storage directly. trial_id = storage.create_new_trial(study_id) base_storage.set_trial_param( trial_id, "paramA", 1.2, optuna.distributions.FloatDistribution(-0.2, 2.3) ) base_storage.set_trial_state_values(trial_id, TrialState.COMPLETE, values=[0.0]) storage._read_trials_from_remote_storage(study_id) assert storage.get_trial(trial_id).state == TrialState.COMPLETE def test_delete_study() -> None: base_storage = RDBStorage("sqlite:///:memory:") storage = _CachedStorage(base_storage) study_id1 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id1 = storage.create_new_trial(study_id1) storage.set_trial_state_values(trial_id1, state=TrialState.COMPLETE) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id2 = storage.create_new_trial(study_id2) storage.set_trial_state_values(trial_id2, state=TrialState.COMPLETE) # Update _StudyInfo.finished_trial_ids storage._read_trials_from_remote_storage(study_id1) storage._read_trials_from_remote_storage(study_id2) storage.delete_study(study_id1) assert storage._get_cached_trial(trial_id1) is None assert storage._get_cached_trial(trial_id2) is not None optuna-3.5.0/tests/storages_tests/test_heartbeat.py000066400000000000000000000323071453453102400226160ustar00rootroot00000000000000import itertools import multiprocessing import time from typing import Any from typing import Optional from unittest.mock import Mock from unittest.mock import patch import warnings import pytest import optuna from optuna import Study from optuna._callbacks import RetryFailedTrialCallback from optuna.storages import RDBStorage from optuna.storages._heartbeat import BaseHeartbeat from optuna.storages._heartbeat import is_heartbeat_enabled from optuna.testing.storages import STORAGE_MODES_HEARTBEAT from optuna.testing.storages import StorageSupplier from optuna.testing.threading import _TestableThread from optuna.trial import FrozenTrial from optuna.trial import TrialState @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_fail_stale_trials_with_optimize(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study1 = optuna.create_study(storage=storage) study2 = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial1 = study1.ask() trial2 = study2.ask() storage.record_heartbeat(trial1._trial_id) storage.record_heartbeat(trial2._trial_id) time.sleep(grace_period + 1) assert study1.trials[0].state is TrialState.RUNNING assert study2.trials[0].state is TrialState.RUNNING # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study1.optimize(lambda _: 1.0, n_trials=1) assert study1.trials[0].state is TrialState.FAIL # type: ignore [comparison-overlap] assert study2.trials[0].state is TrialState.RUNNING @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_invalid_heartbeat_interval_and_grace_period(storage_mode: str) -> None: with pytest.raises(ValueError): with StorageSupplier(storage_mode, heartbeat_interval=-1): pass with pytest.raises(ValueError): with StorageSupplier(storage_mode, grace_period=-1): pass @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_failed_trial_callback(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 def _failed_trial_callback(study: Study, trial: FrozenTrial) -> None: assert study._storage.get_study_system_attrs(study._study_id)["test"] == "A" assert trial.system_attrs["test"] == "B" failed_trial_callback = Mock(wraps=_failed_trial_callback) with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=failed_trial_callback, ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) study._storage.set_study_system_attr(study._study_id, "test", "A") with pytest.warns(UserWarning): trial = study.ask() trial.storage.set_trial_system_attr(trial._trial_id, "test", "B") storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study.optimize(lambda _: 1.0, n_trials=1) failed_trial_callback.assert_called_once() @pytest.mark.parametrize( "storage_mode,max_retry", itertools.product(STORAGE_MODES_HEARTBEAT, [None, 0, 1]) ) def test_retry_failed_trial_callback(storage_mode: str, max_retry: Optional[int]) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=RetryFailedTrialCallback(max_retry=max_retry), ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial = study.ask() trial.suggest_float("_", -1, -1) trial.report(0.5, 1) storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study.optimize(lambda _: 1.0, n_trials=1) # Test the last trial to see if it was a retry of the first trial or not. # Test max_retry=None to see if trial is retried. # Test max_retry=0 to see if no trials are retried. # Test max_retry=1 to see if trial is retried. assert RetryFailedTrialCallback.retried_trial_number(study.trials[1]) == ( None if max_retry == 0 else 0 ) # Test inheritance of trial fields. if max_retry != 0: assert study.trials[0].params == study.trials[1].params assert study.trials[0].distributions == study.trials[1].distributions assert study.trials[0].user_attrs == study.trials[1].user_attrs # Only `intermediate_values` are not inherited. assert study.trials[1].intermediate_values == {} @pytest.mark.parametrize( "storage_mode,max_retry", itertools.product(STORAGE_MODES_HEARTBEAT, [None, 0, 1]) ) def test_retry_failed_trial_callback_intermediate( storage_mode: str, max_retry: Optional[int] ) -> None: heartbeat_interval = 1 grace_period = 2 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=RetryFailedTrialCallback( max_retry=max_retry, inherit_intermediate_values=True ), ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) with warnings.catch_warnings(): warnings.simplefilter("ignore", category=UserWarning) trial = study.ask() trial.suggest_float("_", -1, -1) trial.report(0.5, 1) storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) # Exceptions raised in spawned threads are caught by `_TestableThread`. with patch("optuna.storages._heartbeat.Thread", _TestableThread): study.optimize(lambda _: 1.0, n_trials=1) # Test the last trial to see if it was a retry of the first trial or not. # Test max_retry=None to see if trial is retried. # Test max_retry=0 to see if no trials are retried. # Test max_retry=1 to see if trial is retried. assert RetryFailedTrialCallback.retried_trial_number(study.trials[1]) == ( None if max_retry == 0 else 0 ) # Test inheritance of trial fields. if max_retry != 0: assert study.trials[0].params == study.trials[1].params assert study.trials[0].distributions == study.trials[1].distributions assert study.trials[0].user_attrs == study.trials[1].user_attrs assert study.trials[0].intermediate_values == study.trials[1].intermediate_values @pytest.mark.parametrize("grace_period", [None, 2]) def test_fail_stale_trials(grace_period: Optional[int]) -> None: storage_mode = "sqlite" heartbeat_interval = 1 _grace_period = (heartbeat_interval * 2) if grace_period is None else grace_period def failed_trial_callback(study: "optuna.Study", trial: FrozenTrial) -> None: assert study._storage.get_study_system_attrs(study._study_id)["test"] == "A" assert trial.system_attrs["test"] == "B" def check_change_trial_state_to_fail(study: "optuna.Study") -> None: assert study.trials[0].state is TrialState.RUNNING optuna.storages.fail_stale_trials(study) assert study.trials[0].state is TrialState.FAIL # type: ignore [comparison-overlap] def check_keep_trial_state_in_running(study: "optuna.Study") -> None: assert study.trials[0].state is TrialState.RUNNING optuna.storages.fail_stale_trials(study) assert study.trials[0].state is TrialState.RUNNING with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) storage.heartbeat_interval = heartbeat_interval storage.grace_period = grace_period storage.failed_trial_callback = failed_trial_callback study = optuna.create_study(storage=storage) study._storage.set_study_system_attr(study._study_id, "test", "A") with pytest.warns(UserWarning): trial = study.ask() trial.storage.set_trial_system_attr(trial._trial_id, "test", "B") time.sleep(_grace_period + 1) check_keep_trial_state_in_running(study) storage.record_heartbeat(trial._trial_id) check_keep_trial_state_in_running(study) time.sleep(_grace_period + 1) check_change_trial_state_to_fail(study) def run_fail_stale_trials(storage_url: str, sleep_time: int) -> None: heartbeat_interval = 1 grace_period = 2 storage = RDBStorage(storage_url) storage.heartbeat_interval = heartbeat_interval storage.grace_period = grace_period original_set_trial_state_values = storage.set_trial_state_values def _set_trial_state_values(*args: Any, **kwargs: Any) -> bool: # The second process fails to set state due to the race condition. time.sleep(sleep_time) return original_set_trial_state_values(*args, **kwargs) storage.set_trial_state_values = _set_trial_state_values # type: ignore[method-assign] study = optuna.load_study(study_name=None, storage=storage) optuna.storages.fail_stale_trials(study) def test_fail_stale_trials_with_race_condition() -> None: grace_period = 2 storage_mode = "sqlite" with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) study = optuna.create_study(storage=storage) trial = study.ask() storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) p1 = multiprocessing.Process(target=run_fail_stale_trials, args=(storage.url, 1)) p1.start() p2 = multiprocessing.Process(target=run_fail_stale_trials, args=(storage.url, 2)) p2.start() p1.join() p2.join() assert p1.exitcode == 0 assert p2.exitcode == 0 assert study.trials[0].state is TrialState.FAIL def test_get_stale_trial_ids() -> None: storage_mode = "sqlite" heartbeat_interval = 1 grace_period = 2 with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) storage.heartbeat_interval = heartbeat_interval storage.grace_period = grace_period study = optuna.create_study(storage=storage) with pytest.warns(UserWarning): trial = study.ask() storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) assert len(storage._get_stale_trial_ids(study._study_id)) == 1 assert storage._get_stale_trial_ids(study._study_id)[0] == trial._trial_id @pytest.mark.parametrize("storage_mode", STORAGE_MODES_HEARTBEAT) def test_retry_failed_trial_callback_repetitive_failure(storage_mode: str) -> None: heartbeat_interval = 1 grace_period = 2 max_retry = 3 n_trials = 5 with StorageSupplier( storage_mode, heartbeat_interval=heartbeat_interval, grace_period=grace_period, failed_trial_callback=RetryFailedTrialCallback(max_retry=max_retry), ) as storage: assert is_heartbeat_enabled(storage) assert isinstance(storage, BaseHeartbeat) study = optuna.create_study(storage=storage) # Make repeatedly failed and retried trials by heartbeat. for _ in range(n_trials): with pytest.warns(UserWarning): trial = study.ask() storage.record_heartbeat(trial._trial_id) time.sleep(grace_period + 1) optuna.storages.fail_stale_trials(study) trials = study.trials assert len(trials) == n_trials + 1 assert "failed_trial" not in trials[0].system_attrs assert "retry_history" not in trials[0].system_attrs # The trials 1-3 are retried ones originating from the trial 0. assert trials[1].system_attrs["failed_trial"] == 0 assert trials[1].system_attrs["retry_history"] == [0] assert trials[2].system_attrs["failed_trial"] == 0 assert trials[2].system_attrs["retry_history"] == [0, 1] assert trials[3].system_attrs["failed_trial"] == 0 assert trials[3].system_attrs["retry_history"] == [0, 1, 2] # Trials 4 and later are the newly started ones and # they are retried after exceeding max_retry. assert "failed_trial" not in trials[4].system_attrs assert "retry_history" not in trials[4].system_attrs assert trials[5].system_attrs["failed_trial"] == 4 assert trials[5].system_attrs["retry_history"] == [4] optuna-3.5.0/tests/storages_tests/test_journal.py000066400000000000000000000200261453453102400223240ustar00rootroot00000000000000from concurrent.futures import as_completed from concurrent.futures import ProcessPoolExecutor from concurrent.futures import ThreadPoolExecutor import pickle from types import TracebackType from typing import Any from typing import IO from typing import Optional from typing import Type from unittest import mock import _pytest.capture from fakeredis import FakeStrictRedis import pytest import optuna from optuna import create_study from optuna.storages import JournalStorage from optuna.storages._journal.base import BaseJournalLogSnapshot from optuna.storages._journal.file import JournalFileBaseLock from optuna.storages._journal.storage import JournalStorageReplayResult from optuna.testing.storages import StorageSupplier from optuna.testing.tempfile_pool import NamedTemporaryFilePool LOG_STORAGE = [ "file_with_open_lock", "file_with_link_lock", "redis_default", "redis_with_use_cluster", ] JOURNAL_STORAGE_SUPPORTING_SNAPSHOT = ["journal_redis"] class JournalLogStorageSupplier: def __init__(self, storage_type: str) -> None: self.storage_type = storage_type self.tempfile: Optional[IO[Any]] = None def __enter__(self) -> optuna.storages.BaseJournalLogStorage: if self.storage_type.startswith("file"): self.tempfile = NamedTemporaryFilePool().tempfile() lock: JournalFileBaseLock if self.storage_type == "file_with_open_lock": lock = optuna.storages.JournalFileOpenLock(self.tempfile.name) elif self.storage_type == "file_with_link_lock": lock = optuna.storages.JournalFileSymlinkLock(self.tempfile.name) else: raise Exception("Must not reach here") return optuna.storages.JournalFileStorage(self.tempfile.name, lock) elif self.storage_type.startswith("redis"): use_cluster = self.storage_type == "redis_with_use_cluster" journal_redis_storage = optuna.storages.JournalRedisStorage( "redis://localhost", use_cluster ) journal_redis_storage._redis = FakeStrictRedis() # type: ignore[no-untyped-call] return journal_redis_storage else: raise RuntimeError("Unknown log storage type: {}".format(self.storage_type)) def __exit__( self, exc_type: Type[BaseException], exc_val: BaseException, exc_tb: TracebackType ) -> None: if self.tempfile: self.tempfile.close() @pytest.mark.parametrize("log_storage_type", LOG_STORAGE) def test_concurrent_append_logs_for_multi_processes(log_storage_type: str) -> None: if log_storage_type.startswith("redis"): pytest.skip("The `fakeredis` does not support multi process environments.") num_executors = 10 num_records = 200 record = {"key": "value"} with JournalLogStorageSupplier(log_storage_type) as storage: with ProcessPoolExecutor(num_executors) as pool: pool.map(storage.append_logs, [[record] for _ in range(num_records)], timeout=20) assert len(storage.read_logs(0)) == num_records assert all(record == r for r in storage.read_logs(0)) @pytest.mark.parametrize("log_storage_type", LOG_STORAGE) def test_concurrent_append_logs_for_multi_threads(log_storage_type: str) -> None: num_executors = 10 num_records = 200 record = {"key": "value"} with JournalLogStorageSupplier(log_storage_type) as storage: with ThreadPoolExecutor(num_executors) as pool: pool.map(storage.append_logs, [[record] for _ in range(num_records)], timeout=20) assert len(storage.read_logs(0)) == num_records assert all(record == r for r in storage.read_logs(0)) def pop_waiting_trial(file_path: str, study_name: str) -> Optional[int]: file_storage = optuna.storages.JournalFileStorage(file_path) storage = optuna.storages.JournalStorage(file_storage) study = optuna.load_study(storage=storage, study_name=study_name) return study._pop_waiting_trial_id() def test_pop_waiting_trial_multiprocess_safe() -> None: with NamedTemporaryFilePool() as file: file_storage = optuna.storages.JournalFileStorage(file.name) storage = optuna.storages.JournalStorage(file_storage) study = optuna.create_study(storage=storage) num_enqueued = 10 for i in range(num_enqueued): study.enqueue_trial({"i": i}) trial_id_set = set() with ProcessPoolExecutor(10) as pool: futures = [] for i in range(num_enqueued): future = pool.submit(pop_waiting_trial, file.name, study.study_name) futures.append(future) for future in as_completed(futures): trial_id = future.result() if trial_id is not None: trial_id_set.add(trial_id) assert len(trial_id_set) == num_enqueued @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_save_snapshot_per_each_trial(storage_mode: str) -> None: def objective(trial: optuna.Trial) -> float: return trial.suggest_float("x", 0, 10) with StorageSupplier(storage_mode) as storage: assert isinstance(storage, JournalStorage) study = create_study(storage=storage) journal_log_storage = storage._backend assert isinstance(journal_log_storage, BaseJournalLogSnapshot) assert journal_log_storage.load_snapshot() is None with mock.patch("optuna.storages._journal.storage.SNAPSHOT_INTERVAL", 1, create=True): study.optimize(objective, n_trials=2) assert isinstance(journal_log_storage.load_snapshot(), bytes) @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_save_snapshot_per_each_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: assert isinstance(storage, JournalStorage) journal_log_storage = storage._backend assert isinstance(journal_log_storage, BaseJournalLogSnapshot) assert journal_log_storage.load_snapshot() is None with mock.patch("optuna.storages._journal.storage.SNAPSHOT_INTERVAL", 1, create=True): for _ in range(2): create_study(storage=storage) assert isinstance(journal_log_storage.load_snapshot(), bytes) @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_check_replay_result_restored_from_snapshot(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage1: with mock.patch("optuna.storages._journal.storage.SNAPSHOT_INTERVAL", 1, create=True): for _ in range(2): create_study(storage=storage1) assert isinstance(storage1, JournalStorage) storage2 = optuna.storages.JournalStorage(storage1._backend) assert len(storage1.get_all_studies()) == len(storage2.get_all_studies()) assert storage1._replay_result.log_number_read == storage2._replay_result.log_number_read @pytest.mark.parametrize("storage_mode", JOURNAL_STORAGE_SUPPORTING_SNAPSHOT) def test_snapshot_given(storage_mode: str, capsys: _pytest.capture.CaptureFixture) -> None: with StorageSupplier(storage_mode) as storage: assert isinstance(storage, JournalStorage) replay_result = JournalStorageReplayResult("") # Bytes object which is a valid pickled object. storage.restore_replay_result(pickle.dumps(replay_result)) assert replay_result.log_number_read == storage._replay_result.log_number_read # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) # Bytes object which cannot be unpickled is passed. storage.restore_replay_result(b"hoge") _, err = capsys.readouterr() assert err # Bytes object which can be pickled but is not `JournalStorageReplayResult`. storage.restore_replay_result(pickle.dumps("hoge")) _, err = capsys.readouterr() assert err optuna-3.5.0/tests/storages_tests/test_storages.py000066400000000000000000001415351453453102400225120ustar00rootroot00000000000000from __future__ import annotations import copy from datetime import datetime import pickle import random from time import sleep from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple import numpy as np import pytest import optuna from optuna._typing import JSONSerializable from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.storages import _CachedStorage from optuna.storages import BaseStorage from optuna.storages import InMemoryStorage from optuna.storages import RDBStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.study._frozen import FrozenStudy from optuna.study._study_direction import StudyDirection from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import FrozenTrial from optuna.trial import TrialState ALL_STATES = list(TrialState) EXAMPLE_ATTRS: Dict[str, JSONSerializable] = { "dataset": "MNIST", "none": None, "json_serializable": {"baseline_score": 0.001, "tags": ["image", "classification"]}, } def test_get_storage() -> None: assert isinstance(optuna.storages.get_storage(None), InMemoryStorage) assert isinstance(optuna.storages.get_storage("sqlite:///:memory:"), _CachedStorage) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) frozen_studies = storage.get_all_studies() assert len(frozen_studies) == 1 assert frozen_studies[0]._study_id == study_id assert frozen_studies[0].study_name.startswith(DEFAULT_STUDY_NAME_PREFIX) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) # Study id must be unique. assert study_id != study_id2 frozen_studies = storage.get_all_studies() assert len(frozen_studies) == 2 assert {s._study_id for s in frozen_studies} == {study_id, study_id2} assert all(s.study_name.startswith(DEFAULT_STUDY_NAME_PREFIX) for s in frozen_studies) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_study_unique_id(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.delete_study(study_id2) study_id3 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) # Study id must not be reused after deletion. if not isinstance(storage, (RDBStorage, _CachedStorage)): # TODO(ytsmiling) Fix RDBStorage so that it does not reuse study_id. assert len({study_id, study_id2, study_id3}) == 3 frozen_studies = storage.get_all_studies() assert {s._study_id for s in frozen_studies} == {study_id, study_id3} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_study_with_name(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Generate unique study_name from the current function name and storage_mode. function_name = test_create_new_study_with_name.__name__ study_name = function_name + "/" + storage_mode study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name=study_name ) assert study_name == storage.get_study_name_from_id(study_id) with pytest.raises(optuna.exceptions.DuplicatedStudyError): storage.create_new_study(directions=[StudyDirection.MINIMIZE], study_name=study_name) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_delete_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.create_new_trial(study_id) trials = storage.get_all_trials(study_id) assert len(trials) == 1 with pytest.raises(KeyError): # Deletion of non-existent study. storage.delete_study(study_id + 1) storage.delete_study(study_id) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trials = storage.get_all_trials(study_id) assert len(trials) == 0 storage.delete_study(study_id) with pytest.raises(KeyError): # Double free. storage.delete_study(study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_delete_study_after_create_multiple_studies(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id1 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) study_id3 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.delete_study(study_id2) studies = {s._study_id: s for s in storage.get_all_studies()} assert study_id1 in studies assert study_id2 not in studies assert study_id3 in studies @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_study_id_from_name_and_get_study_name_from_id(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Generate unique study_name from the current function name and storage_mode. function_name = test_get_study_id_from_name_and_get_study_name_from_id.__name__ study_name = function_name + "/" + storage_mode study_id = storage.create_new_study( directions=[StudyDirection.MINIMIZE], study_name=study_name ) # Test existing study. assert storage.get_study_name_from_id(study_id) == study_name assert storage.get_study_id_from_name(study_name) == study_id # Test not existing study. with pytest.raises(KeyError): storage.get_study_id_from_name("dummy-name") with pytest.raises(KeyError): storage.get_study_name_from_id(study_id + 1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_and_get_study_directions(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: for target in [ (StudyDirection.MINIMIZE,), (StudyDirection.MAXIMIZE,), (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE), (StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE), [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE], [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE], ]: study_id = storage.create_new_study(directions=target) def check_get() -> None: got_directions = storage.get_study_directions(study_id) assert got_directions == list( target ), "Direction of a study should be a tuple of `StudyDirection` objects." # Test setting value. check_get() # Test non-existent study. non_existent_study_id = study_id + 1 with pytest.raises(KeyError): storage.get_study_directions(non_existent_study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_and_get_study_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) def check_set_and_get(key: str, value: Any) -> None: storage.set_study_user_attr(study_id, key, value) assert storage.get_study_user_attrs(study_id)[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(key, value) assert storage.get_study_user_attrs(study_id) == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get("dataset", "ImageNet") # Non-existent study id. non_existent_study_id = study_id + 1 with pytest.raises(KeyError): storage.get_study_user_attrs(non_existent_study_id) # Non-existent study id. with pytest.raises(KeyError): storage.set_study_user_attr(non_existent_study_id, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_and_get_study_system_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) def check_set_and_get(key: str, value: Any) -> None: storage.set_study_system_attr(study_id, key, value) assert storage.get_study_system_attrs(study_id)[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(key, value) assert storage.get_study_system_attrs(study_id) == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get("dataset", "ImageNet") # Non-existent study id. non_existent_study_id = study_id + 1 with pytest.raises(KeyError): storage.get_study_system_attrs(non_existent_study_id) # Non-existent study id. with pytest.raises(KeyError): storage.set_study_system_attr(non_existent_study_id, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_study_user_and_system_attrs_confusion(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for key, value in EXAMPLE_ATTRS.items(): storage.set_study_system_attr(study_id, key, value) assert storage.get_study_system_attrs(study_id) == EXAMPLE_ATTRS assert storage.get_study_user_attrs(study_id) == {} study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for key, value in EXAMPLE_ATTRS.items(): storage.set_study_user_attr(study_id, key, value) assert storage.get_study_user_attrs(study_id) == EXAMPLE_ATTRS assert storage.get_study_system_attrs(study_id) == {} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_new_trial(storage_mode: str) -> None: def _check_trials( trials: List[FrozenTrial], idx: int, trial_id: int, time_before_creation: datetime, time_after_creation: datetime, ) -> None: assert len(trials) == idx + 1 assert len({t._trial_id for t in trials}) == idx + 1 assert trial_id in {t._trial_id for t in trials} assert {t.number for t in trials} == set(range(idx + 1)) assert all(t.state == TrialState.RUNNING for t in trials) assert all(t.params == {} for t in trials) assert all(t.intermediate_values == {} for t in trials) assert all(t.user_attrs == {} for t in trials) assert all(t.system_attrs == {} for t in trials) assert all( t.datetime_start < time_before_creation for t in trials if t._trial_id != trial_id and t.datetime_start is not None ) assert all( time_before_creation < t.datetime_start < time_after_creation for t in trials if t._trial_id == trial_id and t.datetime_start is not None ) assert all(t.datetime_complete is None for t in trials) assert all(t.value is None for t in trials) with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) n_trial_in_study = 3 for i in range(n_trial_in_study): time_before_creation = datetime.now() sleep(0.001) # Sleep 1ms to avoid faulty assertion on Windows OS. trial_id = storage.create_new_trial(study_id) sleep(0.001) time_after_creation = datetime.now() trials = storage.get_all_trials(study_id) _check_trials(trials, i, trial_id, time_before_creation, time_after_creation) # Create trial in non-existent study. with pytest.raises(KeyError): storage.create_new_trial(study_id + 1) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for i in range(n_trial_in_study): storage.create_new_trial(study_id2) trials = storage.get_all_trials(study_id2) # Check that the offset of trial.number is zero. assert {t.number for t in trials} == set(range(i + 1)) trials = storage.get_all_trials(study_id) + storage.get_all_trials(study_id2) # Check trial_ids are unique across studies. assert len({t._trial_id for t in trials}) == 2 * n_trial_in_study @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "start_time,complete_time", [(datetime.now(), datetime.now()), (datetime(2022, 9, 1), datetime(2022, 9, 2))], ) def test_create_new_trial_with_template_trial( storage_mode: str, start_time: datetime, complete_time: datetime ) -> None: template_trial = FrozenTrial( state=TrialState.COMPLETE, value=10000, datetime_start=start_time, datetime_complete=complete_time, params={"x": 0.5}, distributions={"x": FloatDistribution(0, 1)}, user_attrs={"foo": "bar"}, system_attrs={"baz": 123}, intermediate_values={1: 10, 2: 100, 3: 1000}, number=55, # This entry is ignored. trial_id=-1, # dummy value (unused). ) def _check_trials(trials: List[FrozenTrial], idx: int, trial_id: int) -> None: assert len(trials) == idx + 1 assert len({t._trial_id for t in trials}) == idx + 1 assert trial_id in {t._trial_id for t in trials} assert {t.number for t in trials} == set(range(idx + 1)) assert all(t.state == template_trial.state for t in trials) assert all(t.params == template_trial.params for t in trials) assert all(t.distributions == template_trial.distributions for t in trials) assert all(t.intermediate_values == template_trial.intermediate_values for t in trials) assert all(t.user_attrs == template_trial.user_attrs for t in trials) assert all(t.system_attrs == template_trial.system_attrs for t in trials) assert all(t.datetime_start == template_trial.datetime_start for t in trials) assert all(t.datetime_complete == template_trial.datetime_complete for t in trials) assert all(t.value == template_trial.value for t in trials) with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) n_trial_in_study = 3 for i in range(n_trial_in_study): trial_id = storage.create_new_trial(study_id, template_trial=template_trial) trials = storage.get_all_trials(study_id) _check_trials(trials, i, trial_id) # Create trial in non-existent study. with pytest.raises(KeyError): storage.create_new_trial(study_id + 1) study_id2 = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) for i in range(n_trial_in_study): storage.create_new_trial(study_id2, template_trial=template_trial) trials = storage.get_all_trials(study_id2) assert {t.number for t in trials} == set(range(i + 1)) trials = storage.get_all_trials(study_id) + storage.get_all_trials(study_id2) # Check trial_ids are unique across studies. assert len({t._trial_id for t in trials}) == 2 * n_trial_in_study @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_number_from_id(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Check if trial_number starts from 0. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) assert storage.get_trial_number_from_id(trial_id) == 0 trial_id = storage.create_new_trial(study_id) assert storage.get_trial_number_from_id(trial_id) == 1 with pytest.raises(KeyError): storage.get_trial_number_from_id(trial_id + 1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_state_values_for_state(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_ids = [storage.create_new_trial(study_id) for _ in ALL_STATES] for trial_id, state in zip(trial_ids, ALL_STATES): if state == TrialState.WAITING: continue assert storage.get_trial(trial_id).state == TrialState.RUNNING datetime_start_prev = storage.get_trial(trial_id).datetime_start storage.set_trial_state_values( trial_id, state=state, values=(0.0,) if state.is_finished() else None ) assert storage.get_trial(trial_id).state == state # Repeated state changes to RUNNING should not trigger further datetime_start changes. if state == TrialState.RUNNING: assert storage.get_trial(trial_id).datetime_start == datetime_start_prev if state.is_finished(): assert storage.get_trial(trial_id).datetime_complete is not None else: assert storage.get_trial(trial_id).datetime_complete is None # Non-existent study. with pytest.raises(KeyError): non_existent_trial_id = max(trial_ids) + 1 storage.set_trial_state_values( non_existent_trial_id, state=TrialState.COMPLETE, ) for state in ALL_STATES: if not state.is_finished(): continue trial_id = storage.create_new_trial(study_id) storage.set_trial_state_values(trial_id, state=state, values=(0.0,)) for state2 in ALL_STATES: # Cannot update states of finished trials. with pytest.raises(RuntimeError): storage.set_trial_state_values(trial_id, state=state2) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_param_and_get_trial_params(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=1) for _, trial_id_to_trial in study_to_trials.items(): for trial_id, expected_trial in trial_id_to_trial.items(): assert storage.get_trial_params(trial_id) == expected_trial.params for key in expected_trial.params.keys(): assert storage.get_trial_param(trial_id, key) == expected_trial.distributions[ key ].to_internal_repr(expected_trial.params[key]) non_existent_trial_id = ( max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 ) with pytest.raises(KeyError): storage.get_trial_params(non_existent_trial_id) with pytest.raises(KeyError): storage.get_trial_param(non_existent_trial_id, "paramA") existent_trial_id = non_existent_trial_id - 1 with pytest.raises(KeyError): storage.get_trial_param(existent_trial_id, "dummy-key") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_param(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Setup test across multiple studies and trials. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) trial_id_2 = storage.create_new_trial(study_id) trial_id_3 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) # Setup distributions. distribution_x = FloatDistribution(low=1.0, high=2.0) distribution_y_1 = CategoricalDistribution(choices=("Shibuya", "Ebisu", "Meguro")) distribution_y_2 = CategoricalDistribution(choices=("Shibuya", "Shinsen")) distribution_z = FloatDistribution(low=1.0, high=100.0, log=True) # Set new params. storage.set_trial_param(trial_id_1, "x", 0.5, distribution_x) storage.set_trial_param(trial_id_1, "y", 2, distribution_y_1) assert storage.get_trial_param(trial_id_1, "x") == 0.5 assert storage.get_trial_param(trial_id_1, "y") == 2 # Check set_param breaks neither get_trial nor get_trial_params. assert storage.get_trial(trial_id_1).params == {"x": 0.5, "y": "Meguro"} assert storage.get_trial_params(trial_id_1) == {"x": 0.5, "y": "Meguro"} # Duplicated registration should overwrite. storage.set_trial_param(trial_id_1, "x", 0.6, distribution_x) assert storage.get_trial_param(trial_id_1, "x") == 0.6 assert storage.get_trial(trial_id_1).params == {"x": 0.6, "y": "Meguro"} assert storage.get_trial_params(trial_id_1) == {"x": 0.6, "y": "Meguro"} # Set params to another trial. storage.set_trial_param(trial_id_2, "x", 0.3, distribution_x) storage.set_trial_param(trial_id_2, "z", 0.1, distribution_z) assert storage.get_trial_param(trial_id_2, "x") == 0.3 assert storage.get_trial_param(trial_id_2, "z") == 0.1 assert storage.get_trial(trial_id_2).params == {"x": 0.3, "z": 0.1} assert storage.get_trial_params(trial_id_2) == {"x": 0.3, "z": 0.1} # Set params with distributions that do not match previous ones. with pytest.raises(ValueError): storage.set_trial_param(trial_id_2, "y", 0.5, distribution_z) # Choices in CategoricalDistribution should match including its order. with pytest.raises(ValueError): storage.set_trial_param( trial_id_2, "y", 2, CategoricalDistribution(choices=("Meguro", "Shibuya", "Ebisu")) ) storage.set_trial_state_values(trial_id_2, state=TrialState.COMPLETE) # Cannot assign params to finished trial. with pytest.raises(RuntimeError): storage.set_trial_param(trial_id_2, "y", 2, distribution_y_1) # Check the previous call does not change the params. with pytest.raises(KeyError): storage.get_trial_param(trial_id_2, "y") # State should be checked prior to distribution compatibility. with pytest.raises(RuntimeError): storage.set_trial_param(trial_id_2, "y", 0.4, distribution_z) # Set params of trials in a different study. storage.set_trial_param(trial_id_3, "y", 1, distribution_y_2) assert storage.get_trial_param(trial_id_3, "y") == 1 assert storage.get_trial(trial_id_3).params == {"y": "Shinsen"} assert storage.get_trial_params(trial_id_3) == {"y": "Shinsen"} # Set params of non-existent trial. non_existent_trial_id = max([trial_id_1, trial_id_2, trial_id_3]) + 1 with pytest.raises(KeyError): storage.set_trial_param(non_existent_trial_id, "x", 0.1, distribution_x) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_state_values_for_values(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Setup test across multiple studies and trials. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) trial_id_2 = storage.create_new_trial(study_id) trial_id_3 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) trial_id_4 = storage.create_new_trial(study_id) trial_id_5 = storage.create_new_trial(study_id) # Test setting new value. storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE, values=(0.5,)) storage.set_trial_state_values( trial_id_3, state=TrialState.COMPLETE, values=(float("inf"),) ) storage.set_trial_state_values( trial_id_4, state=TrialState.WAITING, values=(0.1, 0.2, 0.3) ) storage.set_trial_state_values( trial_id_5, state=TrialState.WAITING, values=[0.1, 0.2, 0.3] ) assert storage.get_trial(trial_id_1).value == 0.5 assert storage.get_trial(trial_id_2).value is None assert storage.get_trial(trial_id_3).value == float("inf") assert storage.get_trial(trial_id_4).values == [0.1, 0.2, 0.3] assert storage.get_trial(trial_id_5).values == [0.1, 0.2, 0.3] non_existent_trial_id = max(trial_id_1, trial_id_2, trial_id_3, trial_id_4, trial_id_5) + 1 with pytest.raises(KeyError): storage.set_trial_state_values( non_existent_trial_id, state=TrialState.COMPLETE, values=(1,) ) # Cannot change values of finished trials. with pytest.raises(RuntimeError): storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE, values=(1,)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_intermediate_value(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Setup test across multiple studies and trials. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) trial_id_2 = storage.create_new_trial(study_id) trial_id_3 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) trial_id_4 = storage.create_new_trial(study_id) # Test setting new values. storage.set_trial_intermediate_value(trial_id_1, 0, 0.3) storage.set_trial_intermediate_value(trial_id_1, 2, 0.4) storage.set_trial_intermediate_value(trial_id_3, 0, 0.1) storage.set_trial_intermediate_value(trial_id_3, 1, 0.4) storage.set_trial_intermediate_value(trial_id_3, 2, 0.5) storage.set_trial_intermediate_value(trial_id_3, 3, float("inf")) storage.set_trial_intermediate_value(trial_id_4, 0, float("nan")) assert storage.get_trial(trial_id_1).intermediate_values == {0: 0.3, 2: 0.4} assert storage.get_trial(trial_id_2).intermediate_values == {} assert storage.get_trial(trial_id_3).intermediate_values == { 0: 0.1, 1: 0.4, 2: 0.5, 3: float("inf"), } assert np.isnan(storage.get_trial(trial_id_4).intermediate_values[0]) # Test setting existing step. storage.set_trial_intermediate_value(trial_id_1, 0, 0.2) assert storage.get_trial(trial_id_1).intermediate_values == {0: 0.2, 2: 0.4} non_existent_trial_id = max(trial_id_1, trial_id_2, trial_id_3, trial_id_4) + 1 with pytest.raises(KeyError): storage.set_trial_intermediate_value(non_existent_trial_id, 0, 0.2) storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE) # Cannot change values of finished trials. with pytest.raises(RuntimeError): storage.set_trial_intermediate_value(trial_id_1, 0, 0.2) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=10) assert all( storage.get_trial_user_attrs(trial_id) == trial.user_attrs for trials in study_to_trials.values() for trial_id, trial in trials.items() ) non_existent_trial = max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 with pytest.raises(KeyError): storage.get_trial_user_attrs(non_existent_trial) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_user_attr(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: trial_id_1 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) def check_set_and_get(trial_id: int, key: str, value: Any) -> None: storage.set_trial_user_attr(trial_id, key, value) assert storage.get_trial(trial_id).user_attrs[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(trial_id_1, key, value) assert storage.get_trial(trial_id_1).user_attrs == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get(trial_id_1, "dataset", "ImageNet") # Test another trial. trial_id_2 = storage.create_new_trial( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) check_set_and_get(trial_id_2, "baseline_score", 0.001) assert len(storage.get_trial(trial_id_2).user_attrs) == 1 assert storage.get_trial(trial_id_2).user_attrs["baseline_score"] == 0.001 # Cannot set attributes of non-existent trials. non_existent_trial_id = max({trial_id_1, trial_id_2}) + 1 with pytest.raises(KeyError): storage.set_trial_user_attr(non_existent_trial_id, "key", "value") # Cannot set attributes of finished trials. storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE) with pytest.raises(RuntimeError): storage.set_trial_user_attr(trial_id_1, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_system_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=10) assert all( storage.get_trial_system_attrs(trial_id) == trial.system_attrs for trials in study_to_trials.values() for trial_id, trial in trials.items() ) non_existent_trial = max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 with pytest.raises(KeyError): storage.get_trial_system_attrs(non_existent_trial) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_set_trial_system_attr(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id_1 = storage.create_new_trial(study_id) def check_set_and_get(trial_id: int, key: str, value: Any) -> None: storage.set_trial_system_attr(trial_id, key, value) assert storage.get_trial_system_attrs(trial_id)[key] == value # Test setting value. for key, value in EXAMPLE_ATTRS.items(): check_set_and_get(trial_id_1, key, value) system_attrs = storage.get_trial(trial_id_1).system_attrs assert system_attrs == EXAMPLE_ATTRS # Test overwriting value. check_set_and_get(trial_id_1, "dataset", "ImageNet") # Test another trial. trial_id_2 = storage.create_new_trial(study_id) check_set_and_get(trial_id_2, "baseline_score", 0.001) system_attrs = storage.get_trial(trial_id_2).system_attrs assert system_attrs == {"baseline_score": 0.001} # Cannot set attributes of non-existent trials. non_existent_trial_id = max({trial_id_1, trial_id_2}) + 1 with pytest.raises(KeyError): storage.set_trial_system_attr(non_existent_trial_id, "key", "value") # Cannot set attributes of finished trials. storage.set_trial_state_values(trial_id_1, state=TrialState.COMPLETE) with pytest.raises(RuntimeError): storage.set_trial_system_attr(trial_id_1, "key", "value") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_studies(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: expected_frozen_studies, _ = _setup_studies(storage, n_study=10, n_trial=10, seed=46) frozen_studies = storage.get_all_studies() assert len(frozen_studies) == len(expected_frozen_studies) for _, expected_frozen_study in expected_frozen_studies.items(): frozen_study: Optional[FrozenStudy] = None for s in frozen_studies: if s.study_name == expected_frozen_study.study_name: frozen_study = s break assert frozen_study is not None assert frozen_study.direction == expected_frozen_study.direction assert frozen_study.study_name == expected_frozen_study.study_name assert frozen_study.user_attrs == expected_frozen_study.user_attrs assert frozen_study.system_attrs == expected_frozen_study.system_attrs @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=20, seed=47) for _, expected_trials in study_to_trials.items(): for expected_trial in expected_trials.values(): trial = storage.get_trial(expected_trial._trial_id) assert trial == expected_trial non_existent_trial_id = ( max(tid for ts in study_to_trials.values() for tid in ts.keys()) + 1 ) with pytest.raises(KeyError): storage.get_trial(non_existent_trial_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=20, seed=48) for study_id, expected_trials in study_to_trials.items(): trials = storage.get_all_trials(study_id) for trial in trials: expected_trial = expected_trials[trial._trial_id] assert trial == expected_trial non_existent_study_id = max(study_to_trials.keys()) + 1 with pytest.raises(KeyError): storage.get_all_trials(non_existent_study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("param_names", [["a", "b"], ["b", "a"]]) def test_get_all_trials_params_order(storage_mode: str, param_names: list[str]) -> None: # We don't actually require that all storages to preserve the order of parameters, # but all current implementations do, so we test this property. with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial( study_id, optuna.trial.create_trial(state=TrialState.RUNNING) ) for param_name in param_names: storage.set_trial_param( trial_id, param_name, 1.0, distribution=FloatDistribution(0.0, 2.0) ) trials = storage.get_all_trials(study_id) assert list(trials[0].params.keys()) == param_names @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials_deepcopy_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: frozen_studies, study_to_trials = _setup_studies(storage, n_study=2, n_trial=5, seed=49) for study_id in frozen_studies: trials0 = storage.get_all_trials(study_id, deepcopy=True) assert len(trials0) == len(study_to_trials[study_id]) # Check modifying output does not break the internal state of the storage. trials0_original = copy.deepcopy(trials0) trials0[0].params["x"] = 0.1 trials1 = storage.get_all_trials(study_id, deepcopy=False) assert trials0_original == trials1 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials_state_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MAXIMIZE]) generator = random.Random(51) states = ( TrialState.COMPLETE, TrialState.COMPLETE, TrialState.PRUNED, ) for state in states: t = _generate_trial(generator) t.state = state storage.create_new_trial(study_id, template_trial=t) trials = storage.get_all_trials(study_id, states=None) assert len(trials) == 3 trials = storage.get_all_trials(study_id, states=(TrialState.COMPLETE,)) assert len(trials) == 2 assert all(t.state == TrialState.COMPLETE for t in trials) trials = storage.get_all_trials(study_id, states=(TrialState.COMPLETE, TrialState.PRUNED)) assert len(trials) == 3 assert all(t.state in (TrialState.COMPLETE, TrialState.PRUNED) for t in trials) trials = storage.get_all_trials(study_id, states=()) assert len(trials) == 0 other_states = [ s for s in ALL_STATES if s != TrialState.COMPLETE and s != TrialState.PRUNED ] for state in other_states: trials = storage.get_all_trials(study_id, states=(state,)) assert len(trials) == 0 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_trials_not_modified(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: _, study_to_trials = _setup_studies(storage, n_study=2, n_trial=20, seed=48) for study_id in study_to_trials.keys(): trials = storage.get_all_trials(study_id, deepcopy=False) deepcopied_trials = copy.deepcopy(trials) for trial in trials: if not trial.state.is_finished(): storage.set_trial_param(trial._trial_id, "paramX", 0, FloatDistribution(0, 1)) storage.set_trial_user_attr(trial._trial_id, "usr_attrX", 0) storage.set_trial_system_attr(trial._trial_id, "sys_attrX", 0) if trial.state == TrialState.RUNNING: if trial.number % 3 == 0: storage.set_trial_state_values(trial._trial_id, TrialState.COMPLETE, [0]) elif trial.number % 3 == 1: storage.set_trial_intermediate_value(trial._trial_id, 0, 0) storage.set_trial_state_values(trial._trial_id, TrialState.PRUNED, [0]) else: storage.set_trial_state_values(trial._trial_id, TrialState.FAIL) elif trial.state == TrialState.WAITING: storage.set_trial_state_values(trial._trial_id, TrialState.RUNNING) assert trials == deepcopied_trials @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_n_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id_to_frozen_studies, _ = _setup_studies(storage, n_study=2, n_trial=7, seed=50) for study_id in study_id_to_frozen_studies: assert storage.get_n_trials(study_id) == 7 non_existent_study_id = max(study_id_to_frozen_studies.keys()) + 1 with pytest.raises(KeyError): assert storage.get_n_trials(non_existent_study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_n_trials_state_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=(StudyDirection.MAXIMIZE,)) generator = random.Random(51) states = [ TrialState.COMPLETE, TrialState.COMPLETE, TrialState.PRUNED, ] for s in states: t = _generate_trial(generator) t.state = s storage.create_new_trial(study_id, template_trial=t) assert storage.get_n_trials(study_id, TrialState.COMPLETE) == 2 assert storage.get_n_trials(study_id, TrialState.PRUNED) == 1 other_states = [ s for s in ALL_STATES if s != TrialState.COMPLETE and s != TrialState.PRUNED ] for s in other_states: assert storage.get_n_trials(study_id, s) == 0 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("direction", [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE]) @pytest.mark.parametrize( "values", [ [0.0, 1.0, 2.0], [0.0, float("inf"), 1.0], [0.0, float("-inf"), 1.0], [float("inf"), 0.0, 1.0, float("-inf")], [float("inf")], [float("-inf")], ], ) def test_get_best_trial(storage_mode: str, direction: StudyDirection, values: List[float]) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[direction]) with pytest.raises(ValueError): storage.get_best_trial(study_id) with pytest.raises(KeyError): storage.get_best_trial(study_id + 1) generator = random.Random(51) for v in values: template_trial = _generate_trial(generator) template_trial.state = TrialState.COMPLETE template_trial.value = v storage.create_new_trial(study_id, template_trial=template_trial) expected_value = max(values) if direction == StudyDirection.MAXIMIZE else min(values) assert storage.get_best_trial(study_id).value == expected_value def test_get_trials_excluded_trial_ids() -> None: storage_mode = "sqlite" with StorageSupplier(storage_mode) as storage: assert isinstance(storage, RDBStorage) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.create_new_trial(study_id) trials = storage._get_trials(study_id, states=None, excluded_trial_ids=set()) assert len(trials) == 1 # A large exclusion list used to raise errors. Check that it is not an issue. # See https://github.com/optuna/optuna/issues/1457. trials = storage._get_trials(study_id, states=None, excluded_trial_ids=set(range(500000))) assert len(trials) == 0 def _setup_studies( storage: BaseStorage, n_study: int, n_trial: int, seed: int, direction: Optional[StudyDirection] = None, ) -> Tuple[Dict[int, FrozenStudy], Dict[int, Dict[int, FrozenTrial]]]: generator = random.Random(seed) study_id_to_frozen_study: Dict[int, FrozenStudy] = {} study_id_to_trials: Dict[int, Dict[int, FrozenTrial]] = {} for i in range(n_study): study_name = "test-study-name-{}".format(i) if direction is None: direction = generator.choice([StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) study_id = storage.create_new_study(directions=(direction,), study_name=study_name) storage.set_study_user_attr(study_id, "u", i) storage.set_study_system_attr(study_id, "s", i) trials = {} for j in range(n_trial): trial = _generate_trial(generator) trial.number = j trial._trial_id = storage.create_new_trial(study_id, trial) trials[trial._trial_id] = trial study_id_to_trials[study_id] = trials study_id_to_frozen_study[study_id] = FrozenStudy( study_name=study_name, direction=direction, user_attrs={"u": i}, system_attrs={"s": i}, study_id=study_id, ) return study_id_to_frozen_study, study_id_to_trials def _generate_trial(generator: random.Random) -> FrozenTrial: example_params = { "paramA": (generator.uniform(0, 1), FloatDistribution(0, 1)), "paramB": (generator.uniform(1, 2), FloatDistribution(1, 2, log=True)), "paramC": ( generator.choice(["CatA", "CatB", "CatC"]), CategoricalDistribution(("CatA", "CatB", "CatC")), ), "paramD": (generator.uniform(-3, 0), FloatDistribution(-3, 0)), "paramE": (generator.choice([0.1, 0.2]), CategoricalDistribution((0.1, 0.2))), } example_attrs = { "attrA": "valueA", "attrB": 1, "attrC": None, "attrD": {"baseline_score": 0.001, "tags": ["image", "classification"]}, } state = generator.choice(ALL_STATES) params = {} distributions = {} user_attrs = {} system_attrs: Dict[str, Any] = {} intermediate_values = {} for key, (value, dist) in example_params.items(): if generator.choice([True, False]): params[key] = value distributions[key] = dist for key, value in example_attrs.items(): if generator.choice([True, False]): user_attrs["usr_" + key] = value if generator.choice([True, False]): system_attrs["sys_" + key] = value for i in range(generator.randint(4, 10)): if generator.choice([True, False]): intermediate_values[i] = generator.uniform(-10, 10) return FrozenTrial( number=0, # dummy state=state, value=generator.uniform(-10, 10), datetime_start=datetime.now(), datetime_complete=datetime.now() if state.is_finished() else None, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, trial_id=0, # dummy ) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_best_trial_for_multi_objective_optimization(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study( directions=(StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE) ) generator = random.Random(51) for i in range(3): template_trial = _generate_trial(generator) template_trial.state = TrialState.COMPLETE template_trial.values = [i, i + 1] storage.create_new_trial(study_id, template_trial=template_trial) with pytest.raises(RuntimeError): storage.get_best_trial(study_id) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trial_id_from_study_id_trial_number(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: with pytest.raises(KeyError): # Matching study does not exist. storage.get_trial_id_from_study_id_trial_number(study_id=0, trial_number=0) study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) with pytest.raises(KeyError): # Matching trial does not exist. storage.get_trial_id_from_study_id_trial_number(study_id, trial_number=0) trial_id = storage.create_new_trial(study_id) assert trial_id == storage.get_trial_id_from_study_id_trial_number( study_id, trial_number=0 ) # Trial IDs are globally unique within a storage but numbers are only unique within a # study. Create a second study within the same storage. study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) assert trial_id == storage.get_trial_id_from_study_id_trial_number( study_id, trial_number=0 ) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_pickle_storage(storage_mode: str) -> None: if "redis" in storage_mode: pytest.skip("The `fakeredis` does not support multi instances.") with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.set_study_system_attr(study_id, "key", "pickle") restored_storage = pickle.loads(pickle.dumps(storage)) storage_system_attrs = storage.get_study_system_attrs(study_id) restored_storage_system_attrs = restored_storage.get_study_system_attrs(study_id) assert storage_system_attrs == restored_storage_system_attrs == {"key": "pickle"} if isinstance(storage, RDBStorage): assert storage.url == restored_storage.url assert storage.engine_kwargs == restored_storage.engine_kwargs assert storage.skip_compatibility_check == restored_storage.skip_compatibility_check assert storage.engine != restored_storage.engine assert storage.scoped_session != restored_storage.scoped_session assert storage._version_manager != restored_storage._version_manager @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_trial_is_updatable(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) storage.check_trial_is_updatable(trial_id, TrialState.RUNNING) storage.check_trial_is_updatable(trial_id, TrialState.WAITING) with pytest.raises(RuntimeError): storage.check_trial_is_updatable(trial_id, TrialState.FAIL) with pytest.raises(RuntimeError): storage.check_trial_is_updatable(trial_id, TrialState.PRUNED) with pytest.raises(RuntimeError): storage.check_trial_is_updatable(trial_id, TrialState.COMPLETE) optuna-3.5.0/tests/storages_tests/test_with_server.py000066400000000000000000000146431453453102400232230ustar00rootroot00000000000000from concurrent.futures import ProcessPoolExecutor from concurrent.futures import ThreadPoolExecutor import os import pickle from typing import Sequence import numpy as np import pytest import optuna from optuna.storages import BaseStorage from optuna.study import StudyDirection from optuna.trial import TrialState _STUDY_NAME = "_test_multiprocess" def f(x: float, y: float) -> float: return (x - 3) ** 2 + y def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -10, 10) trial.report(x, 0) trial.report(y, 1) trial.set_user_attr("x", x) return f(x, y) def get_storage() -> BaseStorage: if "TEST_DB_URL" not in os.environ: pytest.skip("This test requires TEST_DB_URL.") storage_url = os.environ["TEST_DB_URL"] storage_mode = os.environ.get("TEST_DB_MODE", "") storage: BaseStorage if storage_mode == "": storage = optuna.storages.RDBStorage(url=storage_url) elif storage_mode == "journal-redis": journal_redis_storage = optuna.storages.JournalRedisStorage(storage_url) storage = optuna.storages.JournalStorage(journal_redis_storage) else: assert False, f"The mode {storage_mode} is not supported." return storage def run_optimize(study_name: str, n_trials: int) -> None: storage = get_storage() # Create a study study = optuna.load_study(study_name=study_name, storage=storage) # Run optimization study.optimize(objective, n_trials=n_trials) def _check_trials(trials: Sequence[optuna.trial.FrozenTrial]) -> None: # Check trial states. assert all(trial.state == TrialState.COMPLETE for trial in trials) # Check trial values and params. assert all("x" in trial.params for trial in trials) assert all("y" in trial.params for trial in trials) assert all( np.isclose( np.asarray([trial.value for trial in trials]), [f(trial.params["x"], trial.params["y"]) for trial in trials], atol=1e-4, ).tolist() ) # Check intermediate values. assert all(len(trial.intermediate_values) == 2 for trial in trials) assert all(trial.params["x"] == trial.intermediate_values[0] for trial in trials) assert all(trial.params["y"] == trial.intermediate_values[1] for trial in trials) # Check attrs. assert all( np.isclose( [trial.user_attrs["x"] for trial in trials], [trial.params["x"] for trial in trials], atol=1e-4, ).tolist() ) def test_loaded_trials() -> None: # Please create the tables by placing this function before the multi-process tests. storage = get_storage() try: optuna.delete_study(study_name=_STUDY_NAME, storage=storage) except KeyError: pass N_TRIALS = 20 study = optuna.create_study(study_name=_STUDY_NAME, storage=storage) # Run optimization study.optimize(objective, n_trials=N_TRIALS) trials = study.trials assert len(trials) == N_TRIALS _check_trials(trials) # Create a new study to confirm the study can load trial properly. loaded_study = optuna.load_study(study_name=_STUDY_NAME, storage=storage) _check_trials(loaded_study.trials) @pytest.mark.parametrize( "input_value,expected", [ (float("inf"), float("inf")), (-float("inf"), -float("inf")), ], ) def test_store_infinite_values(input_value: float, expected: float) -> None: storage = get_storage() study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) storage.set_trial_intermediate_value(trial_id, 1, input_value) storage.set_trial_state_values(trial_id, state=TrialState.COMPLETE, values=(input_value,)) assert storage.get_trial(trial_id).value == expected assert storage.get_trial(trial_id).intermediate_values[1] == expected def test_store_nan_intermediate_values() -> None: storage = get_storage() study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) trial_id = storage.create_new_trial(study_id) value = float("nan") storage.set_trial_intermediate_value(trial_id, 1, value) got_value = storage.get_trial(trial_id).intermediate_values[1] assert np.isnan(got_value) def test_multithread_create_study() -> None: storage = get_storage() with ThreadPoolExecutor(10) as pool: for _ in range(10): pool.submit( optuna.create_study, storage=storage, study_name="test-multithread-create-study", load_if_exists=True, ) def test_multiprocess_run_optimize() -> None: n_workers = 8 n_trials = 20 storage = get_storage() try: optuna.delete_study(study_name=_STUDY_NAME, storage=storage) except KeyError: pass optuna.create_study(storage=storage, study_name=_STUDY_NAME) with ProcessPoolExecutor(n_workers) as pool: pool.map(run_optimize, *zip(*[[_STUDY_NAME, n_trials]] * n_workers)) study = optuna.load_study(study_name=_STUDY_NAME, storage=storage) trials = study.trials assert len(trials) == n_workers * n_trials _check_trials(trials) def test_pickle_storage() -> None: storage = get_storage() study_id = storage.create_new_study(directions=[StudyDirection.MINIMIZE]) storage.set_study_system_attr(study_id, "key", "pickle") restored_storage = pickle.loads(pickle.dumps(storage)) storage_system_attrs = storage.get_study_system_attrs(study_id) restored_storage_system_attrs = restored_storage.get_study_system_attrs(study_id) assert storage_system_attrs == restored_storage_system_attrs == {"key": "pickle"} @pytest.mark.parametrize("direction", [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE]) @pytest.mark.parametrize( "values", [ [0.0, 1.0, 2.0], [0.0, float("inf"), 1.0], [0.0, float("-inf"), 1.0], [float("inf"), 0.0, 1.0, float("-inf")], [float("inf")], [float("-inf")], ], ) def test_get_best_trial(direction: StudyDirection, values: Sequence[float]) -> None: storage = get_storage() study = optuna.create_study(direction=direction, storage=storage) study.add_trials( [optuna.create_trial(params={}, distributions={}, value=value) for value in values] ) expected_value = max(values) if direction == StudyDirection.MAXIMIZE else min(values) assert study.best_value == expected_value optuna-3.5.0/tests/study_tests/000077500000000000000000000000001453453102400165625ustar00rootroot00000000000000optuna-3.5.0/tests/study_tests/__init__.py000066400000000000000000000000001453453102400206610ustar00rootroot00000000000000optuna-3.5.0/tests/study_tests/test_dataframe.py000066400000000000000000000172061453453102400221250ustar00rootroot00000000000000from __future__ import annotations import pandas as pd import pytest from optuna import create_study from optuna import create_trial from optuna import Trial from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import TrialState def test_study_trials_dataframe_with_no_trials() -> None: study_with_no_trials = create_study() trials_df = study_with_no_trials.trials_dataframe() assert trials_df.empty @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "attrs", [ ( "number", "value", "datetime_start", "datetime_complete", "params", "user_attrs", "system_attrs", "state", ), ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "system_attrs", "state", "intermediate_values", "_trial_id", "distributions", ), ], ) @pytest.mark.parametrize("multi_index", [True, False]) def test_trials_dataframe(storage_mode: str, attrs: tuple[str, ...], multi_index: bool) -> None: def f(trial: Trial) -> float: x = trial.suggest_int("x", 1, 1) y = trial.suggest_categorical("y", (2.5,)) trial.set_user_attr("train_loss", 3) trial.storage.set_trial_system_attr(trial._trial_id, "foo", "bar") value = x + y # 3.5 # Test reported intermediate values, although it in practice is not "intermediate". trial.report(value, step=0) return value with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=3) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) # Change index to access rows via trial number. if multi_index: df.set_index(("number", ""), inplace=True, drop=False) else: df.set_index("number", inplace=True, drop=False) assert len(df) == 3 # Number columns are as follows (total of 13): # non-nested: 6 (number, value, state, datetime_start, datetime_complete, duration) # params: 2 # distributions: 2 # user_attrs: 1 # system_attrs: 1 # intermediate_values: 1 expected_n_columns = len(attrs) if "params" in attrs: expected_n_columns += 1 if "distributions" in attrs: expected_n_columns += 1 assert len(df.columns) == expected_n_columns for i in range(3): assert df.number[i] == i assert df.state[i] == "COMPLETE" assert df.value[i] == 3.5 assert isinstance(df.datetime_start[i], pd.Timestamp) assert isinstance(df.datetime_complete[i], pd.Timestamp) if multi_index: if "distributions" in attrs: assert ("distributions", "x") in df.columns assert ("distributions", "y") in df.columns if "_trial_id" in attrs: assert ("trial_id", "") in df.columns # trial_id depends on other tests. if "duration" in attrs: assert ("duration", "") in df.columns assert df.params.x[i] == 1 assert df.params.y[i] == 2.5 assert df.user_attrs.train_loss[i] == 3 assert df.system_attrs.foo[i] == "bar" else: if "distributions" in attrs: assert "distributions_x" in df.columns assert "distributions_y" in df.columns if "_trial_id" in attrs: assert "trial_id" in df.columns # trial_id depends on other tests. if "duration" in attrs: assert "duration" in df.columns assert df.params_x[i] == 1 assert df.params_y[i] == 2.5 assert df.user_attrs_train_loss[i] == 3 assert df.system_attrs_foo[i] == "bar" @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_trials_dataframe_with_failure(storage_mode: str) -> None: def f(trial: Trial) -> float: x = trial.suggest_int("x", 1, 1) y = trial.suggest_categorical("y", (2.5,)) trial.set_user_attr("train_loss", 3) raise ValueError() return x + y # 3.5 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=3, catch=(ValueError,)) df = study.trials_dataframe() # Change index to access rows via trial number. df.set_index("number", inplace=True, drop=False) assert len(df) == 3 # non-nested: 6, params: 2, user_attrs: 1 system_attrs: 0 assert len(df.columns) == 9 for i in range(3): assert df.number[i] == i assert df.state[i] == "FAIL" assert df.value[i] is None assert isinstance(df.datetime_start[i], pd.Timestamp) assert isinstance(df.datetime_complete[i], pd.Timestamp) assert isinstance(df.duration[i], pd.Timedelta) assert df.params_x[i] == 1 assert df.params_y[i] == 2.5 assert df.user_attrs_train_loss[i] == 3 @pytest.mark.parametrize("attrs", [("value",), ("values",)]) @pytest.mark.parametrize("multi_index", [True, False]) def test_trials_dataframe_with_multi_objective_optimization( attrs: tuple[str, ...], multi_index: bool ) -> None: def f(trial: Trial) -> tuple[float, float]: x = trial.suggest_float("x", 1, 1) y = trial.suggest_float("y", 2, 2) return x + y, x**2 + y**2 # 3, 5 # without set_metric_names() study = create_study(directions=["minimize", "maximize"]) study.optimize(f, n_trials=1) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) if multi_index: assert df.get("values")[0][0] == 3 assert df.get("values")[1][0] == 5 else: assert df.values_0[0] == 3 assert df.values_1[0] == 5 # with set_metric_names() study.set_metric_names(["v0", "v1"]) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) if multi_index: assert df.get("values")["v0"][0] == 3 assert df.get("values")["v1"][0] == 5 else: assert df.get("values_v0")[0] == 3 assert df.get("values_v1")[0] == 5 @pytest.mark.parametrize("attrs", [("value",), ("values",)]) @pytest.mark.parametrize("multi_index", [True, False]) def test_trials_dataframe_with_multi_objective_optimization_with_fail_and_pruned( attrs: tuple[str, ...], multi_index: bool ) -> None: study = create_study(directions=["minimize", "maximize"]) study.add_trial(create_trial(state=TrialState.FAIL)) study.add_trial(create_trial(state=TrialState.PRUNED)) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) # without set_metric_names() if multi_index: for i in range(2): assert df.get("values")[0][i] is None assert df.get("values")[1][i] is None else: for i in range(2): assert df.values_0[i] is None assert df.values_1[i] is None # with set_metric_names() study.set_metric_names(["v0", "v1"]) df = study.trials_dataframe(attrs=attrs, multi_index=multi_index) if multi_index: assert df.get("values")["v0"][0] is None assert df.get("values")["v1"][0] is None else: assert df.get("values_v0")[0] is None assert df.get("values_v1")[0] is None optuna-3.5.0/tests/study_tests/test_multi_objective.py000066400000000000000000000117711453453102400233660ustar00rootroot00000000000000import pytest from optuna.study import StudyDirection from optuna.study._multi_objective import _dominates from optuna.trial import create_trial from optuna.trial import TrialState @pytest.mark.parametrize( ("v1", "v2"), [(-1, 1), (-float("inf"), 0), (0, float("inf")), (-float("inf"), float("inf"))] ) def test_dominates_1d_not_equal(v1: float, v2: float) -> None: t1 = create_trial(values=[v1]) t2 = create_trial(values=[v2]) assert _dominates(t1, t2, [StudyDirection.MINIMIZE]) assert not _dominates(t2, t1, [StudyDirection.MINIMIZE]) assert _dominates(t2, t1, [StudyDirection.MAXIMIZE]) assert not _dominates(t1, t2, [StudyDirection.MAXIMIZE]) @pytest.mark.parametrize("v", [0, -float("inf"), float("inf")]) @pytest.mark.parametrize("direction", [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE]) def test_dominates_1d_equal(v: float, direction: StudyDirection) -> None: assert not _dominates(create_trial(values=[v]), create_trial(values=[v]), [direction]) def test_dominates_2d() -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] # Check all pairs of trials consisting of these values, i.e., # [-inf, -inf], [-inf, -1], [-inf, 1], [-inf, inf], [-1, -inf], ... # These values should be specified in ascending order. vals = [-float("inf"), -1, 1, float("inf")] # The following table illustrates an example of dominance relations. # "d" cells in the table dominates the "t" cell in (MINIMIZE, MAXIMIZE) setting. # # value1 # â•”â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•¤â•â•â•â•â•â•— # â•‘ │ -∞ │ -1 │ 1 │ ∞ â•‘ # ╟─────┼─────┼─────┼─────┼─────╢ # â•‘ -∞ │ │ │ d │ d â•‘ # ╟─────┼─────┼─────┼─────┼─────╢ # â•‘ -1 │ │ │ d │ d â•‘ # value0 ╟─────┼─────┼─────┼─────┼─────╢ # â•‘ 1 │ │ │ t │ d â•‘ # ╟─────┼─────┼─────┼─────┼─────╢ # â•‘ ∞ │ │ │ │ â•‘ # â•šâ•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â•§â•â•â•â•â•â• # # In the following code, we check that for each position of "t" cell, the relation # above holds. # Generate the set of all possible indices. all_indices = set((i, j) for i in range(len(vals)) for j in range(len(vals))) for t_i, t_j in all_indices: # Generate the set of all indices that dominates the current index. dominating_indices = set( (d_i, d_j) for d_i in range(t_i + 1) for d_j in range(t_j, len(vals)) ) dominating_indices -= {(t_i, t_j)} for d_i, d_j in dominating_indices: trial1 = create_trial(values=[vals[t_i], vals[t_j]]) trial2 = create_trial(values=[vals[d_i], vals[d_j]]) assert _dominates(trial2, trial1, directions) for d_i, d_j in all_indices - dominating_indices: trial1 = create_trial(values=[vals[t_i], vals[t_j]]) trial2 = create_trial(values=[vals[d_i], vals[d_j]]) assert not _dominates(trial2, trial1, directions) def test_dominates_invalid() -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] # The numbers of objectives for `t1` and `t2` don't match. t1 = create_trial(values=[1]) # One objective. t2 = create_trial(values=[1, 2]) # Two objectives. with pytest.raises(ValueError): _dominates(t1, t2, directions) # The numbers of objectives and directions don't match. t1 = create_trial(values=[1]) # One objective. t2 = create_trial(values=[1]) # One objective. with pytest.raises(ValueError): _dominates(t1, t2, directions) @pytest.mark.parametrize("t1_state", [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]) @pytest.mark.parametrize("t2_state", [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]) def test_dominates_incomplete_vs_incomplete(t1_state: TrialState, t2_state: TrialState) -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] t1 = create_trial(values=[1, 1], state=t1_state) t2 = create_trial(values=[0, 2], state=t2_state) assert not _dominates(t2, t1, list(directions)) assert not _dominates(t1, t2, list(directions)) @pytest.mark.parametrize("t1_state", [TrialState.FAIL, TrialState.WAITING, TrialState.PRUNED]) def test_dominates_complete_vs_incomplete(t1_state: TrialState) -> None: directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] t1 = create_trial(values=[0, 2], state=t1_state) t2 = create_trial(values=[1, 1], state=TrialState.COMPLETE) assert _dominates(t2, t1, list(directions)) assert not _dominates(t1, t2, list(directions)) optuna-3.5.0/tests/study_tests/test_optimize.py000066400000000000000000000141351453453102400220370ustar00rootroot00000000000000from typing import Callable from typing import Generator from typing import Optional from unittest import mock from _pytest.logging import LogCaptureFixture import pytest from optuna import create_study from optuna import logging from optuna import Trial from optuna import TrialPruned from optuna.study import _optimize from optuna.study._tell import _tell_with_warning from optuna.study._tell import STUDY_TELL_WARNING_KEY from optuna.testing.objectives import fail_objective from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import TrialState @pytest.fixture(autouse=True) def logging_setup() -> Generator[None, None, None]: # We need to reconstruct our default handler to properly capture stderr. logging._reset_library_root_logger() logging.enable_default_handler() logging.set_verbosity(logging.INFO) logging.enable_propagation() yield # After testing, restore default propagation setting. logging.disable_propagation() @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial(storage_mode: str, caplog: LogCaptureFixture) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) caplog.clear() frozen_trial = _optimize._run_trial(study, lambda _: 1.0, catch=()) assert frozen_trial.state == TrialState.COMPLETE assert frozen_trial.value == 1.0 assert "Trial 0 finished with value: 1.0 and parameters" in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, lambda _: float("inf"), catch=()) assert frozen_trial.state == TrialState.COMPLETE assert frozen_trial.value == float("inf") assert "Trial 1 finished with value: inf and parameters" in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, lambda _: -float("inf"), catch=()) assert frozen_trial.state == TrialState.COMPLETE assert frozen_trial.value == -float("inf") assert "Trial 2 finished with value: -inf and parameters" in caplog.text @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_automatically_fail(storage_mode: str, caplog: LogCaptureFixture) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) frozen_trial = _optimize._run_trial(study, lambda _: float("nan"), catch=()) assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None frozen_trial = _optimize._run_trial(study, lambda _: None, catch=()) # type: ignore[arg-type,return-value] # noqa: E501 assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None frozen_trial = _optimize._run_trial(study, lambda _: object(), catch=()) # type: ignore[arg-type,return-value] # noqa: E501 assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None frozen_trial = _optimize._run_trial(study, lambda _: [0, 1], catch=()) assert frozen_trial.state == TrialState.FAIL assert frozen_trial.value is None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_pruned(storage_mode: str, caplog: LogCaptureFixture) -> None: def gen_func(intermediate: Optional[float] = None) -> Callable[[Trial], float]: def func(trial: Trial) -> float: if intermediate is not None: trial.report(step=1, value=intermediate) raise TrialPruned return func with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) caplog.clear() frozen_trial = _optimize._run_trial(study, gen_func(), catch=()) assert frozen_trial.state == TrialState.PRUNED assert frozen_trial.value is None assert "Trial 0 pruned." in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, gen_func(intermediate=1), catch=()) assert frozen_trial.state == TrialState.PRUNED assert frozen_trial.value == 1 assert "Trial 1 pruned." in caplog.text caplog.clear() frozen_trial = _optimize._run_trial(study, gen_func(intermediate=float("nan")), catch=()) assert frozen_trial.state == TrialState.PRUNED assert frozen_trial.value is None assert "Trial 2 pruned." in caplog.text @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_catch_exception(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) frozen_trial = _optimize._run_trial(study, fail_objective, catch=(ValueError,)) assert frozen_trial.state == TrialState.FAIL assert STUDY_TELL_WARNING_KEY not in frozen_trial.system_attrs @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_exception(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) with pytest.raises(ValueError): _optimize._run_trial(study, fail_objective, ()) # Test trial with unacceptable exception. with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) with pytest.raises(ValueError): _optimize._run_trial(study, fail_objective, (ArithmeticError,)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_run_trial_invoke_tell_with_suppressing_warning(storage_mode: str) -> None: def func_numerical(trial: Trial) -> float: return trial.suggest_float("v", 0, 10) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) with mock.patch( "optuna.study._optimize._tell_with_warning", side_effect=_tell_with_warning ) as mock_obj: _optimize._run_trial(study, func_numerical, ()) mock_obj.assert_called_once_with( study=mock.ANY, trial=mock.ANY, value_or_values=mock.ANY, state=mock.ANY, suppress_warning=True, ) optuna-3.5.0/tests/study_tests/test_study.py000066400000000000000000001557271453453102400213640ustar00rootroot00000000000000from __future__ import annotations from concurrent.futures import as_completed from concurrent.futures import ThreadPoolExecutor import copy import multiprocessing import pickle import platform import threading import time from typing import Any from typing import Callable from unittest.mock import Mock from unittest.mock import patch import uuid import warnings import _pytest.capture import pytest from optuna import copy_study from optuna import create_study from optuna import create_trial from optuna import delete_study from optuna import distributions from optuna import get_all_study_names from optuna import get_all_study_summaries from optuna import load_study from optuna import logging from optuna import Study from optuna import Trial from optuna import TrialPruned from optuna.exceptions import DuplicatedStudyError from optuna.exceptions import ExperimentalWarning from optuna.study import StudyDirection from optuna.study.study import _SYSTEM_ATTR_METRIC_NAMES from optuna.testing.objectives import fail_objective from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.trial import FrozenTrial from optuna.trial import TrialState CallbackFuncType = Callable[[Study, FrozenTrial], None] def func(trial: Trial) -> float: x = trial.suggest_float("x", -10.0, 10.0) y = trial.suggest_float("y", 20, 30, log=True) z = trial.suggest_categorical("z", (-1.0, 1.0)) return (x - 2) ** 2 + (y - 25) ** 2 + z class Func: def __init__(self, sleep_sec: float | None = None) -> None: self.n_calls = 0 self.sleep_sec = sleep_sec self.lock = threading.Lock() def __call__(self, trial: Trial) -> float: with self.lock: self.n_calls += 1 # Sleep for testing parallelism. if self.sleep_sec is not None: time.sleep(self.sleep_sec) value = func(trial) check_params(trial.params) return value def check_params(params: dict[str, Any]) -> None: assert sorted(params.keys()) == ["x", "y", "z"] def check_value(value: float | None) -> None: assert isinstance(value, float) assert -1.0 <= value <= 12.0**2 + 5.0**2 + 1.0 def check_frozen_trial(frozen_trial: FrozenTrial) -> None: if frozen_trial.state == TrialState.COMPLETE: check_params(frozen_trial.params) check_value(frozen_trial.value) def check_study(study: Study) -> None: for trial in study.trials: check_frozen_trial(trial) assert not study._is_multi_objective() complete_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) if len(complete_trials) == 0: with pytest.raises(ValueError): study.best_params with pytest.raises(ValueError): study.best_value with pytest.raises(ValueError): study.best_trial else: check_params(study.best_params) check_value(study.best_value) check_frozen_trial(study.best_trial) def stop_objective(threshold_number: int) -> Callable[[Trial], float]: def objective(trial: Trial) -> float: if trial.number >= threshold_number: trial.study.stop() return trial.number return objective def test_optimize_trivial_in_memory_new() -> None: study = create_study() study.optimize(func, n_trials=10) check_study(study) def test_optimize_trivial_in_memory_resume() -> None: study = create_study() study.optimize(func, n_trials=10) study.optimize(func, n_trials=10) check_study(study) def test_optimize_trivial_rdb_resume_study() -> None: study = create_study(storage="sqlite:///:memory:") study.optimize(func, n_trials=10) check_study(study) def test_optimize_with_direction() -> None: study = create_study(direction="minimize") study.optimize(func, n_trials=10) assert study.direction == StudyDirection.MINIMIZE check_study(study) study = create_study(direction="maximize") study.optimize(func, n_trials=10) assert study.direction == StudyDirection.MAXIMIZE check_study(study) with pytest.raises(ValueError): create_study(direction="test") @pytest.mark.parametrize("n_trials", (0, 1, 20)) @pytest.mark.parametrize("n_jobs", (1, 2, -1)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_parallel(n_trials: int, n_jobs: int, storage_mode: str) -> None: f = Func() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=n_trials, n_jobs=n_jobs) assert f.n_calls == len(study.trials) == n_trials check_study(study) def test_optimize_with_thread_pool_executor() -> None: def objective(t: Trial) -> float: return t.suggest_float("x", -10, 10) study = create_study() with ThreadPoolExecutor(max_workers=5) as pool: for _ in range(10): pool.submit(study.optimize, objective, n_trials=10) assert len(study.trials) == 100 @pytest.mark.parametrize("n_trials", (0, 1, 20, None)) @pytest.mark.parametrize("n_jobs", (1, 2, -1)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_parallel_timeout(n_trials: int, n_jobs: int, storage_mode: str) -> None: sleep_sec = 0.1 timeout_sec = 1.0 f = Func(sleep_sec=sleep_sec) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=n_trials, n_jobs=n_jobs, timeout=timeout_sec) assert f.n_calls == len(study.trials) if n_trials is not None: assert f.n_calls <= n_trials # A thread can process at most (timeout_sec / sleep_sec + 1) trials. n_jobs_actual = n_jobs if n_jobs != -1 else multiprocessing.cpu_count() max_calls = (timeout_sec / sleep_sec + 1) * n_jobs_actual assert f.n_calls <= max_calls check_study(study) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_with_catch(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) # Test default exceptions. with pytest.raises(ValueError): study.optimize(fail_objective, n_trials=20) assert len(study.trials) == 1 assert all(trial.state == TrialState.FAIL for trial in study.trials) # Test acceptable exception. study.optimize(fail_objective, n_trials=20, catch=(ValueError,)) assert len(study.trials) == 21 assert all(trial.state == TrialState.FAIL for trial in study.trials) # Test trial with unacceptable exception. with pytest.raises(ValueError): study.optimize(fail_objective, n_trials=20, catch=(ArithmeticError,)) assert len(study.trials) == 22 assert all(trial.state == TrialState.FAIL for trial in study.trials) @pytest.mark.parametrize("catch", [ValueError, (ValueError,), [ValueError], {ValueError}]) def test_optimize_with_catch_valid_type(catch: Any) -> None: study = create_study() study.optimize(fail_objective, n_trials=20, catch=catch) @pytest.mark.parametrize("catch", [None, 1]) def test_optimize_with_catch_invalid_type(catch: Any) -> None: study = create_study() with pytest.raises(TypeError): study.optimize(fail_objective, n_trials=20, catch=catch) @pytest.mark.parametrize("n_jobs", (2, -1)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_optimize_with_reseeding(n_jobs: int, storage_mode: str) -> None: f = Func() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) sampler = study.sampler with patch.object(sampler, "reseed_rng", wraps=sampler.reseed_rng) as mock_object: study.optimize(f, n_trials=1, n_jobs=2) assert mock_object.call_count == 1 def test_call_another_study_optimize_in_optimize() -> None: def inner_objective(t: Trial) -> float: return t.suggest_float("x", -10, 10) def objective(t: Trial) -> float: inner_study = create_study() inner_study.enqueue_trial({"x": t.suggest_int("initial_point", -10, 10)}) inner_study.optimize(inner_objective, n_trials=10) return inner_study.best_value study = create_study() study.optimize(objective, n_trials=10) assert len(study.trials) == 10 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_study_set_and_get_user_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.set_user_attr("dataset", "MNIST") assert study.user_attrs["dataset"] == "MNIST" @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_trial_set_and_get_user_attrs(storage_mode: str) -> None: def f(trial: Trial) -> float: trial.set_user_attr("train_accuracy", 1) assert trial.user_attrs["train_accuracy"] == 1 return 0.0 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(f, n_trials=1) frozen_trial = study.trials[0] assert frozen_trial.user_attrs["train_accuracy"] == 1 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("include_best_trial", [True, False]) def test_get_all_study_summaries(storage_mode: str, include_best_trial: bool) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(func, n_trials=5) summaries = get_all_study_summaries(study._storage, include_best_trial) summary = [s for s in summaries if s._study_id == study._study_id][0] assert summary.study_name == study.study_name assert summary.n_trials == 5 if include_best_trial: assert summary.best_trial is not None else: assert summary.best_trial is None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_study_summaries_with_no_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) summaries = get_all_study_summaries(study._storage) summary = [s for s in summaries if s._study_id == study._study_id][0] assert summary.study_name == study.study_name assert summary.n_trials == 0 assert summary.datetime_start is None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_all_study_names(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: n_studies = 5 studies = [create_study(storage=storage) for _ in range(n_studies)] study_names = get_all_study_names(storage) assert len(study_names) == n_studies for study, study_name in zip(studies, study_names): assert study_name == study.study_name def test_study_pickle() -> None: study_1 = create_study() study_1.optimize(func, n_trials=10) check_study(study_1) assert len(study_1.trials) == 10 dumped_bytes = pickle.dumps(study_1) study_2 = pickle.loads(dumped_bytes) check_study(study_2) assert len(study_2.trials) == 10 study_2.optimize(func, n_trials=10) check_study(study_2) assert len(study_2.trials) == 20 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_create_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Test creating a new study. study = create_study(storage=storage, load_if_exists=False) # Test `load_if_exists=True` with existing study. create_study(study_name=study.study_name, storage=storage, load_if_exists=True) with pytest.raises(DuplicatedStudyError): create_study(study_name=study.study_name, storage=storage, load_if_exists=False) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_load_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if storage is None: # `InMemoryStorage` can not be used with `load_study` function. return study_name = str(uuid.uuid4()) with pytest.raises(KeyError): # Test loading an unexisting study. load_study(study_name=study_name, storage=storage) # Create a new study. created_study = create_study(study_name=study_name, storage=storage) # Test loading an existing study. loaded_study = load_study(study_name=study_name, storage=storage) assert created_study._study_id == loaded_study._study_id @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_load_study_study_name_none(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: if storage is None: # `InMemoryStorage` can not be used with `load_study` function. return study_name = str(uuid.uuid4()) _ = create_study(study_name=study_name, storage=storage) loaded_study = load_study(study_name=None, storage=storage) assert loaded_study.study_name == study_name study_name = str(uuid.uuid4()) _ = create_study(study_name=study_name, storage=storage) # Ambiguous study. with pytest.raises(ValueError): load_study(study_name=None, storage=storage) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_delete_study(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: # Test deleting a non-existing study. with pytest.raises(KeyError): delete_study(study_name="invalid-study-name", storage=storage) # Test deleting an existing study. study = create_study(storage=storage, load_if_exists=False) delete_study(study_name=study.study_name, storage=storage) # Test failed to delete the study which is already deleted. with pytest.raises(KeyError): delete_study(study_name=study.study_name, storage=storage) @pytest.mark.parametrize("from_storage_mode", STORAGE_MODES) @pytest.mark.parametrize("to_storage_mode", STORAGE_MODES) def test_copy_study(from_storage_mode: str, to_storage_mode: str) -> None: with StorageSupplier(from_storage_mode) as from_storage, StorageSupplier( to_storage_mode ) as to_storage: from_study = create_study(storage=from_storage, directions=["maximize", "minimize"]) from_study._storage.set_study_system_attr(from_study._study_id, "foo", "bar") from_study.set_user_attr("baz", "qux") from_study.optimize( lambda t: (t.suggest_float("x0", 0, 1), t.suggest_float("x1", 0, 1)), n_trials=3 ) copy_study( from_study_name=from_study.study_name, from_storage=from_storage, to_storage=to_storage, ) to_study = load_study(study_name=from_study.study_name, storage=to_storage) assert to_study.study_name == from_study.study_name assert to_study.directions == from_study.directions to_study_system_attrs = to_study._storage.get_study_system_attrs(to_study._study_id) from_study_system_attrs = from_study._storage.get_study_system_attrs(from_study._study_id) assert to_study_system_attrs == from_study_system_attrs assert to_study.user_attrs == from_study.user_attrs assert len(to_study.trials) == len(from_study.trials) @pytest.mark.parametrize("from_storage_mode", STORAGE_MODES) @pytest.mark.parametrize("to_storage_mode", STORAGE_MODES) def test_copy_study_to_study_name(from_storage_mode: str, to_storage_mode: str) -> None: with StorageSupplier(from_storage_mode) as from_storage, StorageSupplier( to_storage_mode ) as to_storage: from_study = create_study(study_name="foo", storage=from_storage) _ = create_study(study_name="foo", storage=to_storage) with pytest.raises(DuplicatedStudyError): copy_study( from_study_name=from_study.study_name, from_storage=from_storage, to_storage=to_storage, ) copy_study( from_study_name=from_study.study_name, from_storage=from_storage, to_storage=to_storage, to_study_name="bar", ) _ = load_study(study_name="bar", storage=to_storage) def test_nested_optimization() -> None: def objective(trial: Trial) -> float: with pytest.raises(RuntimeError): trial.study.optimize(lambda _: 0.0, n_trials=1) return 1.0 study = create_study() study.optimize(objective, n_trials=10, catch=()) def test_stop_in_objective() -> None: # Test stopping the optimization: it should stop once the trial number reaches 4. study = create_study() study.optimize(stop_objective(4), n_trials=10) assert len(study.trials) == 5 # Test calling `optimize` again: it should stop once the trial number reaches 11. study.optimize(stop_objective(11), n_trials=10) assert len(study.trials) == 12 def test_stop_in_callback() -> None: def callback(study: Study, trial: FrozenTrial) -> None: if trial.number >= 4: study.stop() # Test stopping the optimization inside a callback. study = create_study() study.optimize(lambda _: 1.0, n_trials=10, callbacks=[callback]) assert len(study.trials) == 5 def test_stop_n_jobs() -> None: def callback(study: Study, trial: FrozenTrial) -> None: if trial.number >= 4: study.stop() study = create_study() study.optimize(lambda _: 1.0, n_trials=None, callbacks=[callback], n_jobs=2) assert 5 <= len(study.trials) <= 6 def test_stop_outside_optimize() -> None: # Test stopping outside the optimization: it should raise `RuntimeError`. study = create_study() with pytest.raises(RuntimeError): study.stop() # Test calling `optimize` after the `RuntimeError` is caught. study.optimize(lambda _: 1.0, n_trials=1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_add_trial(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 trial = create_trial(value=0.8) study.add_trial(trial) assert len(study.trials) == 1 assert study.trials[0].number == 0 assert study.best_value == 0.8 def test_add_trial_invalid_values_length() -> None: study = create_study() trial = create_trial(values=[0, 0]) with pytest.raises(ValueError): study.add_trial(trial) study = create_study(directions=["minimize", "minimize"]) trial = create_trial(value=0) with pytest.raises(ValueError): study.add_trial(trial) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_add_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.add_trials([]) assert len(study.trials) == 0 trials = [create_trial(value=i) for i in range(3)] study.add_trials(trials) assert len(study.trials) == 3 for i, trial in enumerate(study.trials): assert trial.number == i assert trial.value == i other_study = create_study(storage=storage) other_study.add_trials(study.trials) assert len(other_study.trials) == 3 for i, trial in enumerate(other_study.trials): assert trial.number == i assert trial.value == i @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_properly_sets_param_values(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": -5, "y": 5}) study.enqueue_trial(params={"x": -1, "y": 0}) def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.optimize(objective, n_trials=2) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 t1 = study.trials[1] assert t1.params["x"] == -1 assert t1.params["y"] == 0 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_with_unfixed_parameters(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": -5}) def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.optimize(objective, n_trials=1) t = study.trials[0] assert t.params["x"] == -5 assert -10 <= t.params["y"] <= 10 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_properly_sets_user_attr(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": -5, "y": 5}, user_attrs={"is_optimal": False}) study.enqueue_trial(params={"x": 0, "y": 0}, user_attrs={"is_optimal": True}) def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.optimize(objective, n_trials=2) t0 = study.trials[0] assert t0.user_attrs == {"is_optimal": False} t1 = study.trials[1] assert t1.user_attrs == {"is_optimal": True} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_with_out_of_range_parameters(storage_mode: str) -> None: fixed_value = 11 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": fixed_value}) def objective(trial: Trial) -> float: return trial.suggest_int("x", -10, 10) with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) t = study.trials[0] assert t.params["x"] == fixed_value # Internal logic might differ when distribution contains a single element. # Test it explicitly. with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 study.enqueue_trial(params={"x": fixed_value}) def objective(trial: Trial) -> float: return trial.suggest_int("x", 1, 1) # Single element. with pytest.warns(UserWarning): study.optimize(objective, n_trials=1) t = study.trials[0] assert t.params["x"] == fixed_value @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_skips_existing_finished(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.enqueue_trial({"x": -5, "y": 5}) study.optimize(objective, n_trials=1) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 before_enqueue = len(study.trials) study.enqueue_trial({"x": -5, "y": 5}, skip_if_exists=True) after_enqueue = len(study.trials) assert before_enqueue == after_enqueue @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueue_trial_skips_existing_waiting(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) return x**2 + y**2 study.enqueue_trial({"x": -5, "y": 5}) before_enqueue = len(study.trials) study.enqueue_trial({"x": -5, "y": 5}, skip_if_exists=True) after_enqueue = len(study.trials) assert before_enqueue == after_enqueue study.optimize(objective, n_trials=1) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "new_params", [{"x": -5, "y": 5, "z": 5}, {"x": -5}, {"x": -5, "z": 5}, {"x": -5, "y": 6}] ) def test_enqueue_trial_skip_existing_allows_unfixed( storage_mode: str, new_params: dict[str, int] ) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) assert len(study.trials) == 0 def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) y = trial.suggest_int("y", -10, 10) if trial.number == 1: z = trial.suggest_int("z", -10, 10) return x**2 + y**2 + z**2 return x**2 + y**2 study.enqueue_trial({"x": -5, "y": 5}) study.optimize(objective, n_trials=1) t0 = study.trials[0] assert t0.params["x"] == -5 assert t0.params["y"] == 5 study.enqueue_trial(new_params, skip_if_exists=True) study.optimize(objective, n_trials=1) unfixed_params = {"x", "y", "z"} - set(new_params) t1 = study.trials[1] assert all(t1.params[k] == new_params[k] for k in new_params) assert all(-10 <= t1.params[k] <= 10 for k in unfixed_params) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "param", ["foo", 1, 1.1, 1e17, 1e-17, float("inf"), float("-inf"), float("nan"), None] ) def test_enqueue_trial_skip_existing_handles_common_types(storage_mode: str, param: Any) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.enqueue_trial({"x": param}) before_enqueue = len(study.trials) study.enqueue_trial({"x": param}, skip_if_exists=True) after_enqueue = len(study.trials) assert before_enqueue == after_enqueue @patch("optuna.study._optimize.gc.collect") def test_optimize_with_gc(collect_mock: Mock) -> None: study = create_study() study.optimize(func, n_trials=10, gc_after_trial=True) check_study(study) assert collect_mock.call_count == 10 @patch("optuna.study._optimize.gc.collect") def test_optimize_without_gc(collect_mock: Mock) -> None: study = create_study() study.optimize(func, n_trials=10, gc_after_trial=False) check_study(study) assert collect_mock.call_count == 0 @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_with_progbar(n_jobs: int, capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs, show_progress_bar=True) _, err = capsys.readouterr() # Search for progress bar elements in stderr. assert "Best trial: 0" in err assert "Best value: 1" in err assert "10/10" in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar(n_jobs: int, capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs) _, err = capsys.readouterr() assert "Best trial: 0" not in err assert "Best value: 1" not in err assert "10/10" not in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" not in err def test_optimize_with_progbar_timeout(capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() study.optimize(lambda _: 1.0, timeout=2.0, show_progress_bar=True) _, err = capsys.readouterr() assert "Best trial: 0" in err assert "Best value: 1" in err assert "00:02/00:02" in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" in err def test_optimize_with_progbar_parallel_timeout(capsys: _pytest.capture.CaptureFixture) -> None: study = create_study() with pytest.warns( UserWarning, match="The timeout-based progress bar is not supported with n_jobs != 1." ): study.optimize(lambda _: 1.0, timeout=2.0, show_progress_bar=True, n_jobs=2) _, err = capsys.readouterr() # Testing for a character that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize( "timeout,expected", [ (59.0, "/00:59"), (60.0, "/01:00"), (60.0 * 60, "/1:00:00"), (60.0 * 60 * 24, "/24:00:00"), (60.0 * 60 * 24 * 10, "/240:00:00"), ], ) def test_optimize_with_progbar_timeout_formats( timeout: float, expected: str, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(stop_objective(5), timeout=timeout, show_progress_bar=True) _, err = capsys.readouterr() assert expected in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar_timeout( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(lambda _: 1.0, timeout=2.0, n_jobs=n_jobs) _, err = capsys.readouterr() assert "Best trial: 0" not in err assert "Best value: 1.0" not in err assert "00:02/00:02" not in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" not in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_progbar_n_trials_prioritized( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs, timeout=10.0, show_progress_bar=True) _, err = capsys.readouterr() assert "Best trial: 0" in err assert "Best value: 1" in err assert "10/10" in err if platform.system() != "Windows": # Skip this assertion because the progress bar sometimes stops at 99% on Windows. assert "100%" in err assert "it" in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar_n_trials_prioritized( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(lambda _: 1.0, n_trials=10, n_jobs=n_jobs, timeout=10.0) _, err = capsys.readouterr() # Testing for a character that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_progbar_no_constraints( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() with warnings.catch_warnings(): warnings.simplefilter("ignore", category=UserWarning) study.optimize(stop_objective(5), n_jobs=n_jobs, show_progress_bar=True) _, err = capsys.readouterr() # We can't simply test if stderr is empty, since we're not sure # what else could write to it. Instead, we are testing for a character # that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize("n_jobs", [1, 2]) def test_optimize_without_progbar_no_constraints( n_jobs: int, capsys: _pytest.capture.CaptureFixture ) -> None: study = create_study() study.optimize(stop_objective(5), n_jobs=n_jobs) _, err = capsys.readouterr() # Testing for a character that forms progress bar borders. assert "|" not in err @pytest.mark.parametrize("n_jobs", [1, 4]) def test_callbacks(n_jobs: int) -> None: lock = threading.Lock() def with_lock(f: CallbackFuncType) -> CallbackFuncType: def callback(study: Study, trial: FrozenTrial) -> None: with lock: f(study, trial) return callback study = create_study() def objective(trial: Trial) -> float: return trial.suggest_int("x", 1, 1) # Empty callback list. study.optimize(objective, callbacks=[], n_trials=10, n_jobs=n_jobs) # One callback. values = [] callbacks = [with_lock(lambda study, trial: values.append(trial.value))] study.optimize(objective, callbacks=callbacks, n_trials=10, n_jobs=n_jobs) assert values == [1] * 10 # Two callbacks. values = [] params = [] callbacks = [ with_lock(lambda study, trial: values.append(trial.value)), with_lock(lambda study, trial: params.append(trial.params)), ] study.optimize(objective, callbacks=callbacks, n_trials=10, n_jobs=n_jobs) assert values == [1] * 10 assert params == [{"x": 1}] * 10 # If a trial is failed with an exception and the exception is caught by the study, # callbacks are invoked. states = [] callbacks = [with_lock(lambda study, trial: states.append(trial.state))] study.optimize( lambda t: 1 / 0, callbacks=callbacks, n_trials=10, n_jobs=n_jobs, catch=(ZeroDivisionError,), ) assert states == [TrialState.FAIL] * 10 # If a trial is failed with an exception and the exception isn't caught by the study, # callbacks aren't invoked. states = [] callbacks = [with_lock(lambda study, trial: states.append(trial.state))] with pytest.raises(ZeroDivisionError): study.optimize(lambda t: 1 / 0, callbacks=callbacks, n_trials=10, n_jobs=n_jobs, catch=()) assert states == [] def test_optimize_infinite_budget_progbar() -> None: def terminate_study(study: Study, trial: FrozenTrial) -> None: study.stop() study = create_study() with pytest.warns(UserWarning): study.optimize( func, n_trials=None, timeout=None, show_progress_bar=True, callbacks=[terminate_study] ) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trials(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(lambda t: t.suggest_int("x", 1, 5), n_trials=5) with patch("copy.deepcopy", wraps=copy.deepcopy) as mock_object: trials0 = study.get_trials(deepcopy=False) assert mock_object.call_count == 0 assert len(trials0) == 5 trials1 = study.get_trials(deepcopy=True) assert mock_object.call_count > 0 assert trials0 == trials1 # `study.trials` is equivalent to `study.get_trials(deepcopy=True)`. old_count = mock_object.call_count trials2 = study.trials assert mock_object.call_count > old_count assert trials0 == trials2 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_get_trials_state_option(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) def objective(trial: Trial) -> float: if trial.number == 0: return 0.0 # TrialState.COMPLETE. elif trial.number == 1: return 0.0 # TrialState.COMPLETE. elif trial.number == 2: raise TrialPruned # TrialState.PRUNED. else: assert False study.optimize(objective, n_trials=3) trials = study.get_trials(states=None) assert len(trials) == 3 trials = study.get_trials(states=(TrialState.COMPLETE,)) assert len(trials) == 2 assert all(t.state == TrialState.COMPLETE for t in trials) trials = study.get_trials(states=(TrialState.COMPLETE, TrialState.PRUNED)) assert len(trials) == 3 assert all(t.state in (TrialState.COMPLETE, TrialState.PRUNED) for t in trials) trials = study.get_trials(states=()) assert len(trials) == 0 other_states = [ s for s in list(TrialState) if s != TrialState.COMPLETE and s != TrialState.PRUNED ] for s in other_states: trials = study.get_trials(states=(s,)) assert len(trials) == 0 def test_log_completed_trial(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. logging._reset_library_root_logger() logging.set_verbosity(logging.INFO) study = create_study() study.optimize(lambda _: 1.0, n_trials=1) _, err = capsys.readouterr() assert "Trial 0" in err logging.set_verbosity(logging.WARNING) study.optimize(lambda _: 1.0, n_trials=1) _, err = capsys.readouterr() assert "Trial 1" not in err logging.set_verbosity(logging.DEBUG) study.optimize(lambda _: 1.0, n_trials=1) _, err = capsys.readouterr() assert "Trial 2" in err def test_log_completed_trial_skip_storage_access() -> None: study = create_study() # Create a trial to retrieve it as the `study.best_trial`. study.optimize(lambda _: 0.0, n_trials=1) frozen_trial = study.best_trial storage = study._storage with patch.object(storage, "get_best_trial", wraps=storage.get_best_trial) as mock_object: study._log_completed_trial(frozen_trial) assert mock_object.call_count == 1 logging.set_verbosity(logging.WARNING) with patch.object(storage, "get_best_trial", wraps=storage.get_best_trial) as mock_object: study._log_completed_trial(frozen_trial) assert mock_object.call_count == 0 logging.set_verbosity(logging.DEBUG) with patch.object(storage, "get_best_trial", wraps=storage.get_best_trial) as mock_object: study._log_completed_trial(frozen_trial) assert mock_object.call_count == 1 def test_create_study_with_multi_objectives() -> None: study = create_study(directions=["maximize"]) assert study.direction == StudyDirection.MAXIMIZE assert not study._is_multi_objective() study = create_study(directions=["maximize", "minimize"]) assert study.directions == [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE] assert study._is_multi_objective() with pytest.raises(ValueError): # Empty `direction` isn't allowed. _ = create_study(directions=[]) with pytest.raises(ValueError): _ = create_study(direction="minimize", directions=["maximize"]) with pytest.raises(ValueError): _ = create_study(direction="minimize", directions=[]) def test_create_study_with_direction_object() -> None: study = create_study(direction=StudyDirection.MAXIMIZE) assert study.direction == StudyDirection.MAXIMIZE study = create_study(directions=[StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE]) assert study.directions == [StudyDirection.MAXIMIZE, StudyDirection.MINIMIZE] @pytest.mark.parametrize("n_objectives", [2, 3]) def test_optimize_with_multi_objectives(n_objectives: int) -> None: directions = ["minimize" for _ in range(n_objectives)] study = create_study(directions=directions) def objective(trial: Trial) -> list[float]: return [trial.suggest_float("v{}".format(i), 0, 5) for i in range(n_objectives)] study.optimize(objective, n_trials=10) assert len(study.trials) == 10 for trial in study.trials: assert trial.values assert len(trial.values) == n_objectives def test_best_trials() -> None: study = create_study(directions=["minimize", "maximize"]) study.optimize(lambda t: [2, 2], n_trials=1) study.optimize(lambda t: [1, 1], n_trials=1) study.optimize(lambda t: [3, 1], n_trials=1) assert {tuple(t.values) for t in study.best_trials} == {(1, 1), (2, 2)} def test_wrong_n_objectives() -> None: n_objectives = 2 directions = ["minimize" for _ in range(n_objectives)] study = create_study(directions=directions) def objective(trial: Trial) -> list[float]: return [trial.suggest_float("v{}".format(i), 0, 5) for i in range(n_objectives + 1)] study.optimize(objective, n_trials=10) for trial in study.trials: assert trial.state is TrialState.FAIL def test_ask() -> None: study = create_study() trial = study.ask() assert isinstance(trial, Trial) def test_ask_enqueue_trial() -> None: study = create_study() study.enqueue_trial({"x": 0.5}, user_attrs={"memo": "this is memo"}) trial = study.ask() assert trial.suggest_float("x", 0, 1) == 0.5 assert trial.user_attrs == {"memo": "this is memo"} def test_ask_fixed_search_space() -> None: fixed_distributions = { "x": distributions.FloatDistribution(0, 1), "y": distributions.CategoricalDistribution(["bacon", "spam"]), } study = create_study() trial = study.ask(fixed_distributions=fixed_distributions) params = trial.params assert len(trial.params) == 2 assert 0 <= params["x"] < 1 assert params["y"] in ["bacon", "spam"] # Deprecated distributions are internally converted to corresponding distributions. @pytest.mark.filterwarnings("ignore::FutureWarning") def test_ask_distribution_conversion() -> None: fixed_distributions = { "ud": distributions.UniformDistribution(low=0, high=10), "dud": distributions.DiscreteUniformDistribution(low=0, high=10, q=2), "lud": distributions.LogUniformDistribution(low=1, high=10), "id": distributions.IntUniformDistribution(low=0, high=10), "idd": distributions.IntUniformDistribution(low=0, high=10, step=2), "ild": distributions.IntLogUniformDistribution(low=1, high=10), } study = create_study() with pytest.warns( FutureWarning, match="See https://github.com/optuna/optuna/issues/2941", ) as record: trial = study.ask(fixed_distributions=fixed_distributions) assert len(record) == 6 expected_distributions = { "ud": distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": distributions.IntDistribution(low=1, high=10, log=True, step=1), } assert trial.distributions == expected_distributions # It confirms that ask doesn't convert non-deprecated distributions. def test_ask_distribution_conversion_noop() -> None: fixed_distributions = { "ud": distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": distributions.IntDistribution(low=1, high=10, log=True, step=1), "cd": distributions.CategoricalDistribution(choices=["a", "b", "c"]), } study = create_study() trial = study.ask(fixed_distributions=fixed_distributions) # Check fixed_distributions doesn't change. assert trial.distributions == fixed_distributions def test_tell() -> None: study = create_study() assert len(study.trials) == 0 trial = study.ask() assert len(study.trials) == 1 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 0 study.tell(trial, 1.0) assert len(study.trials) == 1 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 1 study.tell(study.ask(), [1.0]) assert len(study.trials) == 2 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 2 # `trial` could be int. study.tell(study.ask().number, 1.0) assert len(study.trials) == 3 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 3 # Inf is supported as values. study.tell(study.ask(), float("inf")) assert len(study.trials) == 4 assert len(study.get_trials(states=(TrialState.COMPLETE,))) == 4 study.tell(study.ask(), state=TrialState.PRUNED) assert len(study.trials) == 5 assert len(study.get_trials(states=(TrialState.PRUNED,))) == 1 study.tell(study.ask(), state=TrialState.FAIL) assert len(study.trials) == 6 assert len(study.get_trials(states=(TrialState.FAIL,))) == 1 def test_tell_pruned() -> None: study = create_study() study.tell(study.ask(), state=TrialState.PRUNED) assert study.trials[-1].value is None assert study.trials[-1].state == TrialState.PRUNED # Store the last intermediates as value. trial = study.ask() trial.report(2.0, step=1) study.tell(trial, state=TrialState.PRUNED) assert study.trials[-1].value == 2.0 assert study.trials[-1].state == TrialState.PRUNED # Inf is also supported as a value. trial = study.ask() trial.report(float("inf"), step=1) study.tell(trial, state=TrialState.PRUNED) assert study.trials[-1].value == float("inf") assert study.trials[-1].state == TrialState.PRUNED # NaN is not supported as a value. trial = study.ask() trial.report(float("nan"), step=1) study.tell(trial, state=TrialState.PRUNED) assert study.trials[-1].value is None assert study.trials[-1].state == TrialState.PRUNED def test_tell_automatically_fail() -> None: study = create_study() # Check invalid values, e.g. str cannot be cast to float. with pytest.warns(UserWarning): study.tell(study.ask(), "a") # type: ignore assert len(study.trials) == 1 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Check invalid values, e.g. `None` that cannot be cast to float. with pytest.warns(UserWarning): study.tell(study.ask(), None) assert len(study.trials) == 2 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Check number of values. with pytest.warns(UserWarning): study.tell(study.ask(), []) assert len(study.trials) == 3 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Check wrong number of values, e.g. two values for single direction. with pytest.warns(UserWarning): study.tell(study.ask(), [1.0, 2.0]) assert len(study.trials) == 4 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Both state and values are not specified. with pytest.warns(UserWarning): study.tell(study.ask()) assert len(study.trials) == 5 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None # Nan is not supported. with pytest.warns(UserWarning): study.tell(study.ask(), float("nan")) assert len(study.trials) == 6 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None def test_tell_multi_objective() -> None: study = create_study(directions=["minimize", "maximize"]) study.tell(study.ask(), [1.0, 2.0]) assert len(study.trials) == 1 def test_tell_multi_objective_automatically_fail() -> None: # Number of values doesn't match the length of directions. study = create_study(directions=["minimize", "maximize"]) with pytest.warns(UserWarning): study.tell(study.ask(), []) assert len(study.trials) == 1 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [1.0]) assert len(study.trials) == 2 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [1.0, 2.0, 3.0]) assert len(study.trials) == 3 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [1.0, None]) # type: ignore assert len(study.trials) == 4 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), [None, None]) # type: ignore assert len(study.trials) == 5 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None with pytest.warns(UserWarning): study.tell(study.ask(), 1.0) assert len(study.trials) == 6 assert study.trials[-1].state == TrialState.FAIL assert study.trials[-1].values is None def test_tell_invalid() -> None: study = create_study() # Missing values for completions. with pytest.raises(ValueError): study.tell(study.ask(), state=TrialState.COMPLETE) # Invalid values for completions. with pytest.raises(ValueError): study.tell(study.ask(), "a", state=TrialState.COMPLETE) # type: ignore with pytest.raises(ValueError): study.tell(study.ask(), None, state=TrialState.COMPLETE) with pytest.raises(ValueError): study.tell(study.ask(), [], state=TrialState.COMPLETE) with pytest.raises(ValueError): study.tell(study.ask(), [1.0, 2.0], state=TrialState.COMPLETE) with pytest.raises(ValueError): study.tell(study.ask(), float("nan"), state=TrialState.COMPLETE) # `state` must be None or finished state. with pytest.raises(ValueError): study.tell(study.ask(), state=TrialState.RUNNING) # `state` must be None or finished state. with pytest.raises(ValueError): study.tell(study.ask(), state=TrialState.WAITING) # `value` must be None for `TrialState.PRUNED`. with pytest.raises(ValueError): study.tell(study.ask(), values=1, state=TrialState.PRUNED) # `value` must be None for `TrialState.FAIL`. with pytest.raises(ValueError): study.tell(study.ask(), values=1, state=TrialState.FAIL) # Trial that has not been asked for cannot be told. with pytest.raises(ValueError): study.tell(study.ask().number + 1, 1.0) # Waiting trial cannot be told. with pytest.raises(ValueError): study.enqueue_trial({}) study.tell(study.trials[-1].number, 1.0) # It must be Trial or int for trial. with pytest.raises(TypeError): study.tell("1", 1.0) # type: ignore def test_tell_duplicate_tell() -> None: study = create_study() trial = study.ask() study.tell(trial, 1.0) # Should not panic when passthrough is enabled. study.tell(trial, 1.0, skip_if_finished=True) with pytest.raises(ValueError): study.tell(trial, 1.0, skip_if_finished=False) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_enqueued_trial_datetime_start(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) def objective(trial: Trial) -> float: time.sleep(1) x = trial.suggest_int("x", -10, 10) return x study.enqueue_trial(params={"x": 1}) assert study.trials[0].datetime_start is None study.optimize(objective, n_trials=1) assert study.trials[0].datetime_start is not None @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_study_summary_datetime_start_calculation(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: def objective(trial: Trial) -> float: x = trial.suggest_int("x", -10, 10) return x # StudySummary datetime_start tests. study = create_study(storage=storage) study.enqueue_trial(params={"x": 1}) # Study summary with only enqueued trials should have null datetime_start. summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert summaries[0].datetime_start is None # Study summary with completed trials should have nonnull datetime_start. study.optimize(objective, n_trials=1) study.enqueue_trial(params={"x": 1}, skip_if_exists=False) summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert summaries[0].datetime_start is not None def _process_tell(study: Study, trial: Trial | int, values: float) -> None: study.tell(trial, values) def test_tell_from_another_process() -> None: pool = multiprocessing.Pool() with StorageSupplier("sqlite") as storage: # Create a study and ask for a new trial. study = create_study(storage=storage) trial0 = study.ask() # Test normal behaviour. pool.starmap(_process_tell, [(study, trial0, 1.2)]) assert len(study.trials) == 1 assert study.best_trial.state == TrialState.COMPLETE assert study.best_value == 1.2 # Test study.tell using trial number. trial = study.ask() pool.starmap(_process_tell, [(study, trial.number, 1.5)]) assert len(study.trials) == 2 assert study.best_trial.state == TrialState.COMPLETE assert study.best_value == 1.2 # Should fail because the trial0 is already finished. with pytest.raises(ValueError): pool.starmap(_process_tell, [(study, trial0, 1.2)]) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_pop_waiting_trial_thread_safe(storage_mode: str) -> None: if "sqlite" == storage_mode or "cached_sqlite" == storage_mode: pytest.skip("study._pop_waiting_trial is not thread-safe on SQLite3") num_enqueued = 10 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) for i in range(num_enqueued): study.enqueue_trial({"i": i}) trial_id_set = set() with ThreadPoolExecutor(10) as pool: futures = [] for i in range(num_enqueued): future = pool.submit(study._pop_waiting_trial_id) futures.append(future) for future in as_completed(futures): trial_id_set.add(future.result()) assert len(trial_id_set) == num_enqueued def test_set_metric_names() -> None: metric_names = ["v0", "v1"] study = create_study(directions=["minimize", "minimize"]) study.set_metric_names(metric_names) got_metric_names = study._storage.get_study_system_attrs(study._study_id).get( _SYSTEM_ATTR_METRIC_NAMES ) assert got_metric_names is not None assert metric_names == got_metric_names def test_set_metric_names_experimental_warning() -> None: study = create_study() with pytest.warns(ExperimentalWarning): study.set_metric_names(["v0"]) def test_set_invalid_metric_names() -> None: metric_names = ["v0", "v1", "v2"] study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): study.set_metric_names(metric_names) def test_get_metric_names() -> None: study = create_study() assert study.metric_names is None study.set_metric_names(["v0"]) assert study.metric_names == ["v0"] study.set_metric_names(["v1"]) assert study.metric_names == ["v1"] optuna-3.5.0/tests/study_tests/test_study_summary.py000066400000000000000000000026471453453102400231310ustar00rootroot00000000000000import copy import pytest from optuna import create_study from optuna import get_all_study_summaries from optuna.storages import RDBStorage def test_study_summary_eq_ne() -> None: storage = RDBStorage("sqlite:///:memory:") create_study(storage=storage) study = create_study(storage=storage) summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert len(summaries) == 2 assert summaries[0] == copy.deepcopy(summaries[0]) assert not summaries[0] != copy.deepcopy(summaries[0]) assert not summaries[0] == summaries[1] assert summaries[0] != summaries[1] assert not summaries[0] == 1 assert summaries[0] != 1 def test_study_summary_lt_le() -> None: storage = RDBStorage("sqlite:///:memory:") create_study(storage=storage) study = create_study(storage=storage) summaries = get_all_study_summaries(study._storage, include_best_trial=True) assert len(summaries) == 2 summary_0 = summaries[0] summary_1 = summaries[1] assert summary_0 < summary_1 assert not summary_1 < summary_0 with pytest.raises(TypeError): summary_0 < 1 assert summary_0 <= summary_0 assert not summary_1 <= summary_0 with pytest.raises(TypeError): summary_0 <= 1 # A list of StudySummaries is sortable. summaries.reverse() summaries.sort() assert summaries[0] == summary_0 assert summaries[1] == summary_1 optuna-3.5.0/tests/terminator_tests/000077500000000000000000000000001453453102400175765ustar00rootroot00000000000000optuna-3.5.0/tests/terminator_tests/improvement_tests/000077500000000000000000000000001453453102400233655ustar00rootroot00000000000000optuna-3.5.0/tests/terminator_tests/improvement_tests/gp_tests/000077500000000000000000000000001453453102400252155ustar00rootroot00000000000000optuna-3.5.0/tests/terminator_tests/improvement_tests/gp_tests/test_botorch.py000066400000000000000000000054101453453102400302660ustar00rootroot00000000000000from optuna.distributions import FloatDistribution from optuna.terminator.improvement.gp.botorch import _BoTorchGaussianProcess from optuna.trial import create_trial def test_fit_predict() -> None: # A typical fit-predict scenario is being tested here, where there are more than one trials # and the Gram matrix is a regular one. trials = [ create_trial( value=1.0, distributions={ "bacon": FloatDistribution(-1.0, 1.0), "egg": FloatDistribution(-1.0, 1.0), }, params={ "bacon": 1.0, "egg": 0.0, }, ), create_trial( value=-1.0, distributions={ "bacon": FloatDistribution(-1.0, 1.0), "egg": FloatDistribution(-1.0, 1.0), }, params={ "bacon": 0.0, "egg": 1.0, }, ), ] gp = _BoTorchGaussianProcess() gp.fit(trials) gp.predict_mean_std(trials) def test_fit_predict_single_trial() -> None: trials = [ create_trial( value=1.0, distributions={ "bacon": FloatDistribution(-1.0, 1.0), "egg": FloatDistribution(-1.0, 1.0), }, params={ "bacon": 1.0, "egg": 0.0, }, ), ] gp = _BoTorchGaussianProcess() gp.fit(trials) gp.predict_mean_std(trials) def test_fit_predict_single_param() -> None: trials = [ create_trial( value=1.0, distributions={ "spam": FloatDistribution(-1.0, 1.0), }, params={ "spam": 1.0, }, ), ] gp = _BoTorchGaussianProcess() gp.fit(trials) gp.predict_mean_std(trials) def test_fit_predict_non_regular_gram_matrix() -> None: # This test case validates that the GP class works even when the Gram matrix is non-regular, # which typically orrcurs when multiple trials share the same parameters. trials = [ create_trial( value=1.0, distributions={ "bacon": FloatDistribution(-1.0, 1.0), "egg": FloatDistribution(-1.0, 1.0), }, params={ "bacon": 1.0, "egg": 0.0, }, ), create_trial( value=1.0, distributions={ "bacon": FloatDistribution(-1.0, 1.0), "egg": FloatDistribution(-1.0, 1.0), }, params={ "bacon": 1.0, "egg": 0.0, }, ), ] gp = _BoTorchGaussianProcess() gp.fit(trials) gp.predict_mean_std(trials) optuna-3.5.0/tests/terminator_tests/improvement_tests/test_evaluator.py000066400000000000000000000103201453453102400267740ustar00rootroot00000000000000from typing import List from typing import Tuple from unittest import mock import numpy as np import pytest from optuna.distributions import FloatDistribution from optuna.study import StudyDirection from optuna.terminator import BestValueStagnationEvaluator from optuna.terminator import RegretBoundEvaluator from optuna.terminator.improvement._preprocessing import NullPreprocessing from optuna.terminator.improvement.evaluator import BaseImprovementEvaluator from optuna.terminator.improvement.gp.base import _get_beta from optuna.terminator.improvement.gp.base import BaseGaussianProcess from optuna.trial import create_trial from optuna.trial import FrozenTrial class _StaticGaussianProcess(BaseGaussianProcess): """A dummy BaseGaussianProcess class always returning 0.0 mean and 1.0 std This class is introduced to make the GP class and the evaluator class loosely-coupled in unit testing. """ def fit( self, trials: List[FrozenTrial], ) -> None: pass def predict_mean_std(self, trials: List[FrozenTrial]) -> Tuple[np.ndarray, np.ndarray]: mean = np.zeros(len(trials)) std = np.ones(len(trials)) return mean, std # TODO(g-votte): test the following edge cases # TODO(g-votte): - the user specifies non-default top_trials_ratio or min_n_trials def test_regret_bound_evaluate() -> None: trials = [ create_trial( value=0, distributions={"a": FloatDistribution(-1.0, 1.0)}, params={"a": 0.0}, ) ] # The purpose of the following mock scope is to maintain loose coupling between the tests for # preprocessing and those for the `RegretBoundEvaluator` class. The preprocessing logic is # thoroughly tested in another file: # tests/terminator_tests/improvement_tests/test_preprocessing.py. with mock.patch.object( RegretBoundEvaluator, "get_preprocessing", return_value=NullPreprocessing() ): evaluator = RegretBoundEvaluator(gp=_StaticGaussianProcess()) regret_bound = evaluator.evaluate(trials, study_direction=StudyDirection.MAXIMIZE) assert regret_bound == 2.0 * np.sqrt(_get_beta(n_params=1, n_trials=len(trials))) def test_best_value_stagnation_invalid_argument() -> None: with pytest.raises(ValueError): # Test that a negative `max_stagnation_trials` raises ValueError. BestValueStagnationEvaluator(max_stagnation_trials=-1) def test_best_value_stagnation_evaluate() -> None: evaluator = BestValueStagnationEvaluator(max_stagnation_trials=1) # A case of monotonical improvement (best step is the latest element). trials = [create_trial(value=value) for value in [0, 1, 2]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) == 1 trials = [create_trial(value=value) for value in [2, 1, 0]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MINIMIZE) == 1 # A case of jagged improvement (best step is the second element). trials = [create_trial(value=value) for value in [0, 1, 0]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) == 0 trials = [create_trial(value=value) for value in [1, 0, 1]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MINIMIZE) == 0 # A case of flat improvement (best step is the first element). trials = [create_trial(value=value) for value in [0, 0, 0]] assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) == -1 assert evaluator.evaluate(trials=trials, study_direction=StudyDirection.MINIMIZE) == -1 @pytest.mark.parametrize("evaluator", [RegretBoundEvaluator(), BestValueStagnationEvaluator()]) def test_evaluate_with_no_trial(evaluator: BaseImprovementEvaluator) -> None: with pytest.raises(ValueError): evaluator.evaluate(trials=[], study_direction=StudyDirection.MAXIMIZE) def test_evaluate_with_empty_intersection_search_space() -> None: evaluator = RegretBoundEvaluator() trials = [ create_trial( value=0, distributions={}, params={}, ) ] with pytest.raises(ValueError): evaluator.evaluate(trials=trials, study_direction=StudyDirection.MAXIMIZE) optuna-3.5.0/tests/terminator_tests/improvement_tests/test_preprocessing.py000066400000000000000000000202511453453102400276610ustar00rootroot00000000000000from __future__ import annotations from unittest import mock import numpy as np import pytest from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.study._study_direction import StudyDirection from optuna.terminator.improvement import _preprocessing from optuna.trial import create_trial from optuna.trial import FrozenTrial def _get_trial_values(trials: list[FrozenTrial]) -> list[float]: values = [] for t in trials: assert t.value is not None values.append(t.value) return values @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) def test_preprocessing_pipeline(direction: StudyDirection) -> None: p1 = _preprocessing.NullPreprocessing() p2 = _preprocessing.NullPreprocessing() pipeline = _preprocessing.PreprocessingPipeline([p1, p2]) with mock.patch.object(p1, "apply") as a1: with mock.patch.object(p2, "apply") as a2: pipeline.apply([], direction) a1.assert_called_once() a2.assert_called_once() @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) def test_null_preprocessing(direction: StudyDirection) -> None: trials_before = [ create_trial(params={"x": 0.0}, distributions={"x": FloatDistribution(-1, 1)}, value=1.0) ] p = _preprocessing.NullPreprocessing() trials_after = p.apply(trials_before, direction) assert trials_before == trials_after @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) def test_unscale_log(direction: StudyDirection) -> None: trials_before = [ create_trial( params={ "categorical": 0, "float": 1, "float_log": 2, "int": 3, "int_log": 4, }, distributions={ "categorical": CategoricalDistribution((0, 1, 2)), "float": FloatDistribution(1, 10), "float_log": FloatDistribution(1, 10, log=True), "int": IntDistribution(1, 10), "int_log": IntDistribution(1, 10, log=True), }, value=1.0, ), ] p = _preprocessing.UnscaleLog() trials_after = p.apply(trials_before, direction) assert len(trials_before) == len(trials_after) == 1 assert trials_before[0].value == trials_after[0].value # Assert that the trial has no log distribution. for d in trials_after[0].distributions: if isinstance(d, (IntDistribution, FloatDistribution)): assert not d.log # Assert that the parameters and distributions are identical for the non-log distributions. for name in ["float", "int", "categorical"]: assert trials_before[0].params[name] == trials_after[0].params[name] assert trials_before[0].distributions[name] == trials_after[0].distributions[name] # Assert that the parameters and distributions are unscaled for the log distributions. for name in ["float_log", "int_log"]: assert np.log(trials_before[0].params[name]) == trials_after[0].params[name] dist_before = trials_before[0].distributions[name] dist_after = trials_after[0].distributions[name] assert isinstance(dist_before, (IntDistribution, FloatDistribution)) assert isinstance(dist_after, (IntDistribution, FloatDistribution)) assert np.log(dist_before.low) == dist_after.low assert np.log(dist_before.high) == dist_after.high @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) @pytest.mark.parametrize( "top_trials_ratio,min_n_trials,n_trials_expected", [ (0.4, 3, 3), # `n_trials` * `top_trials_ratio` is less than `min_n_trials`. (0.8, 3, 4), # `n_trials` * `top_trials_ratio` is greater than `min_n_trials`. ], ) def test_select_top_trials( direction: StudyDirection, top_trials_ratio: float, min_n_trials: int, n_trials_expected: int, ) -> None: values = [1, 0, 0, 2, 1] trials_before = [create_trial(value=v) for v in values] values_in_order = sorted(values, reverse=(direction == StudyDirection.MAXIMIZE)) p = _preprocessing.SelectTopTrials( top_trials_ratio=top_trials_ratio, min_n_trials=min_n_trials, ) trials_after = p.apply(trials_before, direction) assert len(trials_after) == n_trials_expected assert _get_trial_values(trials_after) == values_in_order[:n_trials_expected] @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) def test_to_minimize(direction: StudyDirection) -> None: values = [1, 0, 0, 2, 1] trials_before = [create_trial(value=v) for v in values] expected_values = values[:] if direction == StudyDirection.MAXIMIZE: expected_values = [-1 * v for v in expected_values] p = _preprocessing.ToMinimize() trials_after = p.apply(trials_before, direction) assert _get_trial_values(trials_after) == expected_values @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) def test_one_to_hot(direction: StudyDirection) -> None: # Scenario: categorical parameters are mapped to the float distribution. trials_before = [ create_trial( params={"categorical": 0}, distributions={"categorical": CategoricalDistribution((0, 1))}, value=1.0, ), create_trial( params={"categorical": 1}, distributions={"categorical": CategoricalDistribution((0, 1))}, value=1.0, ), ] p = _preprocessing.OneToHot() trials_after = p.apply(trials_before, direction) for t in trials_after: # Distributions are expected to be mapped to `FloatDistribution(0, 1)` for each choice. assert t.distributions == { "i0_categorical": FloatDistribution(0, 1), "i1_categorical": FloatDistribution(0, 1), } assert len(trials_after) == 2 assert trials_after[0].params == {"i0_categorical": 1.0, "i1_categorical": 0.0} assert trials_after[1].params == {"i0_categorical": 0.0, "i1_categorical": 1.0} # Scenario: int and float parameters are identical even after the preprocessing. trials_before = [ create_trial( params={ "float": 1, "int": 3, }, distributions={ "float": FloatDistribution(1, 10), "int": IntDistribution(1, 10), }, value=1.0, ), ] p = _preprocessing.OneToHot() trials_after = p.apply(trials_before, direction) assert len(trials_after) == 1 # Note that the param/distribution names are modified with "i0_" prefix to be consistent # with categorical parameters. assert trials_after[0].params == { "i0_float": 1, "i0_int": 3, } assert trials_after[0].distributions == { "i0_float": FloatDistribution(1, 10), "i0_int": IntDistribution(1, 10), } @pytest.mark.parametrize("direction", (StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE)) def test_add_random_inputs(direction: StudyDirection) -> None: n_additional_trials = 3 dummy_value = -1 distributions = { "bacon": CategoricalDistribution((0, 1, 2)), "egg": FloatDistribution(1, 10), "spam": IntDistribution(1, 10), } trials_before = [ create_trial( params={"bacon": 0, "egg": 1, "spam": 10}, distributions=distributions, value=1.0, ), ] p = _preprocessing.AddRandomInputs( n_additional_trials=n_additional_trials, dummy_value=dummy_value, ) trials_after = p.apply(trials_before, direction) assert len(trials_after) == len(trials_before) + n_additional_trials assert trials_before[0] == trials_after[0] for t in trials_after[1:]: assert t.value == dummy_value assert t.distributions == distributions assert set(t.params.keys()) == set(distributions.keys()) for name, distribution in distributions.items(): assert distribution._contains(t.params[name]) optuna-3.5.0/tests/terminator_tests/test_callback.py000066400000000000000000000022441453453102400227450ustar00rootroot00000000000000from optuna.study import create_study from optuna.study import Study from optuna.terminator import BaseTerminator from optuna.terminator import TerminatorCallback from optuna.trial import TrialState class _DeterministicTerminator(BaseTerminator): def __init__(self, termination_trial_number: int) -> None: self._termination_trial_number = termination_trial_number def should_terminate(self, study: Study) -> bool: trials = study.get_trials(states=[TrialState.COMPLETE]) latest_number = max([t.number for t in trials]) if latest_number >= self._termination_trial_number: return True else: return False def test_terminator_callback_terminator() -> None: # This test case validates that the study is stopped when the `should_terminate` method of the # terminator returns `True` for the first time. termination_trial_number = 10 callback = TerminatorCallback( terminator=_DeterministicTerminator(termination_trial_number), ) study = create_study() study.optimize(lambda _: 0.0, callbacks=[callback], n_trials=100) assert len(study.trials) == termination_trial_number + 1 optuna-3.5.0/tests/terminator_tests/test_erroreval.py000066400000000000000000000056441453453102400232210ustar00rootroot00000000000000from __future__ import annotations import math import pytest from optuna.study.study import create_study from optuna.terminator import CrossValidationErrorEvaluator from optuna.terminator import report_cross_validation_scores from optuna.terminator import StaticErrorEvaluator from optuna.terminator.erroreval import _CROSS_VALIDATION_SCORES_KEY from optuna.trial import create_trial from optuna.trial import FrozenTrial def _create_trial(value: float, cv_scores: list[float]) -> FrozenTrial: return create_trial( params={}, distributions={}, value=value, system_attrs={_CROSS_VALIDATION_SCORES_KEY: cv_scores}, ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_cross_validation_evaluator(direction: str) -> None: study = create_study(direction=direction) sign = 1 if direction == "minimize" else -1 study.add_trials( [ _create_trial( value=sign * 2.0, cv_scores=[1.0, -1.0] ), # Second best trial with 1.0 var. _create_trial(value=sign * 1.0, cv_scores=[2.0, -2.0]), # Best trial with 4.0 var. ] ) evaluator = CrossValidationErrorEvaluator() serror = evaluator.evaluate(study.trials, study.direction) expected_scale = 1.5 assert serror == math.sqrt(4.0 * expected_scale) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_cross_validation_evaluator_without_cv_scores(direction: str) -> None: study = create_study(direction=direction) study.add_trial( # Note that the CV score is not reported with the system attr. create_trial(params={}, distributions={}, value=0.0) ) evaluator = CrossValidationErrorEvaluator() with pytest.raises(ValueError): evaluator.evaluate(study.trials, study.direction) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_report_cross_validation_scores(direction: str) -> None: scores = [1.0, 2.0] study = create_study(direction=direction) trial = study.ask() report_cross_validation_scores(trial, scores) study.tell(trial, 0.0) assert study.trials[0].system_attrs[_CROSS_VALIDATION_SCORES_KEY] == scores @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_report_cross_validation_scores_with_illegal_scores_length(direction: str) -> None: scores = [1.0] study = create_study(direction=direction) trial = study.ask() with pytest.raises(ValueError): report_cross_validation_scores(trial, scores) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_static_evaluator(direction: str) -> None: study = create_study(direction=direction) study.add_trials( [ _create_trial(value=2.0, cv_scores=[1.0, -1.0]), ] ) evaluator = StaticErrorEvaluator(constant=100.0) serror = evaluator.evaluate(study.trials, study.direction) assert serror == 100.0 optuna-3.5.0/tests/terminator_tests/test_terminator.py000066400000000000000000000044251453453102400234000ustar00rootroot00000000000000from __future__ import annotations import pytest from optuna.study._study_direction import StudyDirection from optuna.study.study import create_study from optuna.terminator import BaseImprovementEvaluator from optuna.terminator import StaticErrorEvaluator from optuna.terminator import Terminator from optuna.terminator.improvement.evaluator import BestValueStagnationEvaluator from optuna.trial import FrozenTrial class _StaticImprovementEvaluator(BaseImprovementEvaluator): def __init__(self, constant: float) -> None: super().__init__() self._constant = constant def evaluate(self, trials: list[FrozenTrial], study_direction: StudyDirection) -> float: return self._constant def test_init() -> None: # Test that a positive `min_n_trials` does not raise any error. Terminator(min_n_trials=1) with pytest.raises(ValueError): # Test that a non-positive `min_n_trials` raises ValueError. Terminator(min_n_trials=0) Terminator(BestValueStagnationEvaluator(), min_n_trials=1) def test_should_terminate() -> None: study = create_study() # Add dummy trial because Terminator needs at least one trial. trial = study.ask() study.tell(trial, 0.0) # Improvement is greater than error. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(3), error_evaluator=StaticErrorEvaluator(0), min_n_trials=1, ) assert not terminator.should_terminate(study) # Improvement is less than error. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(-1), error_evaluator=StaticErrorEvaluator(0), min_n_trials=1, ) assert terminator.should_terminate(study) # Improvement is less than error. However, the number of trials is less than `min_n_trials`. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(-1), error_evaluator=StaticErrorEvaluator(0), min_n_trials=2, ) assert not terminator.should_terminate(study) # Improvement is equal to error. terminator = Terminator( improvement_evaluator=_StaticImprovementEvaluator(0), error_evaluator=StaticErrorEvaluator(0), min_n_trials=1, ) assert not terminator.should_terminate(study) optuna-3.5.0/tests/test_callbacks.py000066400000000000000000000013001453453102400175120ustar00rootroot00000000000000from optuna import create_study from optuna.study import MaxTrialsCallback from optuna.testing.objectives import pruned_objective from optuna.trial import TrialState def test_stop_with_MaxTrialsCallback() -> None: # Test stopping the optimization with MaxTrialsCallback. study = create_study() study.optimize(lambda _: 1.0, n_trials=10, callbacks=[MaxTrialsCallback(5)]) assert len(study.trials) == 5 # Test stopping the optimization with MaxTrialsCallback with pruned trials study = create_study() study.optimize( pruned_objective, n_trials=10, callbacks=[MaxTrialsCallback(5, states=(TrialState.PRUNED,))], ) assert len(study.trials) == 5 optuna-3.5.0/tests/test_cli.py000066400000000000000000001666711453453102400163710ustar00rootroot00000000000000import json import os import platform import re import subprocess from subprocess import CalledProcessError import tempfile from typing import Any from typing import Callable from typing import Optional from typing import Tuple from unittest.mock import MagicMock from unittest.mock import patch import fakeredis import numpy as np from pandas import Timedelta from pandas import Timestamp import pytest import yaml import optuna import optuna.cli from optuna.exceptions import CLIUsageError from optuna.exceptions import ExperimentalWarning from optuna.storages import JournalFileStorage from optuna.storages import JournalRedisStorage from optuna.storages import JournalStorage from optuna.storages import RDBStorage from optuna.storages._base import DEFAULT_STUDY_NAME_PREFIX from optuna.study import StudyDirection from optuna.testing.storages import StorageSupplier from optuna.testing.tempfile_pool import NamedTemporaryFilePool from optuna.trial import Trial from optuna.trial import TrialState # An example of objective functions def objective_func(trial: Trial) -> float: x = trial.suggest_float("x", -10, 10) return (x + 5) ** 2 # An example of objective functions for branched search spaces def objective_func_branched_search_space(trial: Trial) -> float: c = trial.suggest_categorical("c", ("A", "B")) if c == "A": x = trial.suggest_float("x", -10, 10) return (x + 5) ** 2 else: y = trial.suggest_float("y", -10, 10) return (y + 5) ** 2 # An example of objective functions for multi-objective optimization def objective_func_multi_objective(trial: Trial) -> Tuple[float, float]: x = trial.suggest_float("x", -10, 10) return (x + 5) ** 2, (x - 5) ** 2 def _parse_output(output: str, output_format: str) -> Any: """Parse CLI output. Args: output: The output of command. output_format: The format of output specified by command. Returns: For table format, a list of dict formatted rows. For JSON or YAML format, a list or a dict corresponding to ``output``. """ if output_format == "value": # Currently, _parse_output with output_format="value" is used only for # `study-names` command. return [{"name": values} for values in output.split(os.linesep)] elif output_format == "table": rows = output.split(os.linesep) assert all(len(rows[0]) == len(row) for row in rows) # Check ruled lines. assert rows[0] == rows[2] == rows[-1] keys = [r.strip() for r in rows[1].split("|")[1:-1]] ret = [] for record in rows[3:-1]: attrs = {} for key, attr in zip(keys, record.split("|")[1:-1]): attrs[key] = attr.strip() ret.append(attrs) return ret elif output_format == "json": return json.loads(output) elif output_format == "yaml": return yaml.safe_load(output) else: assert False @pytest.mark.skip_coverage def test_create_study_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. command = ["optuna", "create-study", "--storage", storage_url] subprocess.check_call(command) # Command output should be in name string format (no-name + UUID). study_name = str(subprocess.check_output(command).decode().strip()) name_re = r"^no-name-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$" assert re.match(name_re, study_name) is not None # study_name should be stored in storage. study_id = storage.get_study_id_from_name(study_name) assert study_id == 2 @pytest.mark.skip_coverage def test_create_study_command_with_study_name() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" # Create study with name. command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] study_name = str(subprocess.check_output(command).decode().strip()) # Check if study_name is stored in the storage. study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_name_from_id(study_id) == study_name @pytest.mark.skip_coverage def test_create_study_command_without_storage_url() -> None: with pytest.raises(subprocess.CalledProcessError) as err: subprocess.check_output( ["optuna", "create-study"], env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) usage = err.value.output.decode() assert usage.startswith("usage:") @pytest.mark.skip_coverage def test_create_study_command_with_storage_env() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. command = ["optuna", "create-study"] env = {**os.environ, "OPTUNA_STORAGE": storage_url} subprocess.check_call(command, env=env) # Command output should be in name string format (no-name + UUID). study_name = str(subprocess.check_output(command, env=env).decode().strip()) name_re = r"^no-name-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$" assert re.match(name_re, study_name) is not None # study_name should be stored in storage. study_id = storage.get_study_id_from_name(study_name) assert study_id == 2 @pytest.mark.skip_coverage def test_create_study_command_with_direction() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) command = ["optuna", "create-study", "--storage", storage_url, "--direction", "minimize"] study_name = str(subprocess.check_output(command).decode().strip()) study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_directions(study_id) == [StudyDirection.MINIMIZE] command = ["optuna", "create-study", "--storage", storage_url, "--direction", "maximize"] study_name = str(subprocess.check_output(command).decode().strip()) study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_directions(study_id) == [StudyDirection.MAXIMIZE] command = ["optuna", "create-study", "--storage", storage_url, "--direction", "test"] # --direction should be either 'minimize' or 'maximize'. with pytest.raises(subprocess.CalledProcessError): subprocess.check_call(command) @pytest.mark.skip_coverage def test_create_study_command_with_multiple_directions() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) command = [ "optuna", "create-study", "--storage", storage_url, "--directions", "minimize", "maximize", ] study_name = str(subprocess.check_output(command).decode().strip()) study_id = storage.get_study_id_from_name(study_name) expected_directions = [StudyDirection.MINIMIZE, StudyDirection.MAXIMIZE] assert storage.get_study_directions(study_id) == expected_directions command = [ "optuna", "create-study", "--storage", storage_url, "--directions", "minimize", "maximize", "test", ] # Each direction in --directions should be either `minimize` or `maximize`. with pytest.raises(subprocess.CalledProcessError): subprocess.check_call(command) command = [ "optuna", "create-study", "--storage", storage_url, "--direction", "minimize", "--directions", "minimize", "maximize", "test", ] # It can't specify both --direction and --directions with pytest.raises(subprocess.CalledProcessError): subprocess.check_call(command) @pytest.mark.skip_coverage def test_delete_study_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "delete-study-test" # Create study. command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] subprocess.check_call(command) assert study_name in {s.study_name: s for s in storage.get_all_studies()} # Delete study. command = ["optuna", "delete-study", "--storage", storage_url, "--study-name", study_name] subprocess.check_call(command) assert study_name not in {s.study_name: s for s in storage.get_all_studies()} @pytest.mark.skip_coverage def test_delete_study_command_without_storage_url() -> None: with pytest.raises(subprocess.CalledProcessError): subprocess.check_output( ["optuna", "delete-study", "--study-name", "dummy_study"], env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) @pytest.mark.skip_coverage def test_study_set_user_attr_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. study_name = storage.get_study_name_from_id( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) base_command = [ "optuna", "study", "set-user-attr", "--study-name", study_name, "--storage", storage_url, ] example_attrs = {"architecture": "ResNet", "baselen_score": "0.002"} for key, value in example_attrs.items(): subprocess.check_call(base_command + ["--key", key, "--value", value]) # Attrs should be stored in storage. study_id = storage.get_study_id_from_name(study_name) study_user_attrs = storage.get_study_user_attrs(study_id) assert len(study_user_attrs) == 2 assert all(study_user_attrs[k] == v for k, v in example_attrs.items()) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_study_names_command(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) expected_study_names = ["study-names-test1", "study-names-test2"] expected_column_name = "name" # Create a study. command = [ "optuna", "create-study", "--storage", storage_url, "--study-name", expected_study_names[0], ] subprocess.check_output(command) # Get study names. command = ["optuna", "study-names", "--storage", storage_url] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) study_names = _parse_output(output, output_format or "value") # Check user_attrs are not printed. assert len(study_names) == 1 assert study_names[0]["name"] == expected_study_names[0] # Create another study. command = [ "optuna", "create-study", "--storage", storage_url, "--study-name", expected_study_names[1], ] subprocess.check_output(command) # Get study names. command = ["optuna", "study-names", "--storage", storage_url] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) study_names = _parse_output(output, output_format or "value") assert len(study_names) == 2 for i, study_name in enumerate(study_names): assert list(study_name.keys()) == [expected_column_name] assert study_name["name"] == expected_study_names[i] @pytest.mark.skip_coverage def test_study_names_command_without_storage_url() -> None: with pytest.raises(subprocess.CalledProcessError): subprocess.check_output( ["optuna", "study-names", "--study-name", "dummy_study"], env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_studies_command(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # First study. study_1 = optuna.create_study(storage=storage) # Run command. command = ["optuna", "studies", "--storage", storage_url] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") expected_keys = ["name", "direction", "n_trials", "datetime_start"] # Check user_attrs are not printed. if output_format is None or output_format == "table": assert list(studies[0].keys()) == expected_keys else: assert set(studies[0].keys()) == set(expected_keys) # Add a second study. study_2 = optuna.create_study( storage=storage, study_name="study_2", directions=["minimize", "maximize"] ) study_2.optimize(objective_func_multi_objective, n_trials=10) study_2.set_user_attr("key_1", "value_1") study_2.set_user_attr("key_2", "value_2") # Run command again to include second study. output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") expected_keys = ["name", "direction", "n_trials", "datetime_start", "user_attrs"] assert len(studies) == 2 for study in studies: if output_format is None or output_format == "table": assert list(study.keys()) == expected_keys else: assert set(study.keys()) == set(expected_keys) # Check study_name, direction, n_trials and user_attrs for the first study. assert studies[0]["name"] == study_1.study_name if output_format is None or output_format == "table": assert studies[0]["n_trials"] == "0" assert eval(studies[0]["direction"]) == ("MINIMIZE",) assert eval(studies[0]["user_attrs"]) == {} else: assert studies[0]["n_trials"] == 0 assert studies[0]["direction"] == ["MINIMIZE"] assert studies[0]["user_attrs"] == {} # Check study_name, direction, n_trials and user_attrs for the second study. assert studies[1]["name"] == study_2.study_name if output_format is None or output_format == "table": assert studies[1]["n_trials"] == "10" assert eval(studies[1]["direction"]) == ("MINIMIZE", "MAXIMIZE") assert eval(studies[1]["user_attrs"]) == {"key_1": "value_1", "key_2": "value_2"} else: assert studies[1]["n_trials"] == 10 assert studies[1]["direction"] == ["MINIMIZE", "MAXIMIZE"] assert studies[1]["user_attrs"] == {"key_1": "value_1", "key_2": "value_2"} @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_studies_command_flatten(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # First study. study_1 = optuna.create_study(storage=storage) # Run command. command = ["optuna", "studies", "--storage", storage_url, "--flatten"] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") expected_keys_1 = [ "name", "direction_0", "n_trials", "datetime_start", ] # Check user_attrs are not printed. if output_format is None or output_format == "table": assert list(studies[0].keys()) == expected_keys_1 else: assert set(studies[0].keys()) == set(expected_keys_1) # Add a second study. study_2 = optuna.create_study( storage=storage, study_name="study_2", directions=["minimize", "maximize"] ) study_2.optimize(objective_func_multi_objective, n_trials=10) study_2.set_user_attr("key_1", "value_1") study_2.set_user_attr("key_2", "value_2") # Run command again to include second study. output = str(subprocess.check_output(command).decode().strip()) studies = _parse_output(output, output_format or "table") if output_format is None or output_format == "table": expected_keys_1 = expected_keys_2 = [ "name", "direction_0", "direction_1", "n_trials", "datetime_start", "user_attrs", ] else: expected_keys_1 = ["name", "direction_0", "n_trials", "datetime_start", "user_attrs"] expected_keys_2 = [ "name", "direction_0", "direction_1", "n_trials", "datetime_start", "user_attrs", ] assert len(studies) == 2 if output_format is None or output_format == "table": assert list(studies[0].keys()) == expected_keys_1 assert list(studies[1].keys()) == expected_keys_2 else: assert set(studies[0].keys()) == set(expected_keys_1) assert set(studies[1].keys()) == set(expected_keys_2) # Check study_name, direction, n_trials and user_attrs for the first study. assert studies[0]["name"] == study_1.study_name if output_format is None or output_format == "table": assert studies[0]["n_trials"] == "0" assert studies[0]["user_attrs"] == "{}" else: assert studies[0]["n_trials"] == 0 assert studies[0]["user_attrs"] == {} assert studies[0]["direction_0"] == "MINIMIZE" # Check study_name, direction, n_trials and user_attrs for the second study. assert studies[1]["name"] == study_2.study_name if output_format is None or output_format == "table": assert studies[1]["n_trials"] == "10" assert studies[1]["user_attrs"] == "{'key_1': 'value_1', 'key_2': 'value_2'}" else: assert studies[1]["n_trials"] == 10 assert studies[1]["user_attrs"] == {"key_1": "value_1", "key_2": "value_2"} assert studies[1]["direction_0"] == "MINIMIZE" assert studies[1]["direction_1"] == "MAXIMIZE" @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_trials_command(objective: Callable[[Trial], float], output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "trials", "--storage", storage_url, "--study-name", study_name, ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") assert len(trials) == n_trials df = study.trials_dataframe(attrs, multi_index=True) for i, trial in enumerate(trials): for key in df.columns: expected_value = df.loc[i][key] # The param may be NaN when the objective function has branched search space. if ( key[0] == "params" and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert key[1] not in eval(trial["params"]) else: assert key[1] not in trial["params"] continue if key[1] == "": value = trial[key[0]] else: if output_format is None or output_format == "table": value = eval(trial[key[0]])[key[1]] else: value = trial[key[0]][key[1]] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_trials_command_flatten( objective: Callable[[Trial], float], output_format: Optional[str] ) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "trials", "--storage", storage_url, "--study-name", study_name, "--flatten", ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") assert len(trials) == n_trials df = study.trials_dataframe(attrs) for i, trial in enumerate(trials): assert set(trial.keys()) <= set(df.columns) for key in df.columns: expected_value = df.loc[i][key] # The param may be NaN when the objective function has branched search space. if ( key.startswith("params_") and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert trial[key] == "" else: assert key not in trial continue value = trial[key] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trial_command( objective: Callable[[Trial], float], output_format: Optional[str] ) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trial", "--storage", storage_url, "--study-name", study_name, ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) best_trial = _parse_output(output, output_format or "table") if output_format is None or output_format == "table": assert len(best_trial) == 1 best_trial = best_trial[0] df = study.trials_dataframe(attrs, multi_index=True) for key in df.columns: expected_value = df.loc[study.best_trial.number][key] # The param may be NaN when the objective function has branched search space. if ( key[0] == "params" and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert key[1] not in eval(best_trial["params"]) else: assert key[1] not in best_trial["params"] continue if key[1] == "": value = best_trial[key[0]] else: if output_format is None or output_format == "table": value = eval(best_trial[key[0]])[key[1]] else: value = best_trial[key[0]][key[1]] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("objective", (objective_func, objective_func_branched_search_space)) @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trial_command_flatten( objective: Callable[[Trial], float], output_format: Optional[str] ) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study(storage=storage, study_name=study_name) study.optimize(objective, n_trials=n_trials) attrs = ( "number", "value", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trial", "--storage", storage_url, "--study-name", study_name, "--flatten", ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) best_trial = _parse_output(output, output_format or "table") if output_format is None or output_format == "table": assert len(best_trial) == 1 best_trial = best_trial[0] df = study.trials_dataframe(attrs) assert set(best_trial.keys()) <= set(df.columns) for key in df.columns: expected_value = df.loc[study.best_trial.number][key] # The param may be NaN when the objective function has branched search space. if ( key.startswith("params_") and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert best_trial[key] == "" else: assert key not in best_trial continue value = best_trial[key] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trials_command(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study( storage=storage, study_name=study_name, directions=("minimize", "minimize") ) study.optimize(objective_func_multi_objective, n_trials=n_trials) attrs = ( "number", "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trials", "--storage", storage_url, "--study-name", study_name, ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") best_trials = [trial.number for trial in study.best_trials] assert len(trials) == len(best_trials) df = study.trials_dataframe(attrs, multi_index=True) for trial in trials: number = int(trial["number"]) if output_format in (None, "table") else trial["number"] assert number in best_trials for key in df.columns: expected_value = df.loc[number][key] # The param may be NaN when the objective function has branched search space. if ( key[0] == "params" and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert key[1] not in eval(trial["params"]) else: assert key[1] not in trial["params"] continue if key[1] == "": value = trial[key[0]] else: if output_format is None or output_format == "table": value = eval(trial[key[0]])[key[1]] else: value = trial[key[0]][key[1]] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_best_trials_command_flatten(output_format: Optional[str]) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" n_trials = 10 study = optuna.create_study( storage=storage, study_name=study_name, directions=("minimize", "minimize") ) study.optimize(objective_func_multi_objective, n_trials=n_trials) attrs = ( "number", "values", "datetime_start", "datetime_complete", "duration", "params", "user_attrs", "state", ) # Run command. command = [ "optuna", "best-trials", "--storage", storage_url, "--study-name", study_name, "--flatten", ] if output_format is not None: command += ["--format", output_format] output = str(subprocess.check_output(command).decode().strip()) trials = _parse_output(output, output_format or "table") best_trials = [trial.number for trial in study.best_trials] assert len(trials) == len(best_trials) df = study.trials_dataframe(attrs) for trial in trials: assert set(trial.keys()) <= set(df.columns) number = int(trial["number"]) if output_format in (None, "table") else trial["number"] for key in df.columns: expected_value = df.loc[number][key] # The param may be NaN when the objective function has branched search space. if ( key.startswith("params_") and isinstance(expected_value, float) and np.isnan(expected_value) ): if output_format is None or output_format == "table": assert trial[key] == "" else: assert key not in trial continue value = trial[key] if isinstance(value, (int, float)): if np.isnan(expected_value): assert np.isnan(value) else: assert value == expected_value elif isinstance(expected_value, Timestamp): assert value == expected_value.strftime("%Y-%m-%d %H:%M:%S") elif isinstance(expected_value, Timedelta): assert value == str(expected_value.to_pytimedelta()) else: assert value == str(expected_value) @pytest.mark.skip_coverage def test_create_study_command_with_skip_if_exists() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = "test_study" # Create study with name. command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] study_name = str(subprocess.check_output(command).decode().strip()) # Check if study_name is stored in the storage. study_id = storage.get_study_id_from_name(study_name) assert storage.get_study_name_from_id(study_id) == study_name # Try to create the same name study without `--skip-if-exists` flag (error). command = ["optuna", "create-study", "--storage", storage_url, "--study-name", study_name] with pytest.raises(subprocess.CalledProcessError): subprocess.check_output(command) # Try to create the same name study with `--skip-if-exists` flag (OK). command = [ "optuna", "create-study", "--storage", storage_url, "--study-name", study_name, "--skip-if-exists", ] study_name = str(subprocess.check_output(command).decode().strip()) new_study_id = storage.get_study_id_from_name(study_name) assert study_id == new_study_id # The existing study instance is reused. @pytest.mark.skip_coverage def test_study_optimize_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) study_name = storage.get_study_name_from_id( storage.create_new_study(directions=[StudyDirection.MINIMIZE]) ) command = [ "optuna", "study", "optimize", "--study-name", study_name, "--n-trials", "10", __file__, "objective_func", "--storage", storage_url, ] subprocess.check_call(command) study = optuna.load_study(storage=storage_url, study_name=study_name) assert len(study.trials) == 10 assert "x" in study.best_params # Check if a default value of study_name is stored in the storage. assert storage.get_study_name_from_id(study._study_id).startswith( DEFAULT_STUDY_NAME_PREFIX ) @pytest.mark.skip_coverage def test_study_optimize_command_inconsistent_args() -> None: with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) # --study-name argument is missing. with pytest.raises(subprocess.CalledProcessError): subprocess.check_call( [ "optuna", "study", "optimize", "--storage", db_url, "--n-trials", "10", __file__, "objective_func", ] ) @pytest.mark.skip_coverage def test_empty_argv() -> None: command_empty = ["optuna"] command_empty_output = str(subprocess.check_output(command_empty)) command_help = ["optuna", "help"] command_help_output = str(subprocess.check_output(command_help)) assert command_empty_output == command_help_output def test_check_storage_url() -> None: storage_in_args = "sqlite:///args.db" assert storage_in_args == optuna.cli._check_storage_url(storage_in_args) with pytest.warns(ExperimentalWarning): with patch.dict("optuna.cli.os.environ", {"OPTUNA_STORAGE": "sqlite:///args.db"}): optuna.cli._check_storage_url(None) with pytest.raises(CLIUsageError): optuna.cli._check_storage_url(None) @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @patch("optuna.storages._journal.redis.redis") def test_get_storage_without_storage_class(mock_redis: MagicMock) -> None: with tempfile.NamedTemporaryFile(suffix=".db") as fp: storage = optuna.cli._get_storage(f"sqlite:///{fp.name}", storage_class=None) assert isinstance(storage, RDBStorage) with tempfile.NamedTemporaryFile(suffix=".log") as fp: storage = optuna.cli._get_storage(fp.name, storage_class=None) assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalFileStorage) mock_redis.Redis = fakeredis.FakeRedis storage = optuna.cli._get_storage("redis://localhost:6379", storage_class=None) assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalRedisStorage) with pytest.raises(CLIUsageError): optuna.cli._get_storage("./file-not-found.log", storage_class=None) @pytest.mark.skipif(platform.system() == "Windows", reason="Skip on Windows") @patch("optuna.storages._journal.redis.redis") def test_get_storage_with_storage_class(mock_redis: MagicMock) -> None: with tempfile.NamedTemporaryFile(suffix=".db") as fp: storage = optuna.cli._get_storage(f"sqlite:///{fp.name}", storage_class=None) assert isinstance(storage, RDBStorage) with tempfile.NamedTemporaryFile(suffix=".log") as fp: storage = optuna.cli._get_storage(fp.name, storage_class="JournalFileStorage") assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalFileStorage) mock_redis.Redis = fakeredis.FakeRedis storage = optuna.cli._get_storage( "redis:///localhost:6379", storage_class="JournalRedisStorage" ) assert isinstance(storage, JournalStorage) assert isinstance(storage._backend, JournalRedisStorage) with pytest.raises(CLIUsageError): with tempfile.NamedTemporaryFile(suffix=".db") as fp: optuna.cli._get_storage(f"sqlite:///{fp.name}", storage_class="InMemoryStorage") @pytest.mark.skip_coverage def test_storage_upgrade_command() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) command = ["optuna", "storage", "upgrade"] with pytest.raises(CalledProcessError): subprocess.check_call( command, env={k: v for k, v in os.environ.items() if k != "OPTUNA_STORAGE"}, ) command.extend(["--storage", storage_url]) subprocess.check_call(command) @pytest.mark.skip_coverage def test_storage_upgrade_command_with_invalid_url() -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) command = ["optuna", "storage", "upgrade", "--storage", "invalid-storage-url"] with pytest.raises(CalledProcessError): subprocess.check_call(command) @pytest.mark.skip_coverage @pytest.mark.parametrize( "direction,directions,sampler,sampler_kwargs,output_format", [ (None, None, None, None, None), ("minimize", None, None, None, None), (None, "minimize maximize", None, None, None), (None, None, "RandomSampler", None, None), (None, None, "TPESampler", '{"multivariate": true}', None), (None, None, None, None, "json"), (None, None, None, None, "yaml"), ], ) def test_ask( direction: Optional[str], directions: Optional[str], sampler: Optional[str], sampler_kwargs: Optional[str], output_format: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, ] if direction is not None: args += ["--direction", direction] if directions is not None: args += ["--directions"] + directions.split() if sampler is not None: args += ["--sampler", sampler] if sampler_kwargs is not None: args += ["--sampler-kwargs", sampler_kwargs] if output_format is not None: args += ["--format", output_format] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = str(result.stdout.decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" params = eval(trial["params"]) assert len(params) == 2 assert 0 <= params["x"] <= 1 assert params["y"] == "foo" else: assert trial["number"] == 0 assert 0 <= trial["params"]["x"] <= 1 assert trial["params"]["y"] == "foo" if direction is not None or directions is not None: warning_message = result.stderr.decode() assert "FutureWarning" in warning_message @pytest.mark.skip_coverage @pytest.mark.parametrize( "direction,directions,sampler,sampler_kwargs,output_format", [ (None, None, None, None, None), ("minimize", None, None, None, None), (None, "minimize maximize", None, None, None), (None, None, "RandomSampler", None, None), (None, None, "TPESampler", '{"multivariate": true}', None), (None, None, None, None, "json"), (None, None, None, None, "yaml"), ], ) def test_ask_flatten( direction: Optional[str], directions: Optional[str], sampler: Optional[str], sampler_kwargs: Optional[str], output_format: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, "--flatten", ] if direction is not None: args += ["--direction", direction] if directions is not None: args += ["--directions"] + directions.split() if sampler is not None: args += ["--sampler", sampler] if sampler_kwargs is not None: args += ["--sampler-kwargs", sampler_kwargs] if output_format is not None: args += ["--format", output_format] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = str(result.stdout.decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" assert 0 <= float(trial["params_x"]) <= 1 assert trial["params_y"] == "foo" else: assert trial["number"] == 0 assert 0 <= trial["params_x"] <= 1 assert trial["params_y"] == "foo" if direction is not None or directions is not None: warning_message = result.stderr.decode() assert "FutureWarning" in warning_message @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_ask_empty_search_space(output_format: str) -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, ] if output_format is not None: args += ["--format", output_format] output = str(subprocess.check_output(args).decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" assert trial["params"] == "{}" else: assert trial["number"] == 0 assert trial["params"] == {} @pytest.mark.skip_coverage @pytest.mark.parametrize("output_format", (None, "table", "json", "yaml")) def test_ask_empty_search_space_flatten(output_format: str) -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--flatten", ] if output_format is not None: args += ["--format", output_format] output = str(subprocess.check_output(args).decode().strip()) trial = _parse_output(output, output_format or "json") if output_format == "table": assert len(trial) == 1 trial = trial[0] assert trial["number"] == "0" assert "params" not in trial else: assert trial["number"] == 0 assert "params" not in trial @pytest.mark.skip_coverage def test_ask_sampler_kwargs_without_sampler() -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, "--sampler-kwargs", '{"multivariate": true}', ] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert "`--sampler_kwargs` is set without `--sampler`." in error_message @pytest.mark.skip_coverage @pytest.mark.parametrize( "direction,directions,sampler,sampler_kwargs", [ (None, None, None, None), ("minimize", None, None, None), (None, "minimize maximize", None, None), (None, None, "RandomSampler", None), (None, None, "TPESampler", '{"multivariate": true}'), ], ) def test_create_study_and_ask( direction: Optional[str], directions: Optional[str], sampler: Optional[str], sampler_kwargs: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) create_study_args = [ "optuna", "create-study", "--storage", db_url, "--study-name", study_name, ] if direction is not None: create_study_args += ["--direction", direction] if directions is not None: create_study_args += ["--directions"] + directions.split() subprocess.check_call(create_study_args) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, ] if sampler is not None: args += ["--sampler", sampler] if sampler_kwargs is not None: args += ["--sampler-kwargs", sampler_kwargs] output = str(subprocess.check_output(args).decode().strip()) trial = _parse_output(output, "json") assert trial["number"] == 0 assert 0 <= trial["params"]["x"] <= 1 assert trial["params"]["y"] == "foo" @pytest.mark.skip_coverage @pytest.mark.parametrize( "direction,directions,ask_direction,ask_directions", [ (None, None, "maximize", None), ("minimize", None, "maximize", None), ("minimize", None, None, "minimize minimize"), (None, "minimize maximize", None, "maximize minimize"), (None, "minimize maximize", "minimize", None), ], ) def test_create_study_and_ask_with_inconsistent_directions( direction: Optional[str], directions: Optional[str], ask_direction: Optional[str], ask_directions: Optional[str], ) -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) create_study_args = [ "optuna", "create-study", "--storage", db_url, "--study-name", study_name, ] if direction is not None: create_study_args += ["--direction", direction] if directions is not None: create_study_args += ["--directions"] + directions.split() subprocess.check_call(create_study_args) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, ] if ask_direction is not None: args += ["--direction", ask_direction] if ask_directions is not None: args += ["--directions"] + ask_directions.split() result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert "Cannot overwrite study direction" in error_message @pytest.mark.skip_coverage def test_ask_with_both_direction_and_directions() -> None: study_name = "test_study" search_space = ( '{"x": {"name": "FloatDistribution", "attributes": {"low": 0.0, "high": 1.0}}, ' '"y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo"]}}}' ) with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) create_study_args = [ "optuna", "create-study", "--storage", db_url, "--study-name", study_name, ] subprocess.check_call(create_study_args) args = [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--search-space", search_space, "--direction", "minimize", "--directions", "minimize", ] result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert "Specify only one of `direction` and `directions`." in error_message @pytest.mark.skip_coverage def test_tell() -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) output: Any = subprocess.check_output( [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--format", "json", ] ) output = output.decode("utf-8") output = json.loads(output) trial_number = output["number"] subprocess.check_output( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "1.2", ] ) study = optuna.load_study(storage=db_url, study_name=study_name) assert len(study.trials) == 1 assert study.trials[0].state == TrialState.COMPLETE assert study.trials[0].values == [1.2] # Error when updating a finished trial. ret = subprocess.run( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "1.2", ] ) assert ret.returncode != 0 # Passing `--skip-if-finished` to a finished trial for a noop. subprocess.check_output( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "1.3", # Setting a different value and make sure it's not persisted. "--skip-if-finished", ] ) study = optuna.load_study(storage=db_url, study_name=study_name) assert len(study.trials) == 1 assert study.trials[0].state == TrialState.COMPLETE assert study.trials[0].values == [1.2] @pytest.mark.skip_coverage def test_tell_with_nan() -> None: study_name = "test_study" with NamedTemporaryFilePool() as tf: db_url = "sqlite:///{}".format(tf.name) output: Any = subprocess.check_output( [ "optuna", "ask", "--storage", db_url, "--study-name", study_name, "--format", "json", ] ) output = output.decode("utf-8") output = json.loads(output) trial_number = output["number"] subprocess.check_output( [ "optuna", "tell", "--storage", db_url, "--trial-number", str(trial_number), "--values", "nan", ] ) study = optuna.load_study(storage=db_url, study_name=study_name) assert len(study.trials) == 1 assert study.trials[0].state == TrialState.FAIL assert study.trials[0].values is None @pytest.mark.skip_coverage @pytest.mark.parametrize( "verbosity, expected", [ ("--verbose", True), ("--quiet", False), ], ) def test_configure_logging_verbosity(verbosity: str, expected: bool) -> None: with StorageSupplier("sqlite") as storage: assert isinstance(storage, RDBStorage) storage_url = str(storage.engine.url) # Create study. args = ["optuna", "create-study", "--storage", storage_url, verbosity] # `--verbose` makes the log level DEBUG. # `--quiet` makes the log level WARNING. result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) error_message = result.stderr.decode() assert ("A new study created in RDB with name" in error_message) == expected optuna-3.5.0/tests/test_convert_positional_args.py000066400000000000000000000103311453453102400225340ustar00rootroot00000000000000import re from typing import List import pytest from optuna._convert_positional_args import convert_positional_args def _sample_func(*, a: int, b: int, c: int) -> int: return a + b + c class _SimpleClass: @convert_positional_args(previous_positional_arg_names=["self", "a", "b"]) def simple_method(self, a: int, *, b: int, c: int = 1) -> None: pass def test_convert_positional_args_decorator() -> None: previous_positional_arg_names: List[str] = [] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) decorated_func = decorator_converter(_sample_func) assert decorated_func.__name__ == _sample_func.__name__ def test_convert_positional_args_future_warning_for_methods() -> None: simple_class = _SimpleClass() with pytest.warns(FutureWarning) as record: simple_class.simple_method(1, 2, c=3) # type: ignore simple_class.simple_method(1, b=2, c=3) # No warning. simple_class.simple_method(a=1, b=2, c=3) # No warning. assert len(record) == 1 for warn in record.list: assert isinstance(warn.message, FutureWarning) assert "simple_method" in str(warn.message) def test_convert_positional_args_future_warning() -> None: previous_positional_arg_names: List[str] = ["a", "b"] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) decorated_func = decorator_converter(_sample_func) with pytest.warns(FutureWarning) as record: decorated_func(1, 2, c=3) # type: ignore decorated_func(1, b=2, c=3) # type: ignore decorated_func(a=1, b=2, c=3) # No warning. assert len(record) == 2 for warn in record.list: assert isinstance(warn.message, FutureWarning) assert _sample_func.__name__ in str(warn.message) def test_convert_positional_args_mypy_type_inference() -> None: previous_positional_arg_names: List[str] = [] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) class _Sample: def __init__(self) -> None: pass def method(self) -> bool: return True def _func_sample() -> _Sample: return _Sample() def _func_none() -> None: pass ret_none = decorator_converter(_func_none)() assert ret_none is None ret_sample = decorator_converter(_func_sample)() assert isinstance(ret_sample, _Sample) assert ret_sample.method() @pytest.mark.parametrize( "previous_positional_arg_names, raise_error", [(["a", "b", "c", "d"], True), (["a", "d"], True), (["b", "a"], False)], ) def test_convert_positional_args_invalid_previous_positional_arg_names( previous_positional_arg_names: List[str], raise_error: bool ) -> None: decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) if raise_error: with pytest.raises(AssertionError) as record: decorator_converter(_sample_func) res = re.findall(r"({.+?}|set\(\))", str(record.value)) assert len(res) == 2 assert eval(res[0]) == set(previous_positional_arg_names) assert eval(res[1]) == set(["a", "b", "c"]) else: decorator_converter(_sample_func) def test_convert_positional_args_invalid_positional_args() -> None: previous_positional_arg_names: List[str] = ["a", "b"] decorator_converter = convert_positional_args( previous_positional_arg_names=previous_positional_arg_names ) assert callable(decorator_converter) decorated_func = decorator_converter(_sample_func) with pytest.warns(FutureWarning): with pytest.raises(TypeError) as record: decorated_func(1, 2, 3) # type: ignore assert str(record.value) == "_sample_func() takes 2 positional arguments but 3 were given." with pytest.raises(TypeError) as record: decorated_func(1, 3, b=2) # type: ignore assert str(record.value) == "_sample_func() got multiple values for arguments {'b'}." optuna-3.5.0/tests/test_deprecated.py000066400000000000000000000150221453453102400177010ustar00rootroot00000000000000from typing import Any from typing import Optional import pytest from optuna import _deprecated class _Sample: def __init__(self, a: Any, b: Any, c: Any) -> None: pass def _method(self) -> None: """summary detail """ pass def _method_expected(self) -> None: """summary detail .. warning:: Deprecated in v1.1.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v3.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v1.1.0. """ pass @pytest.mark.parametrize("deprecated_version", ["1.1", 100, None, "2.0.0"]) @pytest.mark.parametrize("removed_version", ["1.1", 10, "1.0.0"]) def test_deprecation_raises_error_for_invalid_version( deprecated_version: Any, removed_version: Any ) -> None: with pytest.raises(ValueError): _deprecated.deprecated_func(deprecated_version, removed_version) with pytest.raises(ValueError): _deprecated.deprecated_class(deprecated_version, removed_version) def test_deprecation_decorator() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_func(deprecated_version, removed_version) assert callable(decorator_deprecation) def _func() -> int: return 10 decorated_func = decorator_deprecation(_func) assert decorated_func.__name__ == _func.__name__ assert decorated_func.__doc__ == _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) with pytest.warns(FutureWarning): decorated_func() def test_deprecation_instance_method_decorator() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_func(deprecated_version, removed_version) assert callable(decorator_deprecation) decorated_method = decorator_deprecation(_Sample._method) assert decorated_method.__name__ == _Sample._method.__name__ assert decorated_method.__doc__ == _Sample._method_expected.__doc__ with pytest.warns(FutureWarning): decorated_method(None) # type: ignore def test_deprecation_class_decorator() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_class(deprecated_version, removed_version) assert callable(decorator_deprecation) decorated_class = decorator_deprecation(_Sample) assert decorated_class.__name__ == "_Sample" assert decorated_class.__init__.__name__ == "__init__" assert decorated_class.__doc__ == _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) with pytest.warns(FutureWarning): decorated_class("a", "b", "c") def test_deprecation_class_decorator_name() -> None: name = "foo" decorator_deprecation = _deprecated.deprecated_class("1.1.0", "3.0.0", name=name) decorated_sample = decorator_deprecation(_Sample) with pytest.warns(FutureWarning) as record: decorated_sample("a", "b", "c") assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] def test_deprecation_decorator_name() -> None: def _func() -> int: return 10 name = "bar" decorator_deprecation = _deprecated.deprecated_func("1.1.0", "3.0.0", name=name) decorated_sample_func = decorator_deprecation(_func) with pytest.warns(FutureWarning) as record: decorated_sample_func() assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] @pytest.mark.parametrize("text", [None, "", "test", "test" * 100]) def test_deprecation_text_specified(text: Optional[str]) -> None: def _func() -> int: return 10 decorator_deprecation = _deprecated.deprecated_func("1.1.0", "3.0.0", text=text) decorated_func = decorator_deprecation(_func) expected_func_doc = _deprecated._DEPRECATION_NOTE_TEMPLATE.format(d_ver="1.1.0", r_ver="3.0.0") if text is None: pass elif len(text) > 0: expected_func_doc += "\n\n " + text + "\n" else: expected_func_doc += "\n\n\n" assert decorated_func.__name__ == _func.__name__ assert decorated_func.__doc__ == expected_func_doc with pytest.warns(FutureWarning) as record: decorated_func() assert isinstance(record.list[0].message, Warning) expected_warning_message = _deprecated._DEPRECATION_WARNING_TEMPLATE.format( name="_func", d_ver="1.1.0", r_ver="3.0.0" ) if text is not None: expected_warning_message += " " + text assert record.list[0].message.args[0] == expected_warning_message @pytest.mark.parametrize("text", [None, "", "test", "test" * 100]) def test_deprecation_class_text_specified(text: Optional[str]) -> None: class _Class: def __init__(self, a: Any, b: Any, c: Any) -> None: pass decorator_deprecation = _deprecated.deprecated_class("1.1.0", "3.0.0", text=text) decorated_class = decorator_deprecation(_Class) expected_class_doc = _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver="1.1.0", r_ver="3.0.0" ) if text is None: pass elif len(text) > 0: expected_class_doc += "\n\n " + text + "\n" else: expected_class_doc += "\n\n\n" assert decorated_class.__name__ == _Class.__name__ assert decorated_class.__doc__ == expected_class_doc with pytest.warns(FutureWarning) as record: decorated_class(None, None, None) assert isinstance(record.list[0].message, Warning) expected_warning_message = _deprecated._DEPRECATION_WARNING_TEMPLATE.format( name="_Class", d_ver="1.1.0", r_ver="3.0.0" ) if text is not None: expected_warning_message += " " + text assert record.list[0].message.args[0] == expected_warning_message def test_deprecation_decorator_default_removed_version() -> None: deprecated_version = "1.1.0" removed_version = "3.0.0" decorator_deprecation = _deprecated.deprecated_func(deprecated_version, removed_version) assert callable(decorator_deprecation) def _func() -> int: return 10 decorated_func = decorator_deprecation(_func) assert decorated_func.__name__ == _func.__name__ assert decorated_func.__doc__ == _deprecated._DEPRECATION_NOTE_TEMPLATE.format( d_ver=deprecated_version, r_ver=removed_version ) with pytest.warns(FutureWarning): decorated_func() optuna-3.5.0/tests/test_distributions.py000066400000000000000000000560521453453102400205130ustar00rootroot00000000000000import copy import json from typing import Any from typing import cast from typing import Dict from typing import Optional import warnings import numpy as np import pytest from optuna import distributions _choices = (None, True, False, 0, 1, 0.0, 1.0, float("nan"), float("inf"), -float("inf"), "", "a") _choices_json = '[null, true, false, 0, 1, 0.0, 1.0, NaN, Infinity, -Infinity, "", "a"]' EXAMPLE_DISTRIBUTIONS: Dict[str, Any] = { "i": distributions.IntDistribution(low=1, high=9, log=False), # i2 and i3 are identical to i, and tested for cases when `log` and `step` are omitted in json. "i2": distributions.IntDistribution(low=1, high=9, log=False), "i3": distributions.IntDistribution(low=1, high=9, log=False), "il": distributions.IntDistribution(low=2, high=12, log=True), "il2": distributions.IntDistribution(low=2, high=12, log=True), "id": distributions.IntDistribution(low=1, high=9, log=False, step=2), "id2": distributions.IntDistribution(low=1, high=9, log=False, step=2), "f": distributions.FloatDistribution(low=1.0, high=2.0, log=False), "fl": distributions.FloatDistribution(low=0.001, high=100.0, log=True), "fd": distributions.FloatDistribution(low=1.0, high=9.0, log=False, step=2.0), "c1": distributions.CategoricalDistribution(choices=_choices), "c2": distributions.CategoricalDistribution(choices=("Roppongi", "Azabu")), "c3": distributions.CategoricalDistribution(choices=["Roppongi", "Azabu"]), } EXAMPLE_JSONS = { "i": '{"name": "IntDistribution", "attributes": {"low": 1, "high": 9}}', "i2": '{"name": "IntDistribution", "attributes": {"low": 1, "high": 9, "log": false}}', "i3": '{"name": "IntDistribution", ' '"attributes": {"low": 1, "high": 9, "log": false, "step": 1}}', "il": '{"name": "IntDistribution", ' '"attributes": {"low": 2, "high": 12, "log": true}}', "il2": '{"name": "IntDistribution", ' '"attributes": {"low": 2, "high": 12, "log": true, "step": 1}}', "id": '{"name": "IntDistribution", ' '"attributes": {"low": 1, "high": 9, "step": 2}}', "id2": '{"name": "IntDistribution", ' '"attributes": {"low": 1, "high": 9, "log": false, "step": 2}}', "f": '{"name": "FloatDistribution", ' '"attributes": {"low": 1.0, "high": 2.0, "log": false, "step": null}}', "fl": '{"name": "FloatDistribution", ' '"attributes": {"low": 0.001, "high": 100.0, "log": true, "step": null}}', "fd": '{"name": "FloatDistribution", ' '"attributes": {"low": 1.0, "high": 9.0, "step": 2.0, "log": false}}', "c1": f'{{"name": "CategoricalDistribution", "attributes": {{"choices": {_choices_json}}}}}', "c2": '{"name": "CategoricalDistribution", "attributes": {"choices": ["Roppongi", "Azabu"]}}', "c3": '{"name": "CategoricalDistribution", "attributes": {"choices": ["Roppongi", "Azabu"]}}', } EXAMPLE_ABBREVIATED_JSONS = { "i": '{"type": "int", "low": 1, "high": 9}', "i2": '{"type": "int", "low": 1, "high": 9, "log": false}', "i3": '{"type": "int", "low": 1, "high": 9, "log": false, "step": 1}', "il": '{"type": "int", "low": 2, "high": 12, "log": true}', "il2": '{"type": "int", "low": 2, "high": 12, "log": true, "step": 1}', "id": '{"type": "int", "low": 1, "high": 9, "step": 2}', "id2": '{"type": "int", "low": 1, "high": 9, "log": false, "step": 2}', "f": '{"type": "float", "low": 1.0, "high": 2.0, "log": false, "step": null}', "fl": '{"type": "float", "low": 0.001, "high": 100, "log": true, "step": null}', "fd": '{"type": "float", "low": 1.0, "high": 9.0, "log": false, "step": 2.0}', "c1": f'{{"type": "categorical", "choices": {_choices_json}}}', "c2": '{"type": "categorical", "choices": ["Roppongi", "Azabu"]}', "c3": '{"type": "categorical", "choices": ["Roppongi", "Azabu"]}', } def test_json_to_distribution() -> None: for key in EXAMPLE_JSONS: distribution_actual = distributions.json_to_distribution(EXAMPLE_JSONS[key]) assert distribution_actual == EXAMPLE_DISTRIBUTIONS[key] unknown_json = '{"name": "UnknownDistribution", "attributes": {"low": 1.0, "high": 2.0}}' pytest.raises(ValueError, lambda: distributions.json_to_distribution(unknown_json)) def test_abbreviated_json_to_distribution() -> None: for key in EXAMPLE_ABBREVIATED_JSONS: distribution_actual = distributions.json_to_distribution(EXAMPLE_ABBREVIATED_JSONS[key]) assert distribution_actual == EXAMPLE_DISTRIBUTIONS[key] unknown_json = '{"type": "unknown", "low": 1.0, "high": 2.0}' pytest.raises(ValueError, lambda: distributions.json_to_distribution(unknown_json)) invalid_distribution = ( '{"type": "float", "low": 0.0, "high": -100.0}', '{"type": "float", "low": 7.3, "high": 7.2, "log": true}', '{"type": "float", "low": -30.0, "high": -40.0, "step": 3.0}', '{"type": "float", "low": 1.0, "high": 100.0, "step": 0.0}', '{"type": "float", "low": 1.0, "high": 100.0, "step": -1.0}', '{"type": "int", "low": 123, "high": 100}', '{"type": "int", "low": 123, "high": 100, "step": 2}', '{"type": "int", "low": 123, "high": 100, "log": true}', '{"type": "int", "low": 1, "high": 100, "step": 0}', '{"type": "int", "low": 1, "high": 100, "step": -1}', '{"type": "categorical", "choices": []}', ) for distribution in invalid_distribution: pytest.raises(ValueError, lambda: distributions.json_to_distribution(distribution)) def test_distribution_to_json() -> None: for key in EXAMPLE_JSONS: json_actual = json.loads(distributions.distribution_to_json(EXAMPLE_DISTRIBUTIONS[key])) json_expect = json.loads(EXAMPLE_JSONS[key]) if json_expect["name"] == "IntDistribution" and "step" not in json_expect["attributes"]: json_expect["attributes"]["step"] = 1 if json_expect["name"] == "IntDistribution" and "log" not in json_expect["attributes"]: json_expect["attributes"]["log"] = False assert json_actual == json_expect def test_check_distribution_compatibility() -> None: # Test the same distribution. for key in EXAMPLE_JSONS: distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS[key], EXAMPLE_DISTRIBUTIONS[key] ) # We need to create new objects to compare NaNs. # See https://github.com/optuna/optuna/pull/3567#pullrequestreview-974939837. distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS[key], distributions.json_to_distribution(EXAMPLE_JSONS[key]) ) # Test different distribution classes. pytest.raises( ValueError, lambda: distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["i"], EXAMPLE_DISTRIBUTIONS["fl"] ), ) # Test compatibility between IntDistributions. distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["id"], EXAMPLE_DISTRIBUTIONS["i"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["i"], EXAMPLE_DISTRIBUTIONS["il"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["il"], EXAMPLE_DISTRIBUTIONS["id"] ) # Test compatibility between FloatDistributions. distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fd"], EXAMPLE_DISTRIBUTIONS["f"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["f"], EXAMPLE_DISTRIBUTIONS["fl"] ) with pytest.raises(ValueError): distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fl"], EXAMPLE_DISTRIBUTIONS["fd"] ) # Test dynamic value range (CategoricalDistribution). pytest.raises( ValueError, lambda: distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["c2"], distributions.CategoricalDistribution(choices=("Roppongi", "Akasaka")), ), ) # Test dynamic value range (except CategoricalDistribution). distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["i"], distributions.IntDistribution(low=-3, high=2) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["il"], distributions.IntDistribution(low=1, high=13, log=True) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["id"], distributions.IntDistribution(low=-3, high=1, step=2) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["f"], distributions.FloatDistribution(low=-3.0, high=-2.0) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fl"], distributions.FloatDistribution(low=0.1, high=1.0, log=True) ) distributions.check_distribution_compatibility( EXAMPLE_DISTRIBUTIONS["fd"], distributions.FloatDistribution(low=-1.0, high=11.0, step=0.5) ) @pytest.mark.parametrize("value", (0, 1, 4, 10, 11, 1.1, "1", "1.1", "-1.0", True, False)) def test_int_internal_representation(value: Any) -> None: i = distributions.IntDistribution(low=1, high=10) if isinstance(value, int): expected_value = value else: expected_value = int(float(value)) assert i.to_external_repr(i.to_internal_repr(value)) == expected_value @pytest.mark.parametrize( "value, kwargs", [ ("foo", {}), ((), {}), ([], {}), ({}, {}), (set(), {}), (np.ones(2), {}), (np.nan, {}), (0, dict(log=True)), (-1, dict(log=True)), ], ) def test_int_internal_representation_error(value: Any, kwargs: Dict[str, Any]) -> None: i = distributions.IntDistribution(low=1, high=10, **kwargs) with pytest.raises(ValueError): i.to_internal_repr(value) @pytest.mark.parametrize( "value", (1.99, 2.0, 4.5, 7, 7.1, 1, "1", "1.1", "-1.0", True, False), ) def test_float_internal_representation(value: Any) -> None: f = distributions.FloatDistribution(low=2.0, high=7.0) if isinstance(value, float): expected_value = value else: expected_value = float(value) assert f.to_external_repr(f.to_internal_repr(value)) == expected_value @pytest.mark.parametrize( "value, kwargs", [ ("foo", {}), ((), {}), ([], {}), ({}, {}), (set(), {}), (np.ones(2), {}), (np.nan, {}), (0.0, dict(log=True)), (-1.0, dict(log=True)), ], ) def test_float_internal_representation_error(value: Any, kwargs: Dict[str, Any]) -> None: f = distributions.FloatDistribution(low=2.0, high=7.0, **kwargs) with pytest.raises(ValueError): f.to_internal_repr(value) def test_categorical_internal_representation() -> None: c = EXAMPLE_DISTRIBUTIONS["c1"] for choice in c.choices: if isinstance(choice, float) and np.isnan(choice): assert np.isnan(c.to_external_repr(c.to_internal_repr(choice))) else: assert c.to_external_repr(c.to_internal_repr(choice)) == choice # We need to create new objects to compare NaNs. # See https://github.com/optuna/optuna/pull/3567#pullrequestreview-974939837. c_ = distributions.json_to_distribution(EXAMPLE_JSONS["c1"]) for choice in cast(distributions.CategoricalDistribution, c_).choices: if isinstance(choice, float) and np.isnan(choice): assert np.isnan(c.to_external_repr(c.to_internal_repr(choice))) else: assert c.to_external_repr(c.to_internal_repr(choice)) == choice @pytest.mark.parametrize( ("expected", "value", "step"), [ (False, 0.9, 1), (True, 1, 1), (False, 1.5, 1), (True, 4, 1), (True, 10, 1), (False, 11, 1), (False, 10, 2), (True, 1, 3), (False, 5, 3), (True, 10, 3), ], ) def test_int_contains(expected: bool, value: float, step: int) -> None: with warnings.catch_warnings(): # When `step` is 2, UserWarning will be raised since the range is not divisible by 2. # The range will be replaced with [1, 9]. warnings.simplefilter("ignore", category=UserWarning) i = distributions.IntDistribution(low=1, high=10, step=step) assert i._contains(value) == expected @pytest.mark.parametrize( ("expected", "value", "step"), [ (False, 1.99, None), (True, 2.0, None), (True, 2.5, None), (True, 7, None), (False, 7.1, None), (False, 0.99, 2.0), (True, 2.0, 2.0), (False, 3.0, 2.0), (True, 6, 2.0), (False, 6.1, 2.0), ], ) def test_float_contains(expected: bool, value: float, step: Optional[float]) -> None: with warnings.catch_warnings(): # When `step` is 2.0, UserWarning will be raised since the range is not divisible by 2. # The range will be replaced with [2.0, 6.0]. warnings.simplefilter("ignore", category=UserWarning) f = distributions.FloatDistribution(low=2.0, high=7.0, step=step) assert f._contains(value) == expected def test_categorical_contains() -> None: c = distributions.CategoricalDistribution(choices=("Roppongi", "Azabu")) assert not c._contains(-1) assert c._contains(0) assert c._contains(1) assert c._contains(1.5) assert not c._contains(3) def test_empty_range_contains() -> None: i = distributions.IntDistribution(low=1, high=1) assert not i._contains(0) assert i._contains(1) assert not i._contains(2) iq = distributions.IntDistribution(low=1, high=1, step=2) assert not iq._contains(0) assert iq._contains(1) assert not iq._contains(2) il = distributions.IntDistribution(low=1, high=1, log=True) assert not il._contains(0) assert il._contains(1) assert not il._contains(2) f = distributions.FloatDistribution(low=1.0, high=1.0) assert not f._contains(0.9) assert f._contains(1.0) assert not f._contains(1.1) fd = distributions.FloatDistribution(low=1.0, high=1.0, step=2.0) assert not fd._contains(0.9) assert fd._contains(1.0) assert not fd._contains(1.1) fl = distributions.FloatDistribution(low=1.0, high=1.0, log=True) assert not fl._contains(0.9) assert fl._contains(1.0) assert not fl._contains(1.1) @pytest.mark.parametrize( ("expected", "low", "high", "log", "step"), [ (True, 1, 1, False, 1), (True, 3, 3, False, 2), (True, 2, 2, True, 1), (False, -123, 0, False, 1), (False, -123, 0, False, 123), (False, 2, 4, True, 1), ], ) def test_int_single(expected: bool, low: int, high: int, log: bool, step: int) -> None: distribution = distributions.IntDistribution(low=low, high=high, log=log, step=step) assert distribution.single() == expected @pytest.mark.parametrize( ("expected", "low", "high", "log", "step"), [ (True, 2.0, 2.0, False, None), (True, 2.0, 2.0, True, None), (True, 2.22, 2.22, False, 0.1), (True, 2.22, 2.24, False, 0.3), (False, 1.0, 1.001, False, None), (False, 7.3, 10.0, True, None), (False, -30, -20, False, 2), (False, -30, -20, False, 10), # In Python, "0.3 - 0.2 != 0.1" is True. (False, 0.2, 0.3, False, 0.1), (False, 0.7, 0.8, False, 0.1), ], ) def test_float_single( expected: bool, low: float, high: float, log: bool, step: Optional[float] ) -> None: with warnings.catch_warnings(): # When `step` is 0.3, UserWarning will be raised since the range is not divisible by 0.3. # The range will be replaced with [2.22, 2.24]. warnings.simplefilter("ignore", category=UserWarning) distribution = distributions.FloatDistribution(low=low, high=high, log=log, step=step) assert distribution.single() == expected def test_categorical_single() -> None: assert distributions.CategoricalDistribution(choices=("foo",)).single() assert not distributions.CategoricalDistribution(choices=("foo", "bar")).single() def test_invalid_distribution() -> None: with pytest.warns(UserWarning): distributions.CategoricalDistribution(choices=({"foo": "bar"},)) # type: ignore def test_eq_ne_hash() -> None: # Two instances of a class are regarded as equivalent if the fields have the same values. for d in EXAMPLE_DISTRIBUTIONS.values(): d_copy = copy.deepcopy(d) assert d == d_copy assert hash(d) == hash(d_copy) # Different field values. d0 = distributions.FloatDistribution(low=1, high=2) d1 = distributions.FloatDistribution(low=1, high=3) assert d0 != d1 # Different distribution classes. d2 = distributions.IntDistribution(low=1, high=2) assert d0 != d2 def test_repr() -> None: # The following variables are needed to apply `eval` to distribution # instances that contain `float('nan')` or `float('inf')` as a field value. nan = float("nan") # NOQA inf = float("inf") # NOQA for d in EXAMPLE_DISTRIBUTIONS.values(): assert d == eval("distributions." + repr(d)) @pytest.mark.parametrize( ("key", "low", "high", "log", "step"), [ ("i", 1, 9, False, 1), ("il", 2, 12, True, 1), ("id", 1, 9, False, 2), ], ) def test_int_distribution_asdict(key: str, low: int, high: int, log: bool, step: int) -> None: expected_dict = {"low": low, "high": high, "log": log, "step": step} assert EXAMPLE_DISTRIBUTIONS[key]._asdict() == expected_dict @pytest.mark.parametrize( ("key", "low", "high", "log", "step"), [ ("f", 1.0, 2.0, False, None), ("fl", 0.001, 100.0, True, None), ("fd", 1.0, 9.0, False, 2.0), ], ) def test_float_distribution_asdict( key: str, low: float, high: float, log: bool, step: Optional[float] ) -> None: expected_dict = {"low": low, "high": high, "log": log, "step": step} assert EXAMPLE_DISTRIBUTIONS[key]._asdict() == expected_dict def test_int_init_error() -> None: # Empty distributions cannot be instantiated. with pytest.raises(ValueError): distributions.IntDistribution(low=123, high=100) with pytest.raises(ValueError): distributions.IntDistribution(low=100, high=10, log=True) with pytest.raises(ValueError): distributions.IntDistribution(low=123, high=100, step=2) # 'step' must be 1 when 'log' is True. with pytest.raises(ValueError): distributions.IntDistribution(low=1, high=100, log=True, step=2) # 'step' should be positive. with pytest.raises(ValueError): distributions.IntDistribution(low=1, high=100, step=0) with pytest.raises(ValueError): distributions.IntDistribution(low=1, high=10, step=-1) def test_float_init_error() -> None: # Empty distributions cannot be instantiated. with pytest.raises(ValueError): distributions.FloatDistribution(low=0.0, high=-100.0) with pytest.raises(ValueError): distributions.FloatDistribution(low=7.3, high=7.2, log=True) with pytest.raises(ValueError): distributions.FloatDistribution(low=-30.0, high=-40.0, step=2.5) # 'step' must be None when 'log' is True. with pytest.raises(ValueError): distributions.FloatDistribution(low=1.0, high=100.0, log=True, step=0.5) # 'step' should be positive. with pytest.raises(ValueError): distributions.FloatDistribution(low=1.0, high=10.0, step=0) with pytest.raises(ValueError): distributions.FloatDistribution(low=1.0, high=100.0, step=-1) def test_categorical_init_error() -> None: with pytest.raises(ValueError): distributions.CategoricalDistribution(choices=()) def test_categorical_distribution_different_sequence_types() -> None: c1 = distributions.CategoricalDistribution(choices=("Roppongi", "Azabu")) c2 = distributions.CategoricalDistribution(choices=["Roppongi", "Azabu"]) assert c1 == c2 @pytest.mark.filterwarnings("ignore::FutureWarning") def test_convert_old_distribution_to_new_distribution() -> None: ud = distributions.UniformDistribution(low=0, high=10) assert distributions._convert_old_distribution_to_new_distribution( ud ) == distributions.FloatDistribution(low=0, high=10, log=False, step=None) dud = distributions.DiscreteUniformDistribution(low=0, high=10, q=2) assert distributions._convert_old_distribution_to_new_distribution( dud ) == distributions.FloatDistribution(low=0, high=10, log=False, step=2) lud = distributions.LogUniformDistribution(low=1, high=10) assert distributions._convert_old_distribution_to_new_distribution( lud ) == distributions.FloatDistribution(low=1, high=10, log=True, step=None) id = distributions.IntUniformDistribution(low=0, high=10) assert distributions._convert_old_distribution_to_new_distribution( id ) == distributions.IntDistribution(low=0, high=10, log=False, step=1) idd = distributions.IntUniformDistribution(low=0, high=10, step=2) assert distributions._convert_old_distribution_to_new_distribution( idd ) == distributions.IntDistribution(low=0, high=10, log=False, step=2) ild = distributions.IntLogUniformDistribution(low=1, high=10) assert distributions._convert_old_distribution_to_new_distribution( ild ) == distributions.IntDistribution(low=1, high=10, log=True, step=1) def test_convert_old_distribution_to_new_distribution_noop() -> None: # No conversion happens for CategoricalDistribution. cd = distributions.CategoricalDistribution(choices=["a", "b", "c"]) assert distributions._convert_old_distribution_to_new_distribution(cd) == cd # No conversion happens for new distributions. fd = distributions.FloatDistribution(low=0, high=10, log=False, step=None) assert distributions._convert_old_distribution_to_new_distribution(fd) == fd dfd = distributions.FloatDistribution(low=0, high=10, log=False, step=2) assert distributions._convert_old_distribution_to_new_distribution(dfd) == dfd lfd = distributions.FloatDistribution(low=1, high=10, log=True, step=None) assert distributions._convert_old_distribution_to_new_distribution(lfd) == lfd id = distributions.IntDistribution(low=0, high=10) assert distributions._convert_old_distribution_to_new_distribution(id) == id idd = distributions.IntDistribution(low=0, high=10, step=2) assert distributions._convert_old_distribution_to_new_distribution(idd) == idd ild = distributions.IntDistribution(low=1, high=10, log=True) assert distributions._convert_old_distribution_to_new_distribution(ild) == ild def test_is_distribution_log() -> None: lfd = distributions.FloatDistribution(low=1, high=10, log=True) assert distributions._is_distribution_log(lfd) lid = distributions.IntDistribution(low=1, high=10, log=True) assert distributions._is_distribution_log(lid) fd = distributions.FloatDistribution(low=0, high=10, log=False) assert not distributions._is_distribution_log(fd) id = distributions.IntDistribution(low=0, high=10, log=False) assert not distributions._is_distribution_log(id) cd = distributions.CategoricalDistribution(choices=["a", "b", "c"]) assert not distributions._is_distribution_log(cd) optuna-3.5.0/tests/test_experimental.py000066400000000000000000000063161453453102400203040ustar00rootroot00000000000000from typing import Any import pytest from optuna import _experimental from optuna.exceptions import ExperimentalWarning def _sample_func() -> int: return 10 class _Sample: def __init__(self, a: Any, b: Any, c: Any) -> None: pass def _method(self) -> None: """summary detail """ pass def _method_expected(self) -> None: """summary detail .. note:: Added in v1.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.1.0. """ pass @pytest.mark.parametrize("version", ["1.1", 100, None]) def test_experimental_raises_error_for_invalid_version(version: Any) -> None: with pytest.raises(ValueError): _experimental.experimental_func(version) with pytest.raises(ValueError): _experimental.experimental_class(version) def test_experimental_func_decorator() -> None: version = "1.1.0" decorator_experimental = _experimental.experimental_func(version) assert callable(decorator_experimental) decorated_func = decorator_experimental(_sample_func) assert decorated_func.__name__ == _sample_func.__name__ assert decorated_func.__doc__ == _experimental._EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) with pytest.warns(ExperimentalWarning): decorated_func() def test_experimental_instance_method_decorator() -> None: version = "1.1.0" decorator_experimental = _experimental.experimental_func(version) assert callable(decorator_experimental) decorated_method = decorator_experimental(_Sample._method) assert decorated_method.__name__ == _Sample._method.__name__ assert decorated_method.__doc__ == _Sample._method_expected.__doc__ with pytest.warns(ExperimentalWarning): decorated_method(None) # type: ignore def test_experimental_class_decorator() -> None: version = "1.1.0" decorator_experimental = _experimental.experimental_class(version) assert callable(decorator_experimental) decorated_class = decorator_experimental(_Sample) assert decorated_class.__name__ == "_Sample" assert decorated_class.__init__.__name__ == "__init__" assert decorated_class.__doc__ == _experimental._EXPERIMENTAL_NOTE_TEMPLATE.format(ver=version) with pytest.warns(ExperimentalWarning): decorated_class("a", "b", "c") def test_experimental_class_decorator_name() -> None: name = "foo" decorator_experimental = _experimental.experimental_class("1.1.0", name=name) decorated_sample = decorator_experimental(_Sample) with pytest.warns(ExperimentalWarning) as record: decorated_sample("a", "b", "c") assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] def test_experimental_decorator_name() -> None: name = "bar" decorator_experimental = _experimental.experimental_func("1.1.0", name=name) decorated_sample_func = decorator_experimental(_sample_func) with pytest.warns(ExperimentalWarning) as record: decorated_sample_func() assert isinstance(record.list[0].message, Warning) assert name in record.list[0].message.args[0] optuna-3.5.0/tests/test_imports.py000066400000000000000000000015301453453102400172750ustar00rootroot00000000000000import pytest from optuna._imports import try_import def test_try_import_is_successful() -> None: with try_import() as imports: pass assert imports.is_successful() imports.check() def test_try_import_is_successful_other_error() -> None: with pytest.raises(NotImplementedError): with try_import() as imports: raise NotImplementedError assert imports.is_successful() # No imports failed so `imports` is successful. imports.check() def test_try_import_not_successful() -> None: with try_import() as imports: raise ImportError assert not imports.is_successful() with pytest.raises(ImportError): imports.check() with try_import() as imports: raise SyntaxError assert not imports.is_successful() with pytest.raises(ImportError): imports.check() optuna-3.5.0/tests/test_logging.py000066400000000000000000000062021453453102400172270ustar00rootroot00000000000000import logging import _pytest.capture import _pytest.logging import optuna.logging def test_get_logger(caplog: _pytest.logging.LogCaptureFixture) -> None: # Log propagation is necessary for caplog to capture log outputs. optuna.logging.enable_propagation() logger = optuna.logging.get_logger("optuna.foo") with caplog.at_level(logging.INFO, logger="optuna.foo"): logger.info("hello") assert "hello" in caplog.text def test_default_handler(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.set_verbosity(optuna.logging.INFO) library_root_logger = optuna.logging._get_library_root_logger() example_logger = optuna.logging.get_logger("optuna.bar") # Default handler enabled optuna.logging.enable_default_handler() assert library_root_logger.handlers example_logger.info("hey") _, err = capsys.readouterr() assert "hey" in err # Default handler disabled optuna.logging.disable_default_handler() assert not library_root_logger.handlers example_logger.info("yoyo") _, err = capsys.readouterr() assert "yoyo" not in err def test_verbosity(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() library_root_logger = optuna.logging._get_library_root_logger() example_logger = optuna.logging.get_logger("optuna.hoge") optuna.logging.enable_default_handler() # level INFO optuna.logging.set_verbosity(optuna.logging.INFO) assert library_root_logger.getEffectiveLevel() == logging.INFO example_logger.warning("hello-warning") example_logger.info("hello-info") example_logger.debug("hello-debug") _, err = capsys.readouterr() assert "hello-warning" in err assert "hello-info" in err assert "hello-debug" not in err # level WARNING optuna.logging.set_verbosity(optuna.logging.WARNING) assert library_root_logger.getEffectiveLevel() == logging.WARNING example_logger.warning("bye-warning") example_logger.info("bye-info") example_logger.debug("bye-debug") _, err = capsys.readouterr() assert "bye-warning" in err assert "bye-info" not in err assert "bye-debug" not in err def test_propagation(caplog: _pytest.logging.LogCaptureFixture) -> None: optuna.logging._reset_library_root_logger() logger = optuna.logging.get_logger("optuna.foo") # Propagation is disabled by default. with caplog.at_level(logging.INFO, logger="optuna"): logger.info("no-propagation") assert "no-propagation" not in caplog.text # Enable propagation. optuna.logging.enable_propagation() with caplog.at_level(logging.INFO, logger="optuna"): logger.info("enable-propagate") assert "enable-propagate" in caplog.text # Disable propagation. optuna.logging.disable_propagation() with caplog.at_level(logging.INFO, logger="optuna"): logger.info("disable-propagation") assert "disable-propagation" not in caplog.text optuna-3.5.0/tests/test_multi_objective.py000066400000000000000000000067671453453102400210050ustar00rootroot00000000000000from typing import Tuple from optuna import create_study from optuna.study._multi_objective import _get_pareto_front_trials_2d from optuna.study._multi_objective import _get_pareto_front_trials_nd from optuna.trial import FrozenTrial def _trial_to_values(t: FrozenTrial) -> Tuple[float, ...]: assert t.values is not None return tuple(t.values) def test_get_pareto_front_trials_2d() -> None: study = create_study(directions=["minimize", "maximize"]) assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == set() study.optimize(lambda t: [2, 2], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == {(2, 2)} study.optimize(lambda t: [1, 1], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == {(1, 1), (2, 2)} study.optimize(lambda t: [3, 1], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == {(1, 1), (2, 2)} study.optimize(lambda t: [3, 2], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == {(1, 1), (2, 2)} study.optimize(lambda t: [1, 3], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == {(1, 3)} assert len(_get_pareto_front_trials_2d(study.trials, study.directions)) == 1 study.optimize(lambda t: [1, 3], n_trials=1) # The trial result is the same as the above one. assert { _trial_to_values(t) for t in _get_pareto_front_trials_2d(study.trials, study.directions) } == {(1, 3)} assert len(_get_pareto_front_trials_2d(study.trials, study.directions)) == 2 def test_get_pareto_front_trials_nd() -> None: study = create_study(directions=["minimize", "maximize", "minimize"]) assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == set() study.optimize(lambda t: [2, 2, 2], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == {(2, 2, 2)} study.optimize(lambda t: [1, 1, 1], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == { (1, 1, 1), (2, 2, 2), } study.optimize(lambda t: [3, 1, 3], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == { (1, 1, 1), (2, 2, 2), } study.optimize(lambda t: [3, 2, 3], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == { (1, 1, 1), (2, 2, 2), } study.optimize(lambda t: [1, 3, 1], n_trials=1) assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == {(1, 3, 1)} assert len(_get_pareto_front_trials_nd(study.trials, study.directions)) == 1 study.optimize( lambda t: [1, 3, 1], n_trials=1 ) # The trial result is the same as the above one. assert { _trial_to_values(t) for t in _get_pareto_front_trials_nd(study.trials, study.directions) } == {(1, 3, 1)} assert len(_get_pareto_front_trials_nd(study.trials, study.directions)) == 2 optuna-3.5.0/tests/test_transform.py000066400000000000000000000200171453453102400176140ustar00rootroot00000000000000import math from typing import Any import numpy import pytest from optuna._transform import _SearchSpaceTransform from optuna._transform import _untransform_numerical_param from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution @pytest.mark.parametrize( "param,distribution", [ (0, IntDistribution(0, 3)), (1, IntDistribution(1, 10, log=True)), (2, IntDistribution(0, 10, step=2)), (0.0, FloatDistribution(0, 3)), (1.0, FloatDistribution(1, 10, log=True)), (0.2, FloatDistribution(0, 1, step=0.2)), ("foo", CategoricalDistribution(["foo"])), ("bar", CategoricalDistribution(["foo", "bar", "baz"])), ], ) def test_search_space_transform_shapes_dtypes(param: Any, distribution: BaseDistribution) -> None: trans = _SearchSpaceTransform({"x0": distribution}) trans_params = trans.transform({"x0": param}) if isinstance(distribution, CategoricalDistribution): expected_bounds_shape = (len(distribution.choices), 2) expected_params_shape = (len(distribution.choices),) else: expected_bounds_shape = (1, 2) expected_params_shape = (1,) assert trans.bounds.shape == expected_bounds_shape assert trans.bounds.dtype == numpy.float64 assert trans_params.shape == expected_params_shape assert trans_params.dtype == numpy.float64 def test_search_space_transform_encoding() -> None: trans = _SearchSpaceTransform({"x0": IntDistribution(0, 3)}) assert len(trans.column_to_encoded_columns) == 1 numpy.testing.assert_equal(trans.column_to_encoded_columns[0], numpy.array([0])) numpy.testing.assert_equal(trans.encoded_column_to_column, numpy.array([0])) trans = _SearchSpaceTransform({"x0": CategoricalDistribution(["foo", "bar", "baz"])}) assert len(trans.column_to_encoded_columns) == 1 numpy.testing.assert_equal(trans.column_to_encoded_columns[0], numpy.array([0, 1, 2])) numpy.testing.assert_equal(trans.encoded_column_to_column, numpy.array([0, 0, 0])) trans = _SearchSpaceTransform( { "x0": FloatDistribution(0, 3), "x1": CategoricalDistribution(["foo", "bar", "baz"]), "x3": FloatDistribution(0, 1, step=0.2), } ) assert len(trans.column_to_encoded_columns) == 3 numpy.testing.assert_equal(trans.column_to_encoded_columns[0], numpy.array([0])) numpy.testing.assert_equal(trans.column_to_encoded_columns[1], numpy.array([1, 2, 3])) numpy.testing.assert_equal(trans.column_to_encoded_columns[2], numpy.array([4])) numpy.testing.assert_equal(trans.encoded_column_to_column, numpy.array([0, 1, 1, 1, 2])) @pytest.mark.parametrize("transform_log", [True, False]) @pytest.mark.parametrize("transform_step", [True, False]) @pytest.mark.parametrize("transform_0_1", [True, False]) @pytest.mark.parametrize( "param,distribution", [ (0, IntDistribution(0, 3)), (3, IntDistribution(0, 3)), (1, IntDistribution(1, 10, log=True)), (10, IntDistribution(1, 10, log=True)), (2, IntDistribution(0, 10, step=2)), (10, IntDistribution(0, 10, step=2)), (0.0, FloatDistribution(0, 3)), (3.0, FloatDistribution(0, 3)), (1.0, FloatDistribution(1, 10, log=True)), (10.0, FloatDistribution(1, 10, log=True)), (0.2, FloatDistribution(0, 1, step=0.2)), (1.0, FloatDistribution(0, 1, step=0.2)), ], ) def test_search_space_transform_numerical( transform_log: bool, transform_step: bool, transform_0_1: bool, param: Any, distribution: BaseDistribution, ) -> None: trans = _SearchSpaceTransform( {"x0": distribution}, transform_log=transform_log, transform_step=transform_step, transform_0_1=transform_0_1, ) if transform_0_1: expected_low = 0.0 expected_high = 1.0 else: expected_low = distribution.low # type: ignore expected_high = distribution.high # type: ignore if isinstance(distribution, FloatDistribution): if transform_log and distribution.log: expected_low = math.log(expected_low) expected_high = math.log(expected_high) if transform_step and distribution.step is not None: half_step = 0.5 * distribution.step expected_low -= half_step expected_high += half_step elif isinstance(distribution, IntDistribution): if transform_step: half_step = 0.5 * distribution.step expected_low -= half_step expected_high += half_step if distribution.log and transform_log: expected_low = math.log(expected_low) expected_high = math.log(expected_high) for bound in trans.bounds: assert bound[0] == expected_low assert bound[1] == expected_high trans_params = trans.transform({"x0": param}) assert trans_params.size == 1 assert expected_low <= trans_params <= expected_high assert numpy.isclose(param, trans.untransform(trans_params)["x0"]) @pytest.mark.parametrize( "param,distribution", [ ("foo", CategoricalDistribution(["foo"])), ("bar", CategoricalDistribution(["foo", "bar", "baz"])), ], ) def test_search_space_transform_values_categorical( param: Any, distribution: CategoricalDistribution ) -> None: trans = _SearchSpaceTransform({"x0": distribution}) for bound in trans.bounds: assert bound[0] == 0.0 assert bound[1] == 1.0 trans_params = trans.transform({"x0": param}) for trans_param in trans_params: assert trans_param in (0.0, 1.0) def test_search_space_transform_untransform_params() -> None: search_space = { "x0": CategoricalDistribution(["corge"]), "x1": CategoricalDistribution(["foo", "bar", "baz", "qux"]), "x2": CategoricalDistribution(["quux", "quuz"]), "x3": FloatDistribution(2, 3), "x4": FloatDistribution(-2, 2), "x5": FloatDistribution(1, 10, log=True), "x6": FloatDistribution(1, 1, log=True), "x7": FloatDistribution(0, 1, step=0.2), "x8": IntDistribution(2, 4), "x9": IntDistribution(1, 10, log=True), "x10": IntDistribution(1, 9, step=2), } params = { "x0": "corge", "x1": "qux", "x2": "quux", "x3": 2.0, "x4": -2, "x5": 1.0, "x6": 1.0, "x7": 0.2, "x8": 2, "x9": 1, "x10": 3, } trans = _SearchSpaceTransform(search_space) trans_params = trans.transform(params) untrans_params = trans.untransform(trans_params) for name in params.keys(): assert untrans_params[name] == params[name] @pytest.mark.parametrize("transform_log", [True, False]) @pytest.mark.parametrize("transform_step", [True, False]) @pytest.mark.parametrize( "distribution", [ FloatDistribution(0, 1, step=0.2), IntDistribution(2, 4), IntDistribution(1, 10, log=True), ], ) def test_transform_untransform_params_at_bounds( transform_log: bool, transform_step: bool, distribution: BaseDistribution ) -> None: EPS = 1e-12 # Skip the following two conditions that do not clip in `_untransform_numerical_param`: # 1. `IntDistribution(log=True)` without `transform_log` if not transform_log and (isinstance(distribution, IntDistribution) and distribution.log): return trans = _SearchSpaceTransform({"x0": distribution}, transform_log, transform_step) # Manually create round-off errors. lower_bound = trans.bounds[0][0] - EPS upper_bound = trans.bounds[0][1] + EPS trans_lower_param = _untransform_numerical_param(lower_bound, distribution, transform_log) trans_upper_param = _untransform_numerical_param(upper_bound, distribution, transform_log) assert trans_lower_param == distribution.low # type: ignore assert trans_upper_param == distribution.high # type: ignore optuna-3.5.0/tests/trial_tests/000077500000000000000000000000001453453102400165255ustar00rootroot00000000000000optuna-3.5.0/tests/trial_tests/__init__.py000066400000000000000000000000001453453102400206240ustar00rootroot00000000000000optuna-3.5.0/tests/trial_tests/test_fixed.py000066400000000000000000000017051453453102400212400ustar00rootroot00000000000000from __future__ import annotations import pytest from optuna.trial import FixedTrial def test_params() -> None: params = {"x": 1} trial = FixedTrial(params) assert trial.params == {} assert trial.suggest_float("x", 0, 10) == 1 assert trial.params == params @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. params = {"x": 1} trial = FixedTrial(params) kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. trial.suggest_int("x", -1, 1, *args) def test_number() -> None: params = {"x": 1} trial = FixedTrial(params, 2) assert trial.number == 2 trial = FixedTrial(params) assert trial.number == 0 optuna-3.5.0/tests/trial_tests/test_frozen.py000066400000000000000000000314461453453102400214510ustar00rootroot00000000000000from __future__ import annotations import copy import datetime from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple import pytest from optuna import create_study from optuna.distributions import BaseDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier import optuna.trial from optuna.trial import BaseTrial from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState def _create_trial( *, value: float = 0.2, params: Dict[str, Any] = {"x": 10}, distributions: Dict[str, BaseDistribution] = {"x": FloatDistribution(5, 12)}, ) -> FrozenTrial: trial = optuna.trial.create_trial(value=value, params=params, distributions=distributions) trial.number = 0 return trial def test_eq_ne() -> None: trial = _create_trial() trial_other = copy.copy(trial) assert trial == trial_other trial_other.value = 0.3 assert trial != trial_other def test_lt() -> None: trial = _create_trial() trial_other = copy.copy(trial) assert not trial < trial_other trial_other.number = trial.number + 1 assert trial < trial_other assert not trial_other < trial with pytest.raises(TypeError): trial < 1 assert trial <= trial_other assert not trial_other <= trial with pytest.raises(TypeError): trial <= 1 # A list of FrozenTrials is sortable. trials = [trial_other, trial] trials.sort() assert trials[0] is trial assert trials[1] is trial_other def test_repr() -> None: trial = _create_trial() assert trial == eval(repr(trial)) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_sampling(storage_mode: str) -> None: def objective(trial: BaseTrial) -> float: a = trial.suggest_float("a", 0.0, 10.0) b = trial.suggest_float("b", 0.1, 10.0, log=True) c = trial.suggest_float("c", 0.0, 10.0, step=1.0) d = trial.suggest_int("d", 0, 10) e = trial.suggest_categorical("e", [0, 1, 2]) f = trial.suggest_int("f", 1, 10, log=True) return a + b + c + d + e + f with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=1) best_trial = study.best_trial # re-evaluate objective with the best hyperparameters v = objective(best_trial) assert v == best_trial.value def test_set_value() -> None: trial = _create_trial() trial.value = 0.1 assert trial.value == 0.1 def test_set_values() -> None: trial = _create_trial() trial.values = (0.1, 0.2) assert trial.values == [0.1, 0.2] # type: ignore[comparison-overlap] trial = _create_trial() trial.values = [0.1, 0.2] assert trial.values == [0.1, 0.2] def test_validate() -> None: # Valid. valid_trial = _create_trial() valid_trial._validate() # Invalid: `datetime_start` is not set when the trial is not in the waiting state. for state in [ TrialState.RUNNING, TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL, ]: invalid_trial = copy.copy(valid_trial) invalid_trial.state = state invalid_trial.datetime_start = None with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `RUNNING` and `datetime_complete` is set. invalid_trial = copy.copy(valid_trial) invalid_trial.state = TrialState.RUNNING with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is not `RUNNING` and `datetime_complete` is not set. for state in [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL]: invalid_trial = copy.copy(valid_trial) invalid_trial.state = state invalid_trial.datetime_complete = None with pytest.raises(ValueError): invalid_trial._validate() # Invalid: `state` is `COMPLETE` and `value` is not set. invalid_trial = copy.copy(valid_trial) invalid_trial.value = None with pytest.raises(ValueError): invalid_trial._validate() # Invalid: Inconsistent `params` and `distributions` inconsistent_pairs: List[Tuple[Dict[str, Any], Dict[str, BaseDistribution]]] = [ # `params` has an extra element. ({"x": 0.1, "y": 0.5}, {"x": FloatDistribution(0, 1)}), # `distributions` has an extra element. ({"x": 0.1}, {"x": FloatDistribution(0, 1), "y": FloatDistribution(0.1, 1.0, log=True)}), # The value of `x` isn't contained in the distribution. ({"x": -0.5}, {"x": FloatDistribution(0, 1)}), ] for params, distributions in inconsistent_pairs: invalid_trial = copy.copy(valid_trial) invalid_trial.params = params invalid_trial.distributions = distributions with pytest.raises(ValueError): invalid_trial._validate() def test_number() -> None: trial = _create_trial() assert trial.number == 0 trial.number = 2 assert trial.number == 2 def test_params() -> None: params = {"x": 1} trial = _create_trial( value=0.2, params=params, distributions={"x": FloatDistribution(0, 10)}, ) assert trial.suggest_float("x", 0, 10) == 1 assert trial.params == params params = {"x": 2} trial.params = params assert trial.suggest_float("x", 0, 10) == 2 assert trial.params == params def test_distributions() -> None: distributions = {"x": FloatDistribution(0, 10)} trial = _create_trial( value=0.2, params={"x": 1}, distributions=dict(distributions), ) assert trial.distributions == distributions distributions = {"x": FloatDistribution(1, 9)} trial.distributions = dict(distributions) assert trial.distributions == distributions def test_user_attrs() -> None: trial = _create_trial() assert trial.user_attrs == {} user_attrs = {"data": "MNIST"} trial.user_attrs = user_attrs assert trial.user_attrs == user_attrs def test_system_attrs() -> None: trial = _create_trial() assert trial.system_attrs == {} system_attrs = {"system_message": "test"} trial.system_attrs = system_attrs assert trial.system_attrs == system_attrs def test_called_single_methods_when_multi() -> None: state = TrialState.COMPLETE values = (0.2, 0.3) params = {"x": 10} distributions: Dict[str, BaseDistribution] = {"x": FloatDistribution(5, 12)} user_attrs = {"foo": "bar"} system_attrs = {"baz": "qux"} intermediate_values = {0: 0.0, 1: 0.1, 2: 0.1} trial = optuna.trial.create_trial( state=state, values=values, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) with pytest.raises(RuntimeError): trial.value with pytest.raises(RuntimeError): trial.value = 0.1 with pytest.raises(RuntimeError): trial.value = [0.1] # type: ignore def test_init() -> None: def _create_trial(value: Optional[float], values: Optional[List[float]]) -> FrozenTrial: return FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=value, values=values, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={}, distributions={"x": FloatDistribution(0, 10)}, user_attrs={}, system_attrs={}, intermediate_values={}, ) _ = _create_trial(0.2, None) _ = _create_trial(None, [0.2]) with pytest.raises(ValueError): _ = _create_trial(0.2, [0.2]) with pytest.raises(ValueError): _ = _create_trial(0.2, []) # TODO(hvy): Write exhaustive test include invalid combinations when feature is no longer # experimental. @pytest.mark.parametrize("state", [TrialState.COMPLETE, TrialState.FAIL]) def test_create_trial(state: TrialState) -> None: value = 0.2 params = {"x": 10} distributions: Dict[str, BaseDistribution] = {"x": FloatDistribution(5, 12)} user_attrs = {"foo": "bar"} system_attrs = {"baz": "qux"} intermediate_values = {0: 0.0, 1: 0.1, 2: 0.1} trial = create_trial( state=state, value=value, params=params, distributions=distributions, user_attrs=user_attrs, system_attrs=system_attrs, intermediate_values=intermediate_values, ) assert isinstance(trial, FrozenTrial) assert trial.state == state assert trial.value == value assert trial.params == params assert trial.distributions == distributions assert trial.user_attrs == user_attrs assert trial.system_attrs == system_attrs assert trial.intermediate_values == intermediate_values assert trial.datetime_start is not None assert (trial.datetime_complete is not None) == state.is_finished() with pytest.raises(ValueError): create_trial(state=state, value=value, values=(value,)) # Deprecated distributions are internally converted to corresponding distributions. @pytest.mark.filterwarnings("ignore::FutureWarning") def test_create_trial_distribution_conversion() -> None: fixed_params = { "ud": 0, "dud": 2, "lud": 1, "id": 0, "idd": 2, "ild": 1, } fixed_distributions = { "ud": optuna.distributions.UniformDistribution(low=0, high=10), "dud": optuna.distributions.DiscreteUniformDistribution(low=0, high=10, q=2), "lud": optuna.distributions.LogUniformDistribution(low=1, high=10), "id": optuna.distributions.IntUniformDistribution(low=0, high=10), "idd": optuna.distributions.IntUniformDistribution(low=0, high=10, step=2), "ild": optuna.distributions.IntLogUniformDistribution(low=1, high=10), } with pytest.warns( FutureWarning, match="See https://github.com/optuna/optuna/issues/2941", ) as record: trial = create_trial(params=fixed_params, distributions=fixed_distributions, value=1) assert len(record) == 6 expected_distributions = { "ud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": optuna.distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": optuna.distributions.IntDistribution(low=1, high=10, log=True, step=1), } assert trial.distributions == expected_distributions # It confirms that ask doesn't convert non-deprecated distributions. def test_create_trial_distribution_conversion_noop() -> None: fixed_params = { "ud": 0, "dud": 2, "lud": 1, "id": 0, "idd": 2, "ild": 1, "cd": "a", } fixed_distributions = { "ud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=None), "dud": optuna.distributions.FloatDistribution(low=0, high=10, log=False, step=2), "lud": optuna.distributions.FloatDistribution(low=1, high=10, log=True, step=None), "id": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=1), "idd": optuna.distributions.IntDistribution(low=0, high=10, log=False, step=2), "ild": optuna.distributions.IntDistribution(low=1, high=10, log=True, step=1), "cd": optuna.distributions.CategoricalDistribution(choices=["a", "b", "c"]), } trial = create_trial(params=fixed_params, distributions=fixed_distributions, value=1) # Check fixed_distributions doesn't change. assert trial.distributions == fixed_distributions @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. trial = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.0, values=None, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 1}, distributions={"x": IntDistribution(-1, 1)}, user_attrs={}, system_attrs={}, intermediate_values={}, ) kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. trial.suggest_int("x", -1, 1, *args) optuna-3.5.0/tests/trial_tests/test_trial.py000066400000000000000000000703451453453102400212620ustar00rootroot00000000000000from __future__ import annotations import datetime import math from typing import Any from typing import Dict from typing import List from typing import Optional from typing import Tuple from unittest.mock import Mock from unittest.mock import patch import warnings import numpy as np import pytest import optuna from optuna import create_study from optuna import distributions from optuna import load_study from optuna import samplers from optuna import storages from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.testing.pruners import DeterministicPruner from optuna.testing.samplers import DeterministicSampler from optuna.testing.storages import STORAGE_MODES from optuna.testing.storages import StorageSupplier from optuna.testing.tempfile_pool import NamedTemporaryFilePool from optuna.trial import Trial from optuna.trial._trial import _LazyTrialSystemAttrs @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_float(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) x1 = trial.suggest_float("x1", 10, 20) x2 = trial.suggest_uniform("x1", 10, 20) assert x1 == x2 x3 = trial.suggest_float("x2", 1e-5, 1e-3, log=True) x4 = trial.suggest_loguniform("x2", 1e-5, 1e-3) assert x3 == x4 x5 = trial.suggest_float("x3", 10, 20, step=1.0) x6 = trial.suggest_discrete_uniform("x3", 10, 20, 1.0) assert x5 == x6 with pytest.raises(ValueError): trial.suggest_float("x4", 1e-5, 1e-2, step=1e-5, log=True) with pytest.raises(ValueError): trial.suggest_int("x1", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x1", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_uniform(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_uniform("x", 10, 20) trial.suggest_uniform("x", 10, 20) trial.suggest_uniform("x", 10, 30) # we expect exactly one warning (not counting ones caused by deprecation) assert len([r for r in record if r.category != FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_loguniform(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_loguniform("x", 10, 20) trial.suggest_loguniform("x", 10, 20) trial.suggest_loguniform("x", 10, 30) # we expect exactly one warning (not counting ones caused by deprecation) assert len([r for r in record if r.category != FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_discrete_uniform(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_discrete_uniform("x", 10, 20, 2) trial.suggest_discrete_uniform("x", 10, 20, 2) trial.suggest_discrete_uniform("x", 10, 22, 2) # we expect exactly one warning (not counting ones caused by deprecation) assert len([r for r in record if r.category != FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 10, 20, step=2) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20, step=2) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize("enable_log", [False, True]) def test_check_distribution_suggest_int(storage_mode: str, enable_log: bool) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns() as record: trial.suggest_int("x", 10, 20, log=enable_log) trial.suggest_int("x", 10, 20, log=enable_log) trial.suggest_int("x", 10, 22, log=enable_log) # We expect exactly one warning (not counting ones caused by deprecation). assert len([r for r in record if r.category != FutureWarning]) == 1 with pytest.raises(ValueError): trial.suggest_float("x", 10, 20, log=enable_log) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_float("x", 10, 20, log=enable_log) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_check_distribution_suggest_categorical(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) trial.suggest_categorical("x", [10, 20, 30]) with pytest.raises(ValueError): trial.suggest_categorical("x", [10, 20]) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("x", 10, 20) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_uniform(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1.0, "y": 2.0}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_uniform("x", 0.0, 3.0) == 1.0 # Test suggesting a param. assert trial.suggest_uniform("x", 0.0, 3.0) == 1.0 # Test suggesting the same param. assert trial.suggest_uniform("y", 0.0, 3.0) == 2.0 # Test suggesting a different param. assert trial.params == {"x": 1.0, "y": 2.0} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_loguniform(storage_mode: str) -> None: with pytest.raises(ValueError): FloatDistribution(low=1.0, high=0.9, log=True) with pytest.raises(ValueError): FloatDistribution(low=0.0, high=0.9, log=True) sampler = DeterministicSampler({"x": 1.0, "y": 2.0}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_loguniform("x", 0.1, 4.0) == 1.0 # Test suggesting a param. assert trial.suggest_loguniform("x", 0.1, 4.0) == 1.0 # Test suggesting the same param. assert trial.suggest_loguniform("y", 0.1, 4.0) == 2.0 # Test suggesting a different param. assert trial.params == {"x": 1.0, "y": 2.0} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_discrete_uniform(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1.0, "y": 2.0}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert ( trial.suggest_discrete_uniform("x", 0.0, 3.0, 1.0) == 1.0 ) # Test suggesting a param. assert ( trial.suggest_discrete_uniform("x", 0.0, 3.0, 1.0) == 1.0 ) # Test suggesting the same param. assert ( trial.suggest_discrete_uniform("y", 0.0, 3.0, 1.0) == 2.0 ) # Test suggesting a different param. assert trial.params == {"x": 1.0, "y": 2.0} @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_low_equals_high(storage_mode: str) -> None: with patch.object( distributions, "_get_single_value", wraps=distributions._get_single_value ) as mock_object, StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=samplers.TPESampler(n_startup_trials=0)) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_uniform("a", 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 1 assert trial.suggest_uniform("a", 1.0, 1.0) == 1.0 # Suggesting the same param. assert mock_object.call_count == 1 assert trial.suggest_loguniform("b", 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 2 assert trial.suggest_loguniform("b", 1.0, 1.0) == 1.0 # Suggesting the same param. assert mock_object.call_count == 2 assert trial.suggest_discrete_uniform("c", 1.0, 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 3 assert ( trial.suggest_discrete_uniform("c", 1.0, 1.0, 1.0) == 1.0 ) # Suggesting the same param. assert mock_object.call_count == 3 assert trial.suggest_int("d", 1, 1) == 1 # Suggesting a param. assert mock_object.call_count == 4 assert trial.suggest_int("d", 1, 1) == 1 # Suggesting the same param. assert mock_object.call_count == 4 assert trial.suggest_float("e", 1.0, 1.0) == 1.0 # Suggesting a param. assert mock_object.call_count == 5 assert trial.suggest_float("e", 1.0, 1.0) == 1.0 # Suggesting the same param. assert mock_object.call_count == 5 assert trial.suggest_float("f", 0.5, 0.5, log=True) == 0.5 # Suggesting a param. assert mock_object.call_count == 6 assert trial.suggest_float("f", 0.5, 0.5, log=True) == 0.5 # Suggesting the same param. assert mock_object.call_count == 6 assert trial.suggest_float("g", 0.5, 0.5, log=False) == 0.5 # Suggesting a param. assert mock_object.call_count == 7 assert trial.suggest_float("g", 0.5, 0.5, log=False) == 0.5 # Suggesting the same param. assert mock_object.call_count == 7 assert trial.suggest_float("h", 0.5, 0.5, step=1.0) == 0.5 # Suggesting a param. assert mock_object.call_count == 8 assert trial.suggest_float("h", 0.5, 0.5, step=1.0) == 0.5 # Suggesting the same param. assert mock_object.call_count == 8 assert trial.suggest_int("i", 1, 1, log=True) == 1 # Suggesting a param. assert mock_object.call_count == 9 assert trial.suggest_int("i", 1, 1, log=True) == 1 # Suggesting the same param. assert mock_object.call_count == 9 @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "range_config", [ {"low": 0.0, "high": 10.0, "q": 3.0, "mod_high": 9.0}, {"low": 1.0, "high": 11.0, "q": 3.0, "mod_high": 10.0}, {"low": 64.0, "high": 1312.0, "q": 160.0, "mod_high": 1184.0}, {"low": 0.0, "high": 10.0, "q": math.pi, "mod_high": 3 * math.pi}, {"low": 0.0, "high": 3.45, "q": 0.1, "mod_high": 3.4}, ], ) def test_suggest_discrete_uniform_range(storage_mode: str, range_config: Dict[str, float]) -> None: sampler = samplers.RandomSampler() # Check upper endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.high with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_discrete_uniform( "x", range_config["low"], range_config["high"], range_config["q"] ) assert x == range_config["mod_high"] assert mock_object.call_count == 1 # Check lower endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.low with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_discrete_uniform( "x", range_config["low"], range_config["high"], range_config["q"] ) assert x == range_config["low"] assert mock_object.call_count == 1 def test_suggest_float_invalid_step() -> None: study = create_study() trial = study.ask() with pytest.raises(ValueError): trial.suggest_float("x1", 10, 20, step=0) with pytest.raises(ValueError): trial.suggest_float("x2", 10, 20, step=-1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1, "y": 2}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_int("x", 0, 3) == 1 # Test suggesting a param. assert trial.suggest_int("x", 0, 3) == 1 # Test suggesting the same param. assert trial.suggest_int("y", 0, 3) == 2 # Test suggesting a different param. assert trial.params == {"x": 1, "y": 2} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) @pytest.mark.parametrize( "range_config", [ {"low": 0, "high": 10, "step": 3, "mod_high": 9}, {"low": 1, "high": 11, "step": 3, "mod_high": 10}, {"low": 64, "high": 1312, "step": 160, "mod_high": 1184}, ], ) def test_suggest_int_range(storage_mode: str, range_config: Dict[str, int]) -> None: sampler = samplers.RandomSampler() # Check upper endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.high with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_int( "x", range_config["low"], range_config["high"], step=range_config["step"] ) assert x == range_config["mod_high"] assert mock_object.call_count == 1 # Check lower endpoints. mock = Mock() mock.side_effect = lambda study, trial, param_name, distribution: distribution.low with patch.object(sampler, "sample_independent", mock) as mock_object, StorageSupplier( storage_mode ) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.warns(UserWarning): x = trial.suggest_int( "x", range_config["low"], range_config["high"], step=range_config["step"] ) assert x == range_config["low"] assert mock_object.call_count == 1 def test_suggest_int_invalid_step() -> None: study = create_study() trial = study.ask() with pytest.raises(ValueError): trial.suggest_int("x1", 10, 20, step=0) with pytest.raises(ValueError): trial.suggest_int("x2", 10, 20, step=-1) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int_log(storage_mode: str) -> None: sampler = DeterministicSampler({"x": 1, "y": 2}) with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) assert trial.suggest_int("x", 1, 3, log=True) == 1 # Test suggesting a param. assert trial.suggest_int("x", 1, 3, log=True) == 1 # Test suggesting the same param. assert trial.suggest_int("y", 1, 3, log=True) == 2 # Test suggesting a different param. assert trial.params == {"x": 1, "y": 2} @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_suggest_int_log_invalid_range(storage_mode: str) -> None: sampler = samplers.RandomSampler() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with warnings.catch_warnings(): # UserWarning will be raised since [0.5, 10] is not divisible by 1. warnings.simplefilter("ignore", category=UserWarning) with pytest.raises(ValueError): trial.suggest_int("z", 0.5, 10, log=True) # type: ignore with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) trial = Trial(study, study._storage.create_new_trial(study._study_id)) with pytest.raises(ValueError): trial.suggest_int("w", 1, 3, step=2, log=True) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_distributions(storage_mode: str) -> None: def objective(trial: Trial) -> float: trial.suggest_float("a", 0, 10) trial.suggest_float("b", 0.1, 10, log=True) trial.suggest_float("c", 0, 10, step=1) trial.suggest_int("d", 0, 10) trial.suggest_categorical("e", ["foo", "bar", "baz"]) trial.suggest_int("f", 1, 10, log=True) return 1.0 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=1) assert study.best_trial.distributions == { "a": FloatDistribution(low=0, high=10), "b": FloatDistribution(low=0.1, high=10, log=True), "c": FloatDistribution(low=0, high=10, step=1), "d": IntDistribution(low=0, high=10), "e": CategoricalDistribution(choices=("foo", "bar", "baz")), "f": IntDistribution(low=1, high=10, log=True), } def test_should_prune() -> None: pruner = DeterministicPruner(True) study = create_study(pruner=pruner) trial = Trial(study, study._storage.create_new_trial(study._study_id)) trial.report(1, 1) assert trial.should_prune() @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_relative_parameters(storage_mode: str) -> None: class SamplerStubForTestRelativeParameters(samplers.BaseSampler): def infer_relative_search_space( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial" ) -> Dict[str, distributions.BaseDistribution]: return { "x": FloatDistribution(low=5, high=6), "y": FloatDistribution(low=5, high=6), } def sample_relative( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", search_space: Dict[str, distributions.BaseDistribution], ) -> Dict[str, Any]: return {"x": 5.5, "y": 5.5, "z": 5.5} def sample_independent( self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial", param_name: str, param_distribution: distributions.BaseDistribution, ) -> Any: return 5.0 sampler = SamplerStubForTestRelativeParameters() with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage, sampler=sampler) def create_trial() -> Trial: return Trial(study, study._storage.create_new_trial(study._study_id)) # Suggested by `sample_relative`. trial0 = create_trial() distribution0 = FloatDistribution(low=0, high=100) assert trial0._suggest("x", distribution0) == 5.5 # Not suggested by `sample_relative` (due to unknown parameter name). trial1 = create_trial() distribution1 = distribution0 assert trial1._suggest("w", distribution1) != 5.5 # Not suggested by `sample_relative` (due to incompatible value range). trial2 = create_trial() distribution2 = FloatDistribution(low=0, high=5) assert trial2._suggest("x", distribution2) != 5.5 # Error (due to incompatible distribution class). trial3 = create_trial() distribution3 = IntDistribution(low=1, high=100) with pytest.raises(ValueError): trial3._suggest("y", distribution3) # Error ('z' is included in `sample_relative` but not in `infer_relative_search_space`). trial4 = create_trial() distribution4 = FloatDistribution(low=0, high=10) with pytest.raises(ValueError): trial4._suggest("z", distribution4) # Error (due to incompatible distribution class). trial5 = create_trial() distribution5 = IntDistribution(low=1, high=100, log=True) with pytest.raises(ValueError): trial5._suggest("y", distribution5) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_datetime_start(storage_mode: str) -> None: trial_datetime_start: List[Optional[datetime.datetime]] = [None] def objective(trial: Trial) -> float: trial_datetime_start[0] = trial.datetime_start return 1.0 with StorageSupplier(storage_mode) as storage: study = create_study(storage=storage) study.optimize(objective, n_trials=1) assert study.trials[0].datetime_start == trial_datetime_start[0] def test_report() -> None: study = create_study() trial = Trial(study, study._storage.create_new_trial(study._study_id)) # Report values that can be cast to `float` (OK). trial.report(1.23, 1) trial.report(float("nan"), 2) trial.report("1.23", 3) # type: ignore trial.report("inf", 4) # type: ignore trial.report(1, 5) trial.report(np.array([1], dtype=np.float32)[0], 6) # Report values that cannot be cast to `float` or steps that are negative (Error). with pytest.raises(TypeError): trial.report(None, 7) # type: ignore with pytest.raises(TypeError): trial.report("foo", 7) # type: ignore with pytest.raises(TypeError): trial.report([1, 2, 3], 7) # type: ignore with pytest.raises(TypeError): trial.report("foo", -1) # type: ignore with pytest.raises(ValueError): trial.report(1.23, -1) def test_report_warning() -> None: study = create_study() trial = study.ask() trial.report(1.23, 1) # Warn if multiple times call report method at the same step with pytest.warns(UserWarning): trial.report(1, 1) def test_suggest_with_multi_objectives() -> None: study = create_study(directions=["maximize", "maximize"]) def objective(trial: Trial) -> Tuple[float, float]: p0 = trial.suggest_float("p0", -10, 10) p1 = trial.suggest_float("p1", 3, 5) p2 = trial.suggest_float("p2", 0.00001, 0.1, log=True) p3 = trial.suggest_float("p3", 100, 200, step=5) p4 = trial.suggest_int("p4", -20, -15) p5 = trial.suggest_categorical("p5", [7, 1, 100]) p6 = trial.suggest_float("p6", -10, 10, step=1.0) p7 = trial.suggest_int("p7", 1, 7, log=True) return ( p0 + p1 + p2, p3 + p4 + p5 + p6 + p7, ) study.optimize(objective, n_trials=10) def test_raise_error_for_report_with_multi_objectives() -> None: study = create_study(directions=["maximize", "maximize"]) def objective(trial: Trial) -> Tuple[float, float]: with pytest.raises(NotImplementedError): trial.report(1.0, 0) return 1.0, 1.0 study.optimize(objective, n_trials=1) def test_raise_error_for_should_prune_multi_objectives() -> None: study = create_study(directions=["maximize", "maximize"]) def objective(trial: Trial) -> Tuple[float, float]: with pytest.raises(NotImplementedError): trial.should_prune() return 1.0, 1.0 study.optimize(objective, n_trials=1) def test_persisted_param() -> None: study_name = "my_study" with NamedTemporaryFilePool() as fp: storage = f"sqlite:///{fp.name}" study = create_study(storage=storage, study_name=study_name) assert isinstance(study._storage, storages._CachedStorage), "Pre-condition." # Test more than one trial. The `_CachedStorage` does a cache miss for the first trial and # thus behaves differently for the first trial in comparisons to the following. for _ in range(3): trial = study.ask() trial.suggest_float("x", 0, 1) study = load_study(storage=storage, study_name=study_name) assert all("x" in t.params for t in study.trials) @pytest.mark.parametrize("storage_mode", STORAGE_MODES) def test_lazy_trial_system_attrs(storage_mode: str) -> None: with StorageSupplier(storage_mode) as storage: study = optuna.create_study(storage=storage) trial = study.ask() storage.set_trial_system_attr(trial._trial_id, "int", 0) storage.set_trial_system_attr(trial._trial_id, "str", "A") # _LazyTrialSystemAttrs gets attrs the first time it is needed. # Then, we create the instance for each method, and test the first and second use. system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert system_attrs == {"int": 0, "str": "A"} assert system_attrs == {"int": 0, "str": "A"} system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert len(system_attrs) == 2 assert len(system_attrs) == 2 system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert set(system_attrs.keys()) == {"int", "str"} assert set(system_attrs.keys()) == {"int", "str"} system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert set(system_attrs.values()) == {0, "A"} assert set(system_attrs.values()) == {0, "A"} system_attrs = _LazyTrialSystemAttrs(trial._trial_id, storage) assert set(system_attrs.items()) == {("int", 0), ("str", "A")} assert set(system_attrs.items()) == {("int", 0), ("str", "A")} @pytest.mark.parametrize("positional_args_names", [[], ["step"], ["step", "log"]]) def test_suggest_int_positional_args(positional_args_names: list[str]) -> None: # If log is specified as positional, step must also be provided as positional. study = optuna.create_study() trial = study.ask() kwargs = dict(step=1, log=False) args = [kwargs[name] for name in positional_args_names] # No error should not be raised even if the coding style is old. trial.suggest_int("x", -1, 1, *args) optuna-3.5.0/tests/trial_tests/test_trials.py000066400000000000000000000156341453453102400214450ustar00rootroot00000000000000import datetime import time from typing import Any from typing import Dict from typing import Optional import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution from optuna.study.study import create_study from optuna.trial import BaseTrial from optuna.trial import create_trial from optuna.trial import FixedTrial from optuna.trial import FrozenTrial from optuna.trial import Trial parametrize_trial_type = pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial, Trial]) def _create_trial( trial_type: type, params: Optional[Dict[str, Any]] = None, distributions: Optional[Dict[str, BaseDistribution]] = None, ) -> BaseTrial: if params is None: params = {"x": 10} assert params is not None if distributions is None: distributions = {"x": FloatDistribution(5, 12)} assert distributions is not None if trial_type == FixedTrial: return FixedTrial(params) elif trial_type == FrozenTrial: trial = create_trial(value=0.2, params=params, distributions=distributions) trial.number = 0 return trial elif trial_type == Trial: study = create_study() study.enqueue_trial(params) return study.ask() else: assert False @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_float(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.2}, distributions={"x": FloatDistribution(0.0, 1.0)} ) assert trial.suggest_float("x", 0.0, 1.0) == 0.2 with pytest.raises(ValueError): trial.suggest_float("x", 0.0, 1.0, step=10, log=True) with pytest.raises(ValueError): trial.suggest_float("y", 0.0, 1.0) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_uniform(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.2}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) assert trial.suggest_uniform("x", 0.0, 1.0) == 0.2 with pytest.raises(ValueError): trial.suggest_uniform("y", 0.0, 1.0) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_loguniform(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.99}, distributions={"x": FloatDistribution(0.1, 1.0, log=True)}, ) assert trial.suggest_loguniform("x", 0.1, 1.0) == 0.99 with pytest.raises(ValueError): trial.suggest_loguniform("y", 0.0, 1.0) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_discrete_uniform(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 0.9}, distributions={"x": FloatDistribution(0.0, 1.0, step=0.1)}, ) assert trial.suggest_discrete_uniform("x", 0.0, 1.0, 0.1) == 0.9 with pytest.raises(ValueError): trial.suggest_discrete_uniform("y", 0.0, 1.0, 0.1) @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_int_log(trial_type: type) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 1}, distributions={"x": IntDistribution(1, 10, log=True)}, ) assert trial.suggest_int("x", 1, 10, log=True) == 1 with pytest.raises(ValueError): trial.suggest_int("x", 1, 10, step=2, log=True) with pytest.raises(ValueError): trial.suggest_int("y", 1, 10, log=True) @pytest.mark.parametrize("trial_type", [FixedTrial, FrozenTrial]) def test_suggest_categorical(trial_type: type) -> None: # Integer categories. trial = _create_trial( trial_type=trial_type, params={"x": 1}, distributions={"x": CategoricalDistribution((0, 1, 2, 3))}, ) assert trial.suggest_categorical("x", (0, 1, 2, 3)) == 1 with pytest.raises(ValueError): trial.suggest_categorical("y", [0, 1, 2, 3]) # String categories. trial = _create_trial( trial_type=trial_type, params={"x": "baz"}, distributions={"x": CategoricalDistribution(("foo", "bar", "baz"))}, ) assert trial.suggest_categorical("x", ("foo", "bar", "baz")) == "baz" # Unknown parameter. with pytest.raises(ValueError): trial.suggest_categorical("y", ["foo", "bar", "baz"]) # Not in choices. with pytest.raises(ValueError): trial.suggest_categorical("x", ["foo", "bar"]) # Unknown parameter and bad category type. with pytest.warns(UserWarning): with pytest.raises(ValueError): # Must come after `pytest.warns` to catch failures. trial.suggest_categorical("x", [{"foo": "bar"}]) # type: ignore @parametrize_trial_type @pytest.mark.parametrize( ("suggest_func", "distribution"), [ (lambda trial, *args: trial.suggest_int(*args), IntDistribution(1, 10)), ( lambda trial, *args: trial.suggest_int(*args, log=True), IntDistribution(1, 10, log=True), ), (lambda trial, *args: trial.suggest_int(*args, step=2), IntDistribution(1, 9, step=2)), (lambda trial, *args: trial.suggest_float(*args), FloatDistribution(1, 10)), ( lambda trial, *args: trial.suggest_float(*args, log=True), FloatDistribution(1, 10, log=True), ), ( lambda trial, *args: trial.suggest_float(*args, step=1), FloatDistribution(1, 10, step=1), ), ], ) def test_not_contained_param( trial_type: type, suggest_func: Any, distribution: BaseDistribution ) -> None: trial = _create_trial( trial_type=trial_type, params={"x": 1.0}, distributions={"x": distribution}, ) with pytest.warns(UserWarning): assert suggest_func(trial, "x", 10, 100) == 1 @parametrize_trial_type def test_set_user_attrs(trial_type: type) -> None: trial = _create_trial(trial_type) trial.set_user_attr("data", "MNIST") assert trial.user_attrs["data"] == "MNIST" @parametrize_trial_type def test_report(trial_type: type) -> None: # Ignores reported values. trial = _create_trial(trial_type) trial.report(1.0, 1) trial.report(2.0, 2) @parametrize_trial_type def test_should_prune(trial_type: type) -> None: # Never prunes trials. assert not _create_trial(trial_type).should_prune() @parametrize_trial_type def test_datetime_start(trial_type: type) -> None: trial = _create_trial(trial_type) assert trial.datetime_start is not None old_date_time_start = trial.datetime_start time.sleep(0.001) # Sleep 1ms to avoid faulty assertion on Windows OS. assert datetime.datetime.now() != old_date_time_start optuna-3.5.0/tests/visualization_tests/000077500000000000000000000000001453453102400203135ustar00rootroot00000000000000optuna-3.5.0/tests/visualization_tests/__init__.py000066400000000000000000000000001453453102400224120ustar00rootroot00000000000000optuna-3.5.0/tests/visualization_tests/matplotlib_tests/000077500000000000000000000000001453453102400237045ustar00rootroot00000000000000optuna-3.5.0/tests/visualization_tests/matplotlib_tests/__init__.py000066400000000000000000000000001453453102400260030ustar00rootroot00000000000000optuna-3.5.0/tests/visualization_tests/matplotlib_tests/test_contour.py000066400000000000000000000020001453453102400267760ustar00rootroot00000000000000import numpy as np from optuna.visualization.matplotlib._contour import _create_zmap from optuna.visualization.matplotlib._contour import _interpolate_zmap def test_create_zmap() -> None: x_values = np.arange(10) y_values = np.arange(10) z_values = list(np.random.rand(10)) # we are testing for exact placement of z_values # so also passing x_values and y_values as xi and yi zmap = _create_zmap(x_values.tolist(), y_values.tolist(), z_values, x_values, y_values) assert len(zmap) == len(z_values) for coord, value in zmap.items(): # test if value under coordinate # still refers to original trial value xidx = coord[0] yidx = coord[1] assert xidx == yidx assert z_values[xidx] == value def test_interpolate_zmap() -> None: contour_point_num = 2 zmap = {(0, 0): 1.0, (1, 1): 4.0} expected = np.array([[1.0, 2.5], [2.5, 4.0]]) actual = _interpolate_zmap(zmap, contour_point_num) assert np.allclose(expected, actual) optuna-3.5.0/tests/visualization_tests/matplotlib_tests/test_optimization_history.py000066400000000000000000000022721453453102400316270ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import pytest from optuna.visualization._optimization_history import _OptimizationHistoryInfo from optuna.visualization.matplotlib._matplotlib_imports import plt from optuna.visualization.matplotlib._optimization_history import _get_optimization_history_plot from tests.visualization_tests.test_optimization_history import optimization_history_info_lists @pytest.mark.parametrize("target_name", ["Objective Value", "Target Name"]) @pytest.mark.parametrize("info_list", optimization_history_info_lists) def test_get_optimization_history_plot( target_name: str, info_list: list[_OptimizationHistoryInfo] ) -> None: figure = _get_optimization_history_plot(info_list, target_name=target_name) assert figure.get_ylabel() == target_name expected_legends = [] for info in info_list: expected_legends.append(info.values_info.label_name) if info.best_values_info is not None: expected_legends.append(info.best_values_info.label_name) legends = [legend.get_text() for legend in figure.legend().get_texts()] assert sorted(legends) == sorted(expected_legends) plt.savefig(BytesIO()) plt.close() optuna-3.5.0/tests/visualization_tests/matplotlib_tests/test_timeline.py000066400000000000000000000012101453453102400271150ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import pytest from optuna.trial import TrialState from optuna.visualization.matplotlib._timeline import plot_timeline from tests.visualization_tests.test_timeline import _create_study @pytest.mark.parametrize( "trial_states_list", [ [], [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL], [TrialState.FAIL, TrialState.PRUNED, TrialState.COMPLETE], ], ) def test_get_timeline_plot(trial_states_list: list[TrialState]) -> None: study = _create_study(trial_states_list) fig = plot_timeline(study) fig.get_figure().savefig(BytesIO()) optuna-3.5.0/tests/visualization_tests/test_contour.py000066400000000000000000000534331453453102400234250ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Any from typing import Callable import numpy as np import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_contour as plotly_plot_contour from optuna.visualization._contour import _AxisInfo from optuna.visualization._contour import _ContourInfo from optuna.visualization._contour import _get_contour_info from optuna.visualization._contour import _SubContourInfo from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib import plot_contour as plt_plot_contour from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_contour = pytest.mark.parametrize( "plot_contour", [plotly_plot_contour, plt_plot_contour] ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_log_scale_and_str_category_2d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101"}, distributions=distributions ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100"}, distributions=distributions ) ) return study def _create_study_with_log_scale_and_str_category_3d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), "param_c": CategoricalDistribution(["one", "two"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101", "param_c": "one"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100", "param_c": "two"}, distributions=distributions, ) ) return study def _create_study_mixture_category_types() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution([None, "100"]), "param_b": CategoricalDistribution([101, 102.0]), } study.add_trial( create_trial( value=0.0, params={"param_a": None, "param_b": 101}, distributions=distributions ) ) study.add_trial( create_trial( value=0.5, params={"param_a": "100", "param_b": 102.0}, distributions=distributions ) ) return study def _create_study_with_overlapping_params(direction: str) -> Study: study = create_study(direction=direction) distributions = { "param_a": FloatDistribution(1.0, 2.0), "param_b": CategoricalDistribution(["100", "101"]), "param_c": CategoricalDistribution(["foo", "bar"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1.0, "param_b": "101", "param_c": "foo"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1.0, "param_b": "101", "param_c": "bar"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 2.0, "param_b": "100", "param_c": "foo"}, distributions=distributions, ) ) return study @parametrize_plot_contour def test_plot_contour_customized_target_name(plot_contour: Callable[..., Any]) -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() figure = plot_contour(study, params=params, target_name="Target Name") if isinstance(figure, go.Figure): assert figure.data[0]["colorbar"].title.text == "Target Name" elif isinstance(figure, Axes): assert figure.figure.axes[-1].get_ylabel() == "Target Name" @parametrize_plot_contour @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, []], [create_study, ["param_a"]], [create_study, ["param_a", "param_b"]], [create_study, ["param_a", "param_b", "param_c"]], [create_study, ["param_a", "param_b", "param_c", "param_d"]], [create_study, None], [_create_study_with_failed_trial, []], [_create_study_with_failed_trial, ["param_a"]], [_create_study_with_failed_trial, ["param_a", "param_b"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c", "param_d"]], [_create_study_with_failed_trial, None], [prepare_study_with_trials, []], [prepare_study_with_trials, ["param_a"]], [prepare_study_with_trials, ["param_a", "param_b"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c", "param_d"]], [prepare_study_with_trials, None], [_create_study_with_log_scale_and_str_category_2d, None], [_create_study_with_log_scale_and_str_category_3d, None], [_create_study_mixture_category_types, None], ], ) def test_plot_contour( plot_contour: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_contour(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_contour_info(study) @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_contour_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_contour_info(study, params=params) assert len(info.sorted_params) == 0 assert len(info.sub_plot_infos) == 0 def test_get_contour_info_non_exist_param_error() -> None: study = prepare_study_with_trials() with pytest.raises(ValueError): _get_contour_info(study, ["optuna"]) @pytest.mark.parametrize("params", [[], ["param_a"]]) def test_get_contour_info_too_short_params(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_contour_info(study, params=params) assert len(info.sorted_params) == len(params) assert len(info.sub_plot_infos) == len(params) def test_get_contour_info_2_params() -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() info = _get_contour_info(study, params=params) assert info == _ContourInfo( sorted_params=params, sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(0.925, 2.575), is_log=False, is_cat=False, indices=[0.925, 1.0, 2.5, 2.575], values=[1.0, None, 2.5], ), yaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, indices=[-0.1, 0.0, 1.0, 2.0, 2.1], values=[2.0, 0.0, 1.0], ), z_values={(1, 3): 0.0, (2, 2): 1.0}, constraints=[True, True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_contour_info_more_than_2_params(params: list[str] | None) -> None: study = prepare_study_with_trials() n_params = len(params) if params is not None else 4 info = _get_contour_info(study, params=params) assert len(info.sorted_params) == n_params assert np.shape(np.asarray(info.sub_plot_infos, dtype=object)) == (n_params, n_params, 4) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ], ) def test_get_contour_info_customized_target(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_contour_info( study, params=params, target=lambda t: t.params["param_d"], target_name="param_d" ) n_params = len(params) assert len(info.sorted_params) == n_params plot_shape = (1, 1, 4) if n_params == 2 else (n_params, n_params, 4) assert np.shape(np.asarray(info.sub_plot_infos, dtype=object)) == plot_shape @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # `x_axis` has one observation. ["param_b", "param_a"], # `y_axis` has one observation. ], ) def test_generate_contour_plot_for_few_observations(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_contour_info(study, params=params) assert info == _ContourInfo( sorted_params=sorted(params), sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name=sorted(params)[0], range=(1.0, 1.0), is_log=False, is_cat=False, indices=[1.0], values=[1.0, None], ), yaxis=_AxisInfo( name=sorted(params)[1], range=(-0.1, 2.1), is_log=False, is_cat=False, indices=[-0.1, 0.0, 2.0, 2.1], values=[2.0, 0.0], ), z_values={}, constraints=[], ) ] ], reverse_scale=True, target_name="Objective Value", ) def test_get_contour_info_log_scale_and_str_category_2_params() -> None: # If the search space has two parameters, plot_contour generates a single plot. study = _create_study_with_log_scale_and_str_category_2d() info = _get_contour_info(study) assert info == _ContourInfo( sorted_params=["param_a", "param_b"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(math.pow(10, -6.05), math.pow(10, -4.95)), is_log=True, is_cat=False, indices=[math.pow(10, -6.05), 1e-6, 1e-5, math.pow(10, -4.95)], values=[1e-6, 1e-5], ), yaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=True, indices=["100", "101"], values=["101", "100"], ), z_values={(1, 1): 0.0, (2, 0): 1.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) def test_get_contour_info_log_scale_and_str_category_more_than_2_params() -> None: # If the search space has three parameters, plot_contour generates nine plots. study = _create_study_with_log_scale_and_str_category_3d() info = _get_contour_info(study) params = ["param_a", "param_b", "param_c"] assert info.sorted_params == params assert np.shape(np.asarray(info.sub_plot_infos, dtype=object)) == (3, 3, 4) ranges = { "param_a": (math.pow(10, -6.05), math.pow(10, -4.95)), "param_b": (-0.05, 1.05), "param_c": (-0.05, 1.05), } is_log = {"param_a": True, "param_b": False, "param_c": False} is_cat = {"param_a": False, "param_b": True, "param_c": True} indices: dict[str, list[str | float]] = { "param_a": [math.pow(10, -6.05), 1e-6, 1e-5, math.pow(10, -4.95)], "param_b": ["100", "101"], "param_c": ["one", "two"], } values = {"param_a": [1e-6, 1e-5], "param_b": ["101", "100"], "param_c": ["one", "two"]} def _check_axis(axis: _AxisInfo, name: str) -> None: assert axis.name == name assert axis.range == ranges[name] assert axis.is_log == is_log[name] assert axis.is_cat == is_cat[name] assert axis.indices == indices[name] assert axis.values == values[name] for yi in range(3): for xi in range(3): xaxis = info.sub_plot_infos[yi][xi].xaxis yaxis = info.sub_plot_infos[yi][xi].yaxis x_param = params[xi] y_param = params[yi] _check_axis(xaxis, x_param) _check_axis(yaxis, y_param) z_values = info.sub_plot_infos[yi][xi].z_values if xi == yi: assert z_values == {} else: for i, v in enumerate([0.0, 1.0]): x_value = xaxis.values[i] y_value = yaxis.values[i] assert x_value is not None assert y_value is not None xi = xaxis.indices.index(x_value) yi = yaxis.indices.index(y_value) assert z_values[(xi, yi)] == v def test_get_contour_info_mixture_category_types() -> None: study = _create_study_mixture_category_types() info = _get_contour_info(study) assert info == _ContourInfo( sorted_params=["param_a", "param_b"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(-0.05, 1.05), is_log=False, is_cat=True, indices=["100", "None"], values=["None", "100"], ), yaxis=_AxisInfo( name="param_b", range=(100.95, 102.05), is_log=False, is_cat=False, indices=[100.95, 101, 102, 102.05], values=[101.0, 102.0], ), z_values={(0, 2): 0.5, (1, 1): 0.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("value", [float("inf"), -float("inf")]) def test_get_contour_info_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_contour_info(study, params=["param_b", "param_d"]) assert info == _ContourInfo( sorted_params=["param_b", "param_d"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=False, indices=[-0.05, 0.0, 1.0, 1.05], values=[0.0, 1.0], ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, indices=[1.9, 2.0, 4.0, 4.1], values=[4.0, 2.0], ), z_values={(1, 2): 2.0, (2, 1): 1.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_get_contour_info_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_contour_info( study, params=["param_b", "param_d"], target=lambda t: t.values[objective], target_name="Target Name", ) assert info == _ContourInfo( sorted_params=["param_b", "param_d"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=False, indices=[-0.05, 0.0, 1.0, 1.05], values=[0.0, 1.0], ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, indices=[1.9, 2.0, 4.0, 4.1], values=[4.0, 2.0], ), z_values={(1, 2): 2.0, (2, 1): 1.0}, constraints=[True, True], ) ] ], reverse_scale=True, target_name="Target Name", ) @pytest.mark.parametrize("direction,expected", (("minimize", 0.0), ("maximize", 1.0))) def test_get_contour_info_overlapping_params(direction: str, expected: float) -> None: study = _create_study_with_overlapping_params(direction) info = _get_contour_info(study, params=["param_a", "param_b"]) assert info == _ContourInfo( sorted_params=["param_a", "param_b"], sub_plot_infos=[ [ _SubContourInfo( xaxis=_AxisInfo( name="param_a", range=(0.95, 2.05), is_log=False, is_cat=False, indices=[0.95, 1.0, 2.0, 2.05], values=[1.0, 1.0, 2.0], ), yaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=True, indices=["100", "101"], values=["101", "101", "100"], ), z_values={(1, 1): expected, (2, 0): 1.0}, constraints=[True, True, True], ) ] ], reverse_scale=False if direction == "maximize" else True, target_name="Objective Value", ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(direction=direction) for i in range(3): study.add_trial( create_trial( value=float(i), params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # `target` is `None`. contour = plotly_plot_contour(study).data[0] assert COLOR_SCALE == [v[1] for v in contour["colorscale"]] if direction == "minimize": assert contour["reversescale"] else: assert not contour["reversescale"] # When `target` is not `None`, `reversescale` is always `True`. contour = plotly_plot_contour(study, target=lambda t: t.number, target_name="Number").data[0] assert COLOR_SCALE == [v[1] for v in contour["colorscale"]] assert contour["reversescale"] # Multi-objective optimization. study = create_study(directions=[direction, direction]) for i in range(3): study.add_trial( create_trial( values=[float(i), float(i)], params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) contour = plotly_plot_contour(study, target=lambda t: t.number, target_name="Number").data[0] assert COLOR_SCALE == [v[1] for v in contour["colorscale"]] assert contour["reversescale"] optuna-3.5.0/tests/visualization_tests/test_edf.py000066400000000000000000000154631453453102400224730ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import numpy as np import pytest import optuna from optuna import Study from optuna.study import create_study from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_edf as plotly_plot_edf from optuna.visualization._edf import _EDFInfo from optuna.visualization._edf import _get_edf_info from optuna.visualization._edf import NUM_SAMPLES_X_AXIS from optuna.visualization._plotly_imports import _imports as plotly_imports from optuna.visualization.matplotlib import plot_edf as plt_plot_edf from optuna.visualization.matplotlib._matplotlib_imports import _imports as plt_imports if plotly_imports.is_successful(): from optuna.visualization._plotly_imports import go if plt_imports.is_successful(): from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrized_plot_edf = pytest.mark.parametrize("plot_edf", [plotly_plot_edf, plt_plot_edf]) def save_static_image(figure: go.Figure | Axes | np.ndarray) -> None: if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @parametrized_plot_edf def _validate_edf_values(edf_values: np.ndarray) -> None: np_values = np.array(edf_values) # Confirms that the values are monotonically non-decreasing. assert np.all(np.diff(np_values) >= 0.0) # Confirms that the values are in [0,1]. assert np.all((0 <= np_values) & (np_values <= 1)) @parametrized_plot_edf def test_target_is_none_and_study_is_multi_obj(plot_edf: Callable[..., Any]) -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): plot_edf(study) @parametrized_plot_edf @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_edf_plot_no_trials(plot_edf: Callable[..., Any], direction: str) -> None: figure = plot_edf(create_study(direction=direction)) save_static_image(figure) @parametrized_plot_edf @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("num_studies", [0, 1, 2]) def test_edf_plot_no_trials_studies( plot_edf: Callable[..., Any], direction: str, num_studies: int ) -> None: studies = [create_study(direction=direction) for _ in range(num_studies)] figure = plot_edf(studies) save_static_image(figure) @parametrized_plot_edf @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("num_studies", [0, 1, 2]) def test_plot_edf_with_multiple_studies( plot_edf: Callable[..., Any], direction: str, num_studies: int ) -> None: studies = [] for _ in range(num_studies): study = create_study(direction=direction) study.optimize(lambda t: t.suggest_float("x", 0, 5), n_trials=10) studies.append(study) figure = plot_edf(studies) save_static_image(figure) @parametrized_plot_edf def test_plot_edf_with_target(plot_edf: Callable[..., Any]) -> None: study = create_study() study.optimize(lambda t: t.suggest_float("x", 0, 5), n_trials=10) with pytest.warns(UserWarning): figure = plot_edf(study, target=lambda t: t.params["x"]) save_static_image(figure) @parametrized_plot_edf @pytest.mark.parametrize("target_name", [None, "Target Name"]) def test_plot_edf_with_target_name(plot_edf: Callable[..., Any], target_name: str | None) -> None: study = create_study() study.optimize(lambda t: t.suggest_float("x", 0, 5), n_trials=10) if target_name is None: figure = plot_edf(study) else: figure = plot_edf(study, target_name=target_name) expected = target_name if target_name is not None else "Objective Value" if isinstance(figure, go.Figure): assert figure.layout.xaxis.title.text == expected elif isinstance(figure, Axes): assert figure.xaxis.label.get_text() == expected save_static_image(figure) def test_empty_edf_info() -> None: def _assert_empty(info: _EDFInfo) -> None: assert info.lines == [] np.testing.assert_array_equal(info.x_values, np.array([])) edf_info = _get_edf_info([]) _assert_empty(edf_info) study = create_study() edf_info = _get_edf_info(study) _assert_empty(edf_info) trial = study.ask() study.tell(trial, state=optuna.trial.TrialState.PRUNED) edf_info = _get_edf_info(study) _assert_empty(edf_info) @pytest.mark.parametrize("n_studies", [1, 2, 3]) @pytest.mark.parametrize("target", [None, lambda t: t.params["x"]]) def test_get_edf_info(n_studies: int, target: Callable[[optuna.trial.FrozenTrial], float]) -> None: studies = [] n_trials = 3 max_target = 5 for i in range(n_studies): study = create_study(study_name=str(i)) study.optimize(lambda t: t.suggest_float("x", 0, max_target), n_trials=n_trials) studies.append(study) info = _get_edf_info(studies, target=target, target_name="Target Name") assert info.x_values.shape == (NUM_SAMPLES_X_AXIS,) assert all([0 <= x <= max_target for x in info.x_values]) assert len(info.lines) == n_studies for i, line in enumerate(info.lines): assert str(i) == line.study_name assert line.y_values.shape == (NUM_SAMPLES_X_AXIS,) _validate_edf_values(line.y_values) @pytest.mark.parametrize("value", [float("inf"), -float("inf"), float("nan")]) def test_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) edf_info = _get_edf_info(study) assert all(np.isfinite(edf_info.x_values)) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) edf_info = _get_edf_info( study, target=lambda t: t.values[objective], target_name="Target Name" ) assert all(np.isfinite(edf_info.x_values)) def test_inconsistent_number_of_trial_values() -> None: studies: list[Study] = [] n_studies = 5 for i in range(n_studies): study = prepare_study_with_trials() if i % 2 == 0: study.add_trial(create_trial(value=1.0)) studies.append(study) edf_info = _get_edf_info(studies) x_values = edf_info.x_values min_objective = 0.0 max_objective = 2.0 assert np.min(x_values) == min_objective assert np.max(x_values) == max_objective assert len(x_values) == NUM_SAMPLES_X_AXIS lines = edf_info.lines assert len(lines) == n_studies for line, study in zip(lines, studies): assert line.study_name == study.study_name _validate_edf_values(line.y_values) optuna-3.5.0/tests/visualization_tests/test_hypervolume_history.py000066400000000000000000000036331453453102400260710ustar00rootroot00000000000000from typing import Sequence import numpy as np import pytest from optuna.samplers import NSGAIISampler from optuna.study import create_study from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.visualization._hypervolume_history import _get_hypervolume_history_info from optuna.visualization._hypervolume_history import _HypervolumeHistoryInfo @pytest.mark.parametrize( "directions", [ ["minimize", "minimize"], ["minimize", "maximize"], ["maximize", "minimize"], ["maximize", "maximize"], ], ) def test_get_optimization_history_info(directions: str) -> None: signs = [1 if d == "minimize" else -1 for d in directions] def objective(trial: Trial) -> Sequence[float]: def impl(trial: Trial) -> Sequence[float]: if trial.number == 0: return 1.5, 1.5 # dominated by the reference_point elif trial.number == 1: return 0.75, 0.75 elif trial.number == 2: return 0.5, 0.5 # dominates Trial #1 elif trial.number == 3: return 0.5, 0.5 # dominates Trial #1 elif trial.number == 4: return 0.75, 0.25 # incomparable return 0.0, 0.0 # dominates all values = impl(trial) return signs[0] * values[0], signs[1] * values[1] def constraints(trial: FrozenTrial) -> Sequence[float]: if trial.number == 2: return (1,) # infeasible return (0,) # feasible sampler = NSGAIISampler(constraints_func=constraints) study = create_study(directions=directions, sampler=sampler) study.optimize(objective, n_trials=6) reference_point = np.asarray(signs) info = _get_hypervolume_history_info(study, reference_point) assert info == _HypervolumeHistoryInfo( trial_numbers=[0, 1, 2, 3, 4, 5], values=[0.0, 0.0625, 0.0625, 0.25, 0.3125, 1.0] ) optuna-3.5.0/tests/visualization_tests/test_intermediate_plot.py000066400000000000000000000107261453453102400254420ustar00rootroot00000000000000from io import BytesIO from typing import Any from typing import Callable from typing import Sequence import pytest from optuna.study import create_study from optuna.testing.objectives import fail_objective from optuna.trial import FrozenTrial from optuna.trial import Trial import optuna.visualization._intermediate_values from optuna.visualization._intermediate_values import _get_intermediate_plot_info from optuna.visualization._intermediate_values import _IntermediatePlotInfo from optuna.visualization._intermediate_values import _TrialInfo from optuna.visualization._plotly_imports import go import optuna.visualization.matplotlib._intermediate_values from optuna.visualization.matplotlib._matplotlib_imports import plt def test_intermediate_plot_info() -> None: # Test with no trials. study = create_study(direction="minimize") assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo(trial_infos=[]) # Test with a trial with intermediate values. def objective(trial: Trial, report_intermediate_values: bool) -> float: if report_intermediate_values: trial.report(1.0, step=0) trial.report(2.0, step=1) return 0.0 study = create_study() study.optimize(lambda t: objective(t, True), n_trials=1) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ) ] ) # Test a study with one trial with intermediate values and # one trial without intermediate values. # Expect the trial with no intermediate values to be ignored. study.optimize(lambda t: objective(t, False), n_trials=1) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ) ] ) # Test a study of only one trial that has no intermediate values. study = create_study() study.optimize(lambda t: objective(t, False), n_trials=1) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo(trial_infos=[]) # Ignore failed trials. study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo(trial_infos=[]) # Test a study with constraints def objective_with_constraints(trial: Trial) -> float: trial.set_user_attr("constraint", [trial.number % 2]) trial.report(1.0, step=0) trial.report(2.0, step=1) return 0.0 def constraints(trial: FrozenTrial) -> Sequence[float]: return trial.user_attrs["constraint"] study = create_study(sampler=optuna.samplers.NSGAIIISampler(constraints_func=constraints)) study.optimize(objective_with_constraints, n_trials=2) assert _get_intermediate_plot_info(study) == _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ), _TrialInfo( trial_number=1, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=False ), ] ) @pytest.mark.parametrize( "plotter", [ optuna.visualization._intermediate_values._get_intermediate_plot, optuna.visualization.matplotlib._intermediate_values._get_intermediate_plot, ], ) @pytest.mark.parametrize( "info", [ _IntermediatePlotInfo(trial_infos=[]), _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ) ] ), _IntermediatePlotInfo( trial_infos=[ _TrialInfo( trial_number=0, sorted_intermediate_values=[(0, 1.0), (1, 2.0)], feasible=True ), _TrialInfo( trial_number=1, sorted_intermediate_values=[(1, 2.0), (0, 1.0)], feasible=False ), ] ), ], ) def test_plot_intermediate_values( plotter: Callable[[_IntermediatePlotInfo], Any], info: _IntermediatePlotInfo ) -> None: figure = plotter(info) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() optuna-3.5.0/tests/visualization_tests/test_optimization_history.py000066400000000000000000000354061453453102400262430ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Sequence import numpy as np import pytest from optuna import samplers from optuna import TrialPruned from optuna.study import create_study from optuna.testing.objectives import fail_objective from optuna.testing.objectives import pruned_objective from optuna.trial import FrozenTrial from optuna.trial import Trial from optuna.visualization._optimization_history import _get_optimization_history_info_list from optuna.visualization._optimization_history import _get_optimization_history_plot from optuna.visualization._optimization_history import _OptimizationHistoryInfo from optuna.visualization._optimization_history import _ValuesInfo from optuna.visualization._optimization_history import _ValueState def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=False ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_warn_default_target_name_with_customized_target(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) with pytest.warns(UserWarning): _get_optimization_history_info_list( study, target=lambda t: t.number, target_name="Objective Value", error_bar=error_bar ) # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] with pytest.warns(UserWarning): _get_optimization_history_info_list( studies, target=lambda t: t.number, target_name="Objective Value", error_bar=error_bar ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_info_with_no_trials(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) info_list = _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_ignore_failed_trials(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] for study in studies: study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) @pytest.mark.parametrize("error_bar", [False, True]) def test_ignore_pruned_trials(direction: str, error_bar: bool) -> None: # Single study. study = create_study(direction=direction) study.optimize(pruned_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( study, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] # Multiple studies. studies = [create_study(direction=direction) for _ in range(10)] for study in studies: study.optimize(pruned_objective, n_trials=1, catch=(ValueError,)) info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=error_bar ) assert info_list == [] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list(direction: str) -> None: target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: return 1.0 elif trial.number == 1: return 2.0 elif trial.number == 2: return 0.0 return 0.0 # Test with a trial. study = create_study(direction=direction) study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=False ) best_values = [1.0, 1.0, 0.0] if direction == "minimize" else [1.0, 2.0, 2.0] assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, target_name, [_ValueState.Feasible] * 3), _ValuesInfo(best_values, None, "Best Value", [_ValueState.Feasible] * 3), ) ] # Test customized target. info_list = _get_optimization_history_info_list( study, target=lambda t: t.number, target_name=target_name, error_bar=False ) assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([0.0, 1.0, 2.0], None, target_name, [_ValueState.Feasible] * 3), None, ) ] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_multiple_studies(direction: str) -> None: n_studies = 10 base_values = [1.0, 2.0, 0.0] base_best_values = [1.0, 1.0, 0.0] if direction == "minimize" else [1.0, 2.0, 2.0] target_name = "Target Name" value_states = [_ValueState.Feasible] * 3 # Test with trials. studies = [create_study(direction=direction) for _ in range(n_studies)] for i, study in enumerate(studies): study.optimize(lambda t: base_values[t.number] + i, n_trials=3) info_list = _get_optimization_history_info_list( studies, target=None, target_name=target_name, error_bar=False ) for i, info in enumerate(info_list): values_i = [v + i for v in base_values] best_values_i = [v + i for v in base_best_values] assert info == _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo(values_i, None, f"{target_name} of {studies[i].study_name}", value_states), _ValuesInfo( best_values_i, None, f"Best Value of {studies[i].study_name}", value_states ), ) # Test customized target. info_list = _get_optimization_history_info_list( studies, target=lambda t: t.number, target_name=target_name, error_bar=False ) for i, info in enumerate(info_list): assert info == _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo( [0.0, 1.0, 2.0], None, f"{target_name} of {studies[i].study_name}", value_states ), None, ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_infeasible(direction: str) -> None: target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: trial.set_user_attr("constraint", [0]) return 1.0 elif trial.number == 1: trial.set_user_attr("constraint", [0]) return 2.0 elif trial.number == 2: trial.set_user_attr("constraint", [1]) return 3.0 return 0.0 def constraints_func(trial: FrozenTrial) -> Sequence[float]: return trial.user_attrs["constraint"] sampler = samplers.TPESampler(constraints_func=constraints_func) study = create_study(sampler=sampler, direction=direction) study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=False ) best_values = [1.0, 1.0, 1.0] if direction == "minimize" else [1.0, 2.0, 2.0] states = [_ValueState.Feasible, _ValueState.Feasible, _ValueState.Infeasible] assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo( [1.0, 2.0, 3.0], None, target_name, states, ), _ValuesInfo(best_values, None, "Best Value", states), ) ] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_pruned_trial(direction: str) -> None: target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: return 1.0 elif trial.number == 1: return 2.0 elif trial.number == 2: raise TrialPruned() return 0.0 study = create_study(direction=direction) study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=False ) assert info_list[0].values_info.states == [ _ValueState.Feasible, _ValueState.Feasible, _ValueState.Incomplete, ] assert math.isnan(info_list[0].values_info.values[2]) if info_list[0].best_values_info is not None: assert ( info_list[0].best_values_info.values == [1.0, 1.0, 1.0] if direction == "minimize" else [1.0, 2.0, 2.0] ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_get_optimization_history_info_list_with_error_bar(direction: str) -> None: n_studies = 10 target_name = "Target Name" def objective(trial: Trial) -> float: if trial.number == 0: return 1.0 elif trial.number == 1: return 2.0 elif trial.number == 2: return 0.0 return 0.0 # Test with trials. studies = [create_study(direction=direction) for _ in range(n_studies)] for study in studies: study.optimize(objective, n_trials=3) info_list = _get_optimization_history_info_list( study, target=None, target_name=target_name, error_bar=True ) best_values = [1.0, 1.0, 0.0] if direction == "minimize" else [1.0, 2.0, 2.0] assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], [0.0, 0.0, 0.0], target_name, [_ValueState.Feasible] * 3), _ValuesInfo(best_values, [0.0, 0.0, 0.0], "Best Value", [_ValueState.Feasible] * 3), ) ] # Test customized target. info_list = _get_optimization_history_info_list( study, target=lambda t: t.number, target_name=target_name, error_bar=True ) assert info_list == [ _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([0.0, 1.0, 2.0], [0.0, 0.0, 0.0], target_name, [_ValueState.Feasible] * 3), None, ) ] @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_error_bar_in_optimization_history(direction: str) -> None: def objective(trial: Trial) -> float: return trial.suggest_float("x", 0, 1) studies = [create_study(direction=direction) for _ in range(3)] suggested_params = [0.1, 0.3, 0.2] for x, study in zip(suggested_params, studies): study.enqueue_trial({"x": x}) study.optimize(objective, n_trials=1) info_list = _get_optimization_history_info_list( studies, target=None, target_name="Objective Value", error_bar=True ) mean = np.mean(suggested_params).item() std = np.std(suggested_params).item() value_states = [_ValueState.Feasible] assert info_list == [ _OptimizationHistoryInfo( [0], _ValuesInfo([mean], [std], "Objective Value", value_states), _ValuesInfo([mean], [std], "Best Value", value_states), ) ] optimization_history_info_lists = [ [], # Empty info. [ # Vanilla. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Feasible] * 3), None, ) ], [ # with infeasible trial. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Infeasible] * 3), None, ) ], [ # with incomplete trial. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Incomplete] * 3), None, ) ], [ # Multiple. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Feasible] * 3), None, ), _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 1.0, 1.0], None, "Dummy", [_ValueState.Feasible] * 3), None, ), _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 1.0, 1.0], None, "Dummy", [_ValueState.Infeasible] * 3), None, ), ], [ # With best values. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], None, "Dummy", [_ValueState.Feasible] * 3), _ValuesInfo([1.0, 1.0, 1.0], None, "Best Value", [_ValueState.Feasible] * 3), ) ], [ # With error bar. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], [1.0, 2.0, 0.0], "Dummy", [_ValueState.Feasible] * 3), None, ) ], [ # With best values and error bar. _OptimizationHistoryInfo( [0, 1, 2], _ValuesInfo([1.0, 2.0, 0.0], [1.0, 2.0, 0.0], "Dummy", [_ValueState.Feasible] * 3), _ValuesInfo( [1.0, 1.0, 1.0], [1.0, 2.0, 0.0], "Best Value", [_ValueState.Feasible] * 3 ), ) ], ] @pytest.mark.parametrize("target_name", ["Objective Value", "Target Name"]) @pytest.mark.parametrize("info_list", optimization_history_info_lists) def test_get_optimization_history_plot( target_name: str, info_list: list[_OptimizationHistoryInfo] ) -> None: figure = _get_optimization_history_plot(info_list, target_name=target_name) assert figure.layout.yaxis.title.text == target_name expected_legends = [] for info in info_list: expected_legends.append(info.values_info.label_name) expected_legends.append("Infeasible Trial") if info.best_values_info is not None: expected_legends.append(info.best_values_info.label_name) legends = [scatter.name for scatter in figure.data if scatter.name is not None] assert sorted(legends) == sorted(expected_legends) figure.write_image(BytesIO()) optuna-3.5.0/tests/visualization_tests/test_parallel_coordinate.py000066400000000000000000000604701453453102400257360ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Any from typing import Callable import numpy as np import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import matplotlib from optuna.visualization import plot_parallel_coordinate as plotly_plot_parallel_coordinate from optuna.visualization._parallel_coordinate import _DimensionInfo from optuna.visualization._parallel_coordinate import _get_parallel_coordinate_info from optuna.visualization._parallel_coordinate import _ParallelCoordinateInfo from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_parallel_coordinate = pytest.mark.parametrize( "plot_parallel_coordinate", [plotly_plot_parallel_coordinate, matplotlib.plot_parallel_coordinate], ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_categorical_params() -> Study: study_categorical_params = create_study() distributions: dict[str, BaseDistribution] = { "category_a": CategoricalDistribution(("preferred", "opt")), "category_b": CategoricalDistribution(("net", "una")), } study_categorical_params.add_trial( create_trial( value=0.0, params={"category_a": "preferred", "category_b": "net"}, distributions=distributions, ) ) study_categorical_params.add_trial( create_trial( value=2.0, params={"category_a": "opt", "category_b": "una"}, distributions=distributions, ) ) return study_categorical_params def _create_study_with_numeric_categorical_params() -> Study: study_categorical_params = create_study() distributions: dict[str, BaseDistribution] = { "category_a": CategoricalDistribution((1, 2)), "category_b": CategoricalDistribution((10, 20, 30)), } study_categorical_params.add_trial( create_trial( value=0.0, params={"category_a": 2, "category_b": 20}, distributions=distributions, ) ) study_categorical_params.add_trial( create_trial( value=1.0, params={"category_a": 1, "category_b": 30}, distributions=distributions, ) ) study_categorical_params.add_trial( create_trial( value=2.0, params={"category_a": 2, "category_b": 10}, distributions=distributions, ) ) return study_categorical_params def _create_study_with_log_params() -> Study: study_log_params = create_study() distributions: dict[str, BaseDistribution] = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": FloatDistribution(1, 1000, log=True), } study_log_params.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": 10}, distributions=distributions, ) ) study_log_params.add_trial( create_trial( value=1.0, params={"param_a": 2e-5, "param_b": 200}, distributions=distributions, ) ) study_log_params.add_trial( create_trial( value=0.1, params={"param_a": 1e-4, "param_b": 30}, distributions=distributions, ) ) return study_log_params def _create_study_with_log_scale_and_str_and_numeric_category() -> Study: study_multi_distro_params = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution(("preferred", "opt")), "param_b": CategoricalDistribution((1, 2, 10)), "param_c": FloatDistribution(1, 1000, log=True), "param_d": CategoricalDistribution((1, -1, 2)), } study_multi_distro_params.add_trial( create_trial( value=0.0, params={"param_a": "preferred", "param_b": 2, "param_c": 30, "param_d": 2}, distributions=distributions, ) ) study_multi_distro_params.add_trial( create_trial( value=1.0, params={"param_a": "opt", "param_b": 1, "param_c": 200, "param_d": 2}, distributions=distributions, ) ) study_multi_distro_params.add_trial( create_trial( value=2.0, params={"param_a": "preferred", "param_b": 10, "param_c": 10, "param_d": 1}, distributions=distributions, ) ) study_multi_distro_params.add_trial( create_trial( value=3.0, params={"param_a": "opt", "param_b": 2, "param_c": 10, "param_d": -1}, distributions=distributions, ) ) return study_multi_distro_params def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_parallel_coordinate_info(study) def test_plot_parallel_coordinate_customized_target_name() -> None: study = prepare_study_with_trials() figure = plotly_plot_parallel_coordinate(study, target_name="Target Name") assert figure.data[0]["dimensions"][0]["label"] == "Target Name" figure = matplotlib.plot_parallel_coordinate(study, target_name="Target Name") assert figure.get_figure().axes[1].get_ylabel() == "Target Name" @parametrize_plot_parallel_coordinate @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, None], [prepare_study_with_trials, ["param_a", "param_b"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c", "param_d"]], [_create_study_with_failed_trial, None], [_create_study_with_categorical_params, None], [_create_study_with_numeric_categorical_params, None], [_create_study_with_log_params, None], [_create_study_with_log_scale_and_str_and_numeric_category, None], ], ) def test_plot_parallel_coordinate( plot_parallel_coordinate: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_parallel_coordinate(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() def test_get_parallel_coordinate_info() -> None: # Test with no trial. study = create_study() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(), range=(0, 0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) study = prepare_study_with_trials() # Test with no parameters. info = _get_parallel_coordinate_info(study, params=[]) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 2.0, 1.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) # Test with a trial. info = _get_parallel_coordinate_info(study, params=["param_a", "param_b"]) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 1.0), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), _DimensionInfo( label="param_b", values=(2.0, 1.0), range=(1.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Objective Value", ) # Test with a trial to select parameter. info = _get_parallel_coordinate_info(study, params=["param_a"]) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 1.0), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Objective Value", ) # Test with a customized target value. with pytest.warns(UserWarning): info = _get_parallel_coordinate_info( study, params=["param_a"], target=lambda t: t.params["param_b"] ) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(2.0, 1.0), range=(1.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Objective Value", ) # Test with a customized target name. info = _get_parallel_coordinate_info( study, params=["param_a", "param_b"], target_name="Target Name" ) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Target Name", values=(0.0, 1.0), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1.0, 2.5), range=(1.0, 2.5), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), _DimensionInfo( label="param_b", values=(2.0, 1.0), range=(1.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), ], reverse_scale=True, target_name="Target Name", ) # Test with wrong params that do not exist in trials. with pytest.raises(ValueError, match="Parameter optuna does not exist in your study."): _get_parallel_coordinate_info(study, params=["optuna", "optuna"]) # Ignore failed trials. study = _create_study_with_failed_trial() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(), range=(0, 0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_categorical_params() -> None: # Test with categorical params that cannot be converted to numeral. study = _create_study_with_categorical_params() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 2.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0, 1), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["preferred", "opt"], ), _DimensionInfo( label="category_b", values=(0, 1), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["net", "una"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_categorical_numeric_params() -> None: # Test with categorical params that can be interpreted as numeric params. study = _create_study_with_numeric_categorical_params() # Trials are sorted by using param_a and param_b, i.e., trial#1, trial#2, and trial#0. info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(1.0, 2.0, 0.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0, 1, 1), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["1", "2"], ), _DimensionInfo( label="category_b", values=(2, 0, 1), range=(0, 2), is_log=False, is_cat=True, tickvals=[0, 1, 2], ticktext=["10", "20", "30"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_log_params() -> None: # Test with log params. study = _create_study_with_log_params() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 1.0, 0.1), range=(0.0, 1.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(-6, math.log10(2e-5), -4), range=(-6.0, -4.0), is_log=True, is_cat=False, tickvals=[-6, -5, -4.0], ticktext=["1e-06", "1e-05", "0.0001"], ), _DimensionInfo( label="param_b", values=(1.0, math.log10(200), math.log10(30)), range=(1.0, math.log10(200)), is_log=True, is_cat=False, tickvals=[1.0, 2.0, math.log10(200)], ticktext=["10", "100", "200"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_unique_param() -> None: # Test case when one unique value is suggested during the optimization. study_categorical_params = create_study() distributions: dict[str, BaseDistribution] = { "category_a": CategoricalDistribution(("preferred", "opt")), "param_b": FloatDistribution(1, 1000, log=True), } study_categorical_params.add_trial( create_trial( value=0.0, params={"category_a": "preferred", "param_b": 30}, distributions=distributions, ) ) # Both parameters contain unique values. info = _get_parallel_coordinate_info(study_categorical_params) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0,), range=(0.0, 0.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0.0,), range=(0, 0), is_log=False, is_cat=True, tickvals=[0], ticktext=["preferred"], ), _DimensionInfo( label="param_b", values=(math.log10(30),), range=(math.log10(30), math.log10(30)), is_log=True, is_cat=False, tickvals=[math.log10(30)], ticktext=["30"], ), ], reverse_scale=True, target_name="Objective Value", ) study_categorical_params.add_trial( create_trial( value=2.0, params={"category_a": "preferred", "param_b": 20}, distributions=distributions, ) ) # Still "category_a" contains unique suggested value during the optimization. info = _get_parallel_coordinate_info(study_categorical_params) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(0.0, 2.0), range=(0.0, 2.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="category_a", values=(0, 0), range=(0, 0), is_log=False, is_cat=True, tickvals=[0], ticktext=["preferred"], ), _DimensionInfo( label="param_b", values=(math.log10(30), math.log10(20)), range=(math.log10(20), math.log10(30)), is_log=True, is_cat=False, tickvals=[math.log10(20), math.log10(30)], ticktext=["20", "30"], ), ], reverse_scale=True, target_name="Objective Value", ) def test_get_parallel_coordinate_info_with_log_scale_and_str_and_numeric_category() -> None: # Test with sample from multiple distributions including categorical params # that can be interpreted as numeric params. study = _create_study_with_log_scale_and_str_and_numeric_category() info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(1.0, 3.0, 0.0, 2.0), range=(0.0, 3.0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[ _DimensionInfo( label="param_a", values=(1, 1, 0, 0), range=(0, 1), is_log=False, is_cat=True, tickvals=[0, 1], ticktext=["preferred", "opt"], ), _DimensionInfo( label="param_b", values=(0, 1, 1, 2), range=(0, 2), is_log=False, is_cat=True, tickvals=[0, 1, 2], ticktext=["1", "2", "10"], ), _DimensionInfo( label="param_c", values=(math.log10(200), 1.0, math.log10(30), 1.0), range=(1.0, math.log10(200)), is_log=True, is_cat=False, tickvals=[1, 2, math.log10(200)], ticktext=["10", "100", "200"], ), _DimensionInfo( label="param_d", values=(2, 0, 2, 1), range=(0, 2), is_log=False, is_cat=True, tickvals=[0, 1, 2], ticktext=["-1", "1", "2"], ), ], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(direction=direction) for i in range(3): study.add_trial( create_trial( value=float(i), params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # `target` is `None`. line = plotly_plot_parallel_coordinate(study).data[0]["line"] assert COLOR_SCALE == [v[1] for v in line["colorscale"]] if direction == "minimize": assert line["reversescale"] else: assert not line["reversescale"] # When `target` is not `None`, `reversescale` is always `True`. line = plotly_plot_parallel_coordinate( study, target=lambda t: t.number, target_name="Target Name" ).data[0]["line"] assert COLOR_SCALE == [v[1] for v in line["colorscale"]] assert line["reversescale"] # Multi-objective optimization. study = create_study(directions=[direction, direction]) for i in range(3): study.add_trial( create_trial( values=[float(i), float(i)], params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) line = plotly_plot_parallel_coordinate( study, target=lambda t: t.number, target_name="Target Name" ).data[0]["line"] assert COLOR_SCALE == [v[1] for v in line["colorscale"]] assert line["reversescale"] def test_get_parallel_coordinate_info_only_missing_params() -> None: # When all trials contain only a part of parameters, # the plot returns an empty figure. study = create_study() study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6}, distributions={ "param_a": FloatDistribution(1e-7, 1e-2, log=True), }, ) ) study.add_trial( create_trial( value=1.0, params={"param_b": 200}, distributions={ "param_b": FloatDistribution(1, 1000, log=True), }, ) ) info = _get_parallel_coordinate_info(study) assert info == _ParallelCoordinateInfo( dim_objective=_DimensionInfo( label="Objective Value", values=(), range=(0, 0), is_log=False, is_cat=False, tickvals=[], ticktext=[], ), dims_params=[], reverse_scale=True, target_name="Objective Value", ) @pytest.mark.parametrize("value", [float("inf"), -float("inf"), float("nan")]) def test_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_parallel_coordinate_info(study) assert all(np.isfinite(info.dim_objective.values)) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"), float("nan"))) def test_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_parallel_coordinate_info( study, target=lambda t: t.values[objective], target_name="Target Name" ) assert all(np.isfinite(info.dim_objective.values)) optuna-3.5.0/tests/visualization_tests/test_param_importances.py000066400000000000000000000243251453453102400254360ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import pytest from optuna.distributions import FloatDistribution from optuna.importance import FanovaImportanceEvaluator from optuna.importance import MeanDecreaseImpurityImportanceEvaluator from optuna.importance._base import BaseImportanceEvaluator from optuna.samplers import RandomSampler from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.trial import Trial from optuna.visualization import plot_param_importances as plotly_plot_param_importances from optuna.visualization._param_importances import _get_importances_info from optuna.visualization._param_importances import _get_importances_infos from optuna.visualization._param_importances import _ImportancesInfo from optuna.visualization._plotly_imports import go from optuna.visualization.matplotlib import plot_param_importances as plt_plot_param_importances from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_param_importances = pytest.mark.parametrize( "plot_param_importances", [plotly_plot_param_importances, plt_plot_param_importances] ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_multiobjective_study_with_failed_trial() -> Study: study = create_study(directions=["minimize", "minimize"]) study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_multiobjective_study() -> Study: return prepare_study_with_trials(n_objectives=2) def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_importances_info( study=study, evaluator=None, params=None, target=None, target_name="Objective Value" ) @parametrize_plot_param_importances def test_plot_param_importances_customized_target_name( plot_param_importances: Callable[..., Any] ) -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() figure = plot_param_importances(study, params=params, target_name="Target Name") if isinstance(figure, go.Figure): assert figure.layout.xaxis.title.text == "Hyperparameter Importance" elif isinstance(figure, Axes): assert figure.figure.axes[0].get_xlabel() == "Hyperparameter Importance" @parametrize_plot_param_importances def test_plot_param_importances_multiobjective_all_objectives_displayed( plot_param_importances: Callable[..., Any] ) -> None: n_objectives = 2 params = ["param_a"] study = prepare_study_with_trials(n_objectives) figure = plot_param_importances(study, params=params) if isinstance(figure, go.Figure): assert len(figure.data) == n_objectives elif isinstance(figure, Axes): assert len(figure.patches) == n_objectives * len(params) @parametrize_plot_param_importances @pytest.mark.parametrize( "specific_create_study", [ create_study, _create_multiobjective_study, _create_study_with_failed_trial, _create_multiobjective_study_with_failed_trial, prepare_study_with_trials, ], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], None, ], ) def test_plot_param_importances( plot_param_importances: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_param_importances(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], None, ], ) def test_get_param_importances_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_importances_info( study, None, params=params, target=None, target_name="Objective Value" ) assert info == _ImportancesInfo( importance_values=[], param_names=[], importance_labels=[], target_name="Objective Value" ) @pytest.mark.parametrize( "specific_create_study,objective_names", [(create_study, ["Foo"]), (_create_multiobjective_study, ["Foo", "Bar"])], ) def test_get_param_importances_infos_custom_objective_names( specific_create_study: Callable[[], Study], objective_names: list[str] ) -> None: study = specific_create_study() study.set_metric_names(objective_names) infos = _get_importances_infos( study, evaluator=None, params=["param_a"], target=None, target_name="Objective Value" ) assert len(infos) == len(study.directions) assert all(info.target_name == expected for info, expected in zip(infos, objective_names)) @pytest.mark.parametrize( "specific_create_study,objective_names", [ (create_study, ["Objective Value"]), (_create_multiobjective_study, ["Objective Value 0", "Objective Value 1"]), ], ) def test_get_param_importances_infos_default_objective_names( specific_create_study: Callable[[], Study], objective_names: list[str] ) -> None: study = specific_create_study() infos = _get_importances_infos( study, evaluator=None, params=["param_a"], target=None, target_name="Objective Value" ) assert len(infos) == len(study.directions) assert all(info.target_name == expected for info, expected in zip(infos, objective_names)) def test_switch_label_when_param_insignificant() -> None: def _objective(trial: Trial) -> int: x = trial.suggest_int("x", 0, 2) _ = trial.suggest_int("y", -1, 1) return x**2 study = create_study() for x in range(1, 3): study.enqueue_trial({"x": x, "y": 0}) study.optimize(_objective, n_trials=2) info = _get_importances_info(study, None, None, None, "Objective Value") # Test if label for `y` param has been switched to `<0.01`. assert info.importance_labels == ["<0.01", "1.00"] @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) @pytest.mark.parametrize( "evaluator", [MeanDecreaseImpurityImportanceEvaluator(seed=10), FanovaImportanceEvaluator(seed=10)], ) @pytest.mark.parametrize("n_trials", [0, 10]) def test_get_info_importances_nonfinite_removed( inf_value: float, evaluator: BaseImportanceEvaluator, n_trials: int ) -> None: def _objective(trial: Trial) -> float: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1 + x2 * x3 seed = 13 target_name = "Objective Value" study = create_study(sampler=RandomSampler(seed=seed)) study.optimize(_objective, n_trials=n_trials) # Create param importances info without inf value. info_without_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=None, target_name=target_name ) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( value=inf_value, params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Create param importances info with inf value. info_with_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=None, target_name=target_name ) # Obtained info instances should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert info_with_inf == info_without_inf @pytest.mark.parametrize("target_idx", [0, 1]) @pytest.mark.parametrize("inf_value", [float("inf"), -float("inf")]) @pytest.mark.parametrize( "evaluator", [MeanDecreaseImpurityImportanceEvaluator(seed=10), FanovaImportanceEvaluator(seed=10)], ) @pytest.mark.parametrize("n_trial", [0, 10]) def test_multi_objective_trial_with_infinite_value_ignored( target_idx: int, inf_value: float, evaluator: BaseImportanceEvaluator, n_trial: int ) -> None: def _multi_objective_function(trial: Trial) -> tuple[float, float]: x1 = trial.suggest_float("x1", 0.1, 3) x2 = trial.suggest_float("x2", 0.1, 3, log=True) x3 = trial.suggest_float("x3", 2, 4, log=True) return x1, x2 * x3 seed = 13 target_name = "Target Name" study = create_study(directions=["minimize", "minimize"], sampler=RandomSampler(seed=seed)) study.optimize(_multi_objective_function, n_trials=n_trial) # Create param importances info without inf value. info_without_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=lambda t: t.values[target_idx], target_name=target_name, ) # A trial with an inf value is added into the study manually. study.add_trial( create_trial( values=[inf_value, inf_value], params={"x1": 1.0, "x2": 1.0, "x3": 3.0}, distributions={ "x1": FloatDistribution(low=0.1, high=3), "x2": FloatDistribution(low=0.1, high=3, log=True), "x3": FloatDistribution(low=2, high=4, log=True), }, ) ) # Create param importances info with inf value. info_with_inf = _get_importances_info( study, evaluator=evaluator, params=None, target=lambda t: t.values[target_idx], target_name=target_name, ) # Obtained info instances should be the same between with inf and without inf, # because the last trial whose objective value is an inf is ignored. assert info_with_inf == info_without_inf optuna-3.5.0/tests/visualization_tests/test_pareto_front.py000066400000000000000000000315401453453102400244310ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable from typing import Sequence import warnings import pytest import optuna from optuna import create_study from optuna import create_trial from optuna.distributions import FloatDistribution from optuna.study.study import Study from optuna.trial import FrozenTrial from optuna.visualization import plot_pareto_front import optuna.visualization._pareto_front from optuna.visualization._pareto_front import _get_pareto_front_info from optuna.visualization._pareto_front import _ParetoFrontInfo from optuna.visualization._plotly_imports import go from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib._matplotlib_imports import plt import optuna.visualization.matplotlib._pareto_front def test_get_pareto_front_info_infer_n_targets() -> None: study = optuna.create_study(directions=["minimize", "minimize"]) assert _get_pareto_front_info(study).n_targets == 2 study = optuna.create_study(directions=["minimize"] * 5) assert ( _get_pareto_front_info( study, target_names=["target1", "target2"], targets=lambda _: [0.0, 1.0] ).n_targets == 2 ) study = optuna.create_study(directions=["minimize"] * 5) study.optimize(lambda _: [0] * 5, n_trials=1) assert _get_pareto_front_info(study, targets=lambda _: [0.0, 1.0]).n_targets == 2 study = optuna.create_study(directions=["minimize"] * 2) with pytest.raises(ValueError): _get_pareto_front_info(study, targets=lambda _: [0.0, 1.0]) def create_study_2d() -> Study: study = optuna.create_study(directions=["minimize", "minimize"]) study.enqueue_trial({"x": 1, "y": 2}) study.enqueue_trial({"x": 1, "y": 1}) study.enqueue_trial({"x": 0, "y": 2}) study.enqueue_trial({"x": 1, "y": 0}) study.optimize(lambda t: [t.suggest_int("x", 0, 2), t.suggest_int("y", 0, 2)], n_trials=4) return study def create_study_3d() -> Study: study = optuna.create_study(directions=["minimize", "minimize", "minimize"]) study.enqueue_trial({"x": 1, "y": 2}) study.enqueue_trial({"x": 1, "y": 1}) study.enqueue_trial({"x": 0, "y": 2}) study.enqueue_trial({"x": 1, "y": 0}) study.optimize( lambda t: [t.suggest_int("x", 0, 2), t.suggest_int("y", 0, 2), 1.0], n_trials=4, ) return study @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1], [1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar"]]) @pytest.mark.parametrize("metric_names", [None, ["v0", "v1"]]) def test_get_pareto_front_info_unconstrained( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, metric_names: list[str] | None, ) -> None: if axis_order is not None and targets is not None: pytest.skip("skip using both axis_order and targets") study = create_study_2d() if metric_names is not None: study.set_metric_names(metric_names) trials = study.get_trials(deepcopy=False) with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, ) assert info == _ParetoFrontInfo( n_targets=2, target_names=target_names or metric_names or ["Objective 0", "Objective 1"], best_trials_with_values=[(trials[2], [0, 2]), (trials[3], [1, 0])], non_best_trials_with_values=[(trials[0], [1, 2]), (trials[1], [1, 1])] if include_dominated_trials else [], infeasible_trials_with_values=[], axis_order=axis_order or [0, 1], include_dominated_trials=include_dominated_trials, has_constraints_func=False, ) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1], [1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar"]]) @pytest.mark.parametrize("metric_names", [None, ["v0", "v1"]]) def test_get_pareto_front_info_constrained( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, metric_names: list[str] | None, ) -> None: if axis_order is not None and targets is not None: pytest.skip("skip using both axis_order and targets") study = create_study_2d() if metric_names is not None: study.set_metric_names(metric_names) trials = study.get_trials(deepcopy=False) # (x, y) = (1, 0) is infeasible; others are feasible. def constraints_func(t: FrozenTrial) -> Sequence[float]: return [1.0] if t.params["x"] == 1 and t.params["y"] == 0 else [-1.0] with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, constraints_func=constraints_func, ) assert info == _ParetoFrontInfo( n_targets=2, target_names=target_names or metric_names or ["Objective 0", "Objective 1"], best_trials_with_values=[(trials[1], [1, 1]), (trials[2], [0, 2])], non_best_trials_with_values=[(trials[0], [1, 2])] if include_dominated_trials else [], infeasible_trials_with_values=[(trials[3], [1, 0])], axis_order=axis_order or [0, 1], include_dominated_trials=include_dominated_trials, has_constraints_func=True, ) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("axis_order", [None, [0, 1, 2], [2, 1, 0]]) @pytest.mark.parametrize("targets", [None, lambda t: (t.values[0], t.values[1], t.values[2])]) @pytest.mark.parametrize("target_names", [None, ["Foo", "Bar", "Baz"]]) def test_get_pareto_front_info_3d( include_dominated_trials: bool, axis_order: list[int] | None, targets: Callable[[FrozenTrial], Sequence[float]] | None, target_names: list[str] | None, ) -> None: if axis_order is not None and targets is not None: pytest.skip("skip using both axis_order and targets") study = create_study_3d() trials = study.get_trials(deepcopy=False) with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=FutureWarning) info = _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, targets=targets, target_names=target_names, ) assert info == _ParetoFrontInfo( n_targets=3, target_names=target_names or ["Objective 0", "Objective 1", "Objective 2"], best_trials_with_values=[(trials[2], [0, 2, 1]), (trials[3], [1, 0, 1])], non_best_trials_with_values=[(trials[0], [1, 2, 1]), (trials[1], [1, 1, 1])] if include_dominated_trials else [], infeasible_trials_with_values=[], axis_order=axis_order or [0, 1, 2], include_dominated_trials=include_dominated_trials, has_constraints_func=False, ) def test_get_pareto_front_info_invalid_number_of_target_names() -> None: study = optuna.create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_pareto_front_info(study=study, target_names=["Foo"]) @pytest.mark.parametrize("n_dims", [1, 4]) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_unsupported_dimensions( n_dims: int, include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize"] * n_dims) with pytest.raises(ValueError): _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, constraints_func=constraints_func, ) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("axis_order", [[0, 1, 1], [0, 0], [0, 2], [-1, 1]]) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_invalid_axis_order( axis_order: list[int], include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_pareto_front_info( study=study, include_dominated_trials=include_dominated_trials, axis_order=axis_order, constraints_func=constraints_func, ) @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_invalid_target_values( include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize", "minimize"]) study.optimize(lambda _: [0, 0], n_trials=3) with pytest.raises(ValueError): _get_pareto_front_info( study=study, targets=lambda t: t.values[0], include_dominated_trials=include_dominated_trials, constraints_func=constraints_func, ) @pytest.mark.filterwarnings("ignore::FutureWarning") @pytest.mark.parametrize("include_dominated_trials", [False, True]) @pytest.mark.parametrize("constraints_func", [None, lambda _: [-1.0]]) def test_get_pareto_front_info_using_axis_order_and_targets( include_dominated_trials: bool, constraints_func: Callable[[FrozenTrial], Sequence[float]] | None, ) -> None: study = optuna.create_study(directions=["minimize", "minimize", "minimize"]) with pytest.raises(ValueError): _get_pareto_front_info( study=study, axis_order=[0, 1, 2], targets=lambda t: (t.values[0], t.values[1], t.values[2]), include_dominated_trials=include_dominated_trials, constraints_func=constraints_func, ) def test_constraints_func_experimental_warning() -> None: study = optuna.create_study(directions=["minimize", "minimize"]) with pytest.warns(optuna.exceptions.ExperimentalWarning): _get_pareto_front_info( study=study, constraints_func=lambda _: [1.0], ) @pytest.mark.parametrize( "plotter", [ optuna.visualization._pareto_front._get_pareto_front_plot, optuna.visualization.matplotlib._pareto_front._get_pareto_front_plot, ], ) @pytest.mark.parametrize( "info_template", [ _get_pareto_front_info(create_study_2d()), _get_pareto_front_info(create_study_3d()), ], ) @pytest.mark.parametrize("include_dominated_trials", [True, False]) @pytest.mark.parametrize("has_constraints_func", [True, False]) def test_get_pareto_front_plot( plotter: Callable[[_ParetoFrontInfo], Any], info_template: _ParetoFrontInfo, include_dominated_trials: bool, has_constraints_func: bool, ) -> None: info = info_template if not include_dominated_trials: info = info._replace(include_dominated_trials=False, non_best_trials_with_values=[]) if not has_constraints_func: info = info._replace(has_constraints_func=False, infeasible_trials_with_values=[]) figure = plotter(info) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(directions=[direction, direction]) for i in range(3): study.add_trial( create_trial( values=[float(i), float(i)], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # Since `plot_pareto_front`'s colormap depends on only trial.number, # `reversecale` is not in the plot. marker = plot_pareto_front(study).data[0]["marker"] assert COLOR_SCALE == [v[1] for v in marker["colorscale"]] assert "reversecale" not in marker optuna-3.5.0/tests/visualization_tests/test_rank.py000066400000000000000000000575651453453102400227010ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO import math from typing import Any from typing import Callable import numpy as np import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_rank as plotly_plot_rank from optuna.visualization._plotly_imports import go from optuna.visualization._rank import _AxisInfo from optuna.visualization._rank import _convert_color_idxs_to_scaled_rgb_colors from optuna.visualization._rank import _get_axis_info from optuna.visualization._rank import _get_order_with_same_order_averaging from optuna.visualization._rank import _get_rank_info from optuna.visualization._rank import _RankPlotInfo from optuna.visualization._rank import _RankSubplotInfo parametrize_plot_rank = pytest.mark.parametrize("plot_rank", [plotly_plot_rank]) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_log_scale_and_str_category_2d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101"}, distributions=distributions ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100"}, distributions=distributions ) ) return study def _create_study_with_log_scale_and_str_category_3d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), "param_c": CategoricalDistribution(["one", "two"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101", "param_c": "one"}, distributions=distributions, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100", "param_c": "two"}, distributions=distributions, ) ) return study def _create_study_with_constraints() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": FloatDistribution(0.1, 0.2), "param_b": FloatDistribution(0.3, 0.4), } study.add_trial( create_trial( value=0.0, params={"param_a": 0.11, "param_b": 0.31}, distributions=distributions, system_attrs={_CONSTRAINTS_KEY: [-0.1, 0.0]}, ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 0.19, "param_b": 0.34}, distributions=distributions, system_attrs={_CONSTRAINTS_KEY: [0.1, 0.0]}, ) ) return study def _create_study_mixture_category_types() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution([None, "100"]), "param_b": CategoricalDistribution([101, 102.0]), } study.add_trial( create_trial( value=0.0, params={"param_a": None, "param_b": 101}, distributions=distributions ) ) study.add_trial( create_trial( value=0.5, params={"param_a": "100", "param_b": 102.0}, distributions=distributions ) ) return study def _named_tuple_equal(t1: Any, t2: Any) -> bool: if isinstance(t1, np.ndarray): return bool(np.all(t1 == t2)) elif isinstance(t1, tuple) or isinstance(t1, list): if len(t1) != len(t2): return False for x, y in zip(t1, t2): if not _named_tuple_equal(x, y): return False return True else: return t1 == t2 def _get_nested_list_shape(nested_list: list[list[Any]]) -> tuple[int, int]: assert all(len(nested_list[0]) == len(row) for row in nested_list) return len(nested_list), len(nested_list[0]) @parametrize_plot_rank @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, []], [create_study, ["param_a"]], [create_study, ["param_a", "param_b"]], [create_study, ["param_a", "param_b", "param_c"]], [create_study, ["param_a", "param_b", "param_c", "param_d"]], [create_study, None], [_create_study_with_failed_trial, []], [_create_study_with_failed_trial, ["param_a"]], [_create_study_with_failed_trial, ["param_a", "param_b"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c"]], [_create_study_with_failed_trial, ["param_a", "param_b", "param_c", "param_d"]], [_create_study_with_failed_trial, None], [prepare_study_with_trials, []], [prepare_study_with_trials, ["param_a"]], [prepare_study_with_trials, ["param_a", "param_b"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c"]], [prepare_study_with_trials, ["param_a", "param_b", "param_c", "param_d"]], [prepare_study_with_trials, None], [_create_study_with_log_scale_and_str_category_2d, None], [_create_study_with_log_scale_and_str_category_3d, None], [_create_study_mixture_category_types, None], [_create_study_with_constraints, None], ], ) def test_plot_rank( plot_rank: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_rank(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_rank_info(study, params=None, target=None, target_name="Objective Value") @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_rank_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert len(info.params) == 0 assert len(info.sub_plot_infos) == 0 def test_get_rank_info_non_exist_param_error() -> None: study = prepare_study_with_trials() with pytest.raises(ValueError): _get_rank_info(study, ["optuna"], target=None, target_name="Objective Value") @pytest.mark.parametrize("params", [[], ["param_a"]]) def test_get_rank_info_too_short_params(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert len(info.params) == len(params) assert len(info.sub_plot_infos) == len(params) def test_get_rank_info_2_params() -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert _named_tuple_equal( info, _RankPlotInfo( params=params, sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_a", range=(0.925, 2.575), is_log=False, is_cat=False, ), yaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), xs=[1.0, 2.5], ys=[2.0, 1.0], trials=[study.trials[0], study.trials[2]], zs=np.array([0.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 0.5])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 2.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0, 0.5])), has_custom_target=False, ), ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_rank_info_more_than_2_params(params: list[str] | None) -> None: study = prepare_study_with_trials() n_params = len(params) if params is not None else 4 info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") assert len(info.params) == n_params assert _get_nested_list_shape(info.sub_plot_infos) == (n_params, n_params) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ], ) def test_get_rank_info_customized_target(params: list[str]) -> None: study = prepare_study_with_trials() info = _get_rank_info( study, params=params, target=lambda t: t.params["param_d"], target_name="param_d" ) n_params = len(params) assert len(info.params) == n_params plot_shape = (1, 1) if n_params == 2 else (n_params, n_params) assert _get_nested_list_shape(info.sub_plot_infos) == plot_shape @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # `x_axis` has one observation. ["param_b", "param_a"], # `y_axis` has one observation. ], ) def test_generate_rank_plot_for_no_plots(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") axis_infos = { "param_a": _AxisInfo( name="param_a", range=(1.0, 1.0), is_log=False, is_cat=False, ), "param_b": _AxisInfo( name="param_b", range=(0.0, 0.0), is_log=False, is_cat=False, ), } assert _named_tuple_equal( info, _RankPlotInfo( params=params, sub_plot_infos=[ [ _RankSubplotInfo( xaxis=axis_infos[params[0]], yaxis=axis_infos[params[1]], xs=[], ys=[], trials=[], zs=np.array([]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([])).reshape( -1, 3 ), ) ] ], target_name="Objective Value", zs=np.array([0.0, 2.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # `x_axis` has one observation. ["param_b", "param_a"], # `y_axis` has one observation. ], ) def test_generate_rank_plot_for_few_observations(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_rank_info(study, params=params, target=None, target_name="Objective Value") axis_infos = { "param_a": _AxisInfo( name="param_a", range=(1.0, 1.0), is_log=False, is_cat=False, ), "param_b": _AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), } assert _named_tuple_equal( info, _RankPlotInfo( params=params, sub_plot_infos=[ [ _RankSubplotInfo( xaxis=axis_infos[params[0]], yaxis=axis_infos[params[1]], xs=[study.get_trials()[0].params[params[0]]], ys=[study.get_trials()[0].params[params[1]]], trials=[study.get_trials()[0]], zs=np.array([0.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 2.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) def test_get_rank_info_log_scale_and_str_category_2_params() -> None: # If the search space has two parameters, plot_rank generates a single plot. study = _create_study_with_log_scale_and_str_category_2d() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") assert _named_tuple_equal( info, _RankPlotInfo( params=["param_a", "param_b"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_a", range=(math.pow(10, -6.05), math.pow(10, -4.95)), is_log=True, is_cat=False, ), yaxis=_AxisInfo( name="param_b", range=(-0.05, 1.05), is_log=False, is_cat=True, ), xs=[1e-6, 1e-5], ys=["101", "100"], trials=[study.trials[0], study.trials[1]], zs=np.array([0.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 1.0]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) def test_get_rank_info_log_scale_and_str_category_more_than_2_params() -> None: # If the search space has three parameters, plot_rank generates nine plots. study = _create_study_with_log_scale_and_str_category_3d() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") params = ["param_a", "param_b", "param_c"] assert info.params == params assert _get_nested_list_shape(info.sub_plot_infos) == (3, 3) ranges = { "param_a": (math.pow(10, -6.05), math.pow(10, -4.95)), "param_b": (-0.05, 1.05), "param_c": (-0.05, 1.05), } is_log = {"param_a": True, "param_b": False, "param_c": False} is_cat = {"param_a": False, "param_b": True, "param_c": True} param_values = {"param_a": [1e-6, 1e-5], "param_b": ["101", "100"], "param_c": ["one", "two"]} zs = np.array([0.0, 1.0]) colors = _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])) def _check_axis(axis: _AxisInfo, name: str) -> None: assert axis.name == name assert axis.range == ranges[name] assert axis.is_log == is_log[name] assert axis.is_cat == is_cat[name] for yi in range(3): for xi in range(3): xaxis = info.sub_plot_infos[yi][xi].xaxis yaxis = info.sub_plot_infos[yi][xi].yaxis x_param = params[xi] y_param = params[yi] _check_axis(xaxis, x_param) _check_axis(yaxis, y_param) assert info.sub_plot_infos[yi][xi].xs == param_values[x_param] assert info.sub_plot_infos[yi][xi].ys == param_values[y_param] assert info.sub_plot_infos[yi][xi].trials == study.trials assert np.all(info.sub_plot_infos[yi][xi].zs == zs) assert np.all(info.sub_plot_infos[yi][xi].colors == colors) info.target_name == "Objective Value" assert np.all(info.zs == zs) assert np.all(info.colors == colors) assert not info.has_custom_target def test_get_rank_info_mixture_category_types() -> None: study = _create_study_mixture_category_types() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") assert _named_tuple_equal( info, _RankPlotInfo( params=["param_a", "param_b"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_a", range=(-0.05, 1.05), is_log=False, is_cat=True, ), yaxis=_AxisInfo( name="param_b", range=(100.95, 102.05), is_log=False, is_cat=False, ), xs=[None, "100"], ys=[101, 102.0], trials=study.trials, zs=np.array([0.0, 0.5]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), ) ] ], target_name="Objective Value", zs=np.array([0.0, 0.5]), colors=_convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])), has_custom_target=False, ), ) @pytest.mark.parametrize("value", [float("inf"), float("-inf")]) def test_get_rank_info_nonfinite(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_rank_info( study, params=["param_b", "param_d"], target=None, target_name="Objective Value" ) colors = ( _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0, 0.5])) if value == float("-inf") else _convert_color_idxs_to_scaled_rgb_colors(np.array([1.0, 0.5, 0.0])) ) assert _named_tuple_equal( info, _RankPlotInfo( params=["param_b", "param_d"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, ), xs=[2.0, 0.0, 1.0], ys=[4.0, 4.0, 2.0], trials=study.trials, zs=np.array([value, 2.0, 1.0]), colors=colors, ) ] ], target_name="Objective Value", zs=np.array([value, 2.0, 1.0]), colors=colors, has_custom_target=False, ), ) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), float("-inf"))) def test_get_rank_info_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_rank_info( study, params=["param_b", "param_d"], target=lambda t: t.values[objective], target_name="Target Name", ) colors = ( _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0, 0.5])) if value == float("-inf") else _convert_color_idxs_to_scaled_rgb_colors(np.array([1.0, 0.5, 0.0])) ) assert _named_tuple_equal( info, _RankPlotInfo( params=["param_b", "param_d"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_AxisInfo( name="param_b", range=(-0.1, 2.1), is_log=False, is_cat=False, ), yaxis=_AxisInfo( name="param_d", range=(1.9, 4.1), is_log=False, is_cat=False, ), xs=[2.0, 0.0, 1.0], ys=[4.0, 4.0, 2.0], trials=study.trials, zs=np.array([value, 2.0, 1.0]), colors=colors, ) ] ], target_name="Target Name", zs=np.array([value, 2.0, 1.0]), colors=colors, has_custom_target=True, ), ) def test_generate_rank_info_with_constraints() -> None: study = _create_study_with_constraints() info = _get_rank_info(study, params=None, target=None, target_name="Objective Value") expected_color = _convert_color_idxs_to_scaled_rgb_colors(np.array([0.0, 1.0])) expected_color[1] = [204, 204, 204] assert _named_tuple_equal( info, _RankPlotInfo( params=["param_a", "param_b"], sub_plot_infos=[ [ _RankSubplotInfo( xaxis=_get_axis_info(study.trials, "param_a"), yaxis=_get_axis_info(study.trials, "param_b"), xs=[0.11, 0.19], ys=[0.31, 0.34], trials=study.trials, zs=np.array([0.0, 1.0]), colors=expected_color, ) ] ], target_name="Objective Value", zs=np.array([0.0, 1.0]), colors=expected_color, has_custom_target=False, ), ) def test_get_order_with_same_order_averaging() -> None: x = np.array([6.0, 2.0, 3.0, 1.0, 4.5, 4.5, 8.0, 8.0, 0.0, 8.0]) assert np.all(x == _get_order_with_same_order_averaging(x)) def test_convert_color_idxs_to_scaled_rgb_colors() -> None: x1 = np.array([0.1, 0.2]) result1 = _convert_color_idxs_to_scaled_rgb_colors(x1) np.testing.assert_array_equal(result1, [[69, 117, 180], [116, 173, 209]]) x2 = np.array([]) result2 = _convert_color_idxs_to_scaled_rgb_colors(x2) np.testing.assert_array_equal(result2, []) optuna-3.5.0/tests/visualization_tests/test_slice.py000066400000000000000000000342201453453102400230240ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import pytest from optuna.distributions import BaseDistribution from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.visualization import plot_slice as plotly_plot_slice from optuna.visualization._plotly_imports import go from optuna.visualization._slice import _get_slice_plot_info from optuna.visualization._slice import _SlicePlotInfo from optuna.visualization._slice import _SliceSubplotInfo from optuna.visualization._utils import COLOR_SCALE from optuna.visualization.matplotlib import plot_slice as plt_plot_slice from optuna.visualization.matplotlib._matplotlib_imports import Axes from optuna.visualization.matplotlib._matplotlib_imports import plt parametrize_plot_slice = pytest.mark.parametrize("plot_slice", [plotly_plot_slice, plt_plot_slice]) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _create_study_with_log_scale_and_str_category_2d() -> Study: study = create_study() distributions = { "param_a": FloatDistribution(1e-7, 1e-2, log=True), "param_b": CategoricalDistribution(["100", "101"]), } study.add_trial( create_trial( value=0.0, params={"param_a": 1e-6, "param_b": "101"}, distributions=distributions ) ) study.add_trial( create_trial( value=1.0, params={"param_a": 1e-5, "param_b": "100"}, distributions=distributions ) ) return study def _create_study_mixture_category_types() -> Study: study = create_study() distributions: dict[str, BaseDistribution] = { "param_a": CategoricalDistribution([None, "100"]), "param_b": CategoricalDistribution([101, 102.0]), } study.add_trial( create_trial( value=0.0, params={"param_a": None, "param_b": 101}, distributions=distributions ) ) study.add_trial( create_trial( value=0.5, params={"param_a": "100", "param_b": 102.0}, distributions=distributions ) ) return study @parametrize_plot_slice def test_plot_slice_customized_target_name(plot_slice: Callable[..., Any]) -> None: params = ["param_a", "param_b"] study = prepare_study_with_trials() figure = plot_slice(study, params=params, target_name="Target Name") if isinstance(figure, go.Figure): figure.layout.yaxis.title.text == "Target Name" elif isinstance(figure, Axes): assert figure[0].yaxis.label.get_text() == "Target Name" @parametrize_plot_slice @pytest.mark.parametrize( "specific_create_study, params", [ [create_study, []], [create_study, ["param_a"]], [create_study, None], [prepare_study_with_trials, []], [prepare_study_with_trials, ["param_a"]], [prepare_study_with_trials, None], [_create_study_with_log_scale_and_str_category_2d, None], [_create_study_mixture_category_types, None], ], ) def test_plot_slice( plot_slice: Callable[..., Any], specific_create_study: Callable[[], Study], params: list[str] | None, ) -> None: study = specific_create_study() figure = plot_slice(study, params=params) if isinstance(figure, go.Figure): figure.write_image(BytesIO()) else: plt.savefig(BytesIO()) plt.close() def test_target_is_none_and_study_is_multi_obj() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_slice_plot_info(study, None, target=None, target_name="Objective Value") @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize( "params", [ [], ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_slice_plot_info_empty( specific_create_study: Callable[[], Study], params: list[str] | None ) -> None: study = specific_create_study() info = _get_slice_plot_info(study, params=params, target=None, target_name="Objective Value") assert len(info.subplots) == 0 def test_get_slice_plot_info_non_exist_param_error() -> None: study = prepare_study_with_trials() with pytest.raises(ValueError): _get_slice_plot_info(study, params=["optuna"], target=None, target_name="Objective Value") @pytest.mark.parametrize( "params", [ ["param_a"], ["param_a", "param_b"], ["param_a", "param_b", "param_c"], ["param_a", "param_b", "param_c", "param_d"], None, ], ) def test_get_slice_plot_info_params(params: list[str] | None) -> None: study = prepare_study_with_trials() params = ["param_a", "param_b", "param_c", "param_d"] if params is None else params expected_subplot_infos = { "param_a": _SliceSubplotInfo( param_name="param_a", x=[1.0, 2.5], y=[0.0, 1.0], trial_numbers=[0, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), "param_b": _SliceSubplotInfo( param_name="param_b", x=[2.0, 0.0, 1.0], y=[0.0, 2.0, 1.0], trial_numbers=[0, 1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True, True], ), "param_c": _SliceSubplotInfo( param_name="param_c", x=[3.0, 4.5], y=[0.0, 1.0], trial_numbers=[0, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), "param_d": _SliceSubplotInfo( param_name="param_d", x=[4.0, 4.0, 2.0], y=[0.0, 2.0, 1.0], trial_numbers=[0, 1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True, True], ), } info = _get_slice_plot_info(study, params=params, target=None, target_name="Objective Value") assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[expected_subplot_infos[p] for p in params], ) def test_get_slice_plot_info_customized_target() -> None: params = ["param_a"] study = prepare_study_with_trials() info = _get_slice_plot_info( study, params=params, target=lambda t: t.params["param_d"], target_name="param_d", ) assert info == _SlicePlotInfo( target_name="param_d", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[1.0, 2.5], y=[4.0, 2.0], trial_numbers=[0, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) @pytest.mark.parametrize( "params", [ ["param_a", "param_b"], # First column has 1 observation. ["param_b", "param_a"], # Second column has 1 observation ], ) def test_get_slice_plot_info_for_few_observations(params: list[str]) -> None: study = create_study(direction="minimize") study.add_trial( create_trial( values=[0.0], params={"param_a": 1.0, "param_b": 2.0}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) study.add_trial( create_trial( values=[2.0], params={"param_b": 0.0}, distributions={"param_b": FloatDistribution(0.0, 3.0)}, ) ) info = _get_slice_plot_info(study, params, None, "Objective Value") assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[1.0], y=[0.0], trial_numbers=[0], is_log=False, is_numerical=True, x_labels=None, constraints=[True], ), _SliceSubplotInfo( param_name="param_b", x=[2.0, 0.0], y=[0.0, 2.0], trial_numbers=[0, 1], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) def test_get_slice_plot_info_log_scale_and_str_category_2_params() -> None: study = _create_study_with_log_scale_and_str_category_2d() info = _get_slice_plot_info(study, None, None, "Objective Value") distribution_b = study.trials[0].distributions["param_b"] assert isinstance(distribution_b, CategoricalDistribution) assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[1e-6, 1e-5], y=[0.0, 1.0], trial_numbers=[0, 1], is_log=True, is_numerical=True, x_labels=None, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_b", x=["101", "100"], y=[0.0, 1.0], trial_numbers=[0, 1], is_log=False, is_numerical=False, x_labels=distribution_b.choices, constraints=[True, True], ), ], ) def test_get_slice_plot_info_mixture_category_types() -> None: study = _create_study_mixture_category_types() info = _get_slice_plot_info(study, None, None, "Objective Value") distribution_a = study.trials[0].distributions["param_a"] distribution_b = study.trials[0].distributions["param_b"] assert isinstance(distribution_a, CategoricalDistribution) assert isinstance(distribution_b, CategoricalDistribution) assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_a", x=[None, "100"], y=[0.0, 0.5], trial_numbers=[0, 1], is_log=False, is_numerical=False, x_labels=distribution_a.choices, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_b", x=[101, 102.0], y=[0.0, 0.5], trial_numbers=[0, 1], is_log=False, is_numerical=False, x_labels=distribution_b.choices, constraints=[True, True], ), ], ) @pytest.mark.parametrize("value", [float("inf"), -float("inf")]) def test_get_slice_plot_info_nonfinite_removed(value: float) -> None: study = prepare_study_with_trials(value_for_first_trial=value) info = _get_slice_plot_info( study, params=["param_b", "param_d"], target=None, target_name="Objective Value" ) assert info == _SlicePlotInfo( target_name="Objective Value", subplots=[ _SliceSubplotInfo( param_name="param_b", x=[0.0, 1.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_d", x=[4.0, 2.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) @pytest.mark.parametrize("objective", (0, 1)) @pytest.mark.parametrize("value", (float("inf"), -float("inf"))) def test_get_slice_plot_info_nonfinite_multiobjective(objective: int, value: float) -> None: study = prepare_study_with_trials(n_objectives=2, value_for_first_trial=value) info = _get_slice_plot_info( study, params=["param_b", "param_d"], target=lambda t: t.values[objective], target_name="Target Name", ) assert info == _SlicePlotInfo( target_name="Target Name", subplots=[ _SliceSubplotInfo( param_name="param_b", x=[0.0, 1.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), _SliceSubplotInfo( param_name="param_d", x=[4.0, 2.0], y=[2.0, 1.0], trial_numbers=[1, 2], is_log=False, is_numerical=True, x_labels=None, constraints=[True, True], ), ], ) @pytest.mark.parametrize("direction", ["minimize", "maximize"]) def test_color_map(direction: str) -> None: study = create_study(direction=direction) for i in range(3): study.add_trial( create_trial( value=float(i), params={"param_a": float(i), "param_b": float(i)}, distributions={ "param_a": FloatDistribution(0.0, 3.0), "param_b": FloatDistribution(0.0, 3.0), }, ) ) # Since `plot_slice`'s colormap depends on only trial.number, `reversecale` is not in the plot. marker = plotly_plot_slice(study).data[0]["marker"] assert COLOR_SCALE == [v[1] for v in marker["colorscale"]] assert "reversecale" not in marker optuna-3.5.0/tests/visualization_tests/test_terminator_improvement.py000066400000000000000000000115261453453102400265420ustar00rootroot00000000000000from __future__ import annotations from io import BytesIO from typing import Any from typing import Callable import pytest from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.study import Study from optuna.terminator import BaseErrorEvaluator from optuna.terminator import BaseImprovementEvaluator from optuna.terminator import CrossValidationErrorEvaluator from optuna.terminator import RegretBoundEvaluator from optuna.terminator import report_cross_validation_scores from optuna.terminator import StaticErrorEvaluator from optuna.testing.objectives import fail_objective from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.trial import TrialState from optuna.visualization import plot_terminator_improvement as plotly_plot_terminator_improvement from optuna.visualization._terminator_improvement import _get_improvement_info from optuna.visualization._terminator_improvement import _get_y_range from optuna.visualization._terminator_improvement import _ImprovementInfo parametrize_plot_terminator_improvement = pytest.mark.parametrize( "plot_terminator_improvement", [plotly_plot_terminator_improvement] ) def _create_study_with_failed_trial() -> Study: study = create_study() study.optimize(fail_objective, n_trials=1, catch=(ValueError,)) return study def _prepare_study_with_cross_validation_scores() -> Study: study = create_study() for _ in range(3): trial = study.ask({"x": FloatDistribution(0, 1)}) report_cross_validation_scores(trial, [1.0, 2.0]) study.tell(trial, 0) return study def test_study_is_multi_objective() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _get_improvement_info(study=study) @parametrize_plot_terminator_improvement @pytest.mark.parametrize( "specific_create_study, plot_error", [ (create_study, False), (_create_study_with_failed_trial, False), (prepare_study_with_trials, False), (_prepare_study_with_cross_validation_scores, False), (_prepare_study_with_cross_validation_scores, True), ], ) def test_plot_terminator_improvement( plot_terminator_improvement: Callable[..., Any], specific_create_study: Callable[[], Study], plot_error: bool, ) -> None: study = specific_create_study() figure = plot_terminator_improvement(study, plot_error) figure.write_image(BytesIO()) @pytest.mark.parametrize( "specific_create_study", [create_study, _create_study_with_failed_trial], ) @pytest.mark.parametrize("plot_error", [False, True]) def test_get_terminator_improvement_info_empty( specific_create_study: Callable[[], Study], plot_error: bool ) -> None: study = specific_create_study() info = _get_improvement_info(study, plot_error) assert info == _ImprovementInfo(trial_numbers=[], improvements=[], errors=None) @pytest.mark.parametrize("get_error", [False, True]) @pytest.mark.parametrize( "improvement_evaluator_class", [lambda: RegretBoundEvaluator(), lambda: None] ) @pytest.mark.parametrize( "error_evaluator_class", [ lambda: CrossValidationErrorEvaluator(), lambda: StaticErrorEvaluator(0), lambda: None, ], ) def test_get_improvement_info( get_error: bool, improvement_evaluator_class: Callable[[], BaseImprovementEvaluator | None], error_evaluator_class: Callable[[], BaseErrorEvaluator | None], ) -> None: study = _prepare_study_with_cross_validation_scores() info = _get_improvement_info( study, get_error, improvement_evaluator_class(), error_evaluator_class() ) assert info.trial_numbers == [0, 1, 2] assert len(info.improvements) == 3 if get_error: assert info.errors is not None assert len(info.errors) == 3 assert info.errors[0] == info.errors[1] == info.errors[2] else: assert info.errors is None def test_get_improvement_info_started_with_failed_trials() -> None: study = create_study() for _ in range(3): study.add_trial(create_trial(state=TrialState.FAIL)) trial = study.ask({"x": FloatDistribution(0, 1)}) study.tell(trial, 0) info = _get_improvement_info(study) assert info.trial_numbers == [3] assert len(info.improvements) == 1 assert info.errors is None @pytest.mark.parametrize( "info", [ _ImprovementInfo(trial_numbers=[0], improvements=[0], errors=None), _ImprovementInfo(trial_numbers=[0], improvements=[0], errors=[0]), _ImprovementInfo(trial_numbers=[0, 1], improvements=[0, 1], errors=[0, 1]), ], ) @pytest.mark.parametrize("min_n_trials", [1, 2]) def test_get_y_range(info: _ImprovementInfo, min_n_trials: int) -> None: y_range = _get_y_range(info, min_n_trials) assert len(y_range) == 2 assert y_range[0] <= y_range[1] optuna-3.5.0/tests/visualization_tests/test_timeline.py000066400000000000000000000075721453453102400235450ustar00rootroot00000000000000from __future__ import annotations import datetime from io import BytesIO from typing import Any import _pytest.capture import pytest import optuna from optuna.samplers._base import _CONSTRAINTS_KEY from optuna.study.study import Study from optuna.trial import TrialState from optuna.visualization._plotly_imports import go from optuna.visualization._timeline import _get_timeline_info from optuna.visualization._timeline import plot_timeline def _create_study( trial_states_list: list[TrialState], trial_sys_attrs: dict[str, Any] | None = None, ) -> Study: study = optuna.create_study() fmax = float(len(trial_states_list)) for i, s in enumerate(trial_states_list): study.add_trial( optuna.trial.create_trial( params={"x": float(i)}, distributions={"x": optuna.distributions.FloatDistribution(-1.0, fmax)}, value=0.0, state=s, system_attrs=trial_sys_attrs, ) ) return study def _create_study_negative_elapsed_time() -> Study: start = datetime.datetime.now() complete = start - datetime.timedelta(seconds=1.0) study = optuna.create_study() study.add_trial( optuna.trial.FrozenTrial( number=-1, trial_id=-1, state=TrialState.COMPLETE, value=0.0, values=None, datetime_start=start, datetime_complete=complete, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, ) ) return study def test_get_timeline_info_empty() -> None: study = optuna.create_study() info = _get_timeline_info(study) assert len(info.bars) == 0 @pytest.mark.parametrize( "trial_sys_attrs, infeasible", [ (None, False), ({_CONSTRAINTS_KEY: [1.0]}, True), ({_CONSTRAINTS_KEY: [-1.0]}, False), ], ) def test_get_timeline_info(trial_sys_attrs: dict[str, Any] | None, infeasible: bool) -> None: states = [TrialState.COMPLETE, TrialState.RUNNING, TrialState.WAITING] study = _create_study(states, trial_sys_attrs) info = _get_timeline_info(study) assert len(info.bars) == len(study.get_trials()) for bar, trial in zip(info.bars, study.get_trials()): assert bar.number == trial.number assert bar.state == trial.state assert type(bar.hovertext) is str assert isinstance(bar.start, datetime.datetime) assert isinstance(bar.complete, datetime.datetime) assert bar.start <= bar.complete assert bar.infeasible == infeasible def test_get_timeline_info_negative_elapsed_time(capsys: _pytest.capture.CaptureFixture) -> None: # We need to reconstruct our default handler to properly capture stderr. optuna.logging._reset_library_root_logger() optuna.logging.enable_default_handler() optuna.logging.set_verbosity(optuna.logging.WARNING) study = _create_study_negative_elapsed_time() info = _get_timeline_info(study) _, err = capsys.readouterr() assert err != "" assert len(info.bars) == len(study.get_trials()) for bar, trial in zip(info.bars, study.get_trials()): assert bar.number == trial.number assert bar.state == trial.state assert type(bar.hovertext) is str assert isinstance(bar.start, datetime.datetime) assert isinstance(bar.complete, datetime.datetime) assert bar.complete < bar.start @pytest.mark.parametrize( "trial_states_list", [ [], [TrialState.COMPLETE, TrialState.PRUNED, TrialState.FAIL], [TrialState.FAIL, TrialState.PRUNED, TrialState.COMPLETE], ], ) def test_get_timeline_plot(trial_states_list: list[TrialState]) -> None: study = _create_study(trial_states_list) fig = plot_timeline(study) assert type(fig) is go.Figure fig.write_image(BytesIO()) optuna-3.5.0/tests/visualization_tests/test_utils.py000066400000000000000000000175021453453102400230710ustar00rootroot00000000000000import datetime import logging from textwrap import dedent from typing import cast import numpy as np import pytest from pytest import LogCaptureFixture import optuna from optuna.distributions import FloatDistribution from optuna.study import create_study from optuna.testing.visualization import prepare_study_with_trials from optuna.trial import create_trial from optuna.trial import FrozenTrial from optuna.trial import TrialState from optuna.visualization import is_available from optuna.visualization._utils import _check_plot_args from optuna.visualization._utils import _filter_nonfinite from optuna.visualization._utils import _is_log_scale from optuna.visualization._utils import _make_hovertext def test_is_log_scale() -> None: study = create_study() study.add_trial( create_trial( value=0.0, params={"param_linear": 1.0}, distributions={"param_linear": FloatDistribution(0.0, 3.0)}, ) ) study.add_trial( create_trial( value=2.0, params={"param_linear": 2.0, "param_log": 1e-3}, distributions={ "param_linear": FloatDistribution(0.0, 3.0), "param_log": FloatDistribution(1e-5, 1.0, log=True), }, ) ) assert _is_log_scale(study.trials, "param_log") assert not _is_log_scale(study.trials, "param_linear") def _is_plotly_available() -> bool: try: import plotly # NOQA available = True except Exception: available = False return available def test_visualization_is_available() -> None: assert is_available() == _is_plotly_available() def test_check_plot_args() -> None: study = create_study(directions=["minimize", "minimize"]) with pytest.raises(ValueError): _check_plot_args(study, None, "Objective Value") with pytest.warns(UserWarning): _check_plot_args(study, lambda t: cast(float, t.value), "Objective Value") @pytest.mark.parametrize( "value, expected", [(float("inf"), 1), (-float("inf"), 1), (float("nan"), 1), (0.0, 2)] ) def test_filter_inf_trials(value: float, expected: int) -> None: study = create_study() study.add_trial( create_trial( value=0.0, params={"x": 1.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( value=value, params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) trials = _filter_nonfinite(study.get_trials(states=(TrialState.COMPLETE,))) assert len(trials) == expected assert all([t.number == num for t, num in zip(trials, range(expected))]) @pytest.mark.parametrize( "value,objective_selected,expected", [ (float("inf"), 0, 2), (-float("inf"), 0, 2), (float("nan"), 0, 2), (0.0, 0, 3), (float("inf"), 1, 1), (-float("inf"), 1, 1), (float("nan"), 1, 1), (0.0, 1, 3), ], ) def test_filter_inf_trials_multiobjective( value: float, objective_selected: int, expected: int ) -> None: study = create_study(directions=["minimize", "maximize"]) study.add_trial( create_trial( values=[0.0, 1.0], params={"x": 1.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( values=[0.0, value], params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( values=[value, value], params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) def _target(t: FrozenTrial) -> float: return t.values[objective_selected] trials = _filter_nonfinite(study.get_trials(states=(TrialState.COMPLETE,)), target=_target) assert len(trials) == expected assert all([t.number == num for t, num in zip(trials, range(expected))]) @pytest.mark.parametrize("with_message", [True, False]) def test_filter_inf_trials_message(caplog: LogCaptureFixture, with_message: bool) -> None: study = create_study() study.add_trial( create_trial( value=0.0, params={"x": 1.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) study.add_trial( create_trial( value=float("inf"), params={"x": 0.0}, distributions={"x": FloatDistribution(0.0, 1.0)}, ) ) optuna.logging.enable_propagation() _filter_nonfinite(study.get_trials(states=(TrialState.COMPLETE,)), with_message=with_message) msg = "Trial 1 is omitted in visualization because its objective value is inf or nan." if with_message: assert msg in caplog.text n_filtered_as_inf = 0 for record in caplog.records: if record.msg == msg: assert record.levelno == logging.WARNING n_filtered_as_inf += 1 assert n_filtered_as_inf == 1 else: assert msg not in caplog.text @pytest.mark.filterwarnings("ignore::UserWarning") def test_filter_nonfinite_with_invalid_target() -> None: study = prepare_study_with_trials() trials = study.get_trials(states=(TrialState.COMPLETE,)) with pytest.raises(ValueError): _filter_nonfinite(trials, target=lambda t: "invalid target") # type: ignore def test_make_hovertext() -> None: trial_no_user_attrs = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.2, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 10}, distributions={"x": FloatDistribution(5, 12)}, user_attrs={}, system_attrs={}, intermediate_values={}, ) assert ( _make_hovertext(trial_no_user_attrs) == dedent( """ { "number": 0, "values": [ 0.2 ], "params": { "x": 10 } } """ ) .strip() .replace("\n", "
") ) trial_user_attrs_valid_json = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.2, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 10}, distributions={"x": FloatDistribution(5, 12)}, user_attrs={"a": 42, "b": 3.14}, system_attrs={}, intermediate_values={}, ) assert ( _make_hovertext(trial_user_attrs_valid_json) == dedent( """ { "number": 0, "values": [ 0.2 ], "params": { "x": 10 }, "user_attrs": { "a": 42, "b": 3.14 } } """ ) .strip() .replace("\n", "
") ) trial_user_attrs_invalid_json = FrozenTrial( number=0, trial_id=0, state=TrialState.COMPLETE, value=0.2, datetime_start=datetime.datetime.now(), datetime_complete=datetime.datetime.now(), params={"x": 10}, distributions={"x": FloatDistribution(5, 12)}, user_attrs={"a": 42, "b": 3.14, "c": np.zeros(1), "d": np.nan}, system_attrs={}, intermediate_values={}, ) assert ( _make_hovertext(trial_user_attrs_invalid_json) == dedent( """ { "number": 0, "values": [ 0.2 ], "params": { "x": 10 }, "user_attrs": { "a": 42, "b": 3.14, "c": "[0.]", "d": NaN } } """ ) .strip() .replace("\n", "
") ) optuna-3.5.0/tutorial/000077500000000000000000000000001453453102400146715ustar00rootroot00000000000000optuna-3.5.0/tutorial/10_key_features/000077500000000000000000000000001453453102400176575ustar00rootroot00000000000000optuna-3.5.0/tutorial/10_key_features/001_first.py000066400000000000000000000116151453453102400217440ustar00rootroot00000000000000""" .. _first: Lightweight, versatile, and platform agnostic architecture ========================================================== Optuna is entirely written in Python and has few dependencies. This means that we can quickly move to the real example once you get interested in Optuna. Quadratic Function Example -------------------------- Usually, Optuna is used to optimize hyperparameters, but as an example, let's optimize a simple quadratic function: :math:`(x - 2)^2`. """ ################################################################################################### # First of all, import :mod:`optuna`. import optuna ################################################################################################### # In optuna, conventionally functions to be optimized are named `objective`. def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 ################################################################################################### # This function returns the value of :math:`(x - 2)^2`. Our goal is to find the value of ``x`` # that minimizes the output of the ``objective`` function. This is the "optimization." # During the optimization, Optuna repeatedly calls and evaluates the objective function with # different values of ``x``. # # A :class:`~optuna.trial.Trial` object corresponds to a single execution of the objective # function and is internally instantiated upon each invocation of the function. # # The `suggest` APIs (for example, :func:`~optuna.trial.Trial.suggest_float`) are called # inside the objective function to obtain parameters for a trial. # :func:`~optuna.trial.Trial.suggest_float` selects parameters uniformly within the range # provided. In our example, from :math:`-10` to :math:`10`. # # To start the optimization, we create a study object and pass the objective function to method # :func:`~optuna.study.Study.optimize` as follows. study = optuna.create_study() study.optimize(objective, n_trials=100) ################################################################################################### # You can get the best parameter as follows. best_params = study.best_params found_x = best_params["x"] print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2)) ################################################################################################### # We can see that the ``x`` value found by Optuna is close to the optimal value of ``2``. ################################################################################################### # .. note:: # When used to search for hyperparameters in machine learning, # usually the objective function would return the loss or accuracy # of the model. ################################################################################################### # Study Object # ------------ # # Let us clarify the terminology in Optuna as follows: # # * **Trial**: A single call of the objective function # * **Study**: An optimization session, which is a set of trials # * **Parameter**: A variable whose value is to be optimized, such as ``x`` in the above example # # In Optuna, we use the study object to manage optimization. # Method :func:`~optuna.study.create_study` returns a study object. # A study object has useful properties for analyzing the optimization outcome. ################################################################################################### # To get the dictionary of parameter name and parameter values: study.best_params ################################################################################################### # To get the best observed value of the objective function: study.best_value ################################################################################################### # To get the best trial: study.best_trial ################################################################################################### # To get all trials: study.trials for trial in study.trials[:2]: # Show first two trials print(trial) ################################################################################################### # To get the number of trials: len(study.trials) ################################################################################################### # By executing :func:`~optuna.study.Study.optimize` again, we can continue the optimization. study.optimize(objective, n_trials=100) ################################################################################################### # To get the updated number of trials: len(study.trials) ################################################################################################### # As the objective function is so easy that the last 100 trials don't improve the result. # However, we can check the result again: best_params = study.best_params found_x = best_params["x"] print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2)) optuna-3.5.0/tutorial/10_key_features/002_configurations.py000066400000000000000000000064571453453102400236600ustar00rootroot00000000000000""" .. _configurations: Pythonic Search Space ===================== For hyperparameter sampling, Optuna provides the following features: - :func:`optuna.trial.Trial.suggest_categorical` for categorical parameters - :func:`optuna.trial.Trial.suggest_int` for integer parameters - :func:`optuna.trial.Trial.suggest_float` for floating point parameters With optional arguments of ``step`` and ``log``, we can discretize or take the logarithm of integer and floating point parameters. """ import optuna def objective(trial): # Categorical parameter optimizer = trial.suggest_categorical("optimizer", ["MomentumSGD", "Adam"]) # Integer parameter num_layers = trial.suggest_int("num_layers", 1, 3) # Integer parameter (log) num_channels = trial.suggest_int("num_channels", 32, 512, log=True) # Integer parameter (discretized) num_units = trial.suggest_int("num_units", 10, 100, step=5) # Floating point parameter dropout_rate = trial.suggest_float("dropout_rate", 0.0, 1.0) # Floating point parameter (log) learning_rate = trial.suggest_float("learning_rate", 1e-5, 1e-2, log=True) # Floating point parameter (discretized) drop_path_rate = trial.suggest_float("drop_path_rate", 0.0, 1.0, step=0.1) ################################################################################################### # Defining Parameter Spaces # ------------------------- # # In Optuna, we define search spaces using familiar Python syntax including conditionals and loops. # # Also, you can use branches or loops depending on the parameter values. # # For more various use, see `examples `_. ################################################################################################### # - Branches: import sklearn.ensemble import sklearn.svm def objective(trial): classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"]) if classifier_name == "SVC": svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True) classifier_obj = sklearn.svm.SVC(C=svc_c) else: rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True) classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth) ################################################################################################### # - Loops: # # .. code-block:: python # # import torch # import torch.nn as nn # # # def create_model(trial, in_size): # n_layers = trial.suggest_int("n_layers", 1, 3) # # layers = [] # for i in range(n_layers): # n_units = trial.suggest_int("n_units_l{}".format(i), 4, 128, log=True) # layers.append(nn.Linear(in_size, n_units)) # layers.append(nn.ReLU()) # in_size = n_units # layers.append(nn.Linear(in_size, 10)) # # return nn.Sequential(*layers) ################################################################################################### # Note on the Number of Parameters # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # # The difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters. optuna-3.5.0/tutorial/10_key_features/003_efficient_optimization_algorithms.py000066400000000000000000000216151453453102400276130ustar00rootroot00000000000000""" .. _pruning: Efficient Optimization Algorithms ================================= Optuna enables efficient hyperparameter optimization by adopting state-of-the-art algorithms for sampling hyperparameters and pruning efficiently unpromising trials. Sampling Algorithms ------------------- Samplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values, leading to an optimal search space which giving off parameters leading to better objective values. More detailed explanation of how samplers suggest parameters is in :class:`~optuna.samplers.BaseSampler`. Optuna provides the following sampling algorithms: - Grid Search implemented in :class:`~optuna.samplers.GridSampler` - Random Search implemented in :class:`~optuna.samplers.RandomSampler` - Tree-structured Parzen Estimator algorithm implemented in :class:`~optuna.samplers.TPESampler` - CMA-ES based algorithm implemented in :class:`~optuna.samplers.CmaEsSampler` - Algorithm to enable partial fixed parameters implemented in :class:`~optuna.samplers.PartialFixedSampler` - Nondominated Sorting Genetic Algorithm II implemented in :class:`~optuna.samplers.NSGAIISampler` - A Quasi Monte Carlo sampling algorithm implemented in :class:`~optuna.samplers.QMCSampler` The default sampler is :class:`~optuna.samplers.TPESampler`. Switching Samplers ------------------ """ import optuna ################################################################################################### # By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows. study = optuna.create_study() print(f"Sampler is {study.sampler.__class__.__name__}") ################################################################################################### # If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler` # and :class:`~optuna.samplers.CmaEsSampler`, study = optuna.create_study(sampler=optuna.samplers.RandomSampler()) print(f"Sampler is {study.sampler.__class__.__name__}") study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler()) print(f"Sampler is {study.sampler.__class__.__name__}") ################################################################################################### # Pruning Algorithms # ------------------ # # ``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping). # # Optuna provides the following pruning algorithms: # # - Median pruning algorithm implemented in :class:`~optuna.pruners.MedianPruner` # # - Non-pruning algorithm implemented in :class:`~optuna.pruners.NopPruner` # # - Algorithm to operate pruner with tolerance implemented in :class:`~optuna.pruners.PatientPruner` # # - Algorithm to prune specified percentile of trials implemented in :class:`~optuna.pruners.PercentilePruner` # # - Asynchronous Successive Halving algorithm implemented in :class:`~optuna.pruners.SuccessiveHalvingPruner` # # - Hyperband algorithm implemented in :class:`~optuna.pruners.HyperbandPruner` # # - Threshold pruning algorithm implemented in :class:`~optuna.pruners.ThresholdPruner` # # We use :class:`~optuna.pruners.MedianPruner` in most examples, # though basically it is outperformed by :class:`~optuna.pruners.SuccessiveHalvingPruner` and # :class:`~optuna.pruners.HyperbandPruner` as in `this benchmark result `_. # # # Activating Pruners # ------------------ # To turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training. # :func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values. # :func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition. # # We would recommend using integration modules for major machine learning frameworks. # Exclusive list is :mod:`~optuna.integration` and usecases are available in `~optuna/examples `_. import logging import sys import sklearn.datasets import sklearn.linear_model import sklearn.model_selection def objective(trial): iris = sklearn.datasets.load_iris() classes = list(set(iris.target)) train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split( iris.data, iris.target, test_size=0.25, random_state=0 ) alpha = trial.suggest_float("alpha", 1e-5, 1e-1, log=True) clf = sklearn.linear_model.SGDClassifier(alpha=alpha) for step in range(100): clf.partial_fit(train_x, train_y, classes=classes) # Report intermediate objective value. intermediate_value = 1.0 - clf.score(valid_x, valid_y) trial.report(intermediate_value, step) # Handle pruning based on the intermediate value. if trial.should_prune(): raise optuna.TrialPruned() return 1.0 - clf.score(valid_x, valid_y) ################################################################################################### # Set up the median stopping rule as the pruning condition. # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study = optuna.create_study(pruner=optuna.pruners.MedianPruner()) study.optimize(objective, n_trials=20) ################################################################################################### # As you can see, several trials were pruned (stopped) before they finished all of the iterations. # The format of message is ``"Trial pruned."``. ################################################################################################### # Which Sampler and Pruner Should be Used? # ---------------------------------------- # # From the benchmark results which are available at `optuna/optuna - wiki "Benchmarks with Kurobako" `_, at least for not deep learning tasks, we would say that # # * For :class:`~optuna.samplers.RandomSampler`, :class:`~optuna.pruners.MedianPruner` is the best. # * For :class:`~optuna.samplers.TPESampler`, :class:`~optuna.pruners.HyperbandPruner` is the best. # # However, note that the benchmark is not deep learning. # For deep learning tasks, # consult the below table. # This table is from the `Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 `_ paper, # which is written in Japanese. # # +---------------------------+-----------------------------------------+---------------------------------------------------------------+ # | Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms | # +===========================+=========================================+===============================================================+ # | Limited | No | TPE. GP-EI if search space is low-dimensional and continuous. | # + +-----------------------------------------+---------------------------------------------------------------+ # | | Yes | TPE. GP-EI if search space is low-dimensional and continuous | # +---------------------------+-----------------------------------------+---------------------------------------------------------------+ # | Sufficient | No | CMA-ES, Random Search | # + +-----------------------------------------+---------------------------------------------------------------+ # | | Yes | Random Search or Genetic Algorithm | # +---------------------------+-----------------------------------------+---------------------------------------------------------------+ # ################################################################################################### # Integration Modules for Pruning # ------------------------------- # To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries. # # For the complete list of Optuna's integration modules, see :mod:`~optuna.integration`. # # For example, :class:`~optuna.integration.XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration. # (See also `example `_ for the entire script.) # # .. code-block:: python # # pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error') # bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback]) optuna-3.5.0/tutorial/10_key_features/004_distributed.py000066400000000000000000000067671453453102400231560ustar00rootroot00000000000000""" .. _distributed: Easy Parallelization ==================== It's straightforward to parallelize :func:`optuna.study.Study.optimize`. If you want to manually execute Optuna optimization: 1. start an RDB server (this example uses MySQL) 2. create a study with ``--storage`` argument 3. share the study among multiple nodes and processes Of course, you can use Kubernetes as in `the kubernetes examples `_. To just see how parallel optimization works in Optuna, check the below video. .. raw:: html Create a Study -------------- You can create a study using ``optuna create-study`` command. Alternatively, in Python script you can use :func:`optuna.create_study`. .. code-block:: bash $ mysql -u root -e "CREATE DATABASE IF NOT EXISTS example" $ optuna create-study --study-name "distributed-example" --storage "mysql://root@localhost/example" [I 2020-07-21 13:43:39,642] A new study created with name: distributed-example Then, write an optimization script. Let's assume that ``foo.py`` contains the following code. .. code-block:: python import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 if __name__ == "__main__": study = optuna.load_study( study_name="distributed-example", storage="mysql://root@localhost/example" ) study.optimize(objective, n_trials=100) Share the Study among Multiple Nodes and Processes -------------------------------------------------- Finally, run the shared study from multiple processes. For example, run ``Process 1`` in a terminal, and do ``Process 2`` in another one. They get parameter suggestions based on shared trials' history. Process 1: .. code-block:: bash $ python foo.py [I 2020-07-21 13:45:02,973] Trial 0 finished with value: 45.35553104173011 and parameters: {'x': 8.73465151598285}. Best is trial 0 with value: 45.35553104173011. [I 2020-07-21 13:45:04,013] Trial 2 finished with value: 4.6002397305938905 and parameters: {'x': 4.144816945707463}. Best is trial 1 with value: 0.028194513284051464. ... Process 2 (the same command as process 1): .. code-block:: bash $ python foo.py [I 2020-07-21 13:45:03,748] Trial 1 finished with value: 0.028194513284051464 and parameters: {'x': 1.8320877810162361}. Best is trial 1 with value: 0.028194513284051464. [I 2020-07-21 13:45:05,783] Trial 3 finished with value: 24.45966755098074 and parameters: {'x': 6.945671597566982}. Best is trial 1 with value: 0.028194513284051464. ... .. note:: ``n_trials`` is the number of trials each process will run, not the total number of trials across all processes. For example, the script given above runs 100 trials for each process, 100 trials * 2 processes = 200 trials. :class:`optuna.study.MaxTrialsCallback` can ensure how many times trials will be performed across all processes. .. note:: We do not recommend SQLite for distributed optimizations at scale because it may cause deadlocks and serious performance issues. Please consider to use another database engine like PostgreSQL or MySQL. .. note:: Please avoid putting the SQLite database on NFS when running distributed optimizations. See also: https://www.sqlite.org/faq.html#q5 """ optuna-3.5.0/tutorial/10_key_features/005_visualization.py000066400000000000000000000155511453453102400235250ustar00rootroot00000000000000""" .. _visualization: Quick Visualization for Hyperparameter Optimization Analysis ============================================================ Optuna provides various visualization features in :mod:`optuna.visualization` to analyze optimization results visually. This tutorial walks you through this module by visualizing the history of lightgbm model for breast cancer dataset. For visualizing multi-objective optimization (i.e., the usage of :func:`optuna.visualization.plot_pareto_front`), please refer to the tutorial of :ref:`multi_objective`. .. note:: By using `Optuna Dashboard `_, you can also check the optimization history, hyperparameter importances, hyperparameter relationships, etc. in graphs and tables. Please make your study persistent using :ref:`RDB backend ` and execute following commands to run Optuna Dashboard. .. code-block:: console $ pip install optuna-dashboard $ optuna-dashboard sqlite:///example-study.db Please check out `the GitHub repository `_ for more details. .. list-table:: :header-rows: 1 * - Manage Studies - Visualize with Interactive Graphs * - .. image:: https://user-images.githubusercontent.com/5564044/205545958-305f2354-c7cd-4687-be2f-9e46e7401838.gif - .. image:: https://user-images.githubusercontent.com/5564044/205545965-278cd7f4-da7d-4e2e-ac31-6d81b106cada.gif """ ################################################################################################### import lightgbm as lgb import numpy as np import sklearn.datasets import sklearn.metrics from sklearn.model_selection import train_test_split import optuna # You can use Matplotlib instead of Plotly for visualization by simply replacing `optuna.visualization` with # `optuna.visualization.matplotlib` in the following examples. from optuna.visualization import plot_contour from optuna.visualization import plot_edf from optuna.visualization import plot_intermediate_values from optuna.visualization import plot_optimization_history from optuna.visualization import plot_parallel_coordinate from optuna.visualization import plot_param_importances from optuna.visualization import plot_rank from optuna.visualization import plot_slice from optuna.visualization import plot_timeline SEED = 42 np.random.seed(SEED) ################################################################################################### # Define the objective function. def objective(trial): data, target = sklearn.datasets.load_breast_cancer(return_X_y=True) train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25) dtrain = lgb.Dataset(train_x, label=train_y) dvalid = lgb.Dataset(valid_x, label=valid_y) param = { "objective": "binary", "metric": "auc", "verbosity": -1, "boosting_type": "gbdt", "bagging_fraction": trial.suggest_float("bagging_fraction", 0.4, 1.0), "bagging_freq": trial.suggest_int("bagging_freq", 1, 7), "min_child_samples": trial.suggest_int("min_child_samples", 5, 100), } # Add a callback for pruning. pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc") gbm = lgb.train(param, dtrain, valid_sets=[dvalid], callbacks=[pruning_callback]) preds = gbm.predict(valid_x) pred_labels = np.rint(preds) accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels) return accuracy ################################################################################################### study = optuna.create_study( direction="maximize", sampler=optuna.samplers.TPESampler(seed=SEED), pruner=optuna.pruners.MedianPruner(n_warmup_steps=10), ) study.optimize(objective, n_trials=100, timeout=600) ################################################################################################### # Plot functions # -------------- # Visualize the optimization history. See :func:`~optuna.visualization.plot_optimization_history` for the details. plot_optimization_history(study) ################################################################################################### # Visualize the learning curves of the trials. See :func:`~optuna.visualization.plot_intermediate_values` for the details. plot_intermediate_values(study) ################################################################################################### # Visualize high-dimensional parameter relationships. See :func:`~optuna.visualization.plot_parallel_coordinate` for the details. plot_parallel_coordinate(study) ################################################################################################### # Select parameters to visualize. plot_parallel_coordinate(study, params=["bagging_freq", "bagging_fraction"]) ################################################################################################### # Visualize hyperparameter relationships. See :func:`~optuna.visualization.plot_contour` for the details. plot_contour(study) ################################################################################################### # Select parameters to visualize. plot_contour(study, params=["bagging_freq", "bagging_fraction"]) ################################################################################################### # Visualize individual hyperparameters as slice plot. See :func:`~optuna.visualization.plot_slice` for the details. plot_slice(study) ################################################################################################### # Select parameters to visualize. plot_slice(study, params=["bagging_freq", "bagging_fraction"]) ################################################################################################### # Visualize parameter importances. See :func:`~optuna.visualization.plot_param_importances` for the details. plot_param_importances(study) ################################################################################################### # Learn which hyperparameters are affecting the trial duration with hyperparameter importance. optuna.visualization.plot_param_importances( study, target=lambda t: t.duration.total_seconds(), target_name="duration" ) ################################################################################################### # Visualize empirical distribution function. See :func:`~optuna.visualization.plot_edf` for the details. plot_edf(study) ################################################################################################### # Visualize parameter relations with scatter plots colored by objective values. See :func:`~optuna.visualization.plot_rank` for the details. plot_rank(study) ################################################################################################### # Visualize the optimization timeline of performed trials. See :func:`~optuna.visualization.plot_timeline` for the details. plot_timeline(study) optuna-3.5.0/tutorial/10_key_features/README.rst000066400000000000000000000002301453453102400213410ustar00rootroot00000000000000.. _key_features: Key Features ------------ Showcases Optuna's `Key Features `_. optuna-3.5.0/tutorial/20_recipes/000077500000000000000000000000001453453102400166245ustar00rootroot00000000000000optuna-3.5.0/tutorial/20_recipes/001_rdb.py000066400000000000000000000076401453453102400203340ustar00rootroot00000000000000""" .. _rdb: Saving/Resuming Study with RDB Backend ========================================== An RDB backend enables persistent experiments (i.e., to save and resume a study) as well as access to history of studies. In addition, we can run multi-node optimization tasks with this feature, which is described in :ref:`distributed`. In this section, let's try simple examples running on a local environment with SQLite DB. .. note:: You can also utilize other RDB backends, e.g., PostgreSQL or MySQL, by setting the storage argument to the DB's URL. Please refer to `SQLAlchemy's document `_ for how to set up the URL. New Study --------- We can create a persistent study by calling :func:`~optuna.study.create_study` function as follows. An SQLite file ``example.db`` is automatically initialized with a new study record. """ import logging import sys import optuna # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study_name = "example-study" # Unique identifier of the study. storage_name = "sqlite:///{}.db".format(study_name) study = optuna.create_study(study_name=study_name, storage=storage_name) ################################################################################################### # To run a study, call :func:`~optuna.study.Study.optimize` method passing an objective function. def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study.optimize(objective, n_trials=3) ################################################################################################### # Resume Study # ------------ # # To resume a study, instantiate a :class:`~optuna.study.Study` object # passing the study name ``example-study`` and the DB URL ``sqlite:///example-study.db``. study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True) study.optimize(objective, n_trials=3) ################################################################################################### # Note that the storage doesn't store the state of the instance of :mod:`~optuna.samplers` # and :mod:`~optuna.pruners`. # When we resume a study with a sampler whose ``seed`` argument is specified for # reproducibility, you need to restore the sampler with using ``pickle`` as follows:: # # import pickle # # # Save the sampler with pickle to be loaded later. # with open("sampler.pkl", "wb") as fout: # pickle.dump(study.sampler, fout) # # restored_sampler = pickle.load(open("sampler.pkl", "rb")) # study = optuna.create_study( # study_name=study_name, storage=storage_name, load_if_exists=True, sampler=restored_sampler # ) # study.optimize(objective, n_trials=3) # ################################################################################################### # Experimental History # -------------------- # # We can access histories of studies and trials via the :class:`~optuna.study.Study` class. # For example, we can get all trials of ``example-study`` as: study = optuna.create_study(study_name=study_name, storage=storage_name, load_if_exists=True) df = study.trials_dataframe(attrs=("number", "value", "params", "state")) ################################################################################################### # The method :func:`~optuna.study.Study.trials_dataframe` returns a pandas dataframe like: print(df) ################################################################################################### # A :class:`~optuna.study.Study` object also provides properties # such as :attr:`~optuna.study.Study.trials`, :attr:`~optuna.study.Study.best_value`, # :attr:`~optuna.study.Study.best_params` (see also :ref:`first`). print("Best params: ", study.best_params) print("Best value: ", study.best_value) print("Best Trial: ", study.best_trial) print("Trials: ", study.trials) optuna-3.5.0/tutorial/20_recipes/002_multi_objective.py000066400000000000000000000121221453453102400227410ustar00rootroot00000000000000""" .. _multi_objective: Multi-objective Optimization with Optuna ======================================== This tutorial showcases Optuna's multi-objective optimization feature by optimizing the validation accuracy of Fashion MNIST dataset and the FLOPS of the model implemented in PyTorch. We use `fvcore `_ to measure FLOPS. """ import torch import torch.nn as nn import torch.nn.functional as F import torchvision from fvcore.nn import FlopCountAnalysis import optuna DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu") DIR = ".." BATCHSIZE = 128 N_TRAIN_EXAMPLES = BATCHSIZE * 30 N_VALID_EXAMPLES = BATCHSIZE * 10 def define_model(trial): n_layers = trial.suggest_int("n_layers", 1, 3) layers = [] in_features = 28 * 28 for i in range(n_layers): out_features = trial.suggest_int("n_units_l{}".format(i), 4, 128) layers.append(nn.Linear(in_features, out_features)) layers.append(nn.ReLU()) p = trial.suggest_float("dropout_{}".format(i), 0.2, 0.5) layers.append(nn.Dropout(p)) in_features = out_features layers.append(nn.Linear(in_features, 10)) layers.append(nn.LogSoftmax(dim=1)) return nn.Sequential(*layers) # Defines training and evaluation. def train_model(model, optimizer, train_loader): model.train() for batch_idx, (data, target) in enumerate(train_loader): data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE) optimizer.zero_grad() F.nll_loss(model(data), target).backward() optimizer.step() def eval_model(model, valid_loader): model.eval() correct = 0 with torch.no_grad(): for batch_idx, (data, target) in enumerate(valid_loader): data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE) pred = model(data).argmax(dim=1, keepdim=True) correct += pred.eq(target.view_as(pred)).sum().item() accuracy = correct / N_VALID_EXAMPLES flops = FlopCountAnalysis(model, inputs=(torch.randn(1, 28 * 28).to(DEVICE),)).total() return flops, accuracy ################################################################################################### # Define multi-objective objective function. # Objectives are FLOPS and accuracy. def objective(trial): train_dataset = torchvision.datasets.FashionMNIST( DIR, train=True, download=True, transform=torchvision.transforms.ToTensor() ) train_loader = torch.utils.data.DataLoader( torch.utils.data.Subset(train_dataset, list(range(N_TRAIN_EXAMPLES))), batch_size=BATCHSIZE, shuffle=True, ) val_dataset = torchvision.datasets.FashionMNIST( DIR, train=False, transform=torchvision.transforms.ToTensor() ) val_loader = torch.utils.data.DataLoader( torch.utils.data.Subset(val_dataset, list(range(N_VALID_EXAMPLES))), batch_size=BATCHSIZE, shuffle=True, ) model = define_model(trial).to(DEVICE) optimizer = torch.optim.Adam( model.parameters(), trial.suggest_float("lr", 1e-5, 1e-1, log=True) ) for epoch in range(10): train_model(model, optimizer, train_loader) flops, accuracy = eval_model(model, val_loader) return flops, accuracy ################################################################################################### # Run multi-objective optimization # -------------------------------- # # If your optimization problem is multi-objective, # Optuna assumes that you will specify the optimization direction for each objective. # Specifically, in this example, we want to minimize the FLOPS (we want a faster model) # and maximize the accuracy. So we set ``directions`` to ``["minimize", "maximize"]``. study = optuna.create_study(directions=["minimize", "maximize"]) study.optimize(objective, n_trials=30, timeout=300) print("Number of finished trials: ", len(study.trials)) ################################################################################################### # Check trials on Pareto front visually. optuna.visualization.plot_pareto_front(study, target_names=["FLOPS", "accuracy"]) ################################################################################################### # Fetch the list of trials on the Pareto front with :attr:`~optuna.study.Study.best_trials`. # # For example, the following code shows the number of trials on the Pareto front and picks the trial with the highest accuracy. print(f"Number of trials on the Pareto front: {len(study.best_trials)}") trial_with_highest_accuracy = max(study.best_trials, key=lambda t: t.values[1]) print(f"Trial with highest accuracy: ") print(f"\tnumber: {trial_with_highest_accuracy.number}") print(f"\tparams: {trial_with_highest_accuracy.params}") print(f"\tvalues: {trial_with_highest_accuracy.values}") ################################################################################################### # Learn which hyperparameters are affecting the flops most with hyperparameter importance. optuna.visualization.plot_param_importances( study, target=lambda t: t.values[0], target_name="flops" ) optuna-3.5.0/tutorial/20_recipes/003_attributes.py000066400000000000000000000055451453453102400217570ustar00rootroot00000000000000""" .. _attributes: User Attributes =============== This feature is to annotate experiments with user-defined attributes. """ ################################################################################################### # Adding User Attributes to Studies # --------------------------------- # # A :class:`~optuna.study.Study` object provides :func:`~optuna.study.Study.set_user_attr` method # to register a pair of key and value as an user-defined attribute. # A key is supposed to be a ``str``, and a value be any object serializable with ``json.dumps``. import sklearn.datasets import sklearn.model_selection import sklearn.svm import optuna study = optuna.create_study(storage="sqlite:///example.db") study.set_user_attr("contributors", ["Akiba", "Sano"]) study.set_user_attr("dataset", "MNIST") ################################################################################################### # We can access annotated attributes with :attr:`~optuna.study.Study.user_attrs` property. study.user_attrs # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'} ################################################################################################### # :class:`~optuna.study.StudySummary` object, which can be retrieved by # :func:`~optuna.study.get_all_study_summaries`, also contains user-defined attributes. study_summaries = optuna.get_all_study_summaries("sqlite:///example.db") study_summaries[0].user_attrs # {"contributors": ["Akiba", "Sano"], "dataset": "MNIST"} ################################################################################################### # .. seealso:: # ``optuna study set-user-attr`` command, which sets an attribute via command line interface. ################################################################################################### # Adding User Attributes to Trials # -------------------------------- # # As with :class:`~optuna.study.Study`, a :class:`~optuna.trial.Trial` object provides # :func:`~optuna.trial.Trial.set_user_attr` method. # Attributes are set inside an objective function. def objective(trial): iris = sklearn.datasets.load_iris() x, y = iris.data, iris.target svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True) clf = sklearn.svm.SVC(C=svc_c) accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean() trial.set_user_attr("accuracy", accuracy) return 1.0 - accuracy # return error for minimization study.optimize(objective, n_trials=1) ################################################################################################### # We can access annotated attributes as: study.trials[0].user_attrs ################################################################################################### # Note that, in this example, the attribute is not annotated to a :class:`~optuna.study.Study` # but a single :class:`~optuna.trial.Trial`. optuna-3.5.0/tutorial/20_recipes/004_cli.py000066400000000000000000000060671453453102400203410ustar00rootroot00000000000000""" .. _cli: Command-Line Interface ====================== .. csv-table:: :header: Command, Description :widths: 20, 40 :escape: \\ ask, Create a new trial and suggest parameters. best-trial, Show the best trial. best-trials, Show a list of trials located at the Pareto front. create-study, Create a new study. delete-study, Delete a specified study. storage upgrade, Upgrade the schema of a storage. studies, Show a list of studies. study optimize, Start optimization of a study. study set-user-attr, Set a user attribute to a study. tell, Finish a trial\\, which was created by the ask command. trials, Show a list of trials. Optuna provides command-line interface as shown in the above table. Let us assume you are not in IPython shell and writing Python script files instead. It is totally fine to write scripts like the following: """ import optuna def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 if __name__ == "__main__": study = optuna.create_study() study.optimize(objective, n_trials=100) print("Best value: {} (params: {})\n".format(study.best_value, study.best_params)) ################################################################################################### # However, if we cannot write ``objective`` explicitly in Python code such as developing a new # drug in a lab, an interactive way is suitable. # In Optuna CLI, :ref:`ask_and_tell` style commands provide such an interactive and flexible interface. # # Let us assume we minimize the objective value depending on a parameter ``x`` in :math:`[-10, 10]` # and objective value is calculated via some experiments by hand. # Even so, we can invoke the optimization as follows. # Don't care about ``--storage sqlite:///example.db`` for now, which is described in :ref:`rdb`. # # .. code-block:: bash # # $ STUDY_NAME=`optuna create-study --storage sqlite:///example.db` # $ optuna ask --storage sqlite:///example.db --study-name $STUDY_NAME --sampler TPESampler \ # --search-space '{"x": {"name": "FloatDistribution", "attributes": {"step": null, "low": -10.0, "high": 10.0, "log": false}}}' # # # [I 2022-08-20 06:08:53,158] Asked trial 0 with parameters {'x': 2.512238141966016}. # {"number": 0, "params": {"x": 2.512238141966016}} # # The argument of ``--search-space`` option can be generated by using # :func:`optuna.distributions.distribution_to_json`, for example, # ``optuna.distributions.distribution_to_json(optuna.distributions.FloatDistribution(-10, 10))``. # Please refer to :class:`optuna.distributions.FloatDistribution` and # :class:`optuna.distributions.IntDistribution` for detailed explanations of their arguments. # # After conducting an experiment using the suggested parameter in the lab, # we store the result to Optuna's study as follows: # # .. code-block:: bash # # $ optuna tell --storage sqlite:///example.db --study-name $STUDY_NAME --trial-number 0 --values 0.7 --state complete # [I 2022-08-20 06:22:50,888] Told trial 0 with values [0.7] and state TrialState.COMPLETE. # optuna-3.5.0/tutorial/20_recipes/005_user_defined_sampler.py000066400000000000000000000147271453453102400237540ustar00rootroot00000000000000""" .. _user_defined_sampler: User-Defined Sampler ==================== Thanks to user-defined samplers, you can: - experiment your own sampling algorithms, - implement task-specific algorithms to refine the optimization performance, or - wrap other optimization libraries to integrate them into Optuna pipelines (e.g., :class:`~optuna.integration.BoTorchSampler`). This section describes the internal behavior of sampler classes and shows an example of implementing a user-defined sampler. Overview of Sampler ------------------- A sampler has the responsibility to determine the parameter values to be evaluated in a trial. When a `suggest` API (e.g., :func:`~optuna.trial.Trial.suggest_float`) is called inside an objective function, the corresponding distribution object (e.g., :class:`~optuna.distributions.FloatDistribution`) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the `suggest` API and evaluated in the objective function. To create a new sampler, you need to define a class that inherits :class:`~optuna.samplers.BaseSampler`. The base class has three abstract methods; :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space`, :meth:`~optuna.samplers.BaseSampler.sample_relative`, and :meth:`~optuna.samplers.BaseSampler.sample_independent`. As the method names imply, Optuna supports two types of sampling: one is **relative sampling** that can consider the correlation of the parameters in a trial, and the other is **independent sampling** that samples each parameter independently. At the beginning of a trial, :meth:`~optuna.samplers.BaseSampler.infer_relative_search_space` is called to provide the relative search space for the trial. Then, :meth:`~optuna.samplers.BaseSampler.sample_relative` is invoked to sample relative parameters from the search space. During the execution of the objective function, :meth:`~optuna.samplers.BaseSampler.sample_independent` is used to sample parameters that don't belong to the relative search space. .. note:: Please refer to the document of :class:`~optuna.samplers.BaseSampler` for further details. An Example: Implementing SimulatedAnnealingSampler -------------------------------------------------- For example, the following code defines a sampler based on `Simulated Annealing (SA) `_: """ import numpy as np import optuna class SimulatedAnnealingSampler(optuna.samplers.BaseSampler): def __init__(self, temperature=100): self._rng = np.random.RandomState() self._temperature = temperature # Current temperature. self._current_trial = None # Current state. def sample_relative(self, study, trial, search_space): if search_space == {}: return {} # Simulated Annealing algorithm. # 1. Calculate transition probability. prev_trial = study.trials[-2] if self._current_trial is None or prev_trial.value <= self._current_trial.value: probability = 1.0 else: probability = np.exp( (self._current_trial.value - prev_trial.value) / self._temperature ) self._temperature *= 0.9 # Decrease temperature. # 2. Transit the current state if the previous result is accepted. if self._rng.uniform(0, 1) < probability: self._current_trial = prev_trial # 3. Sample parameters from the neighborhood of the current point. # The sampled parameters will be used during the next execution of # the objective function passed to the study. params = {} for param_name, param_distribution in search_space.items(): if ( not isinstance(param_distribution, optuna.distributions.FloatDistribution) or (param_distribution.step is not None and param_distribution.step != 1) or param_distribution.log ): msg = ( "Only suggest_float() with `step` `None` or 1.0 and" " `log` `False` is supported" ) raise NotImplementedError(msg) current_value = self._current_trial.params[param_name] width = (param_distribution.high - param_distribution.low) * 0.1 neighbor_low = max(current_value - width, param_distribution.low) neighbor_high = min(current_value + width, param_distribution.high) params[param_name] = self._rng.uniform(neighbor_low, neighbor_high) return params # The rest are unrelated to SA algorithm: boilerplate def infer_relative_search_space(self, study, trial): return optuna.search_space.intersection_search_space(study.get_trials(deepcopy=False)) def sample_independent(self, study, trial, param_name, param_distribution): independent_sampler = optuna.samplers.RandomSampler() return independent_sampler.sample_independent(study, trial, param_name, param_distribution) ################################################################################################### # .. note:: # In favor of code simplicity, the above implementation doesn't support some features (e.g., maximization). # If you're interested in how to support those features, please see # `examples/samplers/simulated_annealing.py # `_. # # # You can use ``SimulatedAnnealingSampler`` in the same way as built-in samplers as follows: def objective(trial): x = trial.suggest_float("x", -10, 10) y = trial.suggest_float("y", -5, 5) return x**2 + y sampler = SimulatedAnnealingSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) best_trial = study.best_trial print("Best value: ", best_trial.value) print("Parameters that achieve the best value: ", best_trial.params) ################################################################################################### # In this optimization, the values of ``x`` and ``y`` parameters are sampled by using # ``SimulatedAnnealingSampler.sample_relative`` method. # # .. note:: # Strictly speaking, in the first trial, # ``SimulatedAnnealingSampler.sample_independent`` method is used to sample parameter values. # Because :func:`~optuna.search_space.intersection_search_space` used in # ``SimulatedAnnealingSampler.infer_relative_search_space`` cannot infer the search space # if there are no complete trials. optuna-3.5.0/tutorial/20_recipes/006_user_defined_pruner.py000066400000000000000000000127061453453102400236200ustar00rootroot00000000000000""" .. _user_defined_pruner: User-Defined Pruner =================== In :mod:`optuna.pruners`, we described how an objective function can optionally include calls to a pruning feature which allows Optuna to terminate an optimization trial when intermediate results do not appear promising. In this document, we describe how to implement your own pruner, i.e., a custom strategy for determining when to stop a trial. Overview of Pruning Interface ----------------------------- The :func:`~optuna.study.create_study` constructor takes, as an optional argument, a pruner inheriting from :class:`~optuna.pruners.BasePruner`. The pruner should implement the abstract method :func:`~optuna.pruners.BasePruner.prune`, which takes arguments for the associated :class:`~optuna.study.Study` and :class:`~optuna.trial.Trial` and returns a boolean value: :obj:`True` if the trial should be pruned and :obj:`False` otherwise. Using the Study and Trial objects, you can access all other trials through the :func:`~optuna.study.Study.get_trials` method and, and from a trial, its reported intermediate values through the :func:`~optuna.trial.FrozenTrial.intermediate_values` (a dictionary which maps an integer ``step`` to a float value). You can refer to the source code of the built-in Optuna pruners as templates for building your own. In this document, for illustration, we describe the construction and usage of a simple (but aggressive) pruner which prunes trials that are in last place compared to completed trials at the same step. .. note:: Please refer to the documentation of :class:`~optuna.pruners.BasePruner` or, for example, :class:`~optuna.pruners.ThresholdPruner` or :class:`~optuna.pruners.PercentilePruner` for more robust examples of pruner implementation, including error checking and complex pruner-internal logic. An Example: Implementing ``LastPlacePruner`` -------------------------------------------- We aim to optimize the ``loss`` and ``alpha`` hyperparameters for a stochastic gradient descent classifier (``SGDClassifier``) run on the sklearn iris dataset. We implement a pruner which terminates a trial at a certain step if it is in last place compared to completed trials at the same step. We begin considering pruning after a "warmup" of 1 training step and 5 completed trials. For demonstration purposes, we :func:`print` a diagnostic message from ``prune`` when it is about to return :obj:`True` (indicating pruning). It may be important to note that the ``SGDClassifier`` score, as it is evaluated on a holdout set, decreases with enough training steps due to overfitting. This means that a trial could be pruned even if it had a favorable (high) value on a previous training set. After pruning, Optuna will take the intermediate value last reported as the value of the trial. """ import numpy as np from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.linear_model import SGDClassifier import optuna from optuna.pruners import BasePruner from optuna.trial._state import TrialState class LastPlacePruner(BasePruner): def __init__(self, warmup_steps, warmup_trials): self._warmup_steps = warmup_steps self._warmup_trials = warmup_trials def prune(self, study: "optuna.study.Study", trial: "optuna.trial.FrozenTrial") -> bool: # Get the latest score reported from this trial step = trial.last_step if step: # trial.last_step == None when no scores have been reported yet this_score = trial.intermediate_values[step] # Get scores from other trials in the study reported at the same step completed_trials = study.get_trials(deepcopy=False, states=(TrialState.COMPLETE,)) other_scores = [ t.intermediate_values[step] for t in completed_trials if step in t.intermediate_values ] other_scores = sorted(other_scores) # Prune if this trial at this step has a lower value than all completed trials # at the same step. Note that steps will begin numbering at 0 in the objective # function definition below. if step >= self._warmup_steps and len(other_scores) > self._warmup_trials: if this_score < other_scores[0]: print(f"prune() True: Trial {trial.number}, Step {step}, Score {this_score}") return True return False ################################################################################################### # Lastly, let's confirm the implementation is correct with the simple hyperparameter optimization. def objective(trial): iris = load_iris() classes = np.unique(iris.target) X_train, X_valid, y_train, y_valid = train_test_split( iris.data, iris.target, train_size=100, test_size=50, random_state=0 ) loss = trial.suggest_categorical("loss", ["hinge", "log_loss", "perceptron"]) alpha = trial.suggest_float("alpha", 0.00001, 0.001, log=True) clf = SGDClassifier(loss=loss, alpha=alpha, random_state=0) score = 0 for step in range(0, 5): clf.partial_fit(X_train, y_train, classes=classes) score = clf.score(X_valid, y_valid) trial.report(score, step) if trial.should_prune(): raise optuna.TrialPruned() return score pruner = LastPlacePruner(warmup_steps=1, warmup_trials=5) study = optuna.create_study(direction="maximize", pruner=pruner) study.optimize(objective, n_trials=50) optuna-3.5.0/tutorial/20_recipes/007_optuna_callback.py000066400000000000000000000046361453453102400227170ustar00rootroot00000000000000""" .. _optuna_callback: Callback for Study.optimize =========================== This tutorial showcases how to use & implement Optuna ``Callback`` for :func:`~optuna.study.Study.optimize`. ``Callback`` is called after every evaluation of ``objective``, and it takes :class:`~optuna.study.Study` and :class:`~optuna.trial.FrozenTrial` as arguments, and does some work. :class:`~optuna.integration.MLflowCallback` is a great example. """ ################################################################################################### # Stop optimization after some trials are pruned in a row # ------------------------------------------------------- # # This example implements a stateful callback which stops the optimization # if a certain number of trials are pruned in a row. # The number of trials pruned in a row is specified by ``threshold``. import optuna class StopWhenTrialKeepBeingPrunedCallback: def __init__(self, threshold: int): self.threshold = threshold self._consequtive_pruned_count = 0 def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -> None: if trial.state == optuna.trial.TrialState.PRUNED: self._consequtive_pruned_count += 1 else: self._consequtive_pruned_count = 0 if self._consequtive_pruned_count >= self.threshold: study.stop() ################################################################################################### # This objective prunes all the trials except for the first 5 trials (``trial.number`` starts with 0). def objective(trial): if trial.number > 4: raise optuna.TrialPruned return trial.suggest_float("x", 0, 1) ################################################################################################### # Here, we set the threshold to ``2``: optimization finishes once two trials are pruned in a row. # So, we expect this study to stop after 7 trials. import logging import sys # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study_stop_cb = StopWhenTrialKeepBeingPrunedCallback(2) study = optuna.create_study() study.optimize(objective, n_trials=10, callbacks=[study_stop_cb]) ################################################################################################### # As you can see in the log above, the study stopped after 7 trials as expected. optuna-3.5.0/tutorial/20_recipes/008_specify_params.py000066400000000000000000000124741453453102400226020ustar00rootroot00000000000000""" .. _specify_params: Specify Hyperparameters Manually ================================ It's natural that you have some specific sets of hyperparameters to try first such as initial learning rate values and the number of leaves. Also, it's possible that you've already tried those sets before having Optuna find better sets of hyperparameters. Optuna provides two APIs to support such cases: 1. Passing those sets of hyperparameters and let Optuna evaluate them - :func:`~optuna.study.Study.enqueue_trial` 2. Adding the results of those sets as completed ``Trial``\\s - :func:`~optuna.study.Study.add_trial` .. _enqueue_trial_tutorial: --------------------------------------------------------- First Scenario: Have Optuna evaluate your hyperparameters --------------------------------------------------------- In this scenario, let's assume you have some out-of-box sets of hyperparameters but have not evaluated them yet and decided to use Optuna to find better sets of hyperparameters. Optuna has :func:`optuna.study.Study.enqueue_trial` which lets you pass those sets of hyperparameters to Optuna and Optuna will evaluate them. This section walks you through how to use this lit API with `LightGBM `_. """ import lightgbm as lgb import numpy as np import sklearn.datasets import sklearn.metrics from sklearn.model_selection import train_test_split import optuna ################################################################################################### # Define the objective function. def objective(trial): data, target = sklearn.datasets.load_breast_cancer(return_X_y=True) train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25) dtrain = lgb.Dataset(train_x, label=train_y) dvalid = lgb.Dataset(valid_x, label=valid_y) param = { "objective": "binary", "metric": "auc", "verbosity": -1, "boosting_type": "gbdt", "bagging_fraction": min(trial.suggest_float("bagging_fraction", 0.4, 1.0 + 1e-12), 1), "bagging_freq": trial.suggest_int("bagging_freq", 0, 7), "min_child_samples": trial.suggest_int("min_child_samples", 5, 100), } # Add a callback for pruning. pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc") gbm = lgb.train(param, dtrain, valid_sets=[dvalid], callbacks=[pruning_callback]) preds = gbm.predict(valid_x) pred_labels = np.rint(preds) accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels) return accuracy ################################################################################################### # Then, construct ``Study`` for hyperparameter optimization. study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner()) ################################################################################################### # Here, we get Optuna evaluate some sets with larger ``"bagging_fraq"`` value and # the default values. study.enqueue_trial( { "bagging_fraction": 1.0, "bagging_freq": 0, "min_child_samples": 20, } ) study.enqueue_trial( { "bagging_fraction": 0.75, "bagging_freq": 5, "min_child_samples": 20, } ) import logging import sys # Add stream handler of stdout to show the messages to see Optuna works expectedly. optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study.optimize(objective, n_trials=100, timeout=600) ################################################################################################### # .. _add_trial_tutorial: # # ---------------------------------------------------------------------- # Second scenario: Have Optuna utilize already evaluated hyperparameters # ---------------------------------------------------------------------- # # In this scenario, let's assume you have some out-of-box sets of hyperparameters and # you have already evaluated them but the results are not desirable so that you are thinking of # using Optuna. # # Optuna has :func:`optuna.study.Study.add_trial` which lets you register those results # to Optuna and then Optuna will sample hyperparameters taking them into account. # # In this section, the ``objective`` is the same as the first scenario. study = optuna.create_study(direction="maximize", pruner=optuna.pruners.MedianPruner()) study.add_trial( optuna.trial.create_trial( params={ "bagging_fraction": 1.0, "bagging_freq": 0, "min_child_samples": 20, }, distributions={ "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12), "bagging_freq": optuna.distributions.IntDistribution(0, 7), "min_child_samples": optuna.distributions.IntDistribution(5, 100), }, value=0.94, ) ) study.add_trial( optuna.trial.create_trial( params={ "bagging_fraction": 0.75, "bagging_freq": 5, "min_child_samples": 20, }, distributions={ "bagging_fraction": optuna.distributions.FloatDistribution(0.4, 1.0 + 1e-12), "bagging_freq": optuna.distributions.IntDistribution(0, 7), "min_child_samples": optuna.distributions.IntDistribution(5, 100), }, value=0.95, ) ) study.optimize(objective, n_trials=100, timeout=600) optuna-3.5.0/tutorial/20_recipes/009_ask_and_tell.py000066400000000000000000000213261453453102400222120ustar00rootroot00000000000000""" .. _ask_and_tell: Ask-and-Tell Interface ======================= Optuna has an `Ask-and-Tell` interface, which provides a more flexible interface for hyperparameter optimization. This tutorial explains three use-cases when the ask-and-tell interface is beneficial: - :ref:`Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications` - :ref:`Define-and-Run` - :ref:`Batch-Optimization` .. _Apply-optuna-to-an-existing-optimization-problem-with-minimum-modifications: ---------------------------------------------------------------------------- Apply Optuna to an existing optimization problem with minimum modifications ---------------------------------------------------------------------------- Let's consider the traditional supervised classification problem; you aim to maximize the validation accuracy. To do so, you train `LogisticRegression` as a simple model. """ import numpy as np from sklearn.datasets import make_classification from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split import optuna X, y = make_classification(n_features=10) X_train, X_test, y_train, y_test = train_test_split(X, y) C = 0.01 clf = LogisticRegression(C=C) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) # the objective ################################################################################################### # Then you try to optimize hyperparameters ``C`` and ``solver`` of the classifier by using optuna. # When you introduce optuna naively, you define an ``objective`` function # such that it takes ``trial`` and calls ``suggest_*`` methods of ``trial`` to sample the hyperparameters: def objective(trial): X, y = make_classification(n_features=10) X_train, X_test, y_train, y_test = train_test_split(X, y) C = trial.suggest_float("C", 1e-7, 10.0, log=True) solver = trial.suggest_categorical("solver", ("lbfgs", "saga")) clf = LogisticRegression(C=C, solver=solver) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) return val_accuracy study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=10) ################################################################################################### # This interface is not flexible enough. # For example, if ``objective`` requires additional arguments other than ``trial``, # you need to define a class as in # `How to define objective functions that have own arguments? <../../faq.html#how-to-define-objective-functions-that-have-own-arguments>`_. # The ask-and-tell interface provides a more flexible syntax to optimize hyperparameters. # The following example is equivalent to the previous code block. study = optuna.create_study(direction="maximize") n_trials = 10 for _ in range(n_trials): trial = study.ask() # `trial` is a `Trial` and not a `FrozenTrial`. C = trial.suggest_float("C", 1e-7, 10.0, log=True) solver = trial.suggest_categorical("solver", ("lbfgs", "saga")) clf = LogisticRegression(C=C, solver=solver) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) study.tell(trial, val_accuracy) # tell the pair of trial and objective value ################################################################################################### # The main difference is to use two methods: :func:`optuna.study.Study.ask` # and :func:`optuna.study.Study.tell`. # :func:`optuna.study.Study.ask` creates a trial that can sample hyperparameters, and # :func:`optuna.study.Study.tell` finishes the trial by passing ``trial`` and an objective value. # You can apply Optuna's hyperparameter optimization to your original code # without an ``objective`` function. # # If you want to make your optimization faster with a pruner, you need to explicitly pass the state of trial # to the argument of :func:`optuna.study.Study.tell` method as follows: # # .. code-block:: python # # import numpy as np # from sklearn.datasets import load_iris # from sklearn.linear_model import SGDClassifier # from sklearn.model_selection import train_test_split # # import optuna # # # X, y = load_iris(return_X_y=True) # X_train, X_valid, y_train, y_valid = train_test_split(X, y) # classes = np.unique(y) # n_train_iter = 100 # # # define study with hyperband pruner. # study = optuna.create_study( # direction="maximize", # pruner=optuna.pruners.HyperbandPruner( # min_resource=1, max_resource=n_train_iter, reduction_factor=3 # ), # ) # # for _ in range(20): # trial = study.ask() # # alpha = trial.suggest_float("alpha", 0.0, 1.0) # # clf = SGDClassifier(alpha=alpha) # pruned_trial = False # # for step in range(n_train_iter): # clf.partial_fit(X_train, y_train, classes=classes) # # intermediate_value = clf.score(X_valid, y_valid) # trial.report(intermediate_value, step) # # if trial.should_prune(): # pruned_trial = True # break # # if pruned_trial: # study.tell(trial, state=optuna.trial.TrialState.PRUNED) # tell the pruned state # else: # score = clf.score(X_valid, y_valid) # study.tell(trial, score) # tell objective value ################################################################################################### # .. note:: # # :func:`optuna.study.Study.tell` method can take a trial number rather than the trial object. # ``study.tell(trial.number, y)`` is equivalent to ``study.tell(trial, y)``. ################################################################################################### # .. _Define-and-Run: # # --------------- # Define-and-Run # --------------- # The ask-and-tell interface supports both `define-by-run` and `define-and-run` APIs. # This section shows the example of the `define-and-run` API # in addition to the define-by-run example above. # # Define distributions for the hyperparameters before calling the # :func:`optuna.study.Study.ask` method for define-and-run API. # For example, distributions = { "C": optuna.distributions.FloatDistribution(1e-7, 10.0, log=True), "solver": optuna.distributions.CategoricalDistribution(("lbfgs", "saga")), } ################################################################################################### # Pass ``distributions`` to :func:`optuna.study.Study.ask` method at each call. # The retuned ``trial`` contains the suggested hyperparameters. study = optuna.create_study(direction="maximize") n_trials = 10 for _ in range(n_trials): trial = study.ask(distributions) # pass the pre-defined distributions. # two hyperparameters are already sampled from the pre-defined distributions C = trial.params["C"] solver = trial.params["solver"] clf = LogisticRegression(C=C, solver=solver) clf.fit(X_train, y_train) val_accuracy = clf.score(X_test, y_test) study.tell(trial, val_accuracy) ################################################################################################### # .. _Batch-Optimization: # # ------------------- # Batch Optimization # ------------------- # The ask-and-tell interface enables us to optimize a batched objective for faster optimization. # For example, parallelizable evaluation, operation over vectors, etc. ################################################################################################### # The following objective takes batched hyperparameters ``xs`` and ``ys`` instead of a single # pair of hyperparameters ``x`` and ``y`` and calculates the objective over the full vectors. def batched_objective(xs: np.ndarray, ys: np.ndarray): return xs**2 + ys ################################################################################################### # In the following example, the number of pairs of hyperparameters in a batch is :math:`10`, # and ``batched_objective`` is evaluated three times. # Thus, the number of trials is :math:`30`. # Note that you need to store either ``trial_numbers`` or ``trial`` to call # :func:`optuna.study.Study.tell` method after the batched evaluations. batch_size = 10 study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler()) for _ in range(3): # create batch trial_numbers = [] x_batch = [] y_batch = [] for _ in range(batch_size): trial = study.ask() trial_numbers.append(trial.number) x_batch.append(trial.suggest_float("x", -10, 10)) y_batch.append(trial.suggest_float("y", -10, 10)) # evaluate batched objective x_batch = np.array(x_batch) y_batch = np.array(y_batch) objectives = batched_objective(x_batch, y_batch) # finish all trials in the batch for trial_number, objective in zip(trial_numbers, objectives): study.tell(trial_number, objective) optuna-3.5.0/tutorial/20_recipes/010_reuse_best_trial.py000066400000000000000000000076541453453102400231250ustar00rootroot00000000000000""" .. _reuse_best_trial: Re-use the best trial ====================== In some cases, you may want to re-evaluate the objective function with the best hyperparameters again after the hyperparameter optimization. For example, - You have found good hyperparameters with Optuna and want to run a similar `objective` function using the best hyperparameters found so far to further analyze the results, or - You have optimized with Optuna using a partial dataset to reduce training time. After the hyperparameter tuning, you want to train the model using the whole dataset with the best hyperparameter values found. :class:`~optuna.study.Study.best_trial` provides an interface to re-evaluate the objective function with the current best hyperparameter values. This tutorial shows an example of how to re-run a different `objective` function with the current best values, like the first example above. Investigating the best model further ------------------------------------- Let's consider a classical supervised classification problem with Optuna as follows: """ from sklearn import metrics from sklearn.datasets import make_classification from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split import optuna def objective(trial): X, y = make_classification(n_features=10, random_state=1) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1) C = trial.suggest_float("C", 1e-7, 10.0, log=True) clf = LogisticRegression(C=C) clf.fit(X_train, y_train) return clf.score(X_test, y_test) study = optuna.create_study(direction="maximize") study.optimize(objective, n_trials=10) print(study.best_trial.value) # Show the best value. ################################################################################################### # Suppose after the hyperparameter optimization, you want to calculate other evaluation metrics # such as recall, precision, and f1-score on the same dataset. # You can define another objective function that shares most of the ``objective`` # function to reproduce the model with the best hyperparameters. def detailed_objective(trial): # Use same code objective to reproduce the best model X, y = make_classification(n_features=10, random_state=1) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1) C = trial.suggest_float("C", 1e-7, 10.0, log=True) clf = LogisticRegression(C=C) clf.fit(X_train, y_train) # calculate more evaluation metrics pred = clf.predict(X_test) acc = metrics.accuracy_score(pred, y_test) recall = metrics.recall_score(pred, y_test) precision = metrics.precision_score(pred, y_test) f1 = metrics.f1_score(pred, y_test) return acc, f1, recall, precision ################################################################################################### # Pass ``study.best_trial`` as the argument of ``detailed_objective``. detailed_objective(study.best_trial) # calculate acc, f1, recall, and precision ################################################################################################### # The difference between :class:`~optuna.study.Study.best_trial` and ordinal trials # ---------------------------------------------------------------------------------- # # This uses :class:`~optuna.study.Study.best_trial`, which returns the `best_trial` as a # :class:`~optuna.trial.FrozenTrial`. # The :class:`~optuna.trial.FrozenTrial` is different from an active trial # and behaves differently from :class:`~optuna.trial.Trial` in some situations. # For example, pruning does not work because :class:`~optuna.trial.FrozenTrial.should_prune` # always returns ``False``. # # .. note:: # For multi-objective optimization as demonstrated by :ref:`multi_objective`, # :attr:`~optuna.study.Study.best_trials` returns a list of :class:`~optuna.trial.FrozenTrial` # on Pareto front. So we can re-use each trial in the list by the similar way above. optuna-3.5.0/tutorial/20_recipes/011_journal_storage.py000066400000000000000000000030761453453102400227630ustar00rootroot00000000000000""" .. _journal_storage: (File-based) Journal Storage ============================ Optuna provides :class:`~optuna.storages.JournalStorage`. With this feature, you can easily run a distributed optimization over network using NFS as the shared storage, without need for setting up RDB or Redis. """ import logging import sys import optuna # Add stream handler of stdout to show the messages optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) study_name = "example-study" # Unique identifier of the study. storage = optuna.storages.JournalStorage( optuna.storages.JournalFileStorage("./journal.log"), # NFS path for distributed optimization ) study = optuna.create_study(study_name=study_name, storage=storage) def objective(trial): x = trial.suggest_float("x", -10, 10) return (x - 2) ** 2 study.optimize(objective, n_trials=3) ################################################################################################### # Although the optimization in this example is too short to run in parallel, you can extend this # example to write a optimization script which can be run in parallel. # ################################################################################################### # .. note:: # In a Windows environment, an error message "A required privilege is not held by the client" # may appear. In this case, you can solve the problem with creating storage by specifying # :class:`~optuna.storages.JournalFileOpenLock`. See the reference of # :class:`~optuna.storages.JournalStorage` for any details. optuna-3.5.0/tutorial/20_recipes/012_artifact_tutorial.py000066400000000000000000000477411453453102400233150ustar00rootroot00000000000000""" .. _artifact_tutorial: Optuna Artifacts Tutorial ========================= .. contents:: Table of Contents :depth: 2 The artifact module of Optuna is a module designed for saving comparatively large attributes on a trial-by-trial basis in forms such as files. Introduced from Optuna v3.3, this module finds a broad range of applications, such as utilizing snapshots of large size models for hyperparameter tuning, optimizing massive chemical structures, and even human-in-the-loop optimization employing images or sounds. Use of Optuna's artifact module allows you to handle data that would be too large to store in a database. Furthermore, by integrating with `optuna-dashboard `_, saved artifacts can be automatically visualized with the web UI, which significantly reduces the effort of experiment management. TL;DR ----- - The artifact module provides a simple way to save and use large data associated with trials. - Saved artifacts can be visualized just by accessing the web page using optuna-dashboard, and downloading is also easy. - Thanks to the abstraction of the artifact module, the backend (file system, AWS S3) can be easily switched. - As the artifact module is tightly linked with Optuna, experiment management can be completed with the Optuna ecosystem alone, simplifying the code base. Concepts -------- .. list-table:: :header-rows: 1 * - Fig 1. Concepts of the "artifact". * - .. image:: https://github.com/optuna/optuna/assets/38826298/112e0b75-9d22-474b-85ea-9f3e0d75fa8d An "artifact" is associated with an Optuna trial. In Optuna, the objective function is evaluated sequentially to search for the maximum (or minimum) value. Each evaluation of the sequentially repeated objective function is called a trial. Normally, trials and their associated attributes are saved via storage objects to files or RDBs, etc. For experiment management, you can also save and use `user_attrs` for each trial. However, these attributes are assumed to be integers, short strings, or other small data, which are not suitable for storing large data. With Optuna's artifact module, users can save large data (such as model snapshots, chemical structures, image and audio data, etc.) for each trial. Also, while this tutorial does not touch upon it, it's possible to manage artifacts associated not only with trials but also with studies. Please refer to the `official documentation `_ if you are interested in. Situations where artifacts are useful ------------------------------------- Artifacts are useful when you want to save data that is too large to be stored in RDB for each trial. For example, the artifact module would be handy in situations like the following: - Saving snapshots of machine learning models: Suppose you are tuning hyperparameters for a large-scale machine learning model like an LLM. The model is very large, and each round of learning (which corresponds to one trial in Optuna) takes time. To prepare for unexpected incidents during training (such as blackouts at the data center or a preemption of computation jobs by the scheduler), you may want to save snapshots of the model in the middle of training for each trial. These snapshots often tend to be large and are more suitable to be saved as some kinds of files than to be stored in RDB. In such cases, the artifact module is useful. - Optimizing chemical structures: Suppose you are formulating and exploring a problem of finding stable chemical structures as a black-box optimization problem. Evaluating one chemical structure corresponds to one trial in Optuna, and that chemical structure is a complex and large one. It is not appropriate to store such chemical structure data in RDB. It is conceivable to save the chemical structure data in a specific file format, and in such a case, the artifact module is useful. - Human-in-the-loop optimization of images: Suppose you are optimizing prompts for a generative model that outputs images. You sample the prompts using Optuna, output images using the generative model, and let humans rate the images for a Human-in-the-loop optimization process. Since the output images are large data, it is not appropriate to use RDB to store them, and in such cases, using the artifact module is well suited. How Trials and Artifacts are Recorded ------------------------------------- As explained so far, the artifact module is useful when you want to save large data for each trial. In this section, we explain how artifacts work in the following two scenarios: first when SQLite + local file system-based artifact backend is used (suitable when the entire optimization cycle is completed locally), and second when MySQL + AWS S3-based artifact backend is used (suitable when you want to keep the data in a remote location). Scenario 1: SQLite + file system-based artifact store ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 * - Fig 2. SQLite + file system-based artifact store. * - .. image:: https://github.com/optuna/optuna/assets/38826298/d41d042e-6b78-4615-bf96-05f73a47e9ea First, we explain a simple case where the optimization is completed locally. Normally, Optuna's optimization history is persisted into some kind of a database via storage objects. Here, let's consider a method using SQLite, a lightweight RDB management system, as the backend. With SQLite, data is stored in a single file (e.g., ./example.db). The optimization history comprises what parameters were sampled in each trial, what the evaluation values for those parameters were, when each trial started and ended, etc. This file is in the SQLite format, and it is not suitable for storing large data. Writing large data entries may cause performance degradation. Note that SQLite is not suitable for distributed parallel optimization. If you want to perform that, please use MySQL as we will explain later, or JournalStorage (`example `_). So, let's use the artifact module to save large data in a different format. Suppose the data is generated for each trial and you want to save it in some format (e.g., png format if it's an image). The specific destination for saving the artifacts can be any directory on the local file system (e.g., the ./artifacts directory). When defining the objective function, you only need to save and reference the data using the artifact module. The simple pseudocode for the above case would look something like this: .. code-block:: python import os import optuna from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = FileSystemArtifactStore(base_path=base_path) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) # Creating and writing an artifact. file_path = generate_example(...) # This function returns some kind of file. artifact_id = upload_artifact( trial, file_path, artifact_store ) # The return value is the artifact ID. trial.set_user_attr( "artifact_id", artifact_id ) # Save the ID in RDB so that it can be referenced later. return ... study = optuna.create_study(study_name="test_study", storage="sqlite:///example.db") study.optimize(objective, n_trials=100) # Loading and displaying artifacts associated with the best trial. best_artifact_id = study.best_trial.user_attrs.get("artifact_id") with artifact_store.open_reader(best_artifact_id) as f: content = f.read().decode("utf-8") print(content) Scenario 2: Remote MySQL RDB server + AWS S3 artifact store ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 * - Fig 3. Remote MySQL RDB server + AWS S3 artifact store. * - .. image:: https://github.com/optuna/optuna/assets/38826298/067efc85-1fad-4b46-a2be-626c64439d7b Next, we explain the case where data is read and written remotely. As the scale of optimization increases, it becomes difficult to complete all calculations locally. Optuna's storage objects can persist data remotely by specifying a URL, enabling distributed optimization. Here, we will use MySQL as a remote relational database server. MySQL is an open-source relational database management system and a well-known software used for various purposes. For using MySQL with Optuna, the `tutorial `_ can be a good reference. However, it is also not appropriate to read and write large data in a relational database like MySQL. In Optuna, it is common to use the artifact module when you want to read and write such data for each trial. Unlike Scenario 1, we distribute the optimization across computation nodes, so local file system-based backends will not work. Instead, we will use AWS S3, an online cloud storage service, and Boto3, a framework for interacting with it from Python. As of v3.3, Optuna has a built-in artifact store with this Boto3 backend. The flow of data is shown in Figure 3. The information calculated in each trial, which corresponds to the optimization history (excluding artifact information), is written to the MySQL server. On the other hand, the artifact information is written to AWS S3. All workers conducting distributed optimization can read and write in parallel to each, and issues such as race conditions are automatically resolved by Optuna's storage module and artifact module. As a result, although the actual data location changes between artifact information and non-artifact information (the former is in AWS S3, the latter is in the MySQL RDB), users can read and write data transparently. Translating the above process into simple pseudocode would look something like this: .. code-block:: python import os import boto3 from botocore.config import Config import optuna from optuna.artifact import upload_artifact from optuna.artifact.boto3 import Boto3ArtifactStore artifact_store = Boto3ArtifactStore( client=boto3.client( "s3", aws_access_key_id=os.environ[ "PFS2_AWS_ACCESS_KEY_ID" ], # Assume that these environment variables are set up properly. The same applies below. aws_secret_access_key=os.environ["PFS2_AWS_SECRET_ACCESS_KEY"], endpoint_url=os.environ["PFS2_S3_ENDPOINT"], config=Config(connect_timeout=30, read_timeout=30), ), bucket_name=pfs2_bucket, ) def objective(trial: optuna.Trial) -> float: ... = trial.suggest_float("x", -10, 10) # Creating and writing an artifact. file_path = generate_example(...) # This function returns some kind of file. artifact_id = upload_artifact( trial, file_path, artifact_store ) # The return value is the artifact ID. trial.set_user_attr( "artifact_id", artifact_id ) # Save the ID in RDB so that it can be referenced later. return ... study = optuna.create_study( study_name="test_study", storage="mysql://USER:PASS@localhost:3306/test", # Set the appropriate URL. ) study.optimize(objective, n_trials=100) # Loading and displaying artifacts associated with the best trial. best_artifact_id = study.best_trial.user_attrs.get("artifact_id") with artifact_store.open_reader(best_artifact_id) as f: content = f.read().decode("utf-8") print(content) Example: Optimization of Chemical Structures -------------------------------------------- In this section, we introduce an example of optimizing chemical structure using Optuna by utilizing the artifact module. We will target relatively small structures, but the approach remains the same even for complex structures. Consider the process of a specific molecule adsorbing onto another substance. In this process, the ease of adsorption reaction changes depending on the position of the adsorbing molecule to the substance it is adsorbed onto. The ease of adsorption reaction can be evaluated by the adsorption energy (the difference between the energy of the system after adsorption and before). By formulating the problem as a minimization problem of an objective function that takes the positional relationship of the adsorbing molecule as input and outputs the adsorption energy, this problem is solved as a black-box optimization problem. First, let's import the necessary modules and define some helper functions. You need to install the ASE library for handling chemical structures in addition to Optuna, so please install it with `pip install ase`. """ from __future__ import annotations import io import logging import os import sys import uuid from ase import Atoms from ase.build import bulk, fcc111, molecule, add_adsorbate from ase.calculators.emt import EMT from ase.io import write, read from ase.optimize import LBFGS import numpy as np from optuna.artifacts import FileSystemArtifactStore from optuna.artifacts import upload_artifact from optuna.logging import get_logger from optuna import create_study from optuna import Trial # Add stream handler of stdout to show the messages get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout)) def get_opt_energy(atoms: Atoms, fmax: float = 0.001) -> float: calculator = EMT() atoms.set_calculator(calculator) opt = LBFGS(atoms, logfile=None) opt.run(fmax=fmax) return atoms.get_total_energy() def create_slab() -> tuple[Atoms, float]: calculator = EMT() bulk_atoms = bulk("Pt", cubic=True) bulk_atoms.calc = calculator a = np.mean(np.diag(bulk_atoms.cell)) slab = fcc111("Pt", a=a, size=(4, 4, 4), vacuum=40.0, periodic=True) slab.calc = calculator E_slab = get_opt_energy(slab, fmax=1e-4) return slab, E_slab def create_mol() -> tuple[Atoms, float]: calculator = EMT() mol = molecule("CO") mol.calc = calculator E_mol = get_opt_energy(mol, fmax=1e-4) return mol, E_mol def atoms_to_json(atoms: Atoms) -> str: f = io.StringIO() write(f, atoms, format="json") return f.getvalue() def json_to_atoms(atoms_str: str) -> Atoms: return read(io.StringIO(atoms_str), format="json") ################################################################################################### # Each function is as follows. # # - `get_opt_energy`: Takes a chemical structure, transitions it to a locally stable structure, and returns the energy after the transition. # - `create_slab`: Constructs the substance being adsorbed. # - `create_mol`: Constructs the molecule being adsorbed. # - `atoms_to_json`: Converts the chemical structure to a string. # - `json_to_atoms`: Converts the string to a chemical structure. # # Using these functions, the code to search for adsorption structures using Optuna is as follows. The objective function is defined # as class `Objective` in order to carry the artifact store. In its `__call__` method, it retrieves the substance being adsorbed # (`slab`) and the molecule being adsorbed (`mol`), then after sampling their positional relationship using Optuna (multiple # `trial.suggest_xxx` methods), it triggers an adsorption reaction with the `add_adsorbate` function, transitions to a locally # stable structure, then saves the structure in the artifact store and returns the adsorption energy. # # The `main` function contains the code to create a `Study` and execute optimization. When creating a `Study`, a storage is # specified using SQLite, and a back end using the local file system is used for the artifact store. In other words, it corresponds # to Scenario 1 explained in the previous section. After performing 100 trials of optimization, it displays the information for the # best trial, and finally saves the chemical structure as `best_atoms.png`. The obtained `best_atoms.png` is shown in Figure 4. class Objective: def __init__(self, artifact_store: FileSystemArtifactStore) -> None: self._artifact_store = artifact_store def __call__(self, trial: Trial) -> float: slab = json_to_atoms(trial.study.user_attrs["slab"]) E_slab = trial.study.user_attrs["E_slab"] mol = json_to_atoms(trial.study.user_attrs["mol"]) E_mol = trial.study.user_attrs["E_mol"] phi = 180.0 * trial.suggest_float("phi", -1, 1) theta = np.arccos(trial.suggest_float("theta", -1, 1)) * 180.0 / np.pi psi = 180 * trial.suggest_float("psi", -1, 1) x_pos = trial.suggest_float("x_pos", 0, 0.5) y_pos = trial.suggest_float("y_pos", 0, 0.5) z_hig = trial.suggest_float("z_hig", 1, 5) xy_position = np.matmul([x_pos, y_pos, 0], slab.cell)[:2] mol.euler_rotate(phi=phi, theta=theta, psi=psi) add_adsorbate(slab, mol, z_hig, xy_position) E_slab_mol = get_opt_energy(slab, fmax=1e-2) write(f"./tmp/{trial.number}.json", slab, format="json") artifact_id = upload_artifact(trial, f"./tmp/{trial.number}.json", self._artifact_store) trial.set_user_attr("structure", artifact_id) return E_slab_mol - E_slab - E_mol def main(): study = create_study( study_name="test_study", storage="sqlite:///example.db", load_if_exists=True, ) slab, E_slab = create_slab() study.set_user_attr("slab", atoms_to_json(slab)) study.set_user_attr("E_slab", E_slab) mol, E_mol = create_mol() study.set_user_attr("mol", atoms_to_json(mol)) study.set_user_attr("E_mol", E_mol) os.makedirs("./tmp", exist_ok=True) base_path = "./artifacts" os.makedirs(base_path, exist_ok=True) artifact_store = FileSystemArtifactStore(base_path=base_path) study.optimize(Objective(artifact_store), n_trials=3) print( f"Best trial is #{study.best_trial.number}\n" f" Its adsorption energy is {study.best_value}\n" f" Its adsorption position is\n" f" phi : {study.best_params['phi']}\n" f" theta: {study.best_params['theta']}\n" f" psi. : {study.best_params['psi']}\n" f" x_pos: {study.best_params['x_pos']}\n" f" y_pos: {study.best_params['y_pos']}\n" f" z_hig: {study.best_params['z_hig']}" ) best_artifact_id = study.best_trial.user_attrs["structure"] with artifact_store.open_reader(best_artifact_id) as f: content = f.read().decode("utf-8") best_atoms = json_to_atoms(content) print(best_atoms) write("best_atoms.png", best_atoms, rotation=("315x,0y,0z")) if __name__ == "__main__": main() ################################################################################################### # .. list-table:: # :header-rows: 1 # # * - Fig 4. The chemical structure obtained by the above code. # * - .. image:: https://github.com/optuna/optuna/assets/38826298/c6bd62fd-599a-424e-8c2c-ca88af85cc63 # # As shown above, it is convenient to use the artifact module when performing the optimization of chemical structures with Optuna. # In the case of small structures or fewer trial numbers, it's fine to convert it to a string and save it directly in the RDB. # However, when dealing with complex structures or performing large-scale searches, it's better to save it outside the RDB to # avoid overloading it, such as in an external file system or AWS S3. # # Conclusion # ---------- # # The artifact module is a useful feature when you want to save relatively large data for each trial. It can be used for various # purposes such as saving snapshots of machine learning models, optimizing chemical structures, and human-in-the-loop optimization # of images and sounds. It's a powerful assistant for black-box optimization with Optuna. Also, if there are ways to use it that # we, the Optuna committers, haven't noticed, please let us know on GitHub discussions. Have a great optimization life with Optuna! optuna-3.5.0/tutorial/20_recipes/README.rst000066400000000000000000000001441453453102400203120ustar00rootroot00000000000000.. _recipes: Recipes ------- Showcases the recipes that might help you using Optuna with comfort. optuna-3.5.0/tutorial/README.rst000066400000000000000000000006131453453102400163600ustar00rootroot00000000000000Tutorial ======== If you are new to Optuna or want a general introduction, we highly recommend the below video. .. raw:: html