375 lines
16 KiB
Plaintext
375 lines
16 KiB
Plaintext
Metadata-Version: 2.1
|
|
Name: optuna
|
|
Version: 4.1.0
|
|
Summary: A hyperparameter optimization framework
|
|
Author: Takuya Akiba
|
|
Project-URL: homepage, https://optuna.org/
|
|
Project-URL: repository, https://github.com/optuna/optuna
|
|
Project-URL: documentation, https://optuna.readthedocs.io
|
|
Project-URL: bugtracker, https://github.com/optuna/optuna/issues
|
|
Classifier: Development Status :: 5 - Production/Stable
|
|
Classifier: Intended Audience :: Science/Research
|
|
Classifier: Intended Audience :: Developers
|
|
Classifier: License :: OSI Approved :: MIT License
|
|
Classifier: Programming Language :: Python :: 3
|
|
Classifier: Programming Language :: Python :: 3.8
|
|
Classifier: Programming Language :: Python :: 3.9
|
|
Classifier: Programming Language :: Python :: 3.10
|
|
Classifier: Programming Language :: Python :: 3.11
|
|
Classifier: Programming Language :: Python :: 3.12
|
|
Classifier: Programming Language :: Python :: 3.13
|
|
Classifier: Programming Language :: Python :: 3 :: Only
|
|
Classifier: Topic :: Scientific/Engineering
|
|
Classifier: Topic :: Scientific/Engineering :: Mathematics
|
|
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
Classifier: Topic :: Software Development
|
|
Classifier: Topic :: Software Development :: Libraries
|
|
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
Requires-Python: >=3.8
|
|
Description-Content-Type: text/markdown
|
|
License-File: LICENSE
|
|
License-File: LICENSE_THIRD_PARTY
|
|
Requires-Dist: alembic>=1.5.0
|
|
Requires-Dist: colorlog
|
|
Requires-Dist: numpy
|
|
Requires-Dist: packaging>=20.0
|
|
Requires-Dist: sqlalchemy>=1.4.2
|
|
Requires-Dist: tqdm
|
|
Requires-Dist: PyYAML
|
|
Provides-Extra: benchmark
|
|
Requires-Dist: asv>=0.5.0; extra == "benchmark"
|
|
Requires-Dist: cma; extra == "benchmark"
|
|
Requires-Dist: virtualenv; extra == "benchmark"
|
|
Provides-Extra: checking
|
|
Requires-Dist: black; extra == "checking"
|
|
Requires-Dist: blackdoc; extra == "checking"
|
|
Requires-Dist: flake8; extra == "checking"
|
|
Requires-Dist: isort; extra == "checking"
|
|
Requires-Dist: mypy; extra == "checking"
|
|
Requires-Dist: mypy-boto3-s3; extra == "checking"
|
|
Requires-Dist: types-PyYAML; extra == "checking"
|
|
Requires-Dist: types-redis; extra == "checking"
|
|
Requires-Dist: types-setuptools; extra == "checking"
|
|
Requires-Dist: types-tqdm; extra == "checking"
|
|
Requires-Dist: typing-extensions>=3.10.0.0; extra == "checking"
|
|
Provides-Extra: document
|
|
Requires-Dist: ase; extra == "document"
|
|
Requires-Dist: cmaes>=0.10.0; extra == "document"
|
|
Requires-Dist: fvcore; extra == "document"
|
|
Requires-Dist: kaleido; extra == "document"
|
|
Requires-Dist: lightgbm; extra == "document"
|
|
Requires-Dist: matplotlib!=3.6.0; extra == "document"
|
|
Requires-Dist: pandas; extra == "document"
|
|
Requires-Dist: pillow; extra == "document"
|
|
Requires-Dist: plotly>=4.9.0; extra == "document"
|
|
Requires-Dist: scikit-learn; extra == "document"
|
|
Requires-Dist: sphinx; extra == "document"
|
|
Requires-Dist: sphinx-copybutton; extra == "document"
|
|
Requires-Dist: sphinx-gallery; extra == "document"
|
|
Requires-Dist: sphinx-rtd-theme>=1.2.0; extra == "document"
|
|
Requires-Dist: torch; extra == "document"
|
|
Requires-Dist: torchvision; extra == "document"
|
|
Provides-Extra: optional
|
|
Requires-Dist: boto3; extra == "optional"
|
|
Requires-Dist: cmaes>=0.10.0; extra == "optional"
|
|
Requires-Dist: google-cloud-storage; extra == "optional"
|
|
Requires-Dist: matplotlib!=3.6.0; extra == "optional"
|
|
Requires-Dist: pandas; extra == "optional"
|
|
Requires-Dist: plotly>=4.9.0; extra == "optional"
|
|
Requires-Dist: redis; extra == "optional"
|
|
Requires-Dist: scikit-learn>=0.24.2; extra == "optional"
|
|
Requires-Dist: scipy; extra == "optional"
|
|
Requires-Dist: torch; python_version <= "3.12" and extra == "optional"
|
|
Provides-Extra: test
|
|
Requires-Dist: coverage; extra == "test"
|
|
Requires-Dist: fakeredis[lua]; extra == "test"
|
|
Requires-Dist: kaleido; extra == "test"
|
|
Requires-Dist: moto; extra == "test"
|
|
Requires-Dist: pytest; extra == "test"
|
|
Requires-Dist: scipy>=1.9.2; extra == "test"
|
|
Requires-Dist: torch; python_version <= "3.12" and extra == "test"
|
|
|
|
<div align="center"><img src="https://raw.githubusercontent.com/optuna/optuna/master/docs/image/optuna-logo.png" width="800"/></div>
|
|
|
|
# Optuna: A hyperparameter optimization framework
|
|
|
|
[](https://www.python.org)
|
|
[](https://pypi.python.org/pypi/optuna)
|
|
[](https://anaconda.org/conda-forge/optuna)
|
|
[](https://github.com/optuna/optuna)
|
|
[](https://optuna.readthedocs.io/en/stable/)
|
|
[](https://codecov.io/gh/optuna/optuna)
|
|
|
|
:link: [**Website**](https://optuna.org/)
|
|
| :page_with_curl: [**Docs**](https://optuna.readthedocs.io/en/stable/)
|
|
| :gear: [**Install Guide**](https://optuna.readthedocs.io/en/stable/installation.html)
|
|
| :pencil: [**Tutorial**](https://optuna.readthedocs.io/en/stable/tutorial/index.html)
|
|
| :bulb: [**Examples**](https://github.com/optuna/optuna-examples)
|
|
| [**Twitter**](https://twitter.com/OptunaAutoML)
|
|
| [**LinkedIn**](https://www.linkedin.com/showcase/optuna/)
|
|
| [**Medium**](https://medium.com/optuna)
|
|
|
|
*Optuna* is an automatic hyperparameter optimization software framework, particularly designed
|
|
for machine learning. It features an imperative, *define-by-run* style user API. Thanks to our
|
|
*define-by-run* API, the code written with Optuna enjoys high modularity, and the user of
|
|
Optuna can dynamically construct the search spaces for the hyperparameters.
|
|
|
|
## :loudspeaker: News
|
|
<!-- TODO: when you add a new line, please delete the oldest line -->
|
|
* **Oct 21, 2024**: We posted [an article](https://medium.com/optuna/an-introduction-to-moea-d-and-examples-of-multi-objective-optimization-comparisons-8630565a4e89) introducing [MOEA/D](https://hub.optuna.org/samplers/moead/) and an example comparison with other optimization methods.
|
|
* **Oct 15, 2024**: We posted [an article](https://medium.com/optuna/introducing-a-new-terminator-early-termination-of-black-box-optimization-based-on-expected-9a660774fcdb) about `Terminator`, which is expanded in Optuna 4.0.
|
|
* **Sep 18, 2024**: We posted [an article](https://medium.com/optuna/introducing-the-stabilized-journalstorage-in-optuna-4-0-from-mechanism-to-use-case-e320795ffb61) about `JournalStorage`, which is stabilized in Optuna 4.0.
|
|
* **Sep 2, 2024**: Optuna 4.0 is available! You can install it by `pip install -U optuna`. Find the latest [here](https://github.com/optuna/optuna/releases) and check [our article](https://medium.com/optuna/optuna-4-0-whats-new-in-the-major-release-3325a8420d10).
|
|
* **Aug 30, 2024**: We posted [an article](https://medium.com/optuna/optunahub-a-feature-sharing-platform-for-optuna-now-available-in-official-release-4b99efe9934d) about the official release of [OptunaHub](https://hub.optuna.org/).
|
|
* **Aug 28, 2024**: We posted [an article](https://medium.com/optuna/a-natural-gradient-based-optimization-algorithm-registered-on-optunahub-0dbe17cb0f7d) about [implicit natural gradient optimization (`INGO`)](https://hub.optuna.org/samplers/implicit_natural_gradient/), a sampler newly supported in [OptunaHub](https://hub.optuna.org/).
|
|
|
|
## :fire: Key Features
|
|
|
|
Optuna has modern functionalities as follows:
|
|
|
|
- [Lightweight, versatile, and platform agnostic architecture](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/001_first.html)
|
|
- Handle a wide variety of tasks with a simple installation that has few requirements.
|
|
- [Pythonic search spaces](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html)
|
|
- Define search spaces using familiar Python syntax including conditionals and loops.
|
|
- [Efficient optimization algorithms](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html)
|
|
- Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials.
|
|
- [Easy parallelization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html)
|
|
- Scale studies to tens or hundreds of workers with little or no changes to the code.
|
|
- [Quick visualization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/005_visualization.html)
|
|
- Inspect optimization histories from a variety of plotting functions.
|
|
|
|
|
|
## Basic Concepts
|
|
|
|
We use the terms *study* and *trial* as follows:
|
|
|
|
- Study: optimization based on an objective function
|
|
- Trial: a single execution of the objective function
|
|
|
|
Please refer to the sample code below. The goal of a *study* is to find out the optimal set of
|
|
hyperparameter values (e.g., `regressor` and `svr_c`) through multiple *trials* (e.g.,
|
|
`n_trials=100`). Optuna is a framework designed for automation and acceleration of
|
|
optimization *studies*.
|
|
|
|
<details open>
|
|
<summary>Sample code with scikit-learn</summary>
|
|
|
|
[](http://colab.research.google.com/github/optuna/optuna-examples/blob/main/quickstart.ipynb)
|
|
|
|
```python
|
|
import ...
|
|
|
|
# Define an objective function to be minimized.
|
|
def objective(trial):
|
|
|
|
# Invoke suggest methods of a Trial object to generate hyperparameters.
|
|
regressor_name = trial.suggest_categorical('regressor', ['SVR', 'RandomForest'])
|
|
if regressor_name == 'SVR':
|
|
svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True)
|
|
regressor_obj = sklearn.svm.SVR(C=svr_c)
|
|
else:
|
|
rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
|
|
regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)
|
|
|
|
X, y = sklearn.datasets.fetch_california_housing(return_X_y=True)
|
|
X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)
|
|
|
|
regressor_obj.fit(X_train, y_train)
|
|
y_pred = regressor_obj.predict(X_val)
|
|
|
|
error = sklearn.metrics.mean_squared_error(y_val, y_pred)
|
|
|
|
return error # An objective value linked with the Trial object.
|
|
|
|
study = optuna.create_study() # Create a new study.
|
|
study.optimize(objective, n_trials=100) # Invoke optimization of the objective function.
|
|
```
|
|
</details>
|
|
|
|
> [!NOTE]
|
|
> More examples can be found in [optuna/optuna-examples](https://github.com/optuna/optuna-examples).
|
|
>
|
|
> The examples cover diverse problem setups such as multi-objective optimization, constrained optimization, pruning, and distributed optimization.
|
|
|
|
## Installation
|
|
|
|
Optuna is available at [the Python Package Index](https://pypi.org/project/optuna/) and on [Anaconda Cloud](https://anaconda.org/conda-forge/optuna).
|
|
|
|
```bash
|
|
# PyPI
|
|
$ pip install optuna
|
|
```
|
|
|
|
```bash
|
|
# Anaconda Cloud
|
|
$ conda install -c conda-forge optuna
|
|
```
|
|
|
|
> [!IMPORTANT]
|
|
> Optuna supports Python 3.8 or newer.
|
|
>
|
|
> Also, we provide Optuna docker images on [DockerHub](https://hub.docker.com/r/optuna/optuna).
|
|
|
|
## Integrations
|
|
|
|
Optuna has integration features with various third-party libraries. Integrations can be found in [optuna/optuna-integration](https://github.com/optuna/optuna-integration) and the document is available [here](https://optuna-integration.readthedocs.io/en/stable/index.html).
|
|
|
|
<details>
|
|
<summary>Supported integration libraries</summary>
|
|
|
|
* [Catboost](https://github.com/optuna/optuna-examples/tree/main/catboost/catboost_pruning.py)
|
|
* [Dask](https://github.com/optuna/optuna-examples/tree/main/dask/dask_simple.py)
|
|
* [fastai](https://github.com/optuna/optuna-examples/tree/main/fastai/fastai_simple.py)
|
|
* [Keras](https://github.com/optuna/optuna-examples/tree/main/keras/keras_integration.py)
|
|
* [LightGBM](https://github.com/optuna/optuna-examples/tree/main/lightgbm/lightgbm_integration.py)
|
|
* [MLflow](https://github.com/optuna/optuna-examples/tree/main/mlflow/keras_mlflow.py)
|
|
* [PyTorch](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_simple.py)
|
|
* [PyTorch Ignite](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_ignite_simple.py)
|
|
* [PyTorch Lightning](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_lightning_simple.py)
|
|
* [TensorBoard](https://github.com/optuna/optuna-examples/tree/main/tensorboard/tensorboard_simple.py)
|
|
* [TensorFlow](https://github.com/optuna/optuna-examples/tree/main/tensorflow/tensorflow_estimator_integration.py)
|
|
* [tf.keras](https://github.com/optuna/optuna-examples/tree/main/tfkeras/tfkeras_integration.py)
|
|
* [Weights & Biases](https://github.com/optuna/optuna-examples/tree/main/wandb/wandb_integration.py)
|
|
* [XGBoost](https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py)
|
|
</details>
|
|
|
|
## Web Dashboard
|
|
|
|
[Optuna Dashboard](https://github.com/optuna/optuna-dashboard) is a real-time web dashboard for Optuna.
|
|
You can check the optimization history, hyperparameter importance, etc. in graphs and tables.
|
|
You don't need to create a Python script to call [Optuna's visualization](https://optuna.readthedocs.io/en/stable/reference/visualization/index.html) functions.
|
|
Feature requests and bug reports are welcome!
|
|
|
|
|
|
|
|
`optuna-dashboard` can be installed via pip:
|
|
|
|
```shell
|
|
$ pip install optuna-dashboard
|
|
```
|
|
|
|
> [!TIP]
|
|
> Please check out the convenience of Optuna Dashboard using the sample code below.
|
|
|
|
<details>
|
|
<summary>Sample code to launch Optuna Dashboard</summary>
|
|
|
|
Save the following code as `optimize_toy.py`.
|
|
|
|
```python
|
|
import optuna
|
|
|
|
|
|
def objective(trial):
|
|
x1 = trial.suggest_float("x1", -100, 100)
|
|
x2 = trial.suggest_float("x2", -100, 100)
|
|
return x1 ** 2 + 0.01 * x2 ** 2
|
|
|
|
|
|
study = optuna.create_study(storage="sqlite:///db.sqlite3") # Create a new study with database.
|
|
study.optimize(objective, n_trials=100)
|
|
```
|
|
|
|
Then try the commands below:
|
|
|
|
```shell
|
|
# Run the study specified above
|
|
$ python optimize_toy.py
|
|
|
|
# Launch the dashboard based on the storage `sqlite:///db.sqlite3`
|
|
$ optuna-dashboard sqlite:///db.sqlite3
|
|
...
|
|
Listening on http://localhost:8080/
|
|
Hit Ctrl-C to quit.
|
|
```
|
|
|
|
</details>
|
|
|
|
|
|
## OptunaHub
|
|
|
|
[OptunaHub](https://hub.optuna.org/) is a feature-sharing platform for Optuna.
|
|
You can use the registered features and publish your packages.
|
|
|
|
### Use registered features
|
|
|
|
`optunahub` can be installed via pip:
|
|
|
|
```shell
|
|
$ pip install optunahub
|
|
```
|
|
|
|
You can load registered module with `optunahub.load_module`.
|
|
|
|
```python
|
|
import optuna
|
|
import optunahub
|
|
|
|
|
|
def objective(trial: optuna.Trial) -> float:
|
|
x = trial.suggest_float("x", 0, 1)
|
|
|
|
return x
|
|
|
|
|
|
mod = optunahub.load_module("samplers/simulated_annealing")
|
|
|
|
study = optuna.create_study(sampler=mod.SimulatedAnnealingSampler())
|
|
study.optimize(objective, n_trials=20)
|
|
|
|
print(study.best_trial.value, study.best_trial.params)
|
|
```
|
|
|
|
For more details, please refer to [the optunahub documentation](https://optuna.github.io/optunahub/).
|
|
|
|
### Publish your packages
|
|
|
|
You can publish your package via [optunahub-registry](https://github.com/optuna/optunahub-registry).
|
|
See the [OptunaHub tutorial](https://optuna.github.io/optunahub-registry/index.html).
|
|
|
|
|
|
## Communication
|
|
|
|
- [GitHub Discussions] for questions.
|
|
- [GitHub Issues] for bug reports and feature requests.
|
|
|
|
[GitHub Discussions]: https://github.com/optuna/optuna/discussions
|
|
[GitHub issues]: https://github.com/optuna/optuna/issues
|
|
|
|
|
|
## Contribution
|
|
|
|
Any contributions to Optuna are more than welcome!
|
|
|
|
If you are new to Optuna, please check the [good first issues](https://github.com/optuna/optuna/labels/good%20first%20issue). They are relatively simple, well-defined, and often good starting points for you to get familiar with the contribution workflow and other developers.
|
|
|
|
If you already have contributed to Optuna, we recommend the other [contribution-welcome issues](https://github.com/optuna/optuna/labels/contribution-welcome).
|
|
|
|
For general guidelines on how to contribute to the project, take a look at [CONTRIBUTING.md](./CONTRIBUTING.md).
|
|
|
|
|
|
## Reference
|
|
|
|
If you use Optuna in one of your research projects, please cite [our KDD paper](https://doi.org/10.1145/3292500.3330701) "Optuna: A Next-generation Hyperparameter Optimization Framework":
|
|
|
|
<details open>
|
|
<summary>BibTeX</summary>
|
|
|
|
```bibtex
|
|
@inproceedings{akiba2019optuna,
|
|
title={{O}ptuna: A Next-Generation Hyperparameter Optimization Framework},
|
|
author={Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori},
|
|
booktitle={The 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
|
|
pages={2623--2631},
|
|
year={2019}
|
|
}
|
|
```
|
|
</details>
|
|
|
|
|
|
## License
|
|
|
|
MIT License (see [LICENSE](./LICENSE)).
|
|
|
|
Optuna uses the codes from SciPy and fdlibm projects (see [LICENSE_THIRD_PARTY](./LICENSE_THIRD_PARTY)).
|