Developing algorithms

The documentation here explains how to create new algorithm plugins based on the cookiecutter.


Install Python requirements to use the template:

$ python -m pip install cookiecutter>=1.5 versioneer>=0.18 jinja2

Create a new project directly from the template on GitHub:

$ cookiecutter gh:Epistimio/cookiecutter-orion.algo
plugin_name []: skopt
author_name []: Xavier Bouthillier
author_short [Author Name]:
author_email []:
github_username []: bouthilx
copyright [2019, Author Name]:
short_description [TODO]:
synopsis [TODO]:
algo_name []: BayesianOptimizer
algo_module_name [bayesianoptimizer]: bayes
Field Description  
plugin_name Will be used for orion.algo.plugin_name  
author_name For metadata of python package  
author_short For metadata of python package  
author_email For metadata of python package  
github_username Username to build the url for installation  
copyright For the BSD-3 license (You can change the license)  
short_description For metadata of python package  
synopsis For documentation in algo module  
algo_name Name for the algorithm class  
algo_module_name Name of the algorihtm module  

This will create the following package structure.

├── README.rst
├── setup.cfg
├── LICENSE (BSD License)
├── tox.ini
├── dev-requirements.txt
├── doc
│   ├── requirements.txt
│   └── src
│       ├──
│       └── index.rst
├── tests
│   ├── requirements.txt
│   ├──
│   └── benchmark
│       ├── requirements.txt
│       ├──
│       ├──
│       ├── {algoname}.yaml
│       ├── bayesopt.yaml
│       └── random_search.yaml
└── src
    └── orion
        └── algo
            └── {plugin_name}
                ├── {algoname}.py

The important files to modify are src/orion/algo/{plugin_name}/{module_name}.py to implement the algorithm and tests/benchmark/{algo_name}.yaml to fill the arguments required for the algorithm you implement.


Note that you are free to change the License, copyright is to your name. src/orion/algo/{plugin_name}/

This serves to version automatically your algo, just ignore these if you don’t plan to make releases.


These are the automatically generated tests.


Automatically generated benchmark test using the yaml files created in the same folder.


Automatically generated template for documentation


Tox file defining commands to run tests, build doc and publish code.


requires = 'real'

Some algorithms requires the search space to be real. You can specify this requirement by adding the attribute requires = 'real' at the base of the class definition.In this case, the algorithm wrapper in Orion’s core will convert the search space to real one before passing it to your algorithm. This way the user can define discrete or categorital dimensions while using algorithms that require a real space.

def __init__(self, space, seed=None):

The initialization of the algorithm must pass space and seed to super().__init__, but must also pass any other argument that must be part of the configuration of the algorithm. Any argument passed to super() will be assigned as an attribute to the algorithm and will be included in algo.configuration, which is used to save the configuration of the algorithm in the database.

def seed_rng(self, seed=None):

This method must seed the internal state of the algorithm so that it would always sample the same sequence of points.

def state_dict(self):

The state dict is used to copy algorithms within the parallel strategy. All algorithms must provide a state dict to ensure that we reset it to a previous state.

def set_state(self, state_dict):

Stateful attributes of the algorithm are reset using the given state_dict. Note that set_state must be compliant with state_dict and use the same structure.

def suggest(self, num=1):

The method to suggest new trials. The argument num=1 request the number of trials that the algorithm must sample. Note that it is possible to only support num=1 and raise ValueError otherwise.

def observe(self, points, results):

The method to observe results of suggested trials. Note that observe may be called several times for the same points. Make sure to handle this properly within your algorithm if this is problematic. Points are passed as a list of lists, each list representing the value of the params in the order defined in


To test the freshly built package, you must first install the requirements. From within the new package, run

$ pip install -r tests/requirements.txt

You can then run the unit-tests with

$ pytest tests/

or using tox

$ tox -e py36

Note that the algorithm pre-built is random search so that you can start from a fully working environment and test your way through the modifications.

There is also the option of running the toy-benchmark to compare the performance of your algorithm with random search and bayesian optimization. First install the requirements.

$ pip install -r tests/benchmark/requirements.txt

And then execute the benchmark

$ pytest tests/benchmark/

or using tox

$ tox -e benchmark

Finally, official plugins must follow the same code quality standards than orion.core. Therefore there is tests included in the pre-built package for flake8 and pylint. You can execute them with

$ tox -e flake8


$ tox -e pylint