Hyperparameter optimization

In the sections on Feature engineering and Predicting we learned how to train both the feature learning algorithm and the machine learning algorithm used for prediction in the getML engine. However, there are lots of parameters involved. MultirelModel, RelboostModel, MultirelTimeSeries, RelboostTimeSeries, LinearRegression, LogisticRegression, XGBoostClassifier, and XGBoostRegressor all have their own settings. That is why you might want to use hyperparameter optimization.

The most relevant parameters of these classes can be chosen to constitute individual dimensions of a parameter space. For each parameter, a lower and upper bound has to be provided and the hyperparameter optimization will search the space within these bounds. This will be done iteratively by drawing a specific parameter combination, overwriting the corresponding parameters in a base pipeline, and then fitting and scoring it. The algorithm used to draw from the parameter space is represented by the different classes of hyperopt.

While RandomSearch and LatinHypercubeSearch are purely statistical approaches, GaussianHyperparameterSearch uses prior knowledge obtained from evaluations of previous parameter combinations to select the next one.