{"action":"closed","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12332","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12332/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12332/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12332/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/12332","id":367960018,"node_id":"MDU6SXNzdWUzNjc5NjAwMTg=","number":12332,"title":"GridSearchCV occasionally crashes with OverflowError","user":{"login":"adamjstewart","id":12021217,"node_id":"MDQ6VXNlcjEyMDIxMjE3","avatar_url":"https://avatars2.githubusercontent.com/u/12021217?v=4","gravatar_id":"","url":"https://api.github.com/users/adamjstewart","html_url":"https://github.com/adamjstewart","followers_url":"https://api.github.com/users/adamjstewart/followers","following_url":"https://api.github.com/users/adamjstewart/following{/other_user}","gists_url":"https://api.github.com/users/adamjstewart/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjstewart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjstewart/subscriptions","organizations_url":"https://api.github.com/users/adamjstewart/orgs","repos_url":"https://api.github.com/users/adamjstewart/repos","events_url":"https://api.github.com/users/adamjstewart/events{/privacy}","received_events_url":"https://api.github.com/users/adamjstewart/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-10-08T21:30:50Z","updated_at":"2018-10-10T02:14:17Z","closed_at":"2018-10-10T02:14:17Z","author_association":"NONE","body":"#### Description\r\n\r\nI'm trying to tune a `MLPClassifier` using `GridSearchCV` and the Adam optimizer. Most of the time, it works fine, but no matter what hyperparameters I search over, it will occasionally crash with an OverflowError.\r\n\r\n#### Steps/Code to Reproduce\r\n\r\nHere is a snippet of the relevant code showing which hyperparameters I'm trying to tune.\r\n```python\r\nmodel = MLPClassifier(early_stopping=True)\r\n\r\n# Perform a grid search over the hyperparameter space\r\nhidden_layer_sizes = []\r\nfor num_layer in range(1, 6):\r\n   for size_exp in range(5, 11):\r\n        hidden_layer_sizes.append([2 ** size_exp] * num_layer)\r\n\r\nparam_grid = {\r\n    'hidden_layer_sizes': hidden_layer_sizes,\r\n    'activation': ['logistic', 'tanh', 'relu'],\r\n    'learning_rate_init': [10**i for i in range(-5, 0)],\r\n    'beta_1': [0.9, 0.99, 0.999],\r\n    'beta_2': [0.99, 0.999, 0.9999],\r\n}\r\n\r\nclf = GridSearchCV(model, param_grid, n_jobs=-1)\r\nclf.fit(train_data, train_labels)\r\n```\r\nI realize that you can't actually reproduce this without the dataset, but the dataset is rather large.\r\n\r\n#### Expected Results\r\n\r\nI realize that OverflowError is OverflowError, but why must this crash the entire grid search? I would expect all hyperparameter combinations that result in an error to be handled internally with a try-except block. At the end, the best combination that didn't result in an error should be reported.\r\n\r\n#### Actual Results\r\n\r\nIf even a single hyperparameter combination out of thousands results in an OverflowError, the entire program crashes, wasting dozens of CPU hours. I see an error message like the following:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py\", line 350, in __call__\r\n    return self.func(*args, **kwargs)\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py\", line 131, in __call__\r\n    return [func(*args, **kwargs) for func, args, kwargs in self.items]\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py\", line 131, in <listcomp>\r\n    return [func(*args, **kwargs) for func, args, kwargs in self.items]\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/model_selection/_validation.py\", line 437, in _fit_and_score\r\n    estimator.fit(X_train, y_train, **fit_params)\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py\", line 973, in fit\r\n    hasattr(self, \"classes_\")))\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py\", line 378, in _fit\r\n    intercept_grads, layer_units, incremental)\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py\", line 522, in _fit_stochastic\r\n    self._optimizer.update_params(grads)\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/neural_network/_stochastic_optimizers.py\", line 43, in update_params\r\n    updates = self._get_updates(grads)\r\n  File \"/mnt/bwpy/single/usr/lib/python3.6/site-packages/sklearn/neural_network/_stochastic_optimizers.py\", line 263, in _get_updates\r\n    (1 - self.beta_1 ** self.t))\r\nOverflowError: (34, 'Numerical result out of range')\r\n```\r\nI tried skipping the search over `beta_1` and `beta_2`, but that also resulted in an OverflowError. I can keep narrowing my search, but that runs the risk of finding suboptimal hyperparameters.\r\n\r\n#### Versions\r\n\r\n```python\r\n>>> import platform; print(platform.platform())\r\nLinux-3.0.101-0.47.106.29-default-x86_64-with-SuSE-11-x86_64\r\n>>> import sys; print(\"Python\", sys.version)\r\nPython 3.5.5 (default, May 29 2018, 17:40:43) \r\n[GCC 4.9.3 20150626 (Cray Inc.)]\r\n>>> import numpy; print(\"NumPy\", numpy.__version__)\r\nNumPy 1.14.2\r\n>>> import scipy; print(\"SciPy\", scipy.__version__)\r\nSciPy 1.0.0\r\n>>> import sklearn; print(\"Scikit-Learn\", sklearn.__version__)\r\nScikit-Learn 0.19.0\r\n```"}}
{"actor":{"display_login":"jnothman"}}