{"action":"created","issue":{"url":"https://api.github.com/repos/fmfn/BayesianOptimization/issues/10","repository_url":"https://api.github.com/repos/fmfn/BayesianOptimization","labels_url":"https://api.github.com/repos/fmfn/BayesianOptimization/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/fmfn/BayesianOptimization/issues/10/comments","events_url":"https://api.github.com/repos/fmfn/BayesianOptimization/issues/10/events","html_url":"https://github.com/fmfn/BayesianOptimization/issues/10","id":134902934,"number":10,"title":"Same observation being generatad","user":{"login":"brunodoamaral","id":1158343,"avatar_url":"https://avatars.githubusercontent.com/u/1158343?v=3","gravatar_id":"","url":"https://api.github.com/users/brunodoamaral","html_url":"https://github.com/brunodoamaral","followers_url":"https://api.github.com/users/brunodoamaral/followers","following_url":"https://api.github.com/users/brunodoamaral/following{/other_user}","gists_url":"https://api.github.com/users/brunodoamaral/gists{/gist_id}","starred_url":"https://api.github.com/users/brunodoamaral/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brunodoamaral/subscriptions","organizations_url":"https://api.github.com/users/brunodoamaral/orgs","repos_url":"https://api.github.com/users/brunodoamaral/repos","events_url":"https://api.github.com/users/brunodoamaral/events{/privacy}","received_events_url":"https://api.github.com/users/brunodoamaral/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-19T16:10:58Z","updated_at":"2016-02-19T17:07:17Z","closed_at":null,"body":"Hi,\r\n\r\nI tried to run the code below to optimize a XGBoost classifier, but get stuck with same observation being tested all time. I expected some new observation being generated... or am I wrong?\r\n\r\nConsole output (after initial points generated). Notice that all iterations generates the same observation:\r\n```\r\n('XGB', {'num_round': 20.0, 'subsample': 0.25, 'eta': 0.01, 'colsample_bytree': 0.25, 'max_depth': 2.0})\r\nIteration:   1 | Last sampled value:   -0.680226 | with parameters:  {'num_round': 20.0, 'subsample': 0.25, 'eta': 0.01, 'colsample_bytree': 0.25, 'max_depth': 2.0}\r\n               | Current maximum:      -0.245901 | with parameters:  {'num_round': 28.712248896201515, 'subsample': 0.88492808306639748, 'eta': 0.78136949498158781, 'colsample_bytree': 0.99625386365127699, 'max_depth': 5.3806033554623252}\r\n               | Time taken: 0 minutes and 10.953415 seconds\r\n\r\n('XGB', {'num_round': 20.0, 'subsample': 0.25, 'eta': 0.01, 'colsample_bytree': 0.25, 'max_depth': 2.0})\r\nIteration:   2 | Last sampled value:   -0.680226 | with parameters:  {'num_round': 20.0, 'subsample': 0.25, 'eta': 0.01, 'colsample_bytree': 0.25, 'max_depth': 2.0}\r\n               | Current maximum:      -0.245901 | with parameters:  {'num_round': 28.712248896201515, 'subsample': 0.88492808306639748, 'eta': 0.78136949498158781, 'colsample_bytree': 0.99625386365127699, 'max_depth': 5.3806033554623252}\r\n               | Time taken: 0 minutes and 10.790525 seconds\r\n\r\n('XGB', {'num_round': 20.0, 'subsample': 0.25, 'eta': 0.01, 'colsample_bytree': 0.25, 'max_depth': 2.0})\r\nIteration:   3 | Last sampled value:   -0.680226 | with parameters:  {'num_round': 20.0, 'subsample': 0.25, 'eta': 0.01, 'colsample_bytree': 0.25, 'max_depth': 2.0}\r\n               | Current maximum:      -0.245901 | with parameters:  {'num_round': 28.712248896201515, 'subsample': 0.88492808306639748, 'eta': 0.78136949498158781, 'colsample_bytree': 0.99625386365127699, 'max_depth': 5.3806033554623252}\r\n               | Time taken: 0 minutes and 10.6884 seconds\r\n```\r\n\r\nFull code for the program (uses the xgboost library)\r\n```python\r\nimport xgboost as xgb\r\nfrom bayes_opt import BayesianOptimization\r\nfrom sklearn.datasets import make_classification\r\n\r\nX, y = make_classification(n_samples=2500, n_features=45, n_informative=12, n_redundant=7, n_classes=2, random_state=42)\r\n\r\n\r\ndef xgbcv(max_depth, eta, colsample_bytree, subsample, num_round):\r\n    print(\"XGB\", locals())\r\n\r\n    dtrain = xgb.DMatrix(X, label=y)\r\n\r\n    params = {\r\n        'booster': 'gbtree',\r\n        'objective': 'multi:softprob',\r\n        'silent': 1,\r\n        'max_depth': int(round(max_depth)),\r\n        'eta': eta,\r\n        'colsample_bytree': colsample_bytree,\r\n        'subsample': subsample,\r\n        'num_class': 2,\r\n        'eval_metric': 'mlogloss',\r\n        'seed': 42\r\n    }\r\n\r\n    r = xgb.cv(params, dtrain, int(round(num_round)), nfold=4, metrics={'mlogloss'}, seed=45, show_stdv=False)\r\n\r\n    return -r['test-mlogloss-mean'].mean()\r\n\r\n\r\nxgbBO = BayesianOptimization(xgbcv, {\r\n    'max_depth': (2, 6),\r\n    'eta': (0.01, 0.8),\r\n    'colsample_bytree': (0.25, 1.0),\r\n    'subsample': (0.25, 1.0),\r\n    'num_round': (20, 30),\r\n}, verbose=True)\r\n\r\nxgbBO.maximize(init_points=32, n_iter=6)\r\n\r\n```\r\n\r\nThanks in advice!"},"comment":{"url":"https://api.github.com/repos/fmfn/BayesianOptimization/issues/comments/186307544","html_url":"https://github.com/fmfn/BayesianOptimization/issues/10#issuecomment-186307544","issue_url":"https://api.github.com/repos/fmfn/BayesianOptimization/issues/10","id":186307544,"user":{"login":"fmfn","id":7664252,"avatar_url":"https://avatars.githubusercontent.com/u/7664252?v=3","gravatar_id":"","url":"https://api.github.com/users/fmfn","html_url":"https://github.com/fmfn","followers_url":"https://api.github.com/users/fmfn/followers","following_url":"https://api.github.com/users/fmfn/following{/other_user}","gists_url":"https://api.github.com/users/fmfn/gists{/gist_id}","starred_url":"https://api.github.com/users/fmfn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmfn/subscriptions","organizations_url":"https://api.github.com/users/fmfn/orgs","repos_url":"https://api.github.com/users/fmfn/repos","events_url":"https://api.github.com/users/fmfn/events{/privacy}","received_events_url":"https://api.github.com/users/fmfn/received_events","type":"User","site_admin":false},"created_at":"2016-02-19T17:07:17Z","updated_at":"2016-02-19T17:07:17Z","body":"Ahh, I've seen this (and promptly ignored it) in the past. I believe it is a combination of UCB's obsession with edges, the bounded optimization of the acquisition function that goes slightly over the specified bounds and some rounding errors.\r\n\r\nIf you change your acquisition function to Expected Improvement, you should be ok:\r\n    \r\n    xgbBO.maximize(init_points=2, n_iter=10, acq='ei')\r\n\r\nHowever I will investigate this further, I've been willing to fix this for a while now, and I guess now is the time.\r\n\r\nThanks for raising the issue."}}