{"action":"reopened","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/3397","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/3397/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/3397/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/3397/events","html_url":"https://github.com/dmlc/xgboost/issues/3397","id":333997754,"node_id":"MDU6SXNzdWUzMzM5OTc3NTQ=","number":3397,"title":"xgboost does a premature early stop when using a correct ndcg@xx- metric","user":{"login":"kretes","id":1224887,"node_id":"MDQ6VXNlcjEyMjQ4ODc=","avatar_url":"https://avatars0.githubusercontent.com/u/1224887?v=4","gravatar_id":"","url":"https://api.github.com/users/kretes","html_url":"https://github.com/kretes","followers_url":"https://api.github.com/users/kretes/followers","following_url":"https://api.github.com/users/kretes/following{/other_user}","gists_url":"https://api.github.com/users/kretes/gists{/gist_id}","starred_url":"https://api.github.com/users/kretes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kretes/subscriptions","organizations_url":"https://api.github.com/users/kretes/orgs","repos_url":"https://api.github.com/users/kretes/repos","events_url":"https://api.github.com/users/kretes/events{/privacy}","received_events_url":"https://api.github.com/users/kretes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-20T09:34:07Z","updated_at":"2018-07-04T19:15:18Z","closed_at":null,"author_association":"NONE","body":"When specifying a metric ndcg@60- in evals as last metric xgboost uses it for early stopping however it stops after specified rounds nevermind training progress.\r\n\r\noutput:\r\n```\r\n[0]\ttest-ndcg@60-:0.5\tvalidation-ndcg@60-:0.5\ttrain-ndcg@60-:0.5\tsomedataset-ndcg@60-:0.5\r\nMultiple eval metrics have been passed: 'somedataset-ndcg@60-' will be used for early stopping.\r\n\r\nWill train until somedataset-ndcg@60- hasn't improved in 3 rounds.\r\n[1]\ttest-ndcg@60-:0.5\tvalidation-ndcg@60-:0.5\ttrain-ndcg@60-:0.5\tsomedataset-ndcg@60-:0.5\r\n[2]\ttest-ndcg@60-:0.63093\tvalidation-ndcg@60-:0.5\ttrain-ndcg@60-:0.5\tsomedataset-ndcg@60-:0.63093\r\n[3]\ttest-ndcg@60-:0.63093\tvalidation-ndcg@60-:0.5\ttrain-ndcg@60-:0.5\tsomedataset-ndcg@60-:0.63093\r\nStopping. Best iteration:\r\n[0]\ttest-ndcg@60-:0.5\tvalidation-ndcg@60-:0.5\ttrain-ndcg@60-:0.5\tsomedataset-ndcg@60-:0.5\r\n```\r\nAs you can see it uses 'somedataset-ndcg@60-' as early stopping metric, but it stops too early saying that iteration [0] was the best athough from iteration [2] the metric has gone up from 0.5 to 0.63.\r\n\r\nEverything works as expected with  'ndcg@60' as a metric but the one with '-' is more correct, see http://xgboost.readthedocs.io/en/latest//parameter.html#learning-task-parameters\r\n\r\nIt works as expected when I explicitly add maximize=True to xgb.train() method.\r\n\r\nI guess the bug comes from the fact that here xgboost is making a split on '-' and it doesn't fit the corrected metric name:\r\nhttps://github.com/dmlc/xgboost/blob/master/python-package/xgboost/callback.py#L195\r\n\r\nour params:\r\n```\r\n{\r\n     \"max_depth\": 6,\r\n     \"tree_method\": \"hist\",\r\n     \"subsample\": 0.5,\r\n     \"colsample_bytree\": 1,\r\n     \"objective\": \"rank:pairwise\",\r\n     \"eval_metric\": [\"ndcg@60-\"],\r\n     \"silent\": 0,\r\n     \"seed\": 42,\r\n     \"booster\": \"gbtree\"\r\n    }\r\n```\r\n\r\n\r\nFor bugs or installation issues, please provide the following information.\r\nThe more information you provide, the more easily we will be able to offer\r\nhelp and advice.\r\n\r\n## Environment info\r\nOperating System:\r\nlinux\r\nCompiler:\r\n\r\nPackage used (python/R/jvm/C++):\r\npython\r\n`xgboost` version used:\r\n0.72\r\n\r\nIf you are using python package, please provide\r\n\r\n1. The python version and distribution: 3.5.2\r\n2. The command to install `xgboost` if you are not installing from source\r\npip\r\n"}}
{"actor":{"display_login":"tqchen"}}