{"action":"opened","issue":{"url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1821","repository_url":"https://api.github.com/repos/Microsoft/LightGBM","labels_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1821/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1821/comments","events_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1821/events","html_url":"https://github.com/Microsoft/LightGBM/issues/1821","id":377354749,"node_id":"MDU6SXNzdWUzNzczNTQ3NDk=","number":1821,"title":"[R-package] Getting rid of default metric without custom loss function","user":{"login":"Laurae2","id":9083669,"node_id":"MDQ6VXNlcjkwODM2Njk=","avatar_url":"https://avatars3.githubusercontent.com/u/9083669?v=4","gravatar_id":"","url":"https://api.github.com/users/Laurae2","html_url":"https://github.com/Laurae2","followers_url":"https://api.github.com/users/Laurae2/followers","following_url":"https://api.github.com/users/Laurae2/following{/other_user}","gists_url":"https://api.github.com/users/Laurae2/gists{/gist_id}","starred_url":"https://api.github.com/users/Laurae2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Laurae2/subscriptions","organizations_url":"https://api.github.com/users/Laurae2/orgs","repos_url":"https://api.github.com/users/Laurae2/repos","events_url":"https://api.github.com/users/Laurae2/events{/privacy}","received_events_url":"https://api.github.com/users/Laurae2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-05T11:18:46Z","updated_at":"2018-11-05T11:18:46Z","closed_at":null,"author_association":"COLLABORATOR","body":"Currently, LightGBM forces to use the default objective loss in addition to the custom loss function in R. This can be seen using the following (taken and adapted from https://github.com/Microsoft/LightGBM/blob/master/R-package/demo/early_stopping.R):\r\n\r\n```r\r\nrequire(lightgbm)\r\nrequire(methods)\r\n\r\n# Load in the agaricus dataset\r\ndata(agaricus.train, package = \"lightgbm\")\r\ndata(agaricus.test, package = \"lightgbm\")\r\n\r\ndtrain <- lgb.Dataset(agaricus.train$data, label = agaricus.train$label)\r\ndtest <- lgb.Dataset(agaricus.test$data, label = agaricus.test$label)\r\n\r\nparam <- list(num_leaves = 4,\r\n              learning_rate = 1)\r\nvalids <- list(eval = dtest)\r\nnum_round <- 20\r\n\r\nevalerror <- function(preds, dtrain) {\r\n  labels <- getinfo(dtrain, \"label\")\r\n  err <- as.numeric(sum(labels != (preds > 0.5))) / length(labels)\r\n  return(list(name = \"error\", value = err, higher_better = FALSE))\r\n}\r\n\r\nbst <- lgb.train(param,\r\n                 dtrain,\r\n                 num_round,\r\n                 valids,\r\n                 objective = \"binary\",\r\n                 eval = evalerror,\r\n                 early_stopping_round = 3)\r\n```\r\n\r\nLeads to the following (unwanted binary_logloss):\r\n\r\n```\r\n[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the \"boost_from_average\" parameter in \"binary\" objective is true.\r\nThis may cause significantly different results comparing to the previous versions of LightGBM.\r\nTry to set boost_from_average=false, if your old models produce bad results\r\n[LightGBM] [Info] Number of positive: 3140, number of negative: 3373\r\n[LightGBM] [Info] Total Bins 128\r\n[LightGBM] [Info] Number of data: 6513, number of used features: 107\r\n[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580\r\n[LightGBM] [Info] Start training from score -0.071580\r\n[1]:\teval's binary_logloss:0.204754\teval's error:0.0335196 \r\n[2]:\teval's binary_logloss:0.113096\teval's error:0.0217256 \r\n[3]:\teval's binary_logloss:0.0476511\teval's error:0.00558659 \r\n[4]:\teval's binary_logloss:0.0234622\teval's error:0 \r\n[5]:\teval's binary_logloss:0.11908\teval's error:0.00372439 \r\n[6]:\teval's binary_logloss:0.0154869\teval's error:0 \r\n[7]:\teval's binary_logloss:0.00882988\teval's error:0 \r\n```\r\n\r\nA workaround is the following, but this is not obvious and is probably an unexpected case:\r\n\r\n```r\r\nbst <- lgb.train(param,\r\n                 dtrain,\r\n                 num_round,\r\n                 valids,\r\n                 objective = \"binary\",\r\n                 eval = evalerror,\r\n                 metric = 0,\r\n                 early_stopping_round = 3)\r\n```\r\n\r\n```\r\n[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the \"boost_from_average\" parameter in \"binary\" objective is true.\r\nThis may cause significantly different results comparing to the previous versions of LightGBM.\r\nTry to set boost_from_average=false, if your old models produce bad results\r\n[LightGBM] [Info] Number of positive: 3140, number of negative: 3373\r\n[LightGBM] [Info] Total Bins 128\r\n[LightGBM] [Info] Number of data: 6513, number of used features: 107\r\n[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580\r\n[LightGBM] [Info] Start training from score -0.071580\r\n[1]:\teval's error:0.0335196 \r\n[2]:\teval's error:0.0217256 \r\n[3]:\teval's error:0.00558659 \r\n[4]:\teval's error:0 \r\n[5]:\teval's error:0.00372439 \r\n[6]:\teval's error:0 \r\n[7]:\teval's error:0 \r\n```"}}
{"actor":{"display_login":"Laurae2"}}