{"action":"closed","issue":{"url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1483","repository_url":"https://api.github.com/repos/Microsoft/LightGBM","labels_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1483/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1483/comments","events_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1483/events","html_url":"https://github.com/Microsoft/LightGBM/issues/1483","id":337200078,"node_id":"MDU6SXNzdWUzMzcyMDAwNzg=","number":1483,"title":"feval in lgb.cv giving unexpected results","user":{"login":"goswami-rahul","id":30523092,"node_id":"MDQ6VXNlcjMwNTIzMDky","avatar_url":"https://avatars1.githubusercontent.com/u/30523092?v=4","gravatar_id":"","url":"https://api.github.com/users/goswami-rahul","html_url":"https://github.com/goswami-rahul","followers_url":"https://api.github.com/users/goswami-rahul/followers","following_url":"https://api.github.com/users/goswami-rahul/following{/other_user}","gists_url":"https://api.github.com/users/goswami-rahul/gists{/gist_id}","starred_url":"https://api.github.com/users/goswami-rahul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goswami-rahul/subscriptions","organizations_url":"https://api.github.com/users/goswami-rahul/orgs","repos_url":"https://api.github.com/users/goswami-rahul/repos","events_url":"https://api.github.com/users/goswami-rahul/events{/privacy}","received_events_url":"https://api.github.com/users/goswami-rahul/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-06-30T09:06:18Z","updated_at":"2018-07-02T08:46:35Z","closed_at":"2018-07-02T08:46:35Z","author_association":"NONE","body":"For bugs and unexpected issues, please provide following information, so that we could reproduce on our system.\r\n\r\n## Environment info\r\nOperating System: Ubuntu 16.04 LTS\r\nCPU: Using Google Colab GPU runtime\r\nC++/Python/R version: Python 3\r\n\r\n## Error Message:\r\nNo error\r\n## Reproducible examples\r\nI used `lgb.cv` as: \r\n```python\r\ncvresult = lgb.cv(alg.get_params(), lgbtrain, num_boost_round=3000, nfold=5,\r\n                 early_stopping_rounds=80, verbose_eval=True, seed=42, metrics=\"multi_logloss\",\r\n                 feval=f1_scorer)\r\n```\r\nI have defined my `f1_scorer` (passed as `feval` to `lgv.cv`) function as:\r\n```python\r\ndef f1_scorer(y_pred, y):\r\n    y = y.get_label().astype(\"int\")\r\n    y_pred = y_pred.reshape((-1, 5)).argmax(axis=1) \r\n    return \"F1_scorer\", metrics.f1_score(y, y_pred, average=\"weighted\"), True\r\n```\r\nI reshaped and argmaxed `y_pred` because I guess `y_pred` were probabilties predicted on cv.  \r\nI have 5 classes in my multi-class model, and `y_pred`'s shape was 5 times of `y` (true labels).  \r\nFor e.g., if I have 10 examples, then   \r\n`y.shape` is `(10,)`. I expected `y_pred.shape` to be `(10, 5)`, but it was `(50,)`. (I know it because `metrics.f1_score` threw error displaying the mismatched shapes of these).  \r\nSo, I thought reshaping it to `(-1, 5)` would do the trick.  \r\nBut it seems there was some mistake by me defining `f1_scorer`, because the `f1_score` doesn't increase, while there is a substantial decrease in `multi_logloss`. Here are my outputs:\r\n```\r\n[1]\tcv_agg's multi_logloss: 1.48182 + 0.000107528\tcv_agg's F1_scorer: 0.208046 + 0.00171022\r\n[2]\tcv_agg's multi_logloss: 1.37942 + 0.000209271\tcv_agg's F1_scorer: 0.20873 + 0.0017343\r\n[3]\tcv_agg's multi_logloss: 1.30399 + 0.000401368\tcv_agg's F1_scorer: 0.209169 + 0.00158037\r\n[4]\tcv_agg's multi_logloss: 1.23172 + 0.00047433\tcv_agg's F1_scorer: 0.209576 + 0.00178056\r\n[5]\tcv_agg's multi_logloss: 1.16928 + 0.000577606\tcv_agg's F1_scorer: 0.209329 + 0.00187392\r\n[6]\tcv_agg's multi_logloss: 1.11477 + 0.000623601\tcv_agg's F1_scorer: 0.209316 + 0.001725\r\n[7]\tcv_agg's multi_logloss: 1.06698 + 0.000639912\tcv_agg's F1_scorer: 0.209314 + 0.00166868\r\n[8]\tcv_agg's multi_logloss: 1.0246 + 0.000678841\tcv_agg's F1_scorer: 0.209319 + 0.0018861\r\n.\r\n. # skipped some outputs\r\n.\r\n[150]\tcv_agg's multi_logloss: 0.615405 + 0.00142301\tcv_agg's F1_scorer: 0.209562 + 0.00165159\r\n[151]\tcv_agg's multi_logloss: 0.615341 + 0.00142724\tcv_agg's F1_scorer: 0.209498 + 0.0015317\r\n[152]\tcv_agg's multi_logloss: 0.615274 + 0.0014286\tcv_agg's F1_scorer: 0.209505 + 0.00161461\r\n[153]\tcv_agg's multi_logloss: 0.615205 + 0.00143131\tcv_agg's F1_scorer: 0.209524 + 0.0016036\r\n[154]\tcv_agg's multi_logloss: 0.61514 + 0.00143731\tcv_agg's F1_scorer: 0.20951 + 0.00160288\r\n[155]\tcv_agg's multi_logloss: 0.615072 + 0.00143254\tcv_agg's F1_scorer: 0.209491 + 0.00158067\r\n```\r\nSee, the `multi_logloss` came down from `1.5` to `0.6`, but the `f1_score` is constant.  \r\n`cv` was eventually stopped due `early_stopping` on `f1_scorer`.  \r\nWhat am I doing wrong?  (I suspect it's from the part where I reshaped `y_pred` in `f1_scorer`, but not sure)"}}
{"actor":{"display_login":"goswami-rahul"}}