{"action":"closed","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12470","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12470/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12470/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12470/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/12470","id":374622292,"node_id":"MDU6SXNzdWUzNzQ2MjIyOTI=","number":12470,"title":"OneHotEncoder ignore unknown error when categories are strings ","user":{"login":"GMarzinotto","id":5233985,"node_id":"MDQ6VXNlcjUyMzM5ODU=","avatar_url":"https://avatars0.githubusercontent.com/u/5233985?v=4","gravatar_id":"","url":"https://api.github.com/users/GMarzinotto","html_url":"https://github.com/GMarzinotto","followers_url":"https://api.github.com/users/GMarzinotto/followers","following_url":"https://api.github.com/users/GMarzinotto/following{/other_user}","gists_url":"https://api.github.com/users/GMarzinotto/gists{/gist_id}","starred_url":"https://api.github.com/users/GMarzinotto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GMarzinotto/subscriptions","organizations_url":"https://api.github.com/users/GMarzinotto/orgs","repos_url":"https://api.github.com/users/GMarzinotto/repos","events_url":"https://api.github.com/users/GMarzinotto/events{/privacy}","received_events_url":"https://api.github.com/users/GMarzinotto/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-27T08:38:53Z","updated_at":"2018-11-18T19:59:08Z","closed_at":"2018-11-18T19:59:08Z","author_association":"NONE","body":"#### Description\r\n\r\nThis bug is very specific, but it happens when you set OneHotEncoder to ignore unknown entries.\r\nand your labels are strings. The memory of the arrays is not handled safely and it can lead to a ValueError\r\n\r\nBasically, when you call the transform method it will sets all the unknown strings on your array to OneHotEncoder.categories_[i][0] which is the first category alphabetically sorted given for fit\r\nIf this OneHotEncoder.categories_[i][0] is a long string, and the array that you want to transform has small strings, then it is impossible to fit the whole  OneHotEncoder.categories_[i][0] into the entries of the array we want to transform. So  OneHotEncoder.categories_[i][0]  is truncated and this raise the ValueError.\r\n\r\n\r\n\r\n#### Steps/Code to Reproduce\r\n```\r\n\r\nimport numpy as np\r\nfrom sklearn.preprocessing import OneHotEncoder\r\n\r\n\r\n# It needs to be numpy arrays, the error does not appear \r\n# is you have lists of lists because it gets treated like an array of objects.\r\ntrain  = np.array([ '22','333','4444','11111111' ]).reshape((-1,1))\r\ntest   = np.array([ '55555',  '22' ]).reshape((-1,1))\r\n\r\nohe = OneHotEncoder(dtype=bool,handle_unknown='ignore')\r\n\r\nohe.fit( train )\r\nenc_test = ohe.transform( test )\r\n\r\n```\r\n\r\n\r\n#### Expected Results\r\nHere we should get an sparse matrix 2x4 false everywhere except at (1,1) the '22' that is known\r\n\r\n#### Actual Results\r\n\r\n> ValueError: y contains previously unseen labels: ['111111']\r\n\r\n\r\n#### Versions\r\nSystem:\r\n    python: 2.7.12 (default, Dec  4 2017, 14:50:18)  [GCC 5.4.0 20160609]\r\n   machine: Linux-4.4.0-138-generic-x86_64-with-Ubuntu-16.04-xenial\r\nexecutable: /usr/bin/python\r\n\r\nBLAS:\r\n    macros: HAVE_CBLAS=None\r\ncblas_libs: openblas, openblas\r\n  lib_dirs: /usr/lib\r\n\r\nPython deps:\r\n    Cython: 0.25.2\r\n     scipy: 0.18.1\r\nsetuptools: 36.7.0\r\n       pip: 9.0.1\r\n     numpy: 1.15.2\r\n    pandas: 0.19.1\r\n   sklearn: 0.21.dev0\r\n\r\n\r\n\r\n#### Comments\r\n\r\nI already implemented a fix for this issue, where I check the size of the elements in the array before, and I cast them into objects if necessary."}}
{"actor":{"display_login":"amueller"}}