{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/issues/101","repository_url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding","labels_url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/issues/101/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/issues/101/comments","events_url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/issues/101/events","html_url":"https://github.com/scikit-learn-contrib/categorical-encoding/issues/101","id":349065518,"node_id":"MDU6SXNzdWUzNDkwNjU1MTg=","number":101,"title":"drop_invariant on string columns","user":{"login":"janmotl","id":4568142,"node_id":"MDQ6VXNlcjQ1NjgxNDI=","avatar_url":"https://avatars0.githubusercontent.com/u/4568142?v=4","gravatar_id":"","url":"https://api.github.com/users/janmotl","html_url":"https://github.com/janmotl","followers_url":"https://api.github.com/users/janmotl/followers","following_url":"https://api.github.com/users/janmotl/following{/other_user}","gists_url":"https://api.github.com/users/janmotl/gists{/gist_id}","starred_url":"https://api.github.com/users/janmotl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janmotl/subscriptions","organizations_url":"https://api.github.com/users/janmotl/orgs","repos_url":"https://api.github.com/users/janmotl/repos","events_url":"https://api.github.com/users/janmotl/events{/privacy}","received_events_url":"https://api.github.com/users/janmotl/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/milestones/3","html_url":"https://github.com/scikit-learn-contrib/categorical-encoding/milestone/3","labels_url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/milestones/3/labels","id":3615893,"node_id":"MDk6TWlsZXN0b25lMzYxNTg5Mw==","number":3,"title":"v1.3.0","description":null,"creator":{"login":"wdm0006","id":1313935,"node_id":"MDQ6VXNlcjEzMTM5MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/1313935?v=4","gravatar_id":"","url":"https://api.github.com/users/wdm0006","html_url":"https://github.com/wdm0006","followers_url":"https://api.github.com/users/wdm0006/followers","following_url":"https://api.github.com/users/wdm0006/following{/other_user}","gists_url":"https://api.github.com/users/wdm0006/gists{/gist_id}","starred_url":"https://api.github.com/users/wdm0006/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wdm0006/subscriptions","organizations_url":"https://api.github.com/users/wdm0006/orgs","repos_url":"https://api.github.com/users/wdm0006/repos","events_url":"https://api.github.com/users/wdm0006/events{/privacy}","received_events_url":"https://api.github.com/users/wdm0006/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":4,"state":"open","created_at":"2018-09-02T14:26:08Z","updated_at":"2018-09-03T18:42:22Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-08-09T10:16:42Z","updated_at":"2018-09-03T18:42:22Z","closed_at":"2018-09-03T18:42:22Z","author_association":"COLLABORATOR","body":"The following code will fail if one of the columns is non-numeric (e.g.: because only some of the string columns were converted):\r\n```\r\n# drop all output columns with 0 variance.\r\n        if self.drop_invariant:\r\n            self.drop_cols = []\r\n            X_temp = self.transform(X)\r\n            self.drop_cols = [x for x in X_temp.columns.values if X_temp[x].var() <= 10e-5]\r\n```\r\n\r\nI see multiple options:\r\n1. Measure variance of only numeric columns\r\n2. Measure variance of only generated columns \r\n3. Measure the length of the `unique()` result\r\n4. Write a function, which compares `X[attribute, i] == X[attribute, i-1]` and returns on the first difference\r\n\r\nDiscussion:\r\n1. It is necessary to decide whether we want to touch columns that are not transformed (either because they are not in the `col` parameter or because of their data type). I am inclined to not touch them. Reasoning: It would make unit testing with `self.verify_inverse_transform(X_t, enc.inverse_transform(enc.transform(X_t)))` easier. Either way, the documentation should mention whether drop_invariant applies only on generated columns or all columns (even if they contain monstrous or arguably uncomparable objects).\r\n2. Calculation of variance just to learn whether the content of a column is invariant is possibly an overkill. I am not sure whether a loop with a possibility of an early return or a vectorized comparison `col[1:]==col[:-1]` plus handling of the edge cases would be faster."},"comment":{"url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/issues/comments/418175666","html_url":"https://github.com/scikit-learn-contrib/categorical-encoding/issues/101#issuecomment-418175666","issue_url":"https://api.github.com/repos/scikit-learn-contrib/categorical-encoding/issues/101","id":418175666,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODE3NTY2Ng==","user":{"login":"janmotl","id":4568142,"node_id":"MDQ6VXNlcjQ1NjgxNDI=","avatar_url":"https://avatars0.githubusercontent.com/u/4568142?v=4","gravatar_id":"","url":"https://api.github.com/users/janmotl","html_url":"https://github.com/janmotl","followers_url":"https://api.github.com/users/janmotl/followers","following_url":"https://api.github.com/users/janmotl/following{/other_user}","gists_url":"https://api.github.com/users/janmotl/gists{/gist_id}","starred_url":"https://api.github.com/users/janmotl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janmotl/subscriptions","organizations_url":"https://api.github.com/users/janmotl/orgs","repos_url":"https://api.github.com/users/janmotl/repos","events_url":"https://api.github.com/users/janmotl/events{/privacy}","received_events_url":"https://api.github.com/users/janmotl/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T18:42:22Z","updated_at":"2018-09-03T18:42:22Z","author_association":"COLLABORATOR","body":"Solved with #110"}}
{"actor":{"display_login":"janmotl"}}