{"action":"opened","issue":{"url":"https://api.github.com/repos/marcotcr/lime/issues/236","repository_url":"https://api.github.com/repos/marcotcr/lime","labels_url":"https://api.github.com/repos/marcotcr/lime/issues/236/labels{/name}","comments_url":"https://api.github.com/repos/marcotcr/lime/issues/236/comments","events_url":"https://api.github.com/repos/marcotcr/lime/issues/236/events","html_url":"https://github.com/marcotcr/lime/issues/236","id":364938225,"node_id":"MDU6SXNzdWUzNjQ5MzgyMjU=","number":236,"title":"Potential Compatibility Issue for LimeTextExplainer with Python 3.7 (includes workaround)","user":{"login":"edkrueger","id":22670796,"node_id":"MDQ6VXNlcjIyNjcwNzk2","avatar_url":"https://avatars2.githubusercontent.com/u/22670796?v=4","gravatar_id":"","url":"https://api.github.com/users/edkrueger","html_url":"https://github.com/edkrueger","followers_url":"https://api.github.com/users/edkrueger/followers","following_url":"https://api.github.com/users/edkrueger/following{/other_user}","gists_url":"https://api.github.com/users/edkrueger/gists{/gist_id}","starred_url":"https://api.github.com/users/edkrueger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edkrueger/subscriptions","organizations_url":"https://api.github.com/users/edkrueger/orgs","repos_url":"https://api.github.com/users/edkrueger/repos","events_url":"https://api.github.com/users/edkrueger/events{/privacy}","received_events_url":"https://api.github.com/users/edkrueger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-28T15:30:10Z","updated_at":"2018-09-28T15:30:10Z","closed_at":null,"author_association":"NONE","body":"When using python=3.7.0, lime==0.1.1.32 and scikit-learn==0.20.0, I get an error when I try to use LimeTextExplainer after building a piplined sklearn model.\r\n\r\nFor example in the doc's `Lime - basic usage, two class case.ipynb`, the pipeline is made as follows:\r\n\r\n```\r\nvectorizer = sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)\r\ntrain_vectors = vectorizer.fit_transform(newsgroups_train.data)\r\ntest_vectors = vectorizer.transform(newsgroups_test.data)\r\nrf = sklearn.ensemble.RandomForestClassifier(n_estimators=500)\r\nrf.fit(train_vectors, newsgroups_train.target)\r\nc = make_pipeline(vectorizer, rf)\r\n```.\r\n\r\nWhen I run,\r\n```\r\nexplainer = LimeTextExplainer(class_names=class_names)\r\nexp = explainer.explain_instance(newsgroups_test.data[idx], c.predict_proba, num_features=6)\r\n```\r\nI get the error\r\n\r\n```\r\nTypeError: object of type 'NoneType' has no len()\r\n```.\r\n\r\nOn python=2.7.15, lime==0.1.1.32, scikit-learn==0.20.0, I don't get any error.\r\n\r\nBy looking in the source code for sklearn.feature_extraction.text.CountVectorizer, I found they were using the regex `r\"(?u)\\b\\w\\w+\\b\"` in their tokenizer and this allowed me a work around by doing\r\n\r\n```\r\ntokenizer = lambda doc: re.compile(r\"(?u)\\b\\w\\w+\\b\").findall(doc)\r\nexplainer = LimeTextExplainer(class_names=class_names, split_expression=tokenizer)\r\nexp = explainer.explain_instance(newsgroups_test.data[idx], c.predict_proba, num_features=6)\r\n```\r\n.\r\n\r\nSince its likely that many user will want to just use the default tokenizer with scikit-learn and lime.\r\nI have some ideas to fix this, but I wanted to make sure y'all found this desirable before I looked into it further and made a PR.\r\n\r\nThanks!"}}
{"actor":{"display_login":"edkrueger"}}