{"action":"created","issue":{"url":"https://api.github.com/repos/jpmml/sklearn2pmml/issues/150","repository_url":"https://api.github.com/repos/jpmml/sklearn2pmml","labels_url":"https://api.github.com/repos/jpmml/sklearn2pmml/issues/150/labels{/name}","comments_url":"https://api.github.com/repos/jpmml/sklearn2pmml/issues/150/comments","events_url":"https://api.github.com/repos/jpmml/sklearn2pmml/issues/150/events","html_url":"https://github.com/jpmml/sklearn2pmml/issues/150","id":439559503,"node_id":"MDU6SXNzdWU0Mzk1NTk1MDM=","number":150,"title":"Not able to create PMMLs from pipelines having custom text processing functions","user":{"login":"dilipsundar","id":50202702,"node_id":"MDQ6VXNlcjUwMjAyNzAy","avatar_url":"https://avatars2.githubusercontent.com/u/50202702?v=4","gravatar_id":"","url":"https://api.github.com/users/dilipsundar","html_url":"https://github.com/dilipsundar","followers_url":"https://api.github.com/users/dilipsundar/followers","following_url":"https://api.github.com/users/dilipsundar/following{/other_user}","gists_url":"https://api.github.com/users/dilipsundar/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipsundar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipsundar/subscriptions","organizations_url":"https://api.github.com/users/dilipsundar/orgs","repos_url":"https://api.github.com/users/dilipsundar/repos","events_url":"https://api.github.com/users/dilipsundar/events{/privacy}","received_events_url":"https://api.github.com/users/dilipsundar/received_events","type":"User","site_admin":false},"labels":[{"id":284743293,"node_id":"MDU6TGFiZWwyODQ3NDMyOTM=","url":"https://api.github.com/repos/jpmml/sklearn2pmml/labels/wontfix","name":"wontfix","color":"ffffff","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2019-05-02T12:13:37Z","updated_at":"2019-05-09T11:36:57Z","closed_at":"2019-05-02T12:46:38Z","author_association":"NONE","body":"Hi @vruusmann \r\n\r\nI'm trying to clean and vectorize my text before feeding it to my model and this is what I've written to do that.\r\n``` Python\r\nclass TextPreprocessor(BaseEstimator, TransformerMixin):\r\n        \r\n    def clean_text(self, text):\r\n        text = re.sub(r'[^\\x00-\\x7F]+',' ', text)\r\n        text = contractions.fix(text)\r\n        text = text.lower()\r\n        text = text.strip()\r\n        text = re.sub(' +', ' ',text)\r\n        text = re.sub('[^A-Za-z]+', ' ', text)\r\n        text = ' '.join([w for w in text.split() if len(w)>2])\r\n        return text\r\n\r\n    def word_tag(self, nltk_tag):\r\n        if nltk_tag.startswith('J'):\r\n            return wordnet.ADJ\r\n        elif nltk_tag.startswith('V'):\r\n            return wordnet.VERB\r\n        elif nltk_tag.startswith('N'):\r\n            return wordnet.NOUN\r\n        elif nltk_tag.startswith('R'):\r\n            return wordnet.ADV\r\n        else:\r\n            return None\r\n    \r\n    def lemmatize_sentence(self, sentence):\r\n        sentence = self.clean_text(str(sentence))\r\n        nltk_tagged = nltk.pos_tag(nltk.word_tokenize(sentence))\r\n        wn_tagged = map(lambda x: (x[0], self.word_tag(x[1])), nltk_tagged)\r\n        res_words = []\r\n        for word, tag in wn_tagged:\r\n            if tag is None:\r\n                res_words.append(word)\r\n            else:\r\n                res_words.append(WordNetLemmatizer().lemmatize(word, tag))\r\n        final = ' '.join(res_words)\r\n        return final\r\n    \r\n    def fit(self, X, y=None):\r\n        return self\r\n\r\n    def transform(self, X, y=None):\r\n        if type(X)==str:\r\n            X = [X]\r\n        return [self.lemmatize_sentence(text) for text in X]\r\n\r\nlogit_pipeline = PMMLPipeline([('preproc', TextPreprocessor()), \r\n                               ('vect', CountVectorizer(ngram_range=(1,2))), \r\n                               ('tfidf', TfidfTransformer(use_idf=True)), \r\n                               ('clf', LogisticRegression(C=11.3))])\r\n\r\nlogit_pipeline.active_fields = np.asarray([\"Input Text\"])\r\nlogit_pipeline.target_fields = np.asarray([\"Output\"])\r\nlogit_pipeline.fit(X, y)\r\n```\r\n```\r\nsklearn2pmml(logit_pipeline, 'logit.pmml', user_classpath = [\"/path/sklearn2pmml-plugin/target/sklearn2pmml-plugin-1.0-SNAPSHOT.jar\"])\r\n```\r\nWhen I run the last line, I'm getting this as an exception.\r\n```\r\nStandard output is empty\r\nStandard error:\r\nMay 02, 2019 1:07:00 AM org.jpmml.sklearn.Main run\r\nINFO: Parsing PKL..\r\nMay 02, 2019 1:07:02 AM org.jpmml.sklearn.Main run\r\nINFO: Parsed PKL in 2150 ms.\r\nMay 02, 2019 1:07:02 AM org.jpmml.sklearn.Main run\r\nINFO: Converting..\r\nMay 02, 2019 1:07:03 AM org.jpmml.sklearn.Main run\r\nSEVERE: Failed to convert\r\njava.lang.IllegalArgumentException: Tuple contains an unsupported value (Python class __main__.TextPreprocessor)\r\n        at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:43)\r\n        at com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:599)\r\n        at sklearn.TransformerUtil.getHead(TransformerUtil.java:35)\r\n        at sklearn2pmml.pipeline.PMMLPipeline.encodePMML(PMMLPipeline.java:187)\r\n        at org.jpmml.sklearn.Main.run(Main.java:145)\r\n        at org.jpmml.sklearn.Main.main(Main.java:94)\r\nCaused by: java.lang.ClassCastException: Cannot cast net.razorvine.pickle.objects.ClassDict to sklearn.Transformer\r\n        at java.lang.Class.cast(Class.java:3369)\r\n        at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:41)\r\n        ... 5 more\r\n\r\nException in thread \"main\" java.lang.IllegalArgumentException: Tuple contains an unsupported value (Python class __main__.TextPreprocessor)\r\n        at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:43)\r\n        at com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:599)\r\n        at sklearn.TransformerUtil.getHead(TransformerUtil.java:35)\r\n        at sklearn2pmml.pipeline.PMMLPipeline.encodePMML(PMMLPipeline.java:187)\r\n        at org.jpmml.sklearn.Main.run(Main.java:145)\r\n        at org.jpmml.sklearn.Main.main(Main.java:94)\r\nCaused by: java.lang.ClassCastException: Cannot cast net.razorvine.pickle.objects.ClassDict to sklearn.Transformer\r\n        at java.lang.Class.cast(Class.java:3369)\r\n        at org.jpmml.sklearn.CastFunction.apply(CastFunction.java:41)\r\n        ... 5 more\r\n\r\nTraceback (most recent call last):\r\n  File \"test-pmml.py\", line 126, in <module>\r\n    sklearn2pmml(logit_pipeline, 'logit.pmml', user_classpath = [\"/path/sklearn2pmml-plugin/target/sklearn2pmml-plugin-1.0-SNAPSHOT.jar\"])\r\n  File \"/home/devuser/.local/lib/python2.7/site-packages/sklearn2pmml/__init__.py\", line 252, in sklearn2pmml\r\n    raise RuntimeError(\"The JPMML-SkLearn conversion application has failed. The Java executable should have printed more information about the failure into its standard output and/or standard error streams\")\r\nRuntimeError: The JPMML-SkLearn conversion application has failed. The Java executable should have printed more information about the failure into its standard output and/or standard error streams\r\n```\r\nCan you please help me understand what I'm doing wrong? Or does sklearn2pmml-plugin not support some of the things I'm trying to do here?\r\n"},"comment":{"url":"https://api.github.com/repos/jpmml/sklearn2pmml/issues/comments/490867515","html_url":"https://github.com/jpmml/sklearn2pmml/issues/150#issuecomment-490867515","issue_url":"https://api.github.com/repos/jpmml/sklearn2pmml/issues/150","id":490867515,"node_id":"MDEyOklzc3VlQ29tbWVudDQ5MDg2NzUxNQ==","user":{"login":"dilipsundar","id":50202702,"node_id":"MDQ6VXNlcjUwMjAyNzAy","avatar_url":"https://avatars2.githubusercontent.com/u/50202702?v=4","gravatar_id":"","url":"https://api.github.com/users/dilipsundar","html_url":"https://github.com/dilipsundar","followers_url":"https://api.github.com/users/dilipsundar/followers","following_url":"https://api.github.com/users/dilipsundar/following{/other_user}","gists_url":"https://api.github.com/users/dilipsundar/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipsundar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipsundar/subscriptions","organizations_url":"https://api.github.com/users/dilipsundar/orgs","repos_url":"https://api.github.com/users/dilipsundar/repos","events_url":"https://api.github.com/users/dilipsundar/events{/privacy}","received_events_url":"https://api.github.com/users/dilipsundar/received_events","type":"User","site_admin":false},"created_at":"2019-05-09T11:36:57Z","updated_at":"2019-05-09T11:36:57Z","author_association":"NONE","body":"\r\n\r\n\r\n> You have a one-dimensional input space, right? Just \"convert\" it temporarily from Numpy array to `pandas.DataFrame`.\r\n\r\nI'm expecting a text input i.e. str object. Is that possible to do this conversion within the pipeline itself?"}}
{"actor":{"display_login":"dilipsundar"}}