{"action":"closed","issue":{"url":"https://api.github.com/repos/JohnSnowLabs/spark-nlp/issues/112","repository_url":"https://api.github.com/repos/JohnSnowLabs/spark-nlp","labels_url":"https://api.github.com/repos/JohnSnowLabs/spark-nlp/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/JohnSnowLabs/spark-nlp/issues/112/comments","events_url":"https://api.github.com/repos/JohnSnowLabs/spark-nlp/issues/112/events","html_url":"https://github.com/JohnSnowLabs/spark-nlp/issues/112","id":299147993,"number":112,"title":"Tokenizer: Composite Tokens not retained when targetPattern is set","user":{"login":"easimadi","id":13008169,"avatar_url":"https://avatars3.githubusercontent.com/u/13008169?v=4","gravatar_id":"","url":"https://api.github.com/users/easimadi","html_url":"https://github.com/easimadi","followers_url":"https://api.github.com/users/easimadi/followers","following_url":"https://api.github.com/users/easimadi/following{/other_user}","gists_url":"https://api.github.com/users/easimadi/gists{/gist_id}","starred_url":"https://api.github.com/users/easimadi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/easimadi/subscriptions","organizations_url":"https://api.github.com/users/easimadi/orgs","repos_url":"https://api.github.com/users/easimadi/repos","events_url":"https://api.github.com/users/easimadi/events{/privacy}","received_events_url":"https://api.github.com/users/easimadi/received_events","type":"User","site_admin":false},"labels":[{"id":701583368,"url":"https://api.github.com/repos/JohnSnowLabs/spark-nlp/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"closed","locked":false,"assignee":{"login":"saifjsl","id":27952699,"avatar_url":"https://avatars3.githubusercontent.com/u/27952699?v=4","gravatar_id":"","url":"https://api.github.com/users/saifjsl","html_url":"https://github.com/saifjsl","followers_url":"https://api.github.com/users/saifjsl/followers","following_url":"https://api.github.com/users/saifjsl/following{/other_user}","gists_url":"https://api.github.com/users/saifjsl/gists{/gist_id}","starred_url":"https://api.github.com/users/saifjsl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saifjsl/subscriptions","organizations_url":"https://api.github.com/users/saifjsl/orgs","repos_url":"https://api.github.com/users/saifjsl/repos","events_url":"https://api.github.com/users/saifjsl/events{/privacy}","received_events_url":"https://api.github.com/users/saifjsl/received_events","type":"User","site_admin":false},"assignees":[{"login":"saifjsl","id":27952699,"avatar_url":"https://avatars3.githubusercontent.com/u/27952699?v=4","gravatar_id":"","url":"https://api.github.com/users/saifjsl","html_url":"https://github.com/saifjsl","followers_url":"https://api.github.com/users/saifjsl/followers","following_url":"https://api.github.com/users/saifjsl/following{/other_user}","gists_url":"https://api.github.com/users/saifjsl/gists{/gist_id}","starred_url":"https://api.github.com/users/saifjsl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saifjsl/subscriptions","organizations_url":"https://api.github.com/users/saifjsl/orgs","repos_url":"https://api.github.com/users/saifjsl/repos","events_url":"https://api.github.com/users/saifjsl/events{/privacy}","received_events_url":"https://api.github.com/users/saifjsl/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-02-21T22:10:30Z","updated_at":"2018-02-23T17:54:46Z","closed_at":"2018-02-23T17:54:46Z","author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\nWhen I `setTargetPatter(\\\\w+)` , composite tokens are not retained\r\n## Description\r\n<!--- Provide a more detailed introduction to the issue itself, and why you consider it to be a bug -->\r\nBelow code returns 3 tokens 'New', 'York', 'City' instead of 'New York City'.\r\nHowever  When I comment out setTargetPattern(\\\\w+) the behaviour is becomes correct.\r\n```scala\r\nval df = spark.createDataFrame(Seq((1,\"New York City\"))).toDF(\"id\",\"text\")\r\n\r\nval documentAssembler = new DocumentAssembler()\r\n.setInputCol(\"text\")\r\n.setOutputCol(\"document\")\r\n.setIdCol(\"id\")\r\n\r\nval doc = documentAssembler.transform(df)\r\n\r\n\r\nval tokenizer = new Tokenizer()\r\n                    .setInputCols(\"document\")\r\n                    .setOutputCol(\"tokens\")\r\n                    .setTargetPattern(\"\\\\w+\")\r\n                    .setCompositeTokens(Array(\"New York City\"))\r\n```\r\n\r\n\r\n\r\n\r\n## Context\r\n<!--- How has this bug affected you? What were you trying to accomplish? -->\r\nI am using Spark-NLP release 1.4\r\n\r\n\r\n"}}
{"actor":{"display_login":"saifjsl"}}