{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11871","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11871/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11871/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11871/events","html_url":"https://github.com/apache/spark/pull/11871","id":142505854,"number":11871,"title":"[SPARK-14050][ML] Add multiple languages support and additional methods for Stop Words Remover","user":{"login":"burakkose","id":5651267,"avatar_url":"https://avatars.githubusercontent.com/u/5651267?v=3","gravatar_id":"","url":"https://api.github.com/users/burakkose","html_url":"https://github.com/burakkose","followers_url":"https://api.github.com/users/burakkose/followers","following_url":"https://api.github.com/users/burakkose/following{/other_user}","gists_url":"https://api.github.com/users/burakkose/gists{/gist_id}","starred_url":"https://api.github.com/users/burakkose/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burakkose/subscriptions","organizations_url":"https://api.github.com/users/burakkose/orgs","repos_url":"https://api.github.com/users/burakkose/repos","events_url":"https://api.github.com/users/burakkose/events{/privacy}","received_events_url":"https://api.github.com/users/burakkose/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-03-22T00:02:41Z","updated_at":"2016-03-22T14:10:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11871","html_url":"https://github.com/apache/spark/pull/11871","diff_url":"https://github.com/apache/spark/pull/11871.diff","patch_url":"https://github.com/apache/spark/pull/11871.patch"},"body":"Apache Spark is a global engine, so It should appeal to everyone as much as possible. I added multiple language support for StopWordRemover, and used nltk's language list except for English(English is still same). I added some facilitator functions such as setLanguage, setAdditionalWords and setIgnoredWords. \r\n\r\nEnglish is the default. If you want to change the language, use setLanguage. For example; setLanguage(\"danish\")."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/199832509","html_url":"https://github.com/apache/spark/pull/11871#issuecomment-199832509","issue_url":"https://api.github.com/repos/apache/spark/issues/11871","id":199832509,"user":{"login":"burakkose","id":5651267,"avatar_url":"https://avatars.githubusercontent.com/u/5651267?v=3","gravatar_id":"","url":"https://api.github.com/users/burakkose","html_url":"https://github.com/burakkose","followers_url":"https://api.github.com/users/burakkose/followers","following_url":"https://api.github.com/users/burakkose/following{/other_user}","gists_url":"https://api.github.com/users/burakkose/gists{/gist_id}","starred_url":"https://api.github.com/users/burakkose/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burakkose/subscriptions","organizations_url":"https://api.github.com/users/burakkose/orgs","repos_url":"https://api.github.com/users/burakkose/repos","events_url":"https://api.github.com/users/burakkose/events{/privacy}","received_events_url":"https://api.github.com/users/burakkose/received_events","type":"User","site_admin":false},"created_at":"2016-03-22T14:10:56Z","updated_at":"2016-03-22T14:10:56Z","body":"@srowen , I got from http://www.nltk.org/nltk_data/ , Stopwords Corpus, and they mentioned that \r\n\"\r\nThey were obtained from:\r\nhttp://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/backend/snowball/stopwords/\r\nThe English list has been augmented\r\nhttps://github.com/nltk/nltk_data/issues/22\r\n\""}}