{"action":"created","issue":{"url":"https://api.github.com/repos/diasks2/pragmatic_tokenizer/issues/17","labels_url":"https://api.github.com/repos/diasks2/pragmatic_tokenizer/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/diasks2/pragmatic_tokenizer/issues/17/comments","events_url":"https://api.github.com/repos/diasks2/pragmatic_tokenizer/issues/17/events","html_url":"https://github.com/diasks2/pragmatic_tokenizer/issues/17","id":127553856,"number":17,"title":"additional specs","user":{"login":"maia","id":9836,"avatar_url":"https://avatars.githubusercontent.com/u/9836?v=3","gravatar_id":"","url":"https://api.github.com/users/maia","html_url":"https://github.com/maia","followers_url":"https://api.github.com/users/maia/followers","following_url":"https://api.github.com/users/maia/following{/other_user}","gists_url":"https://api.github.com/users/maia/gists{/gist_id}","starred_url":"https://api.github.com/users/maia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maia/subscriptions","organizations_url":"https://api.github.com/users/maia/orgs","repos_url":"https://api.github.com/users/maia/repos","events_url":"https://api.github.com/users/maia/events{/privacy}","received_events_url":"https://api.github.com/users/maia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-19T22:28:23Z","updated_at":"2016-01-20T01:22:09Z","closed_at":null,"body":"Here are 14 additional specs that currently do not pass. Please let me know which of these should be moved into separate issues (for further discussion, taking care of them at a later point,…). Thanks!\r\n````ruby\r\n#1\r\nit 'cleans words with symbols' do\r\n  text = \"something.com:article title !!wow look!!1\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, clean: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"something.com\", \"article\", \"title\", \"wow\", \"look\"]\r\n  )\r\nend\r\n\r\n#2\r\nit 'knows what isnt an email address' do\r\n  text = \"the great cook.@someone something else@whoever\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, remove_emails: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"the\", \"great\", \"cook.@someone\", \"something\", \"else@whoever\"]\r\n  )\r\nend\r\n\r\n#3\r\n# this could be tricky. maybe pass the option remove_domains a whitelisting array of tld's like: [\".com\", \".net\", \".org\", \".co.uk\"] and only convert these?\r\n# how does pragmatic_segmenter differenciate betweed the end of a sentence and a domain?\r\nit 'knows what isnt a domain 1' do\r\n  text = \"this is a sentence.and no domain.\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, remove_domains: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"this\", \"is\", \"a\", \"sentence\", \".\", \"and\", \"no\", \"domain\", \".\"]\r\n  )\r\nend\r\n\r\n#4\r\n# even without knowing which TLDs exist, this can never be a valid url as (I believe) no domain may only consist of a single letter\r\nit 'knows what isnt a domain 2' do\r\n  text = \"former president g.w.bush was...\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, remove_domains: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"former\", \"president\", \"g.w.bush\", \"was\", \"...\"]\r\n  )\r\nend\r\n\r\n#5\r\nit 'knows what isnt a domain 3' do\r\n  text = \"2.something-times\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, remove_domains: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"2.something-times\"]\r\n  )\r\nend\r\n\r\n#6\r\nit 'identifies hashtags with numbers too' do\r\n  text = \"this is a sentence.#yay this too.#withnumbers123\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text)\r\n  expect(pt.tokenize).to eq(\r\n    [\"this\", \"is\", \"a\", \"sentence\", \".\", \"#yay\", \"this\", \"too\", \".\", \"#withnumbers123\"]\r\n  )\r\nend\r\n\r\n#7\r\n# alternatively it could split at the ampersand, but at least return something\r\nit 'deals with html escaped entities when cleaning' do\r\n  text = \"you&amp;me\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, clean: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"you&me\"]\r\n  )\r\nend\r\n\r\n#8\r\n# in my understanding :semi should only keep tokens including at least one letter, but when keeping the current logic a similar behavior could be implemented using minimum_length:\r\nit 'requires presence of at least minimum_length letters per token' do\r\n  text = \"conference schedule:14.2. day one 12-01-16 26.12.2015 24°C 355€.\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, numbers: :semi, minimum_length: 4)\r\n  expect(pt.tokenize).to eq(\r\n    [\"conference\", \"schedule\"]\r\n  )\r\nend\r\n\r\n#9\r\nit 'splits at a comma' do\r\n  text = \"16.1. day one,17.2. day two\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text)\r\n  expect(pt.tokenize).to eq(\r\n    [\"16.1\", \".\", \"day\", \"one\", \",\", \"17.2\", \".\", \"day\", \"two\"]\r\n  )\r\nend\r\n\r\n#10\r\nit 'splits at hashtags' do\r\n  text = \"some sentence#RT ... i like u2.#bono\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, punctuation: :none)\r\n  expect(pt.tokenize).to eq(\r\n    [\"some\", \"sentence\", \"#rt\", \"i\", \"like\", \"u2\", \"#bono\"]\r\n  )\r\nend\r\n\r\n#11\r\nit 'optimizes the classic filter' do\r\n  text = \"there ́s something\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, classic_filter: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"there\", \"something\"]\r\n  )\r\nend\r\n\r\n#12\r\nit 'splits emojis' do\r\n  text = \"🤔🙄\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text, classic_filter: true)\r\n  expect(pt.tokenize).to eq(\r\n    [\"🤔\", \"🙄\"]\r\n  )\r\nend\r\n\r\n#13\r\n# alternatively :clean=>true should clean the ':' and not return ':the'\r\nit 'identifies prefixed symbols' do\r\n  text = \"look:the sky is blue\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text)\r\n  expect(pt.tokenize).to eq(\r\n    [\"look\", \":\", \"the\", \"sky\", \"is\", \"blue\"]\r\n  )\r\nend\r\n\r\n#14\r\nit 'identifies single quotes' do\r\n  text = \"Sean Penn Sat for Secret Interview With ‘El Chapo,’ Mexican Drug\"\r\n  pt = PragmaticTokenizer::Tokenizer.new(text)\r\n  expect(pt.tokenize).to eq(\r\n    [\"sean\", \"penn\", \"sat\", \"for\", \"secret\", \"interview\", \"with\", \"‘\", \"el\", \"chapo\", \",\", \"’\", \"mexican\", \"drug\"]\r\n  )\r\nend\r\n\r\n````"},"comment":{"url":"https://api.github.com/repos/diasks2/pragmatic_tokenizer/issues/comments/173048756","html_url":"https://github.com/diasks2/pragmatic_tokenizer/issues/17#issuecomment-173048756","issue_url":"https://api.github.com/repos/diasks2/pragmatic_tokenizer/issues/17","id":173048756,"user":{"login":"diasks2","id":1476952,"avatar_url":"https://avatars.githubusercontent.com/u/1476952?v=3","gravatar_id":"","url":"https://api.github.com/users/diasks2","html_url":"https://github.com/diasks2","followers_url":"https://api.github.com/users/diasks2/followers","following_url":"https://api.github.com/users/diasks2/following{/other_user}","gists_url":"https://api.github.com/users/diasks2/gists{/gist_id}","starred_url":"https://api.github.com/users/diasks2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diasks2/subscriptions","organizations_url":"https://api.github.com/users/diasks2/orgs","repos_url":"https://api.github.com/users/diasks2/repos","events_url":"https://api.github.com/users/diasks2/events{/privacy}","received_events_url":"https://api.github.com/users/diasks2/received_events","type":"User","site_admin":false},"created_at":"2016-01-20T01:22:09Z","updated_at":"2016-01-20T01:22:09Z","body":"Can you clarify examples 11 and 12:\r\n\r\nFor 11 when I copy that spec into a text editor there is a space between `there` and the apostrophe. Is this correct?\r\n\r\nFor 12, github is just showing an unknown character and when I search it on Google I get `thinking face (U+1F914)` but it does not display on my computer. Is this one of the 1012 single-character Unicode Emoji: https://github.com/franklsf95/ruby-emoji-regex ?\r\n"}}