{"action":"created","issue":{"url":"https://api.github.com/repos/TALP-UPC/FreeLing/issues/26","repository_url":"https://api.github.com/repos/TALP-UPC/FreeLing","labels_url":"https://api.github.com/repos/TALP-UPC/FreeLing/issues/26/labels{/name}","comments_url":"https://api.github.com/repos/TALP-UPC/FreeLing/issues/26/comments","events_url":"https://api.github.com/repos/TALP-UPC/FreeLing/issues/26/events","html_url":"https://github.com/TALP-UPC/FreeLing/issues/26","id":158963738,"number":26,"title":"Tokenizer fails if sentence ends in a space","user":{"login":"granteous","id":2024205,"avatar_url":"https://avatars.githubusercontent.com/u/2024205?v=3","gravatar_id":"","url":"https://api.github.com/users/granteous","html_url":"https://github.com/granteous","followers_url":"https://api.github.com/users/granteous/followers","following_url":"https://api.github.com/users/granteous/following{/other_user}","gists_url":"https://api.github.com/users/granteous/gists{/gist_id}","starred_url":"https://api.github.com/users/granteous/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/granteous/subscriptions","organizations_url":"https://api.github.com/users/granteous/orgs","repos_url":"https://api.github.com/users/granteous/repos","events_url":"https://api.github.com/users/granteous/events{/privacy}","received_events_url":"https://api.github.com/users/granteous/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-07T16:11:14Z","updated_at":"2016-06-07T18:40:57Z","closed_at":"2016-06-07T18:40:57Z","body":"I have a database with sentences that come from a variety of sources so the structure of the sentences vary and I can't control what each sentence begins and ends with. \r\n\r\nIn tokenizer.cc at line ~162:\r\n```c++\r\n  while (c!=p.end()) {\r\n      // find first non-white space and erase leading whitespaces\r\n      while (iswspace(*c)) {\r\n        ++c;\r\n        ++offset;\r\n      }\r\n```\r\n\r\nI'm getting an assertion that the iterator is not dereferencable. The cause seems to be that it's reaching the end of the sentence but the very end of the sentence has a space. The interim solution now is to reach into the database in my testing environment and manually erase the white space at the end of each sentence that it fails on but in production I won't be able to do this.\r\n\r\n\r\n![error1](https://cloud.githubusercontent.com/assets/2024205/15865075/fcc12910-2c9e-11e6-84b3-710d2a7a89ae.png)\r\n![screen shot 2016-06-07 at 11 06 20 am](https://cloud.githubusercontent.com/assets/2024205/15865369/16b9ce66-2ca0-11e6-91f0-c0b1821c31f0.png)\r\n\r\ntokenizer.cc\r\n```c++\r\n  ///////////////////////////////////////////////////////////////\r\n  /// Split the string into tokens using RegExps from\r\n  /// configuration file, returning a word object list.\r\n  ///////////////////////////////////////////////////////////////\r\n\r\n  void tokenizer::tokenize(const std::wstring &p, unsigned long &offset, list<word> &v) const \r\n  {\r\n    wstring t[10];\r\n    list<pair<wstring, freeling::regexp> >::const_iterator i;\r\n    bool match;\r\n    int j, substr, len=0;\r\n    vector<wstring> results;  // to store match results\r\n\r\n    v.clear(); \r\n    // Loop until line is completely processed. \r\n    wstring::const_iterator c=p.begin();\r\n    while (c!=p.end()) {\r\n      // find first non-white space and erase leading whitespaces\r\n      while (iswspace(*c)) {\r\n        ++c;\r\n        ++offset;\r\n      }\r\n      \r\n      TRACE(4,L\"Tokenizing [\"+wstring(c,p.end())+L\"]\");\r\n      // find first matching rule\r\n      match=false;    \r\n      for (i=rules.begin(); i!=rules.end() && !match; i++) {\r\n        try {\r\n          TRACE(4,L\"  Checking rule \"+i->first);\r\n          if (i->second.search(c, p.end(), results, true)) {\r\n            // regexp matches, extract substrings\r\n            match=true; len=0;\r\n            substr = matches.find(i->first)->second;\r\n            for (j=(substr==0? 0 : 1); j<=substr && match; j++) {\r\n              // get each requested  substring\r\n              t[j] = results.at(j);\r\n              len += t[j].length();\r\n              TRACE(2,L\"Found match \"+util::int2wstring(j)+L\" [\"+t[j]+L\"] for rule \"+i->first);\r\n              // if special rule, match must be in abbrev file\r\n              if ((i->first)[0]==L'*') {\r\n                wstring lower = util::lowercase(t[j]);\r\n                if (abrevs.find(lower)==abrevs.end()) {\r\n                  match = false;\r\n                  TRACE(2,L\"Special rule and found match not in abbrev list. Rule not satisfied\");\r\n                }\r\n              }\r\n            }\r\n          }\r\n        }\r\n        catch (...) {\r\n          // boost::regexp rejects to match an expression if the matched string is too long\r\n          WARNING(L\"Match too long for boost buffer: Rule \"+i->first+L\" skipped.\");\r\n          WARNING(L\"Provided input doesn't look like text.\");\r\n        }\r\n      }\r\n\r\n.............\r\n```"},"comment":{"url":"https://api.github.com/repos/TALP-UPC/FreeLing/issues/comments/224375093","html_url":"https://github.com/TALP-UPC/FreeLing/issues/26#issuecomment-224375093","issue_url":"https://api.github.com/repos/TALP-UPC/FreeLing/issues/26","id":224375093,"user":{"login":"granteous","id":2024205,"avatar_url":"https://avatars.githubusercontent.com/u/2024205?v=3","gravatar_id":"","url":"https://api.github.com/users/granteous","html_url":"https://github.com/granteous","followers_url":"https://api.github.com/users/granteous/followers","following_url":"https://api.github.com/users/granteous/following{/other_user}","gists_url":"https://api.github.com/users/granteous/gists{/gist_id}","starred_url":"https://api.github.com/users/granteous/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/granteous/subscriptions","organizations_url":"https://api.github.com/users/granteous/orgs","repos_url":"https://api.github.com/users/granteous/repos","events_url":"https://api.github.com/users/granteous/events{/privacy}","received_events_url":"https://api.github.com/users/granteous/received_events","type":"User","site_admin":false},"created_at":"2016-06-07T18:40:57Z","updated_at":"2016-06-07T18:40:57Z","body":"Looks good."}}
{"actor":{"display_login":"granteous"}}