{"action":"created","issue":{"url":"https://api.github.com/repos/HazyResearch/snorkel/issues/180","repository_url":"https://api.github.com/repos/HazyResearch/snorkel","labels_url":"https://api.github.com/repos/HazyResearch/snorkel/issues/180/labels{/name}","comments_url":"https://api.github.com/repos/HazyResearch/snorkel/issues/180/comments","events_url":"https://api.github.com/repos/HazyResearch/snorkel/issues/180/events","html_url":"https://github.com/HazyResearch/snorkel/issues/180","id":155338363,"number":180,"title":"CoreNLP adding non-breaking characters?","user":{"login":"jason-fries","id":348682,"avatar_url":"https://avatars.githubusercontent.com/u/348682?v=3","gravatar_id":"","url":"https://api.github.com/users/jason-fries","html_url":"https://github.com/jason-fries","followers_url":"https://api.github.com/users/jason-fries/followers","following_url":"https://api.github.com/users/jason-fries/following{/other_user}","gists_url":"https://api.github.com/users/jason-fries/gists{/gist_id}","starred_url":"https://api.github.com/users/jason-fries/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jason-fries/subscriptions","organizations_url":"https://api.github.com/users/jason-fries/orgs","repos_url":"https://api.github.com/users/jason-fries/repos","events_url":"https://api.github.com/users/jason-fries/events{/privacy}","received_events_url":"https://api.github.com/users/jason-fries/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/HazyResearch/snorkel/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-05-17T19:12:14Z","updated_at":"2016-07-19T19:07:25Z","closed_at":null,"body":"This is a weird issue. When parsing a large collection of PubMed abstracts, for some reason the string (or similar) `http://dx.doi.org/10.7554/eLife.00802.001.` is parsed and appended with a sequence of non-breaking characters, i.e., `http://dx.doi.org/10.7554/eLife.00802.001.\\x00\\x00\\x00\\x00`\r\n\r\nThis causes `corenlp_to_xmltree_sub` to bail out due to some lxml issues\r\n`ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters`\r\n\r\nSee the attached text for an example error case. This happens in a couple places (usually with URLs). The characters aren't in the original string. Any ideas?\r\n\r\n[malformed.txt](https://github.com/HazyResearch/ddlite/files/268968/malformed.txt)\r\n\r\nHere's the parsed sentence object:\r\n\r\n`Sentence(words=[u'DOI', u':', u'http://dx.doi.org/10.7554/eLife.00802.001.\\x00\\x00\\x00\\x00'], lemmas=[u'DOI', u':', u'http://dx.doi.org/10.7554/elife.00802.001.\\x00\\x00\\x00\\x00'], poses=[u'NNP', u':', u'NN'], dep_parents=[0, 1, 1], dep_labels=[u'ROOT', u'punct', u'dep'], sent_id=8, doc_id='23878724', text=u'DOI:http://dx.doi.org/10.7554/eLife.00802.001.', token_idxs=[1081, 1084, 1085])`\r\n\r\n(The hackfix is to change SentenceParser to strip these characters out, but I don't know why they are there to begin with.)"},"comment":{"url":"https://api.github.com/repos/HazyResearch/snorkel/issues/comments/233734348","html_url":"https://github.com/HazyResearch/snorkel/issues/180#issuecomment-233734348","issue_url":"https://api.github.com/repos/HazyResearch/snorkel/issues/180","id":233734348,"user":{"login":"alekdimi","id":6849058,"avatar_url":"https://avatars.githubusercontent.com/u/6849058?v=3","gravatar_id":"","url":"https://api.github.com/users/alekdimi","html_url":"https://github.com/alekdimi","followers_url":"https://api.github.com/users/alekdimi/followers","following_url":"https://api.github.com/users/alekdimi/following{/other_user}","gists_url":"https://api.github.com/users/alekdimi/gists{/gist_id}","starred_url":"https://api.github.com/users/alekdimi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alekdimi/subscriptions","organizations_url":"https://api.github.com/users/alekdimi/orgs","repos_url":"https://api.github.com/users/alekdimi/repos","events_url":"https://api.github.com/users/alekdimi/events{/privacy}","received_events_url":"https://api.github.com/users/alekdimi/received_events","type":"User","site_admin":false},"created_at":"2016-07-19T19:07:25Z","updated_at":"2016-07-19T19:07:25Z","body":"I'm having the same problem when analyzing the pubmed abstracts dataset. I couldn't pinpoint the exact character that caused it, but I \"fixed\" it by converting everything to ASCII (Unicode did not help). \r\n\r\nHas there been a real fix to this?"}}
{"actor":{"display_login":"alekdimi"}}