{"action":"created","issue":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2349","repository_url":"https://api.github.com/repos/kaldi-asr/kaldi","labels_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2349/labels{/name}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2349/comments","events_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2349/events","html_url":"https://github.com/kaldi-asr/kaldi/issues/2349","id":313120472,"number":2349,"title":"tf_idf.py does not handle non-ASCII characters in words","user":{"login":"gooofy","id":6418859,"avatar_url":"https://avatars1.githubusercontent.com/u/6418859?v=4","gravatar_id":"","url":"https://api.github.com/users/gooofy","html_url":"https://github.com/gooofy","followers_url":"https://api.github.com/users/gooofy/followers","following_url":"https://api.github.com/users/gooofy/following{/other_user}","gists_url":"https://api.github.com/users/gooofy/gists{/gist_id}","starred_url":"https://api.github.com/users/gooofy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gooofy/subscriptions","organizations_url":"https://api.github.com/users/gooofy/orgs","repos_url":"https://api.github.com/users/gooofy/repos","events_url":"https://api.github.com/users/gooofy/events{/privacy}","received_events_url":"https://api.github.com/users/gooofy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-10T23:12:05Z","updated_at":"2018-04-11T16:00:32Z","closed_at":null,"author_association":"NONE","body":"while experimenting with automatic segmentation of german librivox recordings using run_segmentation_long_utts.sh, I came across this small issue:\r\n\r\ntf_idf.py will fail if exp/segment_long_utts_a_train/query_docs/split1/query_tf_idf.1.ark.txt contains words with non-ASCII characters because of this assert:\r\n\r\n```\r\ndiff --git a/egs/wsj/s5/steps/cleanup/internal/tf_idf.py b/egs/wsj/s5/steps/cleanup/internal/tf_idf.py\r\nindex 9b2f4d6..f6e950e 100644\r\n--- a/egs/wsj/s5/steps/cleanup/internal/tf_idf.py\r\n+++ b/egs/wsj/s5/steps/cleanup/internal/tf_idf.py\r\n@@ -402,7 +402,7 @@ def read_key(fd):\r\n     str += char\r\n   str = str.strip()\r\n   if str == '': return None # end of file,\r\n-  assert(re.match('^[\\.a-zA-Z0-9_:-]+$',str) != None) # check format,\r\n+  # assert(re.match('^[\\.a-zA-Z0-9_:-]+$',str) != None) # check format,\r\n   return str\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/comments/380505431","html_url":"https://github.com/kaldi-asr/kaldi/issues/2349#issuecomment-380505431","issue_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2349","id":380505431,"user":{"login":"Ore-an","id":18559156,"avatar_url":"https://avatars1.githubusercontent.com/u/18559156?v=4","gravatar_id":"","url":"https://api.github.com/users/Ore-an","html_url":"https://github.com/Ore-an","followers_url":"https://api.github.com/users/Ore-an/followers","following_url":"https://api.github.com/users/Ore-an/following{/other_user}","gists_url":"https://api.github.com/users/Ore-an/gists{/gist_id}","starred_url":"https://api.github.com/users/Ore-an/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ore-an/subscriptions","organizations_url":"https://api.github.com/users/Ore-an/orgs","repos_url":"https://api.github.com/users/Ore-an/repos","events_url":"https://api.github.com/users/Ore-an/events{/privacy}","received_events_url":"https://api.github.com/users/Ore-an/received_events","type":"User","site_admin":false},"created_at":"2018-04-11T16:00:32Z","updated_at":"2018-04-11T16:00:32Z","author_association":"CONTRIBUTOR","body":"That function reads a document/utterance ID, and there's no real reason to use non-ASCII for that, I think. Anyway, the re module has an Unicode flag that could be used in conjunction with extracting the set of characters used (maybe from the lexicon, from transcriptions or IDs in the wav.scp) and adding them to the regular expression."}}
{"actor":{"display_login":"Ore-an"}}