{"action":"created","comment":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments/220412379","pull_request_review_id":158814176,"id":220412379,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIyMDQxMjM3OQ==","diff_hunk":"@@ -0,0 +1,65 @@\n+#!/bin/bash\n+# Copyright 2014  Johns Hopkins University (Author: Yenda Trmal)\n+# Copyright 2016  Xiaohui Zhang\n+#           2018  Ruizhe Huang\n+# Apache 2.0\n+\n+# This script applies a trained Phonetisarus G2P model to\n+# synthesize pronunciations for missing words (i.e., words in\n+# transcripts but not the lexicon), and output the expanded lexicon.\n+# The user could specify either nbest or pmass option \n+# to determine the number of output pronunciation variants, \n+# or use them together to get the intersection of two options.\n+\n+# Begin configuration section.  \n+stage=0\n+nbest=          # Generate up to $nbest variants\n+pmass=          # Generate so many variants to produce $pmass ammount, like 90%, of the prob mass\n+# End configuration section.\n+\n+echo \"$0 $@\"  # Print the command line for logging\n+\n+[ -f ./path.sh ] && . ./path.sh; # source the path.\n+. utils/parse_options.sh || exit 1;\n+\n+set -u\n+set -e\n+\n+if [ $# != 3 ]; then\n+  echo \"Usage: $0 [options] <g2p-model> <word-list> <lexicon-out>\"\n+  echo \"... where <g2p-model> is the trained g2p model.\"\n+  echo \"          <word-list> is a list of words whose pronunciation is to be generated.\"\n+  echo \"          <lexicon-out> output lexicon, whose format is <word>\\t<prob>\\t<pronunciation> for each line.\"\n+  echo \"e.g.: $0 --nbest 1 exp/g2p/model.fst exp/g2p/oov_words.txt data/local/dict_nosp/lexicon.txt\"\n+  echo \"\"\n+  echo \"main options (for others, see top of script file)\"\n+  echo \"  --nbest <int>    # Maximum number of hypotheses to produce. By default, nbest=20\"\n+  echo \"  --pmass <float>  # Select the maximum number of hypotheses summing to a total mass of pmass amount, within [0, 1], for a word. By default, pmass=1.0\"\n+  exit 1;\n+fi\n+\n+model=$1\n+word_list=$2\n+out_lexicon=$3\n+\n+[ ! -z $nbest ] && [[ ! $nbest =~ ^[1-9][0-9]*$ ]] && echo \"$0: nbest should be a positive integer.\" && exit 1;\n+[ -z $pmass ] && [ -z $nbest ] && echo \"$0: nbest or/and pmass should be specified.\" && exit 1;\n+\n+if [ -z $pmass ]; then","path":"egs/wsj/s5/steps/dict/apply_g2p_phonetisaurus.sh","position":48,"original_position":48,"commit_id":"55227dfcc242cb49ef9cf46e6ff29a3f3f1f1109","original_commit_id":"55227dfcc242cb49ef9cf46e6ff29a3f3f1f1109","user":{"login":"xiaohui-zhang","id":14189585,"node_id":"MDQ6VXNlcjE0MTg5NTg1","avatar_url":"https://avatars2.githubusercontent.com/u/14189585?v=4","gravatar_id":"","url":"https://api.github.com/users/xiaohui-zhang","html_url":"https://github.com/xiaohui-zhang","followers_url":"https://api.github.com/users/xiaohui-zhang/followers","following_url":"https://api.github.com/users/xiaohui-zhang/following{/other_user}","gists_url":"https://api.github.com/users/xiaohui-zhang/gists{/gist_id}","starred_url":"https://api.github.com/users/xiaohui-zhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiaohui-zhang/subscriptions","organizations_url":"https://api.github.com/users/xiaohui-zhang/orgs","repos_url":"https://api.github.com/users/xiaohui-zhang/repos","events_url":"https://api.github.com/users/xiaohui-zhang/events{/privacy}","received_events_url":"https://api.github.com/users/xiaohui-zhang/received_events","type":"User","site_admin":false},"body":"Actually the reason we used 20 by default is that, when the user only sets pmass,  in order to get correct pron-probs, we need to specify an \"nbest\" value large enough, since phonetisaurus only computes pron-probs on the nbest list. When the user wants to rely on nbest, we always leave the responsibility to the user for setting the proper value, and that's why we didn't set it at the beginning. In summary, we want to allow all three ways of specifying constraints (pmass/nbest or both), and let the user to determine the proper values needed.\r\n","created_at":"2018-09-26T02:43:44Z","updated_at":"2018-09-26T02:43:44Z","html_url":"https://github.com/kaldi-asr/kaldi/pull/2730#discussion_r220412379","pull_request_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments/220412379"},"html":{"href":"https://github.com/kaldi-asr/kaldi/pull/2730#discussion_r220412379"},"pull_request":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730"}},"in_reply_to_id":219917389},"pull_request":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730","id":217327676,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE3MzI3Njc2","html_url":"https://github.com/kaldi-asr/kaldi/pull/2730","diff_url":"https://github.com/kaldi-asr/kaldi/pull/2730.diff","patch_url":"https://github.com/kaldi-asr/kaldi/pull/2730.patch","issue_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2730","number":2730,"state":"open","locked":false,"title":"Added phonetisaurus-based g2p scripts","user":{"login":"huangruizhe","id":28626513,"node_id":"MDQ6VXNlcjI4NjI2NTEz","avatar_url":"https://avatars2.githubusercontent.com/u/28626513?v=4","gravatar_id":"","url":"https://api.github.com/users/huangruizhe","html_url":"https://github.com/huangruizhe","followers_url":"https://api.github.com/users/huangruizhe/followers","following_url":"https://api.github.com/users/huangruizhe/following{/other_user}","gists_url":"https://api.github.com/users/huangruizhe/gists{/gist_id}","starred_url":"https://api.github.com/users/huangruizhe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huangruizhe/subscriptions","organizations_url":"https://api.github.com/users/huangruizhe/orgs","repos_url":"https://api.github.com/users/huangruizhe/repos","events_url":"https://api.github.com/users/huangruizhe/events{/privacy}","received_events_url":"https://api.github.com/users/huangruizhe/received_events","type":"User","site_admin":false},"body":"Phonetisaurus-based g2p was added, and corresponding changes are made to multi_en recipe.\r\n\r\n1) The correctness of language modeling script was verified by the same *.arpa file generated for multi_en recipe.\r\n\r\n2) The correctness of g2p result was verified by running multi_en recipe. Our new scripts can generate the same lexicon as the old script, except few entries (5 out of 35,709) which may be due to ties of scores or floating point rounding:\r\n\r\n```\r\n$ diff s5_new/data/local/dict_nosp/lexicon.txt s5_old/data/local/dict_nosp/lexicon.txt\r\n46990c46990\r\n< consubstantial\tk ah n s ah b s t ae n sh ah l\r\n---\r\n> consubstantial\tk ah n s ah b s t ae n ch ah l\r\n153708c153708\r\n< necticut\tn eh t ah k ah t\r\n---\r\n> necticut\tn eh t ih k ah t\r\n168558c168558\r\n< pennyless\tp eh n iy l ih s\r\n---\r\n> pennyless\tp eh n iy l ah s\r\n189949c189949\r\n< rhamni\tr ae m iy\r\n---\r\n> rhamni\tr ae m ay\r\n236164c236164\r\n< unsubstantial\tah n s ah b s t ae n sh ah l\r\n---\r\n> unsubstantial\tah n s ah b s t ae n ch ah l\r\n```\r\n\r\nnumber of original lexicon: 217,594\r\nnumber of missing words in corpus: 35,709\r\nnumber of newly generated pronunciation: 35,708 \r\n(WARNING:phonetisaurus-apply:2018-09-20 22:03:04:  No pronunciation for word: 'eux')","created_at":"2018-09-21T15:51:06Z","updated_at":"2018-09-26T02:43:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2c878d0244e656a98c1b45233c5b9cfbee029e1f","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730/commits","review_comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730/comments","review_comment_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2730/comments","statuses_url":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/55227dfcc242cb49ef9cf46e6ff29a3f3f1f1109","head":{"label":"huangruizhe:g2p_phonetisaurus","ref":"g2p_phonetisaurus","sha":"55227dfcc242cb49ef9cf46e6ff29a3f3f1f1109","user":{"login":"huangruizhe","id":28626513,"node_id":"MDQ6VXNlcjI4NjI2NTEz","avatar_url":"https://avatars2.githubusercontent.com/u/28626513?v=4","gravatar_id":"","url":"https://api.github.com/users/huangruizhe","html_url":"https://github.com/huangruizhe","followers_url":"https://api.github.com/users/huangruizhe/followers","following_url":"https://api.github.com/users/huangruizhe/following{/other_user}","gists_url":"https://api.github.com/users/huangruizhe/gists{/gist_id}","starred_url":"https://api.github.com/users/huangruizhe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huangruizhe/subscriptions","organizations_url":"https://api.github.com/users/huangruizhe/orgs","repos_url":"https://api.github.com/users/huangruizhe/repos","events_url":"https://api.github.com/users/huangruizhe/events{/privacy}","received_events_url":"https://api.github.com/users/huangruizhe/received_events","type":"User","site_admin":false},"repo":{"id":131504108,"node_id":"MDEwOlJlcG9zaXRvcnkxMzE1MDQxMDg=","name":"kaldi","full_name":"huangruizhe/kaldi","private":false,"owner":{"login":"huangruizhe","id":28626513,"node_id":"MDQ6VXNlcjI4NjI2NTEz","avatar_url":"https://avatars2.githubusercontent.com/u/28626513?v=4","gravatar_id":"","url":"https://api.github.com/users/huangruizhe","html_url":"https://github.com/huangruizhe","followers_url":"https://api.github.com/users/huangruizhe/followers","following_url":"https://api.github.com/users/huangruizhe/following{/other_user}","gists_url":"https://api.github.com/users/huangruizhe/gists{/gist_id}","starred_url":"https://api.github.com/users/huangruizhe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huangruizhe/subscriptions","organizations_url":"https://api.github.com/users/huangruizhe/orgs","repos_url":"https://api.github.com/users/huangruizhe/repos","events_url":"https://api.github.com/users/huangruizhe/events{/privacy}","received_events_url":"https://api.github.com/users/huangruizhe/received_events","type":"User","site_admin":false},"html_url":"https://github.com/huangruizhe/kaldi","description":"This is now the official location of the Kaldi project.","fork":true,"url":"https://api.github.com/repos/huangruizhe/kaldi","forks_url":"https://api.github.com/repos/huangruizhe/kaldi/forks","keys_url":"https://api.github.com/repos/huangruizhe/kaldi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/huangruizhe/kaldi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/huangruizhe/kaldi/teams","hooks_url":"https://api.github.com/repos/huangruizhe/kaldi/hooks","issue_events_url":"https://api.github.com/repos/huangruizhe/kaldi/issues/events{/number}","events_url":"https://api.github.com/repos/huangruizhe/kaldi/events","assignees_url":"https://api.github.com/repos/huangruizhe/kaldi/assignees{/user}","branches_url":"https://api.github.com/repos/huangruizhe/kaldi/branches{/branch}","tags_url":"https://api.github.com/repos/huangruizhe/kaldi/tags","blobs_url":"https://api.github.com/repos/huangruizhe/kaldi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/huangruizhe/kaldi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/huangruizhe/kaldi/git/refs{/sha}","trees_url":"https://api.github.com/repos/huangruizhe/kaldi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/huangruizhe/kaldi/statuses/{sha}","languages_url":"https://api.github.com/repos/huangruizhe/kaldi/languages","stargazers_url":"https://api.github.com/repos/huangruizhe/kaldi/stargazers","contributors_url":"https://api.github.com/repos/huangruizhe/kaldi/contributors","subscribers_url":"https://api.github.com/repos/huangruizhe/kaldi/subscribers","subscription_url":"https://api.github.com/repos/huangruizhe/kaldi/subscription","commits_url":"https://api.github.com/repos/huangruizhe/kaldi/commits{/sha}","git_commits_url":"https://api.github.com/repos/huangruizhe/kaldi/git/commits{/sha}","comments_url":"https://api.github.com/repos/huangruizhe/kaldi/comments{/number}","issue_comment_url":"https://api.github.com/repos/huangruizhe/kaldi/issues/comments{/number}","contents_url":"https://api.github.com/repos/huangruizhe/kaldi/contents/{+path}","compare_url":"https://api.github.com/repos/huangruizhe/kaldi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/huangruizhe/kaldi/merges","archive_url":"https://api.github.com/repos/huangruizhe/kaldi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/huangruizhe/kaldi/downloads","issues_url":"https://api.github.com/repos/huangruizhe/kaldi/issues{/number}","pulls_url":"https://api.github.com/repos/huangruizhe/kaldi/pulls{/number}","milestones_url":"https://api.github.com/repos/huangruizhe/kaldi/milestones{/number}","notifications_url":"https://api.github.com/repos/huangruizhe/kaldi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/huangruizhe/kaldi/labels{/name}","releases_url":"https://api.github.com/repos/huangruizhe/kaldi/releases{/id}","deployments_url":"https://api.github.com/repos/huangruizhe/kaldi/deployments","created_at":"2018-04-29T15:02:32Z","updated_at":"2018-09-21T05:06:15Z","pushed_at":"2018-09-23T07:07:35Z","git_url":"git://github.com/huangruizhe/kaldi.git","ssh_url":"git@github.com:huangruizhe/kaldi.git","clone_url":"https://github.com/huangruizhe/kaldi.git","svn_url":"https://github.com/huangruizhe/kaldi","homepage":"http://kaldi-asr.org","size":112359,"stargazers_count":0,"watchers_count":0,"language":"Shell","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kaldi-asr:master","ref":"master","sha":"ebbae44c619d28eaac117c9a5a30c7cddd04cd6a","user":{"login":"kaldi-asr","id":11650082,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExNjUwMDgy","avatar_url":"https://avatars2.githubusercontent.com/u/11650082?v=4","gravatar_id":"","url":"https://api.github.com/users/kaldi-asr","html_url":"https://github.com/kaldi-asr","followers_url":"https://api.github.com/users/kaldi-asr/followers","following_url":"https://api.github.com/users/kaldi-asr/following{/other_user}","gists_url":"https://api.github.com/users/kaldi-asr/gists{/gist_id}","starred_url":"https://api.github.com/users/kaldi-asr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaldi-asr/subscriptions","organizations_url":"https://api.github.com/users/kaldi-asr/orgs","repos_url":"https://api.github.com/users/kaldi-asr/repos","events_url":"https://api.github.com/users/kaldi-asr/events{/privacy}","received_events_url":"https://api.github.com/users/kaldi-asr/received_events","type":"Organization","site_admin":false},"repo":{"id":34275045,"node_id":"MDEwOlJlcG9zaXRvcnkzNDI3NTA0NQ==","name":"kaldi","full_name":"kaldi-asr/kaldi","private":false,"owner":{"login":"kaldi-asr","id":11650082,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExNjUwMDgy","avatar_url":"https://avatars2.githubusercontent.com/u/11650082?v=4","gravatar_id":"","url":"https://api.github.com/users/kaldi-asr","html_url":"https://github.com/kaldi-asr","followers_url":"https://api.github.com/users/kaldi-asr/followers","following_url":"https://api.github.com/users/kaldi-asr/following{/other_user}","gists_url":"https://api.github.com/users/kaldi-asr/gists{/gist_id}","starred_url":"https://api.github.com/users/kaldi-asr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaldi-asr/subscriptions","organizations_url":"https://api.github.com/users/kaldi-asr/orgs","repos_url":"https://api.github.com/users/kaldi-asr/repos","events_url":"https://api.github.com/users/kaldi-asr/events{/privacy}","received_events_url":"https://api.github.com/users/kaldi-asr/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/kaldi-asr/kaldi","description":"This is now the official location of the Kaldi project.","fork":false,"url":"https://api.github.com/repos/kaldi-asr/kaldi","forks_url":"https://api.github.com/repos/kaldi-asr/kaldi/forks","keys_url":"https://api.github.com/repos/kaldi-asr/kaldi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kaldi-asr/kaldi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kaldi-asr/kaldi/teams","hooks_url":"https://api.github.com/repos/kaldi-asr/kaldi/hooks","issue_events_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/events{/number}","events_url":"https://api.github.com/repos/kaldi-asr/kaldi/events","assignees_url":"https://api.github.com/repos/kaldi-asr/kaldi/assignees{/user}","branches_url":"https://api.github.com/repos/kaldi-asr/kaldi/branches{/branch}","tags_url":"https://api.github.com/repos/kaldi-asr/kaldi/tags","blobs_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/refs{/sha}","trees_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/{sha}","languages_url":"https://api.github.com/repos/kaldi-asr/kaldi/languages","stargazers_url":"https://api.github.com/repos/kaldi-asr/kaldi/stargazers","contributors_url":"https://api.github.com/repos/kaldi-asr/kaldi/contributors","subscribers_url":"https://api.github.com/repos/kaldi-asr/kaldi/subscribers","subscription_url":"https://api.github.com/repos/kaldi-asr/kaldi/subscription","commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/commits{/sha}","git_commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/commits{/sha}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/comments{/number}","issue_comment_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/comments{/number}","contents_url":"https://api.github.com/repos/kaldi-asr/kaldi/contents/{+path}","compare_url":"https://api.github.com/repos/kaldi-asr/kaldi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kaldi-asr/kaldi/merges","archive_url":"https://api.github.com/repos/kaldi-asr/kaldi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kaldi-asr/kaldi/downloads","issues_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues{/number}","pulls_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls{/number}","milestones_url":"https://api.github.com/repos/kaldi-asr/kaldi/milestones{/number}","notifications_url":"https://api.github.com/repos/kaldi-asr/kaldi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kaldi-asr/kaldi/labels{/name}","releases_url":"https://api.github.com/repos/kaldi-asr/kaldi/releases{/id}","deployments_url":"https://api.github.com/repos/kaldi-asr/kaldi/deployments","created_at":"2015-04-20T17:23:16Z","updated_at":"2018-09-26T01:43:53Z","pushed_at":"2018-09-26T01:43:43Z","git_url":"git://github.com/kaldi-asr/kaldi.git","ssh_url":"git@github.com:kaldi-asr/kaldi.git","clone_url":"https://github.com/kaldi-asr/kaldi.git","svn_url":"https://github.com/kaldi-asr/kaldi","homepage":"http://kaldi-asr.org","size":112356,"stargazers_count":4494,"watchers_count":4494,"language":"Shell","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2299,"mirror_url":null,"archived":false,"open_issues_count":162,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":2299,"open_issues":162,"watchers":4494,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730"},"html":{"href":"https://github.com/kaldi-asr/kaldi/pull/2730"},"issue":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2730"},"comments":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2730/comments"},"review_comments":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730/comments"},"review_comment":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2730/commits"},"statuses":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/55227dfcc242cb49ef9cf46e6ff29a3f3f1f1109"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"xiaohui-zhang"}}