{"action":"created","comment":{"url":"https://api.github.com/repos/danpovey/pocolm/pulls/comments/75008840","id":75008840,"diff_hunk":"@@ -57,6 +57,18 @@ for order in 3 4 5; do\n     format_arpa_lm.py data/lm_20k_${order}_prune${threshold} | gzip -c > data/arpa/20k_${order}gram_prune${threshold}.arpa.gz\n \n   done\n+\n+# example of pruning by size.","path":"egs/swbd/run.sh","position":null,"original_position":5,"commit_id":"b1a4d8496b21bdbcd7a6a9d9d56510c5dda91000","original_commit_id":"cc94865347ad60c12aa8db89f2c19959f2884edd","user":{"login":"danpovey","id":3298747,"avatar_url":"https://avatars.githubusercontent.com/u/3298747?v=3","gravatar_id":"","url":"https://api.github.com/users/danpovey","html_url":"https://github.com/danpovey","followers_url":"https://api.github.com/users/danpovey/followers","following_url":"https://api.github.com/users/danpovey/following{/other_user}","gists_url":"https://api.github.com/users/danpovey/gists{/gist_id}","starred_url":"https://api.github.com/users/danpovey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danpovey/subscriptions","organizations_url":"https://api.github.com/users/danpovey/orgs","repos_url":"https://api.github.com/users/danpovey/repos","events_url":"https://api.github.com/users/danpovey/events{/privacy}","received_events_url":"https://api.github.com/users/danpovey/received_events","type":"User","site_admin":false},"body":"Thanks for finding these.\nI fixed the issue with the space that should have been tab.\nThe problems you experienced with prune_lm.py, we are aware of and Jian is\nworking on fixing it (it involves a lot of changes and it's currently being\ntested).\nI'll wait for your pull request to fix the remaining issue.\nDan\n\n\n\n> Hello, Dan and Jian There are few possible bugs I believed i) the\n> generated arpa file contains the line <s> (the line after \\1-grams) is\n> separated by \"tab and space”, which should be “tab and tab” ii) in\n> swbd/run.sh lm_dir=\"data/lm/\" arpa_dir=\"data/arpa/“ it results print some\n> thing like data/lm//20000_5.pocolm, which doesn’t really matter, but make\n> it a bit un-beautiful These two are quite trivial and I will fix it\n> shortly. iii) The third one I am not quite sure When I use prune_lm.py to\n> prune lm-dir to certain target n-gram counts, for some threshold it works\n> (from 1003378333 to 408157132), but for some threshold it fails (like from\n> 1003378333 to 7598178). Since I use the exact same way of doing these two,\n> it is kind of weird (I tried this several times). The following is the log:\n> … Fixing lm-state total 6.64527e+06 -> 6.64618e+06 Fixing lm-state total\n> 5.97697e+06 -> 5.97853e+06 Fixing lm-state total 4.62547e+06 -> 4.62753e+06\n> float-counts-prune: logprob change per word was -0.777808 over 3.59103e+08\n> words. float-counts-prune: aside from unigram there were 19595753 nonzero\n> n-grams. Of these 11 were not pruned because they were shadowed by a\n> higher-order n-gram, and 2338106 because they lead to an existing LM-state\n> (according to <protected-counts-input>). Of the 17257636 n-grams eligible\n> for pruning, 17257636 were actually pruned. Traceback (most recent call\n> last): File \"/export/a08/zhouyang/pocolm/scripts/prune_lm_dir.py\", line\n> 647, in <module> (threshold, iter) = FindThreshold() File\n> \"/export/a08/zhouyang/pocolm/scripts/prune_lm_dir.py\", line 599, in\n> FindThreshold log_threshold = ls.Estimate(math.log(args.target_num_ngrams))\n> File \"/export/a08/zhouyang/pocolm/scripts/prune_lm_dir.py\", line 544, in\n> Estimate alpha = (self.y1 - self.y0) / (self.x1 - self.x0)\n> ZeroDivisionError: float division by zero … I believe we may need to be\n> cautious when some this special case when self.x1 == self.x0 occur. Please\n> let me know if I could provide more details or any help on fixing this bug\n> best regards, Zhouyang\n> … <#m_4710640430120309100_>\n> On Aug 15, 2016, at 4:29 PM, Daniel Povey ***@***.***> wrote: In\n> scripts/internal/prune_size_model.py: > + # the num-xgrams decreased on\n> the previous iteration. > + assert prev_num_xgrams >= cur_num_xgrams > +\n> prev_change_factor = (float(cur_num_xgrams) / prev_num_xgrams) ** (1.0/3.0)\n> > + baseline_decrease_factor = 1.0 / 1.5 > + # choose the factor from these\n> two, that's closest to one. > + predicted_decrease_factor =\n> max(prev_change_factor, baseline_decrease_factor) > + > + # the following\n> gives us the predicted num-xgrams if we were > + # to prune again with the\n> same threshold 'cur_threshold'. > + predicted_num_xgrams_if_repeat =\n> predicted_decrease_factor * cur_num_xgrams > + assert next_threshold >=\n> cur_threshold > + > + predicted_extra_factor = (next_threshold /\n> cur_threshold) ** (self.xgrams_change_power) > + return\n> predicted_num_xgrams_if_repeat * predicted_extra_factor > + > + def\n> AdjustModelForOvershoot(self): I'm responding here to your post about the\n> thing not converging right because I think it has to do with this. I notice\n> some issues with the code.. Firstly, you are doing some things in this\n> function that don't have anything to do with adjusting the model. You\n> should have a function Backtrack() that will do the actual backtracking.\n> Adjusting the model is a separate issue. For backtracking, if the two most\n> recent thresholds were identical, then you should remove the two most\n> recent elements from the history [i.e. backtrack twice]. If the history\n> would then be empty, then the user-specified initial threshold was too high\n> and you can exit with failure. Secondly, the way you are adjusting the\n> model needs to be changed to ensure convergence. The model doesn't have\n> just one parameter [the xgrams_change_power], it has three parameters,\n> although I'll suggest to remove one of them. Your code currently reads like\n> this: prev_change_factor = (float(cur_num_xgrams) / prev_num_xgrams) **\n> (1.0/3.0) baseline_decrease_factor = 1.0 / 1.5 I think it would be a good\n> idea to get rid of baseline_decrease_factor entirely, as a simplification,\n> so that there will be no 'max' expression. Be sure to rename variables\n> appropriately. [if we kept the baseline_decrease_factor, we'd have to\n> update it when experiencing overshoot, by increasing it.] Have a variable\n> self.prev_change_power = 0.5 which will replace the 1.0 / 3.0 power in your\n> existing code. And when you adjust the model, replace your code\n> self.xgrams_change_power *= float(cur_modeled_num_xgrams) / cur_num_xgrams\n> with the following: self.xgrams_change_power *= 1.2 self.prev_change_power\n> *= 1.2 if (self.prev_change_power > 1.0) self.prev_change_power = 1.0 The\n> problem you are experiencing in your comment here #45\n> <https://github.com/danpovey/pocolm/pull/45> (comment) is that at the\n> point of failure, the xgrams_change_power is not the important part of the\n> model, the important part is the 'predicted_decrease_factor', which is\n> controlled by other parameters, which you are not updating. Next, the way\n> you are adjusting the model is not very good. You need to adjust the — You\n> are receiving this because you are subscribed to this thread. Reply to this\n> email directly, view it on GitHub, or mute the thread.\n>\n> —\n> You are receiving this because you modified the open/close state.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/danpovey/pocolm/pull/31/files/cc94865347ad60c12aa8db89f2c19959f2884edd#r75007646>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ADJVu-0tSI0OsP6udvooIC4flNxiYnt8ks5qghWngaJpZM4JU463>\n> .\n>\n","created_at":"2016-08-16T20:02:08Z","updated_at":"2016-08-16T20:02:08Z","html_url":"https://github.com/danpovey/pocolm/pull/31#discussion_r75008840","pull_request_url":"https://api.github.com/repos/danpovey/pocolm/pulls/31","_links":{"self":{"href":"https://api.github.com/repos/danpovey/pocolm/pulls/comments/75008840"},"html":{"href":"https://github.com/danpovey/pocolm/pull/31#discussion_r75008840"},"pull_request":{"href":"https://api.github.com/repos/danpovey/pocolm/pulls/31"}}},"pull_request":{"url":"https://api.github.com/repos/danpovey/pocolm/pulls/31","id":78808340,"html_url":"https://github.com/danpovey/pocolm/pull/31","diff_url":"https://github.com/danpovey/pocolm/pull/31.diff","patch_url":"https://github.com/danpovey/pocolm/pull/31.patch","issue_url":"https://api.github.com/repos/danpovey/pocolm/issues/31","number":31,"state":"closed","locked":false,"title":"Prune by size","user":{"login":"wantee","id":6971724,"avatar_url":"https://avatars.githubusercontent.com/u/6971724?v=3","gravatar_id":"","url":"https://api.github.com/users/wantee","html_url":"https://github.com/wantee","followers_url":"https://api.github.com/users/wantee/followers","following_url":"https://api.github.com/users/wantee/following{/other_user}","gists_url":"https://api.github.com/users/wantee/gists{/gist_id}","starred_url":"https://api.github.com/users/wantee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wantee/subscriptions","organizations_url":"https://api.github.com/users/wantee/orgs","repos_url":"https://api.github.com/users/wantee/repos","events_url":"https://api.github.com/users/wantee/events{/privacy}","received_events_url":"https://api.github.com/users/wantee/received_events","type":"User","site_admin":false},"body":"Let user specify a target size for\r\nthe language model (in n-grams) and work out (approximately) the\r\namount of pruning needed to get this.","created_at":"2016-07-26T08:09:48Z","updated_at":"2016-08-16T20:02:08Z","closed_at":"2016-07-28T23:55:33Z","merged_at":"2016-07-28T23:55:33Z","merge_commit_sha":"6824be5a3b9407cda5b4b73f5d1fc9b5a6affdbe","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/danpovey/pocolm/pulls/31/commits","review_comments_url":"https://api.github.com/repos/danpovey/pocolm/pulls/31/comments","review_comment_url":"https://api.github.com/repos/danpovey/pocolm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/danpovey/pocolm/issues/31/comments","statuses_url":"https://api.github.com/repos/danpovey/pocolm/statuses/b1a4d8496b21bdbcd7a6a9d9d56510c5dda91000","head":{"label":"wantee:prune-by-size","ref":"prune-by-size","sha":"b1a4d8496b21bdbcd7a6a9d9d56510c5dda91000","user":{"login":"wantee","id":6971724,"avatar_url":"https://avatars.githubusercontent.com/u/6971724?v=3","gravatar_id":"","url":"https://api.github.com/users/wantee","html_url":"https://github.com/wantee","followers_url":"https://api.github.com/users/wantee/followers","following_url":"https://api.github.com/users/wantee/following{/other_user}","gists_url":"https://api.github.com/users/wantee/gists{/gist_id}","starred_url":"https://api.github.com/users/wantee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wantee/subscriptions","organizations_url":"https://api.github.com/users/wantee/orgs","repos_url":"https://api.github.com/users/wantee/repos","events_url":"https://api.github.com/users/wantee/events{/privacy}","received_events_url":"https://api.github.com/users/wantee/received_events","type":"User","site_admin":false},"repo":{"id":60158342,"name":"pocolm","full_name":"wantee/pocolm","owner":{"login":"wantee","id":6971724,"avatar_url":"https://avatars.githubusercontent.com/u/6971724?v=3","gravatar_id":"","url":"https://api.github.com/users/wantee","html_url":"https://github.com/wantee","followers_url":"https://api.github.com/users/wantee/followers","following_url":"https://api.github.com/users/wantee/following{/other_user}","gists_url":"https://api.github.com/users/wantee/gists{/gist_id}","starred_url":"https://api.github.com/users/wantee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wantee/subscriptions","organizations_url":"https://api.github.com/users/wantee/orgs","repos_url":"https://api.github.com/users/wantee/repos","events_url":"https://api.github.com/users/wantee/events{/privacy}","received_events_url":"https://api.github.com/users/wantee/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/wantee/pocolm","description":"Small language toolkit for creation, interpolation and pruning of ARPA language models","fork":true,"url":"https://api.github.com/repos/wantee/pocolm","forks_url":"https://api.github.com/repos/wantee/pocolm/forks","keys_url":"https://api.github.com/repos/wantee/pocolm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wantee/pocolm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wantee/pocolm/teams","hooks_url":"https://api.github.com/repos/wantee/pocolm/hooks","issue_events_url":"https://api.github.com/repos/wantee/pocolm/issues/events{/number}","events_url":"https://api.github.com/repos/wantee/pocolm/events","assignees_url":"https://api.github.com/repos/wantee/pocolm/assignees{/user}","branches_url":"https://api.github.com/repos/wantee/pocolm/branches{/branch}","tags_url":"https://api.github.com/repos/wantee/pocolm/tags","blobs_url":"https://api.github.com/repos/wantee/pocolm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wantee/pocolm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wantee/pocolm/git/refs{/sha}","trees_url":"https://api.github.com/repos/wantee/pocolm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wantee/pocolm/statuses/{sha}","languages_url":"https://api.github.com/repos/wantee/pocolm/languages","stargazers_url":"https://api.github.com/repos/wantee/pocolm/stargazers","contributors_url":"https://api.github.com/repos/wantee/pocolm/contributors","subscribers_url":"https://api.github.com/repos/wantee/pocolm/subscribers","subscription_url":"https://api.github.com/repos/wantee/pocolm/subscription","commits_url":"https://api.github.com/repos/wantee/pocolm/commits{/sha}","git_commits_url":"https://api.github.com/repos/wantee/pocolm/git/commits{/sha}","comments_url":"https://api.github.com/repos/wantee/pocolm/comments{/number}","issue_comment_url":"https://api.github.com/repos/wantee/pocolm/issues/comments{/number}","contents_url":"https://api.github.com/repos/wantee/pocolm/contents/{+path}","compare_url":"https://api.github.com/repos/wantee/pocolm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wantee/pocolm/merges","archive_url":"https://api.github.com/repos/wantee/pocolm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wantee/pocolm/downloads","issues_url":"https://api.github.com/repos/wantee/pocolm/issues{/number}","pulls_url":"https://api.github.com/repos/wantee/pocolm/pulls{/number}","milestones_url":"https://api.github.com/repos/wantee/pocolm/milestones{/number}","notifications_url":"https://api.github.com/repos/wantee/pocolm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wantee/pocolm/labels{/name}","releases_url":"https://api.github.com/repos/wantee/pocolm/releases{/id}","deployments_url":"https://api.github.com/repos/wantee/pocolm/deployments","created_at":"2016-06-01T08:17:50Z","updated_at":"2016-06-01T08:17:51Z","pushed_at":"2016-08-16T05:20:23Z","git_url":"git://github.com/wantee/pocolm.git","ssh_url":"git@github.com:wantee/pocolm.git","clone_url":"https://github.com/wantee/pocolm.git","svn_url":"https://github.com/wantee/pocolm","homepage":null,"size":422,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"danpovey:master","ref":"master","sha":"807c99a45006ff25414ff8f1e360de0f29ab69f7","user":{"login":"danpovey","id":3298747,"avatar_url":"https://avatars.githubusercontent.com/u/3298747?v=3","gravatar_id":"","url":"https://api.github.com/users/danpovey","html_url":"https://github.com/danpovey","followers_url":"https://api.github.com/users/danpovey/followers","following_url":"https://api.github.com/users/danpovey/following{/other_user}","gists_url":"https://api.github.com/users/danpovey/gists{/gist_id}","starred_url":"https://api.github.com/users/danpovey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danpovey/subscriptions","organizations_url":"https://api.github.com/users/danpovey/orgs","repos_url":"https://api.github.com/users/danpovey/repos","events_url":"https://api.github.com/users/danpovey/events{/privacy}","received_events_url":"https://api.github.com/users/danpovey/received_events","type":"User","site_admin":false},"repo":{"id":56359219,"name":"pocolm","full_name":"danpovey/pocolm","owner":{"login":"danpovey","id":3298747,"avatar_url":"https://avatars.githubusercontent.com/u/3298747?v=3","gravatar_id":"","url":"https://api.github.com/users/danpovey","html_url":"https://github.com/danpovey","followers_url":"https://api.github.com/users/danpovey/followers","following_url":"https://api.github.com/users/danpovey/following{/other_user}","gists_url":"https://api.github.com/users/danpovey/gists{/gist_id}","starred_url":"https://api.github.com/users/danpovey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danpovey/subscriptions","organizations_url":"https://api.github.com/users/danpovey/orgs","repos_url":"https://api.github.com/users/danpovey/repos","events_url":"https://api.github.com/users/danpovey/events{/privacy}","received_events_url":"https://api.github.com/users/danpovey/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/danpovey/pocolm","description":"Small language toolkit for creation, interpolation and pruning of ARPA language models","fork":false,"url":"https://api.github.com/repos/danpovey/pocolm","forks_url":"https://api.github.com/repos/danpovey/pocolm/forks","keys_url":"https://api.github.com/repos/danpovey/pocolm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/danpovey/pocolm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/danpovey/pocolm/teams","hooks_url":"https://api.github.com/repos/danpovey/pocolm/hooks","issue_events_url":"https://api.github.com/repos/danpovey/pocolm/issues/events{/number}","events_url":"https://api.github.com/repos/danpovey/pocolm/events","assignees_url":"https://api.github.com/repos/danpovey/pocolm/assignees{/user}","branches_url":"https://api.github.com/repos/danpovey/pocolm/branches{/branch}","tags_url":"https://api.github.com/repos/danpovey/pocolm/tags","blobs_url":"https://api.github.com/repos/danpovey/pocolm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/danpovey/pocolm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/danpovey/pocolm/git/refs{/sha}","trees_url":"https://api.github.com/repos/danpovey/pocolm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/danpovey/pocolm/statuses/{sha}","languages_url":"https://api.github.com/repos/danpovey/pocolm/languages","stargazers_url":"https://api.github.com/repos/danpovey/pocolm/stargazers","contributors_url":"https://api.github.com/repos/danpovey/pocolm/contributors","subscribers_url":"https://api.github.com/repos/danpovey/pocolm/subscribers","subscription_url":"https://api.github.com/repos/danpovey/pocolm/subscription","commits_url":"https://api.github.com/repos/danpovey/pocolm/commits{/sha}","git_commits_url":"https://api.github.com/repos/danpovey/pocolm/git/commits{/sha}","comments_url":"https://api.github.com/repos/danpovey/pocolm/comments{/number}","issue_comment_url":"https://api.github.com/repos/danpovey/pocolm/issues/comments{/number}","contents_url":"https://api.github.com/repos/danpovey/pocolm/contents/{+path}","compare_url":"https://api.github.com/repos/danpovey/pocolm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/danpovey/pocolm/merges","archive_url":"https://api.github.com/repos/danpovey/pocolm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/danpovey/pocolm/downloads","issues_url":"https://api.github.com/repos/danpovey/pocolm/issues{/number}","pulls_url":"https://api.github.com/repos/danpovey/pocolm/pulls{/number}","milestones_url":"https://api.github.com/repos/danpovey/pocolm/milestones{/number}","notifications_url":"https://api.github.com/repos/danpovey/pocolm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/danpovey/pocolm/labels{/name}","releases_url":"https://api.github.com/repos/danpovey/pocolm/releases{/id}","deployments_url":"https://api.github.com/repos/danpovey/pocolm/deployments","created_at":"2016-04-16T01:24:14Z","updated_at":"2016-07-16T16:27:05Z","pushed_at":"2016-08-16T20:00:56Z","git_url":"git://github.com/danpovey/pocolm.git","ssh_url":"git@github.com:danpovey/pocolm.git","clone_url":"https://github.com/danpovey/pocolm.git","svn_url":"https://github.com/danpovey/pocolm","homepage":null,"size":526,"stargazers_count":6,"watchers_count":6,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":12,"forks":6,"open_issues":12,"watchers":6,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/danpovey/pocolm/pulls/31"},"html":{"href":"https://github.com/danpovey/pocolm/pull/31"},"issue":{"href":"https://api.github.com/repos/danpovey/pocolm/issues/31"},"comments":{"href":"https://api.github.com/repos/danpovey/pocolm/issues/31/comments"},"review_comments":{"href":"https://api.github.com/repos/danpovey/pocolm/pulls/31/comments"},"review_comment":{"href":"https://api.github.com/repos/danpovey/pocolm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/danpovey/pocolm/pulls/31/commits"},"statuses":{"href":"https://api.github.com/repos/danpovey/pocolm/statuses/b1a4d8496b21bdbcd7a6a9d9d56510c5dda91000"}}}}
{"actor":{"display_login":"danpovey"}}