{"action":"created","comment":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments/190385106","pull_request_review_id":122749227,"id":190385106,"diff_hunk":"@@ -0,0 +1,238 @@\n+#!/bin/bash\n+\n+set -e -o pipefail\n+\n+\n+# This script is called from local/nnet3/run_tdnn.sh and local/chain/run_tdnn.sh (and may eventually\n+# be called by more scripts).  It contains the common feature preparation and iVector-related parts\n+# of the script.  See those scripts for examples of usage.\n+\n+\n+stage=0\n+nj=30\n+min_seg_len=1.55  # min length in seconds... we do this because chain training\n+                  # will discard segments shorter than 1.5 seconds.   Must remain in sync\n+                  # with the same option given to prepare_lores_feats_and_alignments.sh\n+train_set=train_cleaned   # you might set this to e.g. train.\n+gmm=tri3_cleaned          # This specifies a GMM-dir from the features of the type you're training the system on;\n+                         # it should contain alignments for 'train_set'.\n+\n+num_threads_ubm=32\n+nnet3_affix=_cleaned     # affix for exp/nnet3 directory to put iVector stuff in, so it\n+                         # becomes exp/nnet3_cleaned or whatever.\n+\n+. ./cmd.sh\n+. ./path.sh\n+. utils/parse_options.sh\n+\n+\n+gmm_dir=exp/${gmm}\n+ali_dir=exp/${gmm}_ali_${train_set}_sp_comb\n+\n+for f in data/${train_set}/feats.scp ${gmm_dir}/final.mdl; do\n+  if [ ! -f $f ]; then\n+    echo \"$0: expected file $f to exist\"\n+    exit 1\n+  fi\n+done\n+\n+\n+\n+if [ $stage -le 2 ] && [ -f data/${train_set}_sp_hires/feats.scp ]; then\n+  echo \"$0: data/${train_set}_sp_hires/feats.scp already exists.\"\n+  echo \" ... Please either remove it, or rerun this script with stage > 2.\"\n+  exit 1\n+fi\n+\n+\n+if [ $stage -le 1 ]; then\n+  echo \"$0: preparing directory for speed-perturbed data\"\n+  utils/data/perturb_data_dir_speed_3way.sh data/${train_set} data/${train_set}_sp\n+fi\n+\n+if [ $stage -le 2 ]; then\n+  echo \"$0: creating high-resolution MFCC features\"\n+\n+  # this shows how you can split across multiple file-systems.  we'll split the\n+  # MFCC dir across multiple locations.  You might want to be careful here, if you\n+  # have multiple copies of Kaldi checked out and run the same recipe, not to let\n+  # them overwrite each other.\n+  mfccdir=data/${train_set}_sp_hires/data\n+  if [[ $(hostname -f) == *.clsp.jhu.edu ]] && [ ! -d $mfccdir/storage ]; then\n+    utils/create_split_dir.pl /export/b0{5,6,7,8}/$USER/kaldi-data/mfcc/tedlium-$(date +'%m_%d_%H_%M')/s5/$mfccdir/storage $mfccdir/storage\n+  fi\n+\n+  for datadir in ${train_set}_sp dev test; do\n+    utils/copy_data_dir.sh data/$datadir data/${datadir}_hires\n+  done\n+\n+  # do volume-perturbation on the training data prior to extracting hires\n+  # features; this helps make trained nnets more invariant to test data volume.\n+  utils/data/perturb_data_dir_volume.sh data/${train_set}_sp_hires\n+\n+  for datadir in ${train_set}_sp dev test; do\n+    steps/make_mfcc.sh --nj $nj --mfcc-config conf/mfcc_hires.conf \\\n+      --cmd \"$train_cmd\" data/${datadir}_hires\n+    steps/compute_cmvn_stats.sh data/${datadir}_hires\n+    utils/fix_data_dir.sh data/${datadir}_hires\n+  done\n+fi\n+\n+if [ $stage -le 3 ]; then\n+  echo \"$0: combining short segments of speed-perturbed high-resolution MFCC training data\"\n+  # we have to combine short segments or we won't be able to train chain models\n+  # on those segments.\n+  utils/data/combine_short_segments.sh \\\n+     data/${train_set}_sp_hires $min_seg_len data/${train_set}_sp_hires_comb\n+\n+  # just copy over the CMVN to avoid having to recompute it.\n+  cp data/${train_set}_sp_hires/cmvn.scp data/${train_set}_sp_hires_comb/\n+  utils/fix_data_dir.sh data/${train_set}_sp_hires_comb/\n+fi\n+\n+if [ $stage -le 4 ]; then\n+  echo \"$0: selecting segments of hires training data that were also present in the\"","path":"egs/tedlium/s5_r3/local/nnet3/run_ivector_common.sh","position":94,"original_position":94,"commit_id":"3855c7b112c564b533380f5f3ec857daeebeb76c","original_commit_id":"3855c7b112c564b533380f5f3ec857daeebeb76c","user":{"login":"danpovey","id":3298747,"avatar_url":"https://avatars1.githubusercontent.com/u/3298747?v=4","gravatar_id":"","url":"https://api.github.com/users/danpovey","html_url":"https://github.com/danpovey","followers_url":"https://api.github.com/users/danpovey/followers","following_url":"https://api.github.com/users/danpovey/following{/other_user}","gists_url":"https://api.github.com/users/danpovey/gists{/gist_id}","starred_url":"https://api.github.com/users/danpovey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danpovey/subscriptions","organizations_url":"https://api.github.com/users/danpovey/orgs","repos_url":"https://api.github.com/users/danpovey/repos","events_url":"https://api.github.com/users/danpovey/events{/privacy}","received_events_url":"https://api.github.com/users/danpovey/received_events","type":"User","site_admin":false},"body":"I think I said something on kaldi-help about this.   You should be using PCA not LDA, and it simplifies the script (you can select the data subset in a simpler way).  You can refer to any up-to-date script e.g. I think the Switchboard one.","created_at":"2018-05-23T20:17:44Z","updated_at":"2018-05-23T20:19:25Z","html_url":"https://github.com/kaldi-asr/kaldi/pull/2442#discussion_r190385106","pull_request_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments/190385106"},"html":{"href":"https://github.com/kaldi-asr/kaldi/pull/2442#discussion_r190385106"},"pull_request":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442"}}},"pull_request":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442","id":189654958,"html_url":"https://github.com/kaldi-asr/kaldi/pull/2442","diff_url":"https://github.com/kaldi-asr/kaldi/pull/2442.diff","patch_url":"https://github.com/kaldi-asr/kaldi/pull/2442.patch","issue_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2442","number":2442,"state":"open","locked":false,"title":"[wip] Tedlium r3 recipe","user":{"login":"francoishernandez","id":14880403,"avatar_url":"https://avatars2.githubusercontent.com/u/14880403?v=4","gravatar_id":"","url":"https://api.github.com/users/francoishernandez","html_url":"https://github.com/francoishernandez","followers_url":"https://api.github.com/users/francoishernandez/followers","following_url":"https://api.github.com/users/francoishernandez/following{/other_user}","gists_url":"https://api.github.com/users/francoishernandez/gists{/gist_id}","starred_url":"https://api.github.com/users/francoishernandez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/francoishernandez/subscriptions","organizations_url":"https://api.github.com/users/francoishernandez/orgs","repos_url":"https://api.github.com/users/francoishernandez/repos","events_url":"https://api.github.com/users/francoishernandez/events{/privacy}","received_events_url":"https://api.github.com/users/francoishernandez/received_events","type":"User","site_admin":false},"body":"Here is the recipe coming with the recent release of [TED-LIUM corpus version 3](https://lium.univ-lemans.fr/ted-lium3/).\r\n\r\nI copied the ability to download the LMs from [David's PR](https://github.com/kaldi-asr/kaldi/pull/2441), and made the same for the RNNLM models. Could you add [these models](https://goo.gl/JSQT5s) to kaldi-asr.org? (I added a dummy link in local/ted_download_rnnlm.sh for now.)\r\n\r\nI did not add what Dan is mentioning about the ivector training [here](https://groups.google.com/forum/#!searchin/kaldi-help/tedlium%7Csort:date/kaldi-help/UnjTZGVwEpw/EvSjuoF1BgAJ) yet.\r\n\r\nAlso, please note that the \"join_suffix\" normalization script present in r2 has been applied to the LM data available in the r3 tarball, which explains its disappearing from this recipe.","created_at":"2018-05-22T13:02:23Z","updated_at":"2018-05-23T20:19:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0cb572dca782d3ec633aa971300bab0fb6d882fe","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442/commits","review_comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442/comments","review_comment_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2442/comments","statuses_url":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/3855c7b112c564b533380f5f3ec857daeebeb76c","head":{"label":"francoishernandez:tedlium_r3_recipe","ref":"tedlium_r3_recipe","sha":"3855c7b112c564b533380f5f3ec857daeebeb76c","user":{"login":"francoishernandez","id":14880403,"avatar_url":"https://avatars2.githubusercontent.com/u/14880403?v=4","gravatar_id":"","url":"https://api.github.com/users/francoishernandez","html_url":"https://github.com/francoishernandez","followers_url":"https://api.github.com/users/francoishernandez/followers","following_url":"https://api.github.com/users/francoishernandez/following{/other_user}","gists_url":"https://api.github.com/users/francoishernandez/gists{/gist_id}","starred_url":"https://api.github.com/users/francoishernandez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/francoishernandez/subscriptions","organizations_url":"https://api.github.com/users/francoishernandez/orgs","repos_url":"https://api.github.com/users/francoishernandez/repos","events_url":"https://api.github.com/users/francoishernandez/events{/privacy}","received_events_url":"https://api.github.com/users/francoishernandez/received_events","type":"User","site_admin":false},"repo":{"id":95011411,"name":"kaldi","full_name":"francoishernandez/kaldi","owner":{"login":"francoishernandez","id":14880403,"avatar_url":"https://avatars2.githubusercontent.com/u/14880403?v=4","gravatar_id":"","url":"https://api.github.com/users/francoishernandez","html_url":"https://github.com/francoishernandez","followers_url":"https://api.github.com/users/francoishernandez/followers","following_url":"https://api.github.com/users/francoishernandez/following{/other_user}","gists_url":"https://api.github.com/users/francoishernandez/gists{/gist_id}","starred_url":"https://api.github.com/users/francoishernandez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/francoishernandez/subscriptions","organizations_url":"https://api.github.com/users/francoishernandez/orgs","repos_url":"https://api.github.com/users/francoishernandez/repos","events_url":"https://api.github.com/users/francoishernandez/events{/privacy}","received_events_url":"https://api.github.com/users/francoishernandez/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/francoishernandez/kaldi","description":"This is now the official location of the Kaldi project.","fork":true,"url":"https://api.github.com/repos/francoishernandez/kaldi","forks_url":"https://api.github.com/repos/francoishernandez/kaldi/forks","keys_url":"https://api.github.com/repos/francoishernandez/kaldi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/francoishernandez/kaldi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/francoishernandez/kaldi/teams","hooks_url":"https://api.github.com/repos/francoishernandez/kaldi/hooks","issue_events_url":"https://api.github.com/repos/francoishernandez/kaldi/issues/events{/number}","events_url":"https://api.github.com/repos/francoishernandez/kaldi/events","assignees_url":"https://api.github.com/repos/francoishernandez/kaldi/assignees{/user}","branches_url":"https://api.github.com/repos/francoishernandez/kaldi/branches{/branch}","tags_url":"https://api.github.com/repos/francoishernandez/kaldi/tags","blobs_url":"https://api.github.com/repos/francoishernandez/kaldi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/francoishernandez/kaldi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/francoishernandez/kaldi/git/refs{/sha}","trees_url":"https://api.github.com/repos/francoishernandez/kaldi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/francoishernandez/kaldi/statuses/{sha}","languages_url":"https://api.github.com/repos/francoishernandez/kaldi/languages","stargazers_url":"https://api.github.com/repos/francoishernandez/kaldi/stargazers","contributors_url":"https://api.github.com/repos/francoishernandez/kaldi/contributors","subscribers_url":"https://api.github.com/repos/francoishernandez/kaldi/subscribers","subscription_url":"https://api.github.com/repos/francoishernandez/kaldi/subscription","commits_url":"https://api.github.com/repos/francoishernandez/kaldi/commits{/sha}","git_commits_url":"https://api.github.com/repos/francoishernandez/kaldi/git/commits{/sha}","comments_url":"https://api.github.com/repos/francoishernandez/kaldi/comments{/number}","issue_comment_url":"https://api.github.com/repos/francoishernandez/kaldi/issues/comments{/number}","contents_url":"https://api.github.com/repos/francoishernandez/kaldi/contents/{+path}","compare_url":"https://api.github.com/repos/francoishernandez/kaldi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/francoishernandez/kaldi/merges","archive_url":"https://api.github.com/repos/francoishernandez/kaldi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/francoishernandez/kaldi/downloads","issues_url":"https://api.github.com/repos/francoishernandez/kaldi/issues{/number}","pulls_url":"https://api.github.com/repos/francoishernandez/kaldi/pulls{/number}","milestones_url":"https://api.github.com/repos/francoishernandez/kaldi/milestones{/number}","notifications_url":"https://api.github.com/repos/francoishernandez/kaldi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/francoishernandez/kaldi/labels{/name}","releases_url":"https://api.github.com/repos/francoishernandez/kaldi/releases{/id}","deployments_url":"https://api.github.com/repos/francoishernandez/kaldi/deployments","created_at":"2017-06-21T14:12:39Z","updated_at":"2017-12-11T09:38:42Z","pushed_at":"2018-05-23T07:43:13Z","git_url":"git://github.com/francoishernandez/kaldi.git","ssh_url":"git@github.com:francoishernandez/kaldi.git","clone_url":"https://github.com/francoishernandez/kaldi.git","svn_url":"https://github.com/francoishernandez/kaldi","homepage":"http://kaldi-asr.org","size":111465,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kaldi-asr:master","ref":"master","sha":"1ab3df05a71db38f57d81471bef9b702015adcc8","user":{"login":"kaldi-asr","id":11650082,"avatar_url":"https://avatars2.githubusercontent.com/u/11650082?v=4","gravatar_id":"","url":"https://api.github.com/users/kaldi-asr","html_url":"https://github.com/kaldi-asr","followers_url":"https://api.github.com/users/kaldi-asr/followers","following_url":"https://api.github.com/users/kaldi-asr/following{/other_user}","gists_url":"https://api.github.com/users/kaldi-asr/gists{/gist_id}","starred_url":"https://api.github.com/users/kaldi-asr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaldi-asr/subscriptions","organizations_url":"https://api.github.com/users/kaldi-asr/orgs","repos_url":"https://api.github.com/users/kaldi-asr/repos","events_url":"https://api.github.com/users/kaldi-asr/events{/privacy}","received_events_url":"https://api.github.com/users/kaldi-asr/received_events","type":"Organization","site_admin":false},"repo":{"id":34275045,"name":"kaldi","full_name":"kaldi-asr/kaldi","owner":{"login":"kaldi-asr","id":11650082,"avatar_url":"https://avatars2.githubusercontent.com/u/11650082?v=4","gravatar_id":"","url":"https://api.github.com/users/kaldi-asr","html_url":"https://github.com/kaldi-asr","followers_url":"https://api.github.com/users/kaldi-asr/followers","following_url":"https://api.github.com/users/kaldi-asr/following{/other_user}","gists_url":"https://api.github.com/users/kaldi-asr/gists{/gist_id}","starred_url":"https://api.github.com/users/kaldi-asr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaldi-asr/subscriptions","organizations_url":"https://api.github.com/users/kaldi-asr/orgs","repos_url":"https://api.github.com/users/kaldi-asr/repos","events_url":"https://api.github.com/users/kaldi-asr/events{/privacy}","received_events_url":"https://api.github.com/users/kaldi-asr/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kaldi-asr/kaldi","description":"This is now the official location of the Kaldi project.","fork":false,"url":"https://api.github.com/repos/kaldi-asr/kaldi","forks_url":"https://api.github.com/repos/kaldi-asr/kaldi/forks","keys_url":"https://api.github.com/repos/kaldi-asr/kaldi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kaldi-asr/kaldi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kaldi-asr/kaldi/teams","hooks_url":"https://api.github.com/repos/kaldi-asr/kaldi/hooks","issue_events_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/events{/number}","events_url":"https://api.github.com/repos/kaldi-asr/kaldi/events","assignees_url":"https://api.github.com/repos/kaldi-asr/kaldi/assignees{/user}","branches_url":"https://api.github.com/repos/kaldi-asr/kaldi/branches{/branch}","tags_url":"https://api.github.com/repos/kaldi-asr/kaldi/tags","blobs_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/refs{/sha}","trees_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/{sha}","languages_url":"https://api.github.com/repos/kaldi-asr/kaldi/languages","stargazers_url":"https://api.github.com/repos/kaldi-asr/kaldi/stargazers","contributors_url":"https://api.github.com/repos/kaldi-asr/kaldi/contributors","subscribers_url":"https://api.github.com/repos/kaldi-asr/kaldi/subscribers","subscription_url":"https://api.github.com/repos/kaldi-asr/kaldi/subscription","commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/commits{/sha}","git_commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/commits{/sha}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/comments{/number}","issue_comment_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/comments{/number}","contents_url":"https://api.github.com/repos/kaldi-asr/kaldi/contents/{+path}","compare_url":"https://api.github.com/repos/kaldi-asr/kaldi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kaldi-asr/kaldi/merges","archive_url":"https://api.github.com/repos/kaldi-asr/kaldi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kaldi-asr/kaldi/downloads","issues_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues{/number}","pulls_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls{/number}","milestones_url":"https://api.github.com/repos/kaldi-asr/kaldi/milestones{/number}","notifications_url":"https://api.github.com/repos/kaldi-asr/kaldi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kaldi-asr/kaldi/labels{/name}","releases_url":"https://api.github.com/repos/kaldi-asr/kaldi/releases{/id}","deployments_url":"https://api.github.com/repos/kaldi-asr/kaldi/deployments","created_at":"2015-04-20T17:23:16Z","updated_at":"2018-05-23T19:43:48Z","pushed_at":"2018-05-23T19:43:34Z","git_url":"git://github.com/kaldi-asr/kaldi.git","ssh_url":"git@github.com:kaldi-asr/kaldi.git","clone_url":"https://github.com/kaldi-asr/kaldi.git","svn_url":"https://github.com/kaldi-asr/kaldi","homepage":"http://kaldi-asr.org","size":111375,"stargazers_count":3728,"watchers_count":3728,"language":"Shell","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1812,"mirror_url":null,"archived":false,"open_issues_count":164,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":1812,"open_issues":164,"watchers":3728,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442"},"html":{"href":"https://github.com/kaldi-asr/kaldi/pull/2442"},"issue":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2442"},"comments":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/issues/2442/comments"},"review_comments":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442/comments"},"review_comment":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/2442/commits"},"statuses":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/3855c7b112c564b533380f5f3ec857daeebeb76c"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"danpovey"}}