{"action":"created","comment":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments/58446557","id":58446557,"diff_hunk":"@@ -0,0 +1,142 @@\n+#!/bin/bash\n+\n+# this script contains some common (shared) parts of the run_nnet*.sh scripts.\n+\n+. cmd.sh\n+\n+\n+stage=0\n+generate_alignments=true # false if doing ctc training\n+speed_perturb=true\n+\n+set -e\n+. cmd.sh\n+. ./path.sh\n+. ./utils/parse_options.sh\n+\n+\n+train_set=train_960\n+if [ \"$speed_perturb\" == \"true\" ]; then\n+  if [ $stage -le 1 ]; then\n+    #Although the nnet will be trained by high resolution data, we still have to perturbe the normal data to get the alignment\n+    # _sp stands for speed-perturbed\n+\n+    for datadir in train_960; do\n+      utils/perturb_data_dir_speed.sh 0.9 data/${datadir} data/temp1\n+      utils/perturb_data_dir_speed.sh 1.1 data/${datadir} data/temp2\n+      utils/combine_data.sh data/${datadir}_tmp data/temp1 data/temp2\n+      utils/validate_data_dir.sh --no-feats data/${datadir}_tmp\n+      rm -r data/temp1 data/temp2\n+\n+      mfccdir=mfcc_perturbed\n+      steps/make_mfcc.sh --cmd \"$train_cmd\" --nj 50 \\\n+        data/${datadir}_tmp exp/make_mfcc/${datadir}_tmp $mfccdir || exit 1;\n+      steps/compute_cmvn_stats.sh data/${datadir}_tmp exp/make_mfcc/${datadir}_tmp $mfccdir || exit 1;\n+      utils/fix_data_dir.sh data/${datadir}_tmp\n+\n+      utils/copy_data_dir.sh --spk-prefix sp1.0- --utt-prefix sp1.0- data/${datadir} data/temp0\n+      utils/combine_data.sh data/${datadir}_sp data/${datadir}_tmp data/temp0\n+      utils/fix_data_dir.sh data/${datadir}_sp\n+      rm -r data/temp0 data/${datadir}_tmp\n+    done\n+  fi\n+\n+  if [ $stage -le 2 ] && [ \"$generate_alignments\" == \"true\" ]; then\n+    #obtain the alignment of the perturbed data\n+    steps/align_fmllr.sh --nj 100 --cmd \"$train_cmd\" \\\n+      data/train_960_sp data/lang exp/tri6b exp/tri6b_sp || exit 1\n+  fi\n+  train_set=train_960_sp\n+fi\n+\n+if [ $stage -le 3 ]; then\n+  # Create high-resolution MFCC features (with 40 cepstra instead of 13).\n+  # this shows how you can split across multiple file-systems.  we'll split the\n+  # MFCC dir across multiple locations.  You might want to be careful here, if you\n+  # have multiple copies of Kaldi checked out and run the same recipe, not to let\n+  # them overwrite each other.\n+  mfccdir=mfcc_hires\n+  if [[ $(hostname -f) == *.clsp.jhu.edu ]] && [ ! -d $mfccdir/storage ]; then\n+    utils/create_split_dir.pl /export/b0{1,2,3,4}/$USER/kaldi-data/egs/librispeech-$(date +'%m_%d_%H_%M')/s5/$mfccdir/storage $mfccdir/storage\n+  fi\n+\n+  for datadir in $train_set test_clean test_other dev_clean dev_other; do\n+    utils/copy_data_dir.sh data/$datadir data/${datadir}_hires\n+    steps/make_mfcc.sh --nj 70 --mfcc-config conf/mfcc_hires.conf \\\n+      --cmd \"$train_cmd\" data/${datadir}_hires exp/make_hires/$datadir $mfccdir || exit 1;\n+    steps/compute_cmvn_stats.sh data/${datadir}_hires exp/make_hires/$datadir $mfccdir || exit 1;\n+  done\n+\n+  # now create some data subsets.\n+  # mixed is the clean+other data.\n+  # 30k is 1/10 of the data (around 100 hours), 60k is 1/5th of it (around 200 hours).\n+  utils/subset_data_dir.sh data/${train_set}_hires 30000 data/${train_set}_mixed_hires_30k\n+  utils/subset_data_dir.sh data/${train_set}_hires 60000 data/${train_set}_mixed_hires_60k\n+fi\n+\n+if [ $stage -le 4 ]; then\n+  # We need to build a small system just because we need the LDA+MLLT transform\n+  # to train the diag-UBM on top of.  We align a subset of training data for\n+  # this purpose.\n+  utils/subset_data_dir.sh --utt-list <(awk '{print $1}' data/${train_set}_mixed_hires_30k/utt2spk) \\\n+     data/${train_set} data/${train_set}_30k\n+\n+  steps/align_fmllr.sh --nj 40 --cmd \"$train_cmd\" \\\n+    data/${train_set}_30k data/lang exp/tri6b exp/nnet3/tri6b_ali_30k\n+fi\n+\n+if [ $stage -le 5 ]; then\n+  # Train a small system just for its LDA+MLLT transform.  We use --num-iters 13\n+  # because after we get the transform (12th iter is the last), any further\n+  # training is pointless.\n+  steps/train_lda_mllt.sh --cmd \"$train_cmd\" --num-iters 13 \\\n+    --realign-iters \"\" \\\n+    --splice-opts \"--left-context=3 --right-context=3\" \\\n+    5000 10000 data/${train_set}_mixed_hires_30k data/lang \\","path":"egs/librispeech/s5/local/nnet3/run_ivector_common.sh","position":95,"original_position":95,"commit_id":"9ea0ff3e250cec5ed8f22b1a837c17791398eaf6","original_commit_id":"9ea0ff3e250cec5ed8f22b1a837c17791398eaf6","user":{"login":"danpovey","id":3298747,"avatar_url":"https://avatars.githubusercontent.com/u/3298747?v=3","gravatar_id":"","url":"https://api.github.com/users/danpovey","html_url":"https://github.com/danpovey","followers_url":"https://api.github.com/users/danpovey/followers","following_url":"https://api.github.com/users/danpovey/following{/other_user}","gists_url":"https://api.github.com/users/danpovey/gists{/gist_id}","starred_url":"https://api.github.com/users/danpovey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danpovey/subscriptions","organizations_url":"https://api.github.com/users/danpovey/orgs","repos_url":"https://api.github.com/users/danpovey/repos","events_url":"https://api.github.com/users/danpovey/events{/privacy}","received_events_url":"https://api.github.com/users/danpovey/received_events","type":"User","site_admin":false},"body":"Be careful about whether or not you are doing data-perturbing.  You\nprobably should be doing it, but last time I looked at those librispeech\nscripts, you were not.\nDan\n\n\nOn Mon, Apr 4, 2016 at 4:43 PM, Yiming Wang <notifications@github.com>\nwrote:\n\n> In egs/librispeech/s5/local/nnet3/run_ivector_common.sh\n> <https://github.com/kaldi-asr/kaldi/pull/645#discussion_r58445291>:\n>\n> > +  # this purpose.\n> > +  utils/subset_data_dir.sh --utt-list <(awk '{print $1}' data/${train_set}_mixed_hires_30k/utt2spk) \\\n> > +     data/${train_set} data/${train_set}_30k\n> > +\n> > +  steps/align_fmllr.sh --nj 40 --cmd \"$train_cmd\" \\\n> > +    data/${train_set}_30k data/lang exp/tri6b exp/nnet3/tri6b_ali_30k\n> > +fi\n> > +\n> > +if [ $stage -le 5 ]; then\n> > +  # Train a small system just for its LDA+MLLT transform.  We use --num-iters 13\n> > +  # because after we get the transform (12th iter is the last), any further\n> > +  # training is pointless.\n> > +  steps/train_lda_mllt.sh --cmd \"$train_cmd\" --num-iters 13 \\\n> > +    --realign-iters \"\" \\\n> > +    --splice-opts \"--left-context=3 --right-context=3\" \\\n> > +    5000 10000 data/${train_set}_mixed_hires_30k data/lang \\\n>\n> OK. Thanks!\n>\n> —\n> You are receiving this because you commented.\n> Reply to this email directly or view it on GitHub\n> <https://github.com/kaldi-asr/kaldi/pull/645/files/9ea0ff3e250cec5ed8f22b1a837c17791398eaf6#r58445291>\n>\n","created_at":"2016-04-04T20:51:15Z","updated_at":"2016-04-04T20:51:15Z","html_url":"https://github.com/kaldi-asr/kaldi/pull/645#discussion_r58446557","pull_request_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645","_links":{"self":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments/58446557"},"html":{"href":"https://github.com/kaldi-asr/kaldi/pull/645#discussion_r58446557"},"pull_request":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645"}}},"pull_request":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645","id":65057223,"html_url":"https://github.com/kaldi-asr/kaldi/pull/645","diff_url":"https://github.com/kaldi-asr/kaldi/pull/645.diff","patch_url":"https://github.com/kaldi-asr/kaldi/pull/645.patch","issue_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/645","number":645,"state":"open","locked":false,"title":"WIP: Librispeech tdnn nnet3 and chain recipe","user":{"login":"freewym","id":3506322,"avatar_url":"https://avatars.githubusercontent.com/u/3506322?v=3","gravatar_id":"","url":"https://api.github.com/users/freewym","html_url":"https://github.com/freewym","followers_url":"https://api.github.com/users/freewym/followers","following_url":"https://api.github.com/users/freewym/following{/other_user}","gists_url":"https://api.github.com/users/freewym/gists{/gist_id}","starred_url":"https://api.github.com/users/freewym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freewym/subscriptions","organizations_url":"https://api.github.com/users/freewym/orgs","repos_url":"https://api.github.com/users/freewym/repos","events_url":"https://api.github.com/users/freewym/events{/privacy}","received_events_url":"https://api.github.com/users/freewym/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-04-03T02:55:36Z","updated_at":"2016-04-04T20:51:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"827710649e6243aa3a00475cf4f1c3f2733c95c6","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645/commits","review_comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645/comments","review_comment_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/645/comments","statuses_url":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/9ea0ff3e250cec5ed8f22b1a837c17791398eaf6","head":{"label":"freewym:librispeech","ref":"librispeech","sha":"9ea0ff3e250cec5ed8f22b1a837c17791398eaf6","user":{"login":"freewym","id":3506322,"avatar_url":"https://avatars.githubusercontent.com/u/3506322?v=3","gravatar_id":"","url":"https://api.github.com/users/freewym","html_url":"https://github.com/freewym","followers_url":"https://api.github.com/users/freewym/followers","following_url":"https://api.github.com/users/freewym/following{/other_user}","gists_url":"https://api.github.com/users/freewym/gists{/gist_id}","starred_url":"https://api.github.com/users/freewym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freewym/subscriptions","organizations_url":"https://api.github.com/users/freewym/orgs","repos_url":"https://api.github.com/users/freewym/repos","events_url":"https://api.github.com/users/freewym/events{/privacy}","received_events_url":"https://api.github.com/users/freewym/received_events","type":"User","site_admin":false},"repo":{"id":41891891,"name":"kaldi","full_name":"freewym/kaldi","owner":{"login":"freewym","id":3506322,"avatar_url":"https://avatars.githubusercontent.com/u/3506322?v=3","gravatar_id":"","url":"https://api.github.com/users/freewym","html_url":"https://github.com/freewym","followers_url":"https://api.github.com/users/freewym/followers","following_url":"https://api.github.com/users/freewym/following{/other_user}","gists_url":"https://api.github.com/users/freewym/gists{/gist_id}","starred_url":"https://api.github.com/users/freewym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freewym/subscriptions","organizations_url":"https://api.github.com/users/freewym/orgs","repos_url":"https://api.github.com/users/freewym/repos","events_url":"https://api.github.com/users/freewym/events{/privacy}","received_events_url":"https://api.github.com/users/freewym/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/freewym/kaldi","description":"This is now the official location of the Kaldi project.","fork":true,"url":"https://api.github.com/repos/freewym/kaldi","forks_url":"https://api.github.com/repos/freewym/kaldi/forks","keys_url":"https://api.github.com/repos/freewym/kaldi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/freewym/kaldi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/freewym/kaldi/teams","hooks_url":"https://api.github.com/repos/freewym/kaldi/hooks","issue_events_url":"https://api.github.com/repos/freewym/kaldi/issues/events{/number}","events_url":"https://api.github.com/repos/freewym/kaldi/events","assignees_url":"https://api.github.com/repos/freewym/kaldi/assignees{/user}","branches_url":"https://api.github.com/repos/freewym/kaldi/branches{/branch}","tags_url":"https://api.github.com/repos/freewym/kaldi/tags","blobs_url":"https://api.github.com/repos/freewym/kaldi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/freewym/kaldi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/freewym/kaldi/git/refs{/sha}","trees_url":"https://api.github.com/repos/freewym/kaldi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/freewym/kaldi/statuses/{sha}","languages_url":"https://api.github.com/repos/freewym/kaldi/languages","stargazers_url":"https://api.github.com/repos/freewym/kaldi/stargazers","contributors_url":"https://api.github.com/repos/freewym/kaldi/contributors","subscribers_url":"https://api.github.com/repos/freewym/kaldi/subscribers","subscription_url":"https://api.github.com/repos/freewym/kaldi/subscription","commits_url":"https://api.github.com/repos/freewym/kaldi/commits{/sha}","git_commits_url":"https://api.github.com/repos/freewym/kaldi/git/commits{/sha}","comments_url":"https://api.github.com/repos/freewym/kaldi/comments{/number}","issue_comment_url":"https://api.github.com/repos/freewym/kaldi/issues/comments{/number}","contents_url":"https://api.github.com/repos/freewym/kaldi/contents/{+path}","compare_url":"https://api.github.com/repos/freewym/kaldi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/freewym/kaldi/merges","archive_url":"https://api.github.com/repos/freewym/kaldi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/freewym/kaldi/downloads","issues_url":"https://api.github.com/repos/freewym/kaldi/issues{/number}","pulls_url":"https://api.github.com/repos/freewym/kaldi/pulls{/number}","milestones_url":"https://api.github.com/repos/freewym/kaldi/milestones{/number}","notifications_url":"https://api.github.com/repos/freewym/kaldi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/freewym/kaldi/labels{/name}","releases_url":"https://api.github.com/repos/freewym/kaldi/releases{/id}","deployments_url":"https://api.github.com/repos/freewym/kaldi/deployments","created_at":"2015-09-04T01:18:16Z","updated_at":"2016-01-12T19:28:02Z","pushed_at":"2016-04-04T16:03:28Z","git_url":"git://github.com/freewym/kaldi.git","ssh_url":"git@github.com:freewym/kaldi.git","clone_url":"https://github.com/freewym/kaldi.git","svn_url":"https://github.com/freewym/kaldi","homepage":"http://kaldi-asr.org","size":101155,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"base":{"label":"kaldi-asr:chain","ref":"chain","sha":"611ca84599e36d910118d928bf9917c57b509c7a","user":{"login":"kaldi-asr","id":11650082,"avatar_url":"https://avatars.githubusercontent.com/u/11650082?v=3","gravatar_id":"","url":"https://api.github.com/users/kaldi-asr","html_url":"https://github.com/kaldi-asr","followers_url":"https://api.github.com/users/kaldi-asr/followers","following_url":"https://api.github.com/users/kaldi-asr/following{/other_user}","gists_url":"https://api.github.com/users/kaldi-asr/gists{/gist_id}","starred_url":"https://api.github.com/users/kaldi-asr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaldi-asr/subscriptions","organizations_url":"https://api.github.com/users/kaldi-asr/orgs","repos_url":"https://api.github.com/users/kaldi-asr/repos","events_url":"https://api.github.com/users/kaldi-asr/events{/privacy}","received_events_url":"https://api.github.com/users/kaldi-asr/received_events","type":"Organization","site_admin":false},"repo":{"id":34275045,"name":"kaldi","full_name":"kaldi-asr/kaldi","owner":{"login":"kaldi-asr","id":11650082,"avatar_url":"https://avatars.githubusercontent.com/u/11650082?v=3","gravatar_id":"","url":"https://api.github.com/users/kaldi-asr","html_url":"https://github.com/kaldi-asr","followers_url":"https://api.github.com/users/kaldi-asr/followers","following_url":"https://api.github.com/users/kaldi-asr/following{/other_user}","gists_url":"https://api.github.com/users/kaldi-asr/gists{/gist_id}","starred_url":"https://api.github.com/users/kaldi-asr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaldi-asr/subscriptions","organizations_url":"https://api.github.com/users/kaldi-asr/orgs","repos_url":"https://api.github.com/users/kaldi-asr/repos","events_url":"https://api.github.com/users/kaldi-asr/events{/privacy}","received_events_url":"https://api.github.com/users/kaldi-asr/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kaldi-asr/kaldi","description":"This is now the official location of the Kaldi project.","fork":false,"url":"https://api.github.com/repos/kaldi-asr/kaldi","forks_url":"https://api.github.com/repos/kaldi-asr/kaldi/forks","keys_url":"https://api.github.com/repos/kaldi-asr/kaldi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kaldi-asr/kaldi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kaldi-asr/kaldi/teams","hooks_url":"https://api.github.com/repos/kaldi-asr/kaldi/hooks","issue_events_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/events{/number}","events_url":"https://api.github.com/repos/kaldi-asr/kaldi/events","assignees_url":"https://api.github.com/repos/kaldi-asr/kaldi/assignees{/user}","branches_url":"https://api.github.com/repos/kaldi-asr/kaldi/branches{/branch}","tags_url":"https://api.github.com/repos/kaldi-asr/kaldi/tags","blobs_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/refs{/sha}","trees_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/{sha}","languages_url":"https://api.github.com/repos/kaldi-asr/kaldi/languages","stargazers_url":"https://api.github.com/repos/kaldi-asr/kaldi/stargazers","contributors_url":"https://api.github.com/repos/kaldi-asr/kaldi/contributors","subscribers_url":"https://api.github.com/repos/kaldi-asr/kaldi/subscribers","subscription_url":"https://api.github.com/repos/kaldi-asr/kaldi/subscription","commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/commits{/sha}","git_commits_url":"https://api.github.com/repos/kaldi-asr/kaldi/git/commits{/sha}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/comments{/number}","issue_comment_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/comments{/number}","contents_url":"https://api.github.com/repos/kaldi-asr/kaldi/contents/{+path}","compare_url":"https://api.github.com/repos/kaldi-asr/kaldi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kaldi-asr/kaldi/merges","archive_url":"https://api.github.com/repos/kaldi-asr/kaldi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kaldi-asr/kaldi/downloads","issues_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues{/number}","pulls_url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls{/number}","milestones_url":"https://api.github.com/repos/kaldi-asr/kaldi/milestones{/number}","notifications_url":"https://api.github.com/repos/kaldi-asr/kaldi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kaldi-asr/kaldi/labels{/name}","releases_url":"https://api.github.com/repos/kaldi-asr/kaldi/releases{/id}","deployments_url":"https://api.github.com/repos/kaldi-asr/kaldi/deployments","created_at":"2015-04-20T17:23:16Z","updated_at":"2016-04-04T13:26:25Z","pushed_at":"2016-04-04T20:48:57Z","git_url":"git://github.com/kaldi-asr/kaldi.git","ssh_url":"git@github.com:kaldi-asr/kaldi.git","clone_url":"https://github.com/kaldi-asr/kaldi.git","svn_url":"https://github.com/kaldi-asr/kaldi","homepage":"http://kaldi-asr.org","size":115186,"stargazers_count":541,"watchers_count":541,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":327,"mirror_url":null,"open_issues_count":51,"forks":327,"open_issues":51,"watchers":541,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645"},"html":{"href":"https://github.com/kaldi-asr/kaldi/pull/645"},"issue":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/issues/645"},"comments":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/issues/645/comments"},"review_comments":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645/comments"},"review_comment":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/645/commits"},"statuses":{"href":"https://api.github.com/repos/kaldi-asr/kaldi/statuses/9ea0ff3e250cec5ed8f22b1a837c17791398eaf6"}}}}