{"action":"created","issue":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/487","repository_url":"https://api.github.com/repos/kaldi-asr/kaldi","labels_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/487/labels{/name}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/487/comments","events_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/487/events","html_url":"https://github.com/kaldi-asr/kaldi/pull/487","id":131592208,"number":487,"title":"Added a tool to compute the confidence interval of WER","user":{"login":"nsmartinez","id":1479074,"avatar_url":"https://avatars.githubusercontent.com/u/1479074?v=3","gravatar_id":"","url":"https://api.github.com/users/nsmartinez","html_url":"https://github.com/nsmartinez","followers_url":"https://api.github.com/users/nsmartinez/followers","following_url":"https://api.github.com/users/nsmartinez/following{/other_user}","gists_url":"https://api.github.com/users/nsmartinez/gists{/gist_id}","starred_url":"https://api.github.com/users/nsmartinez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nsmartinez/subscriptions","organizations_url":"https://api.github.com/users/nsmartinez/orgs","repos_url":"https://api.github.com/users/nsmartinez/repos","events_url":"https://api.github.com/users/nsmartinez/events{/privacy}","received_events_url":"https://api.github.com/users/nsmartinez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-02-05T09:13:18Z","updated_at":"2016-02-29T16:57:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/pulls/487","html_url":"https://github.com/kaldi-asr/kaldi/pull/487","diff_url":"https://github.com/kaldi-asr/kaldi/pull/487.diff","patch_url":"https://github.com/kaldi-asr/kaldi/pull/487.patch"},"body":"Just a slight modification of compute-wer to estimate the confidence interval of WER, so different system can be compared more reliably. It is based on the paper:\r\n\r\nhttp://www-i6.informatik.rwth-aachen.de/PostScript/InterneArbeiten/Bisani_BootstrapEstimatesForConfidenceIntervalsInASRPerformanceEvaluation_ICASSP_2004.pdf\r\n"},"comment":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/comments/190287348","html_url":"https://github.com/kaldi-asr/kaldi/pull/487#issuecomment-190287348","issue_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/487","id":190287348,"user":{"login":"jtrmal","id":11054175,"avatar_url":"https://avatars.githubusercontent.com/u/11054175?v=3","gravatar_id":"","url":"https://api.github.com/users/jtrmal","html_url":"https://github.com/jtrmal","followers_url":"https://api.github.com/users/jtrmal/followers","following_url":"https://api.github.com/users/jtrmal/following{/other_user}","gists_url":"https://api.github.com/users/jtrmal/gists{/gist_id}","starred_url":"https://api.github.com/users/jtrmal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jtrmal/subscriptions","organizations_url":"https://api.github.com/users/jtrmal/orgs","repos_url":"https://api.github.com/users/jtrmal/repos","events_url":"https://api.github.com/users/jtrmal/events{/privacy}","received_events_url":"https://api.github.com/users/jtrmal/received_events","type":"User","site_admin":false},"created_at":"2016-02-29T16:57:33Z","updated_at":"2016-02-29T16:57:33Z","body":"I think the code is in a good shape. Nicolas, thank you for your work.\n\nmy comments:\nI'd prefer to call the binary compute-wer-bootci (I saw bootci used several\ntimes to denote the \"boostrap conf intervals\", so I think it might be more\nself explaining -- compute-wer-confidence might perhaps cause incorrect\nassociation with the confidences of the individual words or something to\nthat extent.\nI'd also welcome extension of the steps/scoring_kaldi.sh to produce the\nconfidence intervals by default for the best system (for example a file in\nthe details/ directory).\n\nThere were two particular  minor issues which I think are not exactly right\n--please review if I'm right or not -- I might be just mistaken.\n\ny\n\n\nOn Sat, Feb 27, 2016 at 5:25 PM, Jan Trmal <jtrmal@gmail.com> wrote:\n\n> Yeah, I apologize, I shouldn't have forgotten about this. I'll get to this\n> ASAP. y.\n>\n> On Sat, Feb 27, 2016 at 5:18 PM, Daniel Povey <notifications@github.com>\n> wrote:\n>\n>> @jtrmal <https://github.com/jtrmal>, don't forget this! I know you are\n>> traveling..\n>>\n>> —\n>> Reply to this email directly or view it on GitHub\n>> <https://github.com/kaldi-asr/kaldi/pull/487#issuecomment-189738604>.\n>>\n>\n>\n"}}