{"action":"created","issue":{"url":"https://api.github.com/repos/syl22-00/pocketsphinx.js/issues/67","repository_url":"https://api.github.com/repos/syl22-00/pocketsphinx.js","labels_url":"https://api.github.com/repos/syl22-00/pocketsphinx.js/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/syl22-00/pocketsphinx.js/issues/67/comments","events_url":"https://api.github.com/repos/syl22-00/pocketsphinx.js/issues/67/events","html_url":"https://github.com/syl22-00/pocketsphinx.js/issues/67","id":166423146,"number":67,"title":"Getting accurate segmentation with audio files, multiple keyphrases","user":{"login":"DDews","id":9905864,"avatar_url":"https://avatars.githubusercontent.com/u/9905864?v=3","gravatar_id":"","url":"https://api.github.com/users/DDews","html_url":"https://github.com/DDews","followers_url":"https://api.github.com/users/DDews/followers","following_url":"https://api.github.com/users/DDews/following{/other_user}","gists_url":"https://api.github.com/users/DDews/gists{/gist_id}","starred_url":"https://api.github.com/users/DDews/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DDews/subscriptions","organizations_url":"https://api.github.com/users/DDews/orgs","repos_url":"https://api.github.com/users/DDews/repos","events_url":"https://api.github.com/users/DDews/events{/privacy}","received_events_url":"https://api.github.com/users/DDews/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-07-19T20:05:39Z","updated_at":"2016-07-19T20:31:19Z","closed_at":"2016-07-19T20:30:38Z","body":"Hello.\r\nI've made a webpage that lets you pipe in files and gets hypothesis on each file, telling you what it thinks the file said, and at what time it thinks each detected phrase starts at.\r\n\r\nFor context, I've compiled pocketsphinx js to detect multiple keyphrases.\r\n\r\nHowever, the segment that is returned is increased every time a new file is processed by the recognizer.\r\nHere is the code for the webpage:\r\n```\r\n<script src=\"callbackManager.js\"></script>\r\n<script type=\"text/javascript\" src=\"processors.js\"></script>\r\n<input type=\"file\" id=\"files\" name=\"files[]\" multiple />\r\n<div id=\"output\"></div>\r\n<script>\r\nvar inputElement = document.getElementById(\"files\");\r\nvar outputDiv = document.getElementById(\"output\");\r\ninputElement.style.display = \"none\";\r\nvar recognizer = new Worker(\"recognizer.js\");\r\nvar callbackManager = new CallbackManager();\r\n\r\nvar inputElement = document.getElementById(\"files\");\r\nvar reader1 = new FileReader();\r\nvar fileList, reader;\r\nfunction spawnWorker(workerURL, onReady) {\r\n  recognizer = new Worker(workerURL);\r\n\r\n  recognizer.onmessage = function(event) {\r\n    onReady(recognizer);\r\n  };\r\n\r\n  recognizer.postMessage('');\r\n}\r\nvar oldCount;\r\nfunction postRecognizerJob(message, callback) {\r\n  var msg = message || {};\r\n\r\n  if (callbackManager) {\r\n    msg.callbackId = callbackManager.add(callback);\r\n  }\r\n\r\n  if (recognizer) {\r\n    recognizer.postMessage(msg);\r\n  }\r\n}\r\nfunction initRecognizer() {\r\n  recognizer.postMessage({\r\n    command: 'load',\r\n    callbackId: callbackManager.add(\r\n        function() {\r\n\r\n          // We pass dictionary and keyword parameters to the recognizer\r\n          postRecognizerJob({\r\n          command: 'load',\r\n          data: ['data.js']\r\n        }, function() {\r\n            postRecognizerJob({\r\n            command: 'initialize',\r\n            data: [\r\n              ['-kws', 'keyphrase.list'],\r\n              ['-dict', 'keyphrase.dict'],\r\n              ['-hmm','en-us']\r\n              //['-remove_noise', 'yes'],\r\n            ]\r\n            }, function() {\r\n              inputElement.style.display = \"\";\r\n              console.log(\"ready\");\r\n            });\r\n        });\r\n\r\n        }),\r\n      data: [\r\n        '/keyphrase-list.js'\r\n      ]\r\n  });\r\n}\r\nvar mytimeout;\r\nspawnWorker('/recognizer.js', function(worker) {\r\n    // This is the onmessage function, once the worker is fully loaded\r\n    worker.onmessage = function(event) {\r\n      //console.log('event', event.data.hyp);\r\n\r\n      // This is the case when we have a callback id to be called\r\n      if (event.data.hasOwnProperty('id')) {\r\n        var callback = callbackManager.get(event.data['id']);\r\n        var data = {};\r\n\r\n        if (event.data.hasOwnProperty('data')) {\r\n          data = event.data.data;\r\n        }\r\n\r\n        if (callback) {\r\n          callback(data);\r\n        }\r\n      }\r\n\r\n      // This is a case when the recognizer has a new hypothesis\r\n      if (event.data.hasOwnProperty('hyp')) {\r\n        var newCount = event.data.hyp;\r\n        var seg = event.data.hypseg;\r\n        console.log(Object.getOwnPropertyNames(seg).length);\r\n        if (event.data.hasOwnProperty('final') &&  event.data.final) {\r\n          var o = Object.getOwnPropertyNames(seg);\r\n          newCount = \"(\" + o.length + \" phrases) \" + newCount;\r\n          var words;\r\n          newCount += \"<br/>Starting at (in seconds): \";\r\n          for (var i = 0; i < o.length; i++) {\r\n            newCount += seg[o[i]] + \", \";\r\n          }\r\n          newCount = newCount.substring(0,newCount.length - 3);\r\n          console.log(newCount);\r\n          outputDiv.innerHTML += newCount + \"<br/>\";\r\n        }\r\n        clearTimeout(mytimeout);\r\n        recognizer.postMessage({command: 'stop'});\r\n        mytimeout = setTimeout(() => { next_file(); }, 1000);\r\n      }\r\n\r\n      // This is the case when we have an error\r\n      if (event.data.hasOwnProperty('status') && (event.data.status === 'error')) {\r\n        //updateStatus('Error in ' + event.data.command + ' with code ' + event.data.code);\r\n      }\r\n    };\r\n\r\n    // Once the worker is fully loaded, we can call the initialize function\r\n    initRecognizer();\r\n  });\r\ninitRecognizer();\r\ninputElement.addEventListener(\"change\", handleFiles, false);\r\nvar buffer;\r\nvar blob;\r\nvar _i = 0;\r\nfunction next_file() {\r\n  recognizer.postMessage({command: 'start', data: null});\r\n  _i++;\r\n  if (_i < fileList.length) {\r\n    blob = fileList[_i].slice(0,fileList[_i].size);\r\n    reader.readAsArrayBuffer(blob);\r\n  }\r\n}\r\nfunction handleFiles(event) {\r\n    recognizer.postMessage({command: 'start', data: null});\r\n    fileList = event.target.files;\r\n    var start = 0;\r\n    _i = 0;\r\n    reader = new FileReader();\r\n\r\n    // If we use onloadend, we need to check the readyState.\r\n    reader.onloadend = function(evt) {\r\n      if (evt.target.readyState == FileReader.DONE) { // DONE == 2\r\n        //console.log(evt);\r\n        /*buffer[_i / 2] = evt.target.result;\r\n        _i += 2;\r\n        if (_i <= file.size) {\r\n            blob = file.slice(_i,Math.min(_i + 1, file.size));\r\n            reader.readAsArrayBuffer(blob);\r\n        } else {\r\n            recognizer.postMessage({ command: 'process', data: buffer});\r\n            recognizer.postMessage({ command: 'stop'});\r\n        }*/\r\n        buffer = new Int16Array(evt.target.result);\r\n        console.log(fileList[_i].name + \":\");\r\n        outputDiv.innerHTML += \"<br/> \" + fileList[_i].name + \":<br/>\";\r\n        recognizer.postMessage({command: 'process', data: buffer});\r\n      }\r\n    };\r\n    blob = fileList[0].slice(0,fileList[0].size);\r\n    reader.readAsArrayBuffer(blob);\r\n    // Gets the final recognized string:\r\n}\r\n</script>\r\n```\r\nAnd in the recognizer.js, I've modified the stop function:\r\n```\r\nfunction stop() {\r\n  if (recognizer) {\r\n    var output = recognizer.stop();\r\n    if (output !== Module.ReturnType.SUCCESS) {\r\n      post({\r\n        status: 'error',\r\n        command: 'stop',\r\n        code: output\r\n      });\r\n    } else {\r\n      var mynum = 0;\r\n      if (recognizer.getHypseg(segmentation) === Module.ReturnType.SUCCESS) {\r\n        var seg = segmentation;\r\n        var numWords = {};\r\n        for (var i= 0; i < seg.size(); i++) {\r\n          var segItem = seg.get(i);\r\n          var s = segItem.start / 100;\r\n          var original = numWords[segItem.end] || 0;\r\n          if (s < original || original === 0) numWords[segItem.end] = s;\r\n        }\r\n        mynum = numWords;\r\n      }\r\n\r\n      post({\r\n        hyp: recognizer.getHyp(),\r\n        hypseg: mynum,\r\n        final: true\r\n      });\r\n    }\r\n  } else {\r\n    post({\r\n      status: 'error',\r\n      command: 'stop',\r\n      code: 'js-no-recognizer'\r\n    });\r\n  }\r\n}\r\n```\r\n\r\nAs you can see, the stop function gets the hypseg at the end of a recording, and for each matched keyphrase it grabs its starting segment, divides it by 100 since to get the time in seconds, then makes an object with this structure:\r\n```\r\n{\r\nend-segment1: earliest start-segment with this end-segment in seconds,\r\nend-segment2: earliest start-segment with this end-segment in seconds\r\n}\r\n```\r\nThis lets me get an idea of the number of phrases matched. If multiple keyphrases are detected for the same stretch of audio, they will all share the same end-segment. So multiple different end-segments means multiple keyphrases matched (most likely).\r\n\r\nHowever, the start-segment in seconds is increasing for every new file read and processed by the recognizer:\r\n```\r\njoinfoxden1.wav:\r\n(1 phrases) YO FOXDEN HEY FOXDEN \r\nStarting at (in seconds): 0.9\r\n\r\njoinfoxden2.wav:\r\n(0 phrases) \r\nStarting at (in seconds\r\n\r\njoinfoxden3.wav:\r\n(1 phrases) SAY FOXDEN MY FOXDEN HI FOXDEN THE FOXDEN OKAY FOXDEN A FOXDEN YO FOXDEN HEY FOXDEN \r\nStarting at (in seconds): 6.\r\n\r\njoinfoxdendan1.wav:\r\n(1 phrases) OKAY FOXDEN A FOXDEN YO FOXDEN HEY FOXDEN \r\nStarting at (in seconds): 9.4\r\n\r\njoinfoxdendan2.wav:\r\n(1 phrases) A FOXDEN OKAY FOXDEN HEY FOXDEN YO FOXDEN \r\nStarting at (in seconds): 13.0\r\n\r\njoinfoxdendan3.wav:\r\n(1 phrases) SAY FOXDEN MY FOXDEN HELLO FOXDEN THE FOXDEN HI FOXDEN A FOXDEN YO FOXDEN OKAY FOXDEN HEY FOXDEN \r\nStarting at (in seconds): 15.8\r\n\r\njoinfoxdennick1.wav:\r\n(1 phrases) MY FOXDEN THE FOXDEN OKAY FOXDEN A FOXDEN YO FOXDEN HEY FOXDEN \r\nStarting at (in seconds): 18.7\r\n\r\njoinfoxdennick2.wav:\r\n(1 phrases) SAY FOXDEN HI FOXDEN MY FOXDEN OKAY FOXDEN HELLO FOXDEN THE FOXDEN A FOXDEN YO FOXDEN HEY FOXDEN \r\nStarting at (in seconds): 22.2\r\n```\r\n\r\nThese audio files are all about 4-6 seconds long. So starting at \"22.2\" makes no sense. I believe its not re-initializing the segmentation object. So I had it do segmentation.delete(); segmentation = new Module.Segmentation();\r\nand it still gave me the same results.\r\n\r\nDoes anyone know a way to reset the segment number for each hypothesis session?\r\n\r\nWhat I want is the segment number to be reset to 0 after the stop command is issued to the recognizer. That way I can get an accurate timestamp of when pocketsphinx js thinks the first keyphrase is detected in seconds.\r\n"},"comment":{"url":"https://api.github.com/repos/syl22-00/pocketsphinx.js/issues/comments/233756326","html_url":"https://github.com/syl22-00/pocketsphinx.js/issues/67#issuecomment-233756326","issue_url":"https://api.github.com/repos/syl22-00/pocketsphinx.js/issues/67","id":233756326,"user":{"login":"nshmyrev","id":2886672,"avatar_url":"https://avatars.githubusercontent.com/u/2886672?v=3","gravatar_id":"","url":"https://api.github.com/users/nshmyrev","html_url":"https://github.com/nshmyrev","followers_url":"https://api.github.com/users/nshmyrev/followers","following_url":"https://api.github.com/users/nshmyrev/following{/other_user}","gists_url":"https://api.github.com/users/nshmyrev/gists{/gist_id}","starred_url":"https://api.github.com/users/nshmyrev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nshmyrev/subscriptions","organizations_url":"https://api.github.com/users/nshmyrev/orgs","repos_url":"https://api.github.com/users/nshmyrev/repos","events_url":"https://api.github.com/users/nshmyrev/events{/privacy}","received_events_url":"https://api.github.com/users/nshmyrev/received_events","type":"User","site_admin":false},"created_at":"2016-07-19T20:31:19Z","updated_at":"2016-07-19T20:31:19Z","body":"Ok, thanks. We target stream processing in ps, so you could have multiple utterances but single stream and you get timings from a stream, not inside the utterance."}}
{"actor":{"display_login":"nshmyrev"}}