{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/issues/137","repository_url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices","labels_url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/issues/137/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/issues/137/comments","events_url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/issues/137/events","html_url":"https://github.com/Microsoft/BotBuilder-CognitiveServices/issues/137","id":359365199,"node_id":"MDU6SXNzdWUzNTkzNjUxOTk=","number":137,"title":"[Node] qnaThreshold parameter","user":{"login":"afnan-ahmad","id":34241491,"node_id":"MDQ6VXNlcjM0MjQxNDkx","avatar_url":"https://avatars2.githubusercontent.com/u/34241491?v=4","gravatar_id":"","url":"https://api.github.com/users/afnan-ahmad","html_url":"https://github.com/afnan-ahmad","followers_url":"https://api.github.com/users/afnan-ahmad/followers","following_url":"https://api.github.com/users/afnan-ahmad/following{/other_user}","gists_url":"https://api.github.com/users/afnan-ahmad/gists{/gist_id}","starred_url":"https://api.github.com/users/afnan-ahmad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/afnan-ahmad/subscriptions","organizations_url":"https://api.github.com/users/afnan-ahmad/orgs","repos_url":"https://api.github.com/users/afnan-ahmad/repos","events_url":"https://api.github.com/users/afnan-ahmad/events{/privacy}","received_events_url":"https://api.github.com/users/afnan-ahmad/received_events","type":"User","site_admin":false},"labels":[{"id":968409986,"node_id":"MDU6TGFiZWw5Njg0MDk5ODY=","url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/labels/Node.js%20SDK","name":"Node.js SDK","color":"8eafe2","default":false}],"state":"open","locked":false,"assignee":{"login":"stevkan","id":18403946,"node_id":"MDQ6VXNlcjE4NDAzOTQ2","avatar_url":"https://avatars2.githubusercontent.com/u/18403946?v=4","gravatar_id":"","url":"https://api.github.com/users/stevkan","html_url":"https://github.com/stevkan","followers_url":"https://api.github.com/users/stevkan/followers","following_url":"https://api.github.com/users/stevkan/following{/other_user}","gists_url":"https://api.github.com/users/stevkan/gists{/gist_id}","starred_url":"https://api.github.com/users/stevkan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevkan/subscriptions","organizations_url":"https://api.github.com/users/stevkan/orgs","repos_url":"https://api.github.com/users/stevkan/repos","events_url":"https://api.github.com/users/stevkan/events{/privacy}","received_events_url":"https://api.github.com/users/stevkan/received_events","type":"User","site_admin":false},"assignees":[{"login":"stevkan","id":18403946,"node_id":"MDQ6VXNlcjE4NDAzOTQ2","avatar_url":"https://avatars2.githubusercontent.com/u/18403946?v=4","gravatar_id":"","url":"https://api.github.com/users/stevkan","html_url":"https://github.com/stevkan","followers_url":"https://api.github.com/users/stevkan/followers","following_url":"https://api.github.com/users/stevkan/following{/other_user}","gists_url":"https://api.github.com/users/stevkan/gists{/gist_id}","starred_url":"https://api.github.com/users/stevkan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevkan/subscriptions","organizations_url":"https://api.github.com/users/stevkan/orgs","repos_url":"https://api.github.com/users/stevkan/repos","events_url":"https://api.github.com/users/stevkan/events{/privacy}","received_events_url":"https://api.github.com/users/stevkan/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-09-12T08:17:30Z","updated_at":"2018-10-17T19:09:42Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nThe **qnaThreshold** parameter specified in the **IQnAMakerOptions** interface does not seem to have any effect on the recognizer. See below for detailed explanation.\r\n\r\n**Example:**\r\nHere I have configured a **QnAMakerRecognizer** with a simple intent dialog and have specified the value of 0.6 in **qnaThreshold** parameter in the options. \r\n```js\r\nvar bot = new builder.UniversalBot(connector);\r\nbot.set(\"storage\", new builder.MemoryBotStorage());\r\n\r\nvar intentRecognizer = new builder.LuisRecognizer(constants.modelUrl);\r\n\r\nvar qnaRecognizer = new cognitiveservices.QnAMakerRecognizer({\r\n  knowledgeBaseId: constants.knowledgeBaseId,\r\n  endpointHostName: constants.qnaEndpoint,\r\n  authKey: constants.qnaAuthKey,\r\n  top: 3,\r\n  qnaThreshold: 0.6\r\n});\r\n\r\nvar intents = new builder.IntentDialog({\r\n  recognizers: [intentRecognizer, qnaRecognizer]\r\n})\r\n  .matches(\"qna\", (session, args) => {\r\n    console.log(args);\r\n    var answer = builder.EntityRecognizer.findEntity(args.entities, \"answer\");\r\n    session.endDialog(answer.entity);\r\n  })\r\n  .onDefault(session => {\r\n    session.endDialog(\"This is the default dialog.\");\r\n  });\r\n\r\nbot.dialog(\"/\", intents);\r\n```\r\n\r\n**Actual Response:**\r\nQnA Maker is also returning answers which are of lower scores than the specified **qnaThreshold**\r\n```js\r\nrestify listening to http://[::]:3978\r\nChatConnector: message received.\r\nUniversalBot(\"*\") routing \"windows\" from \"emulator\"\r\nSession.beginDialog(/)\r\n/ - Session.sendBatch() sending 0 message(s)\r\n/ - IntentDialog.matches(qna)\r\n/ - waterfall() step 1 of 1\r\n{ answers: \r\n   [ { questions: [Array],\r\n       answer: '... omitted ...',\r\n       score: 0.5489,\r\n       id: 26,\r\n       source: 'ChatQnA.pdf',\r\n       metadata: [] },\r\n     { questions: [Array],\r\n       answer: '... omitted ...',\r\n       score: 0.5162,\r\n       id: 13,\r\n       source: 'ChatQnA.pdf',\r\n       metadata: [] },\r\n     { questions: [Array],\r\n       answer: '... omitted ...',\r\n       score: 0.4851,\r\n       id: 14,\r\n       source: 'ChatQnA.pdf',\r\n       metadata: [] } ],\r\n  score: 0.5489,\r\n  entities: \r\n   [ ... omitted ... ],\r\n  intent: 'qna' }\r\n/ - Session.endDialog()\r\nSession.sendBatch() sending 1 message(s)\r\n```\r\n**Expected Response:**\r\nQnA Maker should have ignored answers which scored below specified **qnaThreshold**. (Which is 0.6 in this example)\r\n\r\n**Possible Causes:**\r\nIt seems to me that, the parameter **qnaThreshold** in the **IQnAMakerOptions** interface is not being used in the actual request or anywhere else aside from being declared as an option. So, it is obvious that it has no effect on the responses being returned from QnA Maker.\r\n\r\n**Solutions:**\r\nWe should send the threshold parameter in the request to QnA Maker's **generateAnswer** endpoint if it is supported on the server side (it should be). Since it is basically the same way **top** works, and we'll avoid downloading answers that are not required, thus saving data and time. \r\n\r\nFor this, I tried adding **\"qnaThreshold\"** parameter to the raw request and sent it through Postman. It is still returning the same answers, below threshold. So, maybe filtering based on threshold is not supported in the QnA Maker API v4.0 GA. \r\n\r\nSee: \r\n- https://docs.microsoft.com/en-us/azure/cognitive-services/qnamaker/how-to/metadata-generateanswer-usage \r\n\r\n- https://docs.microsoft.com/en-us/azure/cognitive-services/qnamaker/quickstarts/nodejs#GetAnswers\r\n\r\nAlso, the **strictFilters** parameter in the API is only for using with metadata, so can't filter score through it.\r\n\r\nAside from few small descriptions at the above links, I can't seem to find any docs about the REST spec of the Endpoint (Runtime) API. \r\n\r\nThere are docs for v4 API Reference (https://westus.dev.cognitive.microsoft.com/docs/services/5a93fcf85b4ccd136866eb37/operations/5ac266295b4ccd1554da75ff), but it seems it's only for Management APIs and there is no information about the **generateAnswer** operation.\r\n\r\nI would request that it should be documented clearly whether it is supported on the server side to filter answers based on threshold. And the client libraries should be updated to reflect that. \r\n\r\n**Possible Workarounds:**\r\nIf due to any reason, it is not supported (yet) on the server side then, either the client libraries should filter the answers (for the time being, maybe based on a flag) on the client side after getting the response from QnA Maker (since the parameter is already specified in the **IQnAMakerOptions** interface).\r\n\r\nUntil we get the parameter working on the server side, we can use some kind of wrapper around the recognizer to filter out the answers that scored below the specified threshold before returning the response to the dialog.\r\n\r\nPlease look into this and update as this is an important feature and it is required in many scenarios.\r\n\r\n**EDIT:** It appears that the .NET version of the SDK does actually filter the answers on the client side. (See: https://github.com/Microsoft/BotBuilder-CognitiveServices/blob/master/CSharp/Library/QnAMaker/QnAMaker/QnAMakerService/QnAMakerService.cs#L196). So I guess it would be fine to implement the same in the Node.js SDK as well. I would be happy to submit a PR once I get confirmation.\r\n\r\n**Thanks,**\r\nAfnan\r\n"},"comment":{"url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/issues/comments/430752074","html_url":"https://github.com/Microsoft/BotBuilder-CognitiveServices/issues/137#issuecomment-430752074","issue_url":"https://api.github.com/repos/Microsoft/BotBuilder-CognitiveServices/issues/137","id":430752074,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMDc1MjA3NA==","user":{"login":"stevkan","id":18403946,"node_id":"MDQ6VXNlcjE4NDAzOTQ2","avatar_url":"https://avatars2.githubusercontent.com/u/18403946?v=4","gravatar_id":"","url":"https://api.github.com/users/stevkan","html_url":"https://github.com/stevkan","followers_url":"https://api.github.com/users/stevkan/followers","following_url":"https://api.github.com/users/stevkan/following{/other_user}","gists_url":"https://api.github.com/users/stevkan/gists{/gist_id}","starred_url":"https://api.github.com/users/stevkan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevkan/subscriptions","organizations_url":"https://api.github.com/users/stevkan/orgs","repos_url":"https://api.github.com/users/stevkan/repos","events_url":"https://api.github.com/users/stevkan/events{/privacy}","received_events_url":"https://api.github.com/users/stevkan/received_events","type":"User","site_admin":false},"created_at":"2018-10-17T19:09:42Z","updated_at":"2018-10-17T19:09:42Z","author_association":"CONTRIBUTOR","body":"You are right that IQnAMakerOptions is used in both QnAMakerRecognizer and QnAMakerDialog. Forgive my oversight on that. However, your code above shows you are trying to apply qnaRecognizer to IntentDialog which **does not** accept the qnaMaker recognizer options.\r\n\r\nYou can reference the IntentDialog class [here](https://github.com/Microsoft/BotBuilder/blob/7b70cfd/Node/core/lib/botbuilder.d.ts#L3448) and its associated options [here](https://github.com/Microsoft/BotBuilder/blob/7b70cfd/Node/core/lib/botbuilder.d.ts#L1011)."}}
{"actor":{"display_login":"stevkan"}}