{"action":"created","issue":{"url":"https://api.github.com/repos/RasaHQ/rasa_nlu/issues/1100","repository_url":"https://api.github.com/repos/RasaHQ/rasa_nlu","labels_url":"https://api.github.com/repos/RasaHQ/rasa_nlu/issues/1100/labels{/name}","comments_url":"https://api.github.com/repos/RasaHQ/rasa_nlu/issues/1100/comments","events_url":"https://api.github.com/repos/RasaHQ/rasa_nlu/issues/1100/events","html_url":"https://github.com/RasaHQ/rasa_nlu/issues/1100","id":325205968,"number":1100,"title":"interpreting random sequences of characters","user":{"login":"disimone","id":25104002,"avatar_url":"https://avatars1.githubusercontent.com/u/25104002?v=4","gravatar_id":"","url":"https://api.github.com/users/disimone","html_url":"https://github.com/disimone","followers_url":"https://api.github.com/users/disimone/followers","following_url":"https://api.github.com/users/disimone/following{/other_user}","gists_url":"https://api.github.com/users/disimone/gists{/gist_id}","starred_url":"https://api.github.com/users/disimone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/disimone/subscriptions","organizations_url":"https://api.github.com/users/disimone/orgs","repos_url":"https://api.github.com/users/disimone/repos","events_url":"https://api.github.com/users/disimone/events{/privacy}","received_events_url":"https://api.github.com/users/disimone/received_events","type":"User","site_admin":false},"labels":[{"id":461549562,"url":"https://api.github.com/repos/RasaHQ/rasa_nlu/labels/bug","name":"bug","color":"ee0701","default":true},{"id":461549564,"url":"https://api.github.com/repos/RasaHQ/rasa_nlu/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2018-05-22T09:14:10Z","updated_at":"2018-05-23T07:26:32Z","closed_at":null,"author_association":"NONE","body":"<!-- If you don't know your Rasa NLU version, here is some help: https://goo.gl/g9QQg2. If you are creating a feature request, feel free to remove all the system information stuff. --> \r\n\r\n**Rasa NLU version**:\r\ncommit 3d736227e3439ab745126263e8431e875b1882cd\r\n\r\n**Operating system** (windows, osx, ...):\r\nlinux\r\n\r\n**Issue**:\r\nHallo,\r\n\r\nI have trained a model (intent_featurizer_count_vectors+intent_classifier_tensorflow_embedding) on some German sentences.\r\n\r\nWhile testing the reaction to invalid input, I realized that the intepreter apparently \"understands\" also random sequences of characters. \"fgergq\" for example, is assigned an intent with fairly high confidence (65%). \"RFSSSw\" reaches 96%.\r\n\r\n\"osifhaskdnkauer\" is recognised as greet with 96% confidence, and this is particularly interesting since my \"greet\" training sentences are the ones you would expect (along the lines of \"Hallo\", \"Good *\"), and also mostly very short sentences.\r\n\r\nThe only way I found to actually get back an empty intent with zero confidence is to parse an empty string. Even single-characters never occurring in the training are interpreted: \"j\" gets intent \"bye\" with 80% confidence, while to be fair it would be a lot closer to \"ja\", one of my \"affirm\" examples.\r\n\r\nI have two questions:\r\n\r\n1) for my education, what is actually happening here? How does rasa_nlu+tensorflow_embedding evaluate the intent of a single word never seen before?\r\n\r\n2) how do I protect a chatbot from such things? Downstream, the Fallback policy in rasa_core uses the intent confidence to decide whether the user input is valid or not, but there is not much I can do against a 95% confidence from rasa_nlu, so I am guessing there is something I need to tune on the NLU side. Any suggestions?\r\n\r\nThanks for your help,\r\n\r\nAndrea.\r\n"},"comment":{"url":"https://api.github.com/repos/RasaHQ/rasa_nlu/issues/comments/391247670","html_url":"https://github.com/RasaHQ/rasa_nlu/issues/1100#issuecomment-391247670","issue_url":"https://api.github.com/repos/RasaHQ/rasa_nlu/issues/1100","id":391247670,"user":{"login":"Ghostvv","id":4136956,"avatar_url":"https://avatars1.githubusercontent.com/u/4136956?v=4","gravatar_id":"","url":"https://api.github.com/users/Ghostvv","html_url":"https://github.com/Ghostvv","followers_url":"https://api.github.com/users/Ghostvv/followers","following_url":"https://api.github.com/users/Ghostvv/following{/other_user}","gists_url":"https://api.github.com/users/Ghostvv/gists{/gist_id}","starred_url":"https://api.github.com/users/Ghostvv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ghostvv/subscriptions","organizations_url":"https://api.github.com/users/Ghostvv/orgs","repos_url":"https://api.github.com/users/Ghostvv/repos","events_url":"https://api.github.com/users/Ghostvv/events{/privacy}","received_events_url":"https://api.github.com/users/Ghostvv/received_events","type":"User","site_admin":false},"created_at":"2018-05-23T07:26:31Z","updated_at":"2018-05-23T07:26:31Z","author_association":"CONTRIBUTOR","body":"@disimone thank you for testing. Does it also work, when `spacy` is in the pipeline? We're going to merge it soon, though I cannot tell you precise timeline"}}
{"actor":{"display_login":"Ghostvv"}}