{"action":"created","issue":{"url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN/issues/14","repository_url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN","labels_url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN/issues/14/comments","events_url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN/issues/14/events","html_url":"https://github.com/VT-vision-lab/VQA_LSTM_CNN/issues/14","id":138839734,"number":14,"title":"Ideas for NLP pre-processing and feature engineering","user":{"login":"honnibal","id":8059750,"avatar_url":"https://avatars.githubusercontent.com/u/8059750?v=3","gravatar_id":"","url":"https://api.github.com/users/honnibal","html_url":"https://github.com/honnibal","followers_url":"https://api.github.com/users/honnibal/followers","following_url":"https://api.github.com/users/honnibal/following{/other_user}","gists_url":"https://api.github.com/users/honnibal/gists{/gist_id}","starred_url":"https://api.github.com/users/honnibal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/honnibal/subscriptions","organizations_url":"https://api.github.com/users/honnibal/orgs","repos_url":"https://api.github.com/users/honnibal/repos","events_url":"https://api.github.com/users/honnibal/events{/privacy}","received_events_url":"https://api.github.com/users/honnibal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-03-06T21:18:46Z","updated_at":"2016-03-07T01:15:24Z","closed_at":null,"body":"Hi all,\r\n\r\nI'm excited to do some work on the text processing side of the Visual QA task. I develop the spaCy NLP library. I think we should be able to get some extra accuracy, with some extra NLP logic on the question parsing side. We'll see.\r\n\r\nThe first thing I'd like to try is mapping out of vocabulary words to similar tokens, using a word2vec model. For instance, let's say the word *colour* is OOV. Seems easy to map this to *color*.\r\n\r\n```\r\nInput: What colour is his shirt?\r\nTokens: [\"What\", \"colour\", \"is\", \"his\", \"shirt\", \"?\"]\r\nTransform: [\"What\", \"color\", \"is\", \"his\", \"shirt\", \"?\"]\r\n```\r\n\r\nI think this input normalization trick is novel, but it makes sense to me for this problem. It lets you exploit pre-trained vectors without interfering with the rest of your model choices.\r\n\r\nI think the normalization could be taken a bit further, by using the POS tagger and parser to compute context-specific keys, so that the replacement could be more exact ([sense2vec](https://spacy.io/blog/sense2vec-with-spacy)). I think just the word replacement is probably okay though.\r\n\r\nIt's also easy to calculate auxiliary features with spaCy. It's easy to train a question classifier, of course. I'm not sure the model is making many errors of that type, though.\r\n\r\nIf I had to say one thing was unsatisfying about the model, I'd say it's the multiclass classification output. Have you tried having the model output a vector, and using it to find a nearest neighbour?"},"comment":{"url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN/issues/comments/193031775","html_url":"https://github.com/VT-vision-lab/VQA_LSTM_CNN/issues/14#issuecomment-193031775","issue_url":"https://api.github.com/repos/VT-vision-lab/VQA_LSTM_CNN/issues/14","id":193031775,"user":{"login":"dhruvbatra","id":2941091,"avatar_url":"https://avatars.githubusercontent.com/u/2941091?v=3","gravatar_id":"","url":"https://api.github.com/users/dhruvbatra","html_url":"https://github.com/dhruvbatra","followers_url":"https://api.github.com/users/dhruvbatra/followers","following_url":"https://api.github.com/users/dhruvbatra/following{/other_user}","gists_url":"https://api.github.com/users/dhruvbatra/gists{/gist_id}","starred_url":"https://api.github.com/users/dhruvbatra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhruvbatra/subscriptions","organizations_url":"https://api.github.com/users/dhruvbatra/orgs","repos_url":"https://api.github.com/users/dhruvbatra/repos","events_url":"https://api.github.com/users/dhruvbatra/events{/privacy}","received_events_url":"https://api.github.com/users/dhruvbatra/received_events","type":"User","site_admin":false},"created_at":"2016-03-07T01:15:24Z","updated_at":"2016-03-07T01:15:24Z","body":"I like 4 the best. \r\n\r\n@jiasenlu @abhshkdz @dexter1691: Do you have a preference? "}}