{"action":"created","issue":{"url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch/issues/32","repository_url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch","labels_url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch/issues/32/comments","events_url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch/issues/32/events","html_url":"https://github.com/LuminosoInsight/conceptnet-numberbatch/issues/32","id":157280721,"number":32,"title":"Large number of zero vectors","user":{"login":"shirish93","id":3619659,"avatar_url":"https://avatars.githubusercontent.com/u/3619659?v=3","gravatar_id":"","url":"https://api.github.com/users/shirish93","html_url":"https://github.com/shirish93","followers_url":"https://api.github.com/users/shirish93/followers","following_url":"https://api.github.com/users/shirish93/following{/other_user}","gists_url":"https://api.github.com/users/shirish93/gists{/gist_id}","starred_url":"https://api.github.com/users/shirish93/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shirish93/subscriptions","organizations_url":"https://api.github.com/users/shirish93/orgs","repos_url":"https://api.github.com/users/shirish93/repos","events_url":"https://api.github.com/users/shirish93/events{/privacy}","received_events_url":"https://api.github.com/users/shirish93/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-27T20:01:09Z","updated_at":"2016-05-31T17:57:45Z","closed_at":null,"body":"Hello,\r\n\r\nThis could be the case with my processing, but it appears that 617, 129 out of the 665, 494 english vectors are zero vectors: they are defined in the label, but have all zeros (ie, there are only 48, 365 non-zero vectors for English). I discovered this with the 300-sized dataset. Might this be an issue with the uploaded dataset, or should I recheck my methodology? If you could confirm this is _not_ the issue on your side using the dataset available for download, I can work on fixing on my side.\r\n\r\nFor reference, this is the code I used to count empty vectors:\r\n```\r\nempty = np.zeros(300)\r\ncount = 0\r\nfor each in englishVectors:\r\n if np.array_equal(each, empty):\r\n  count +=1\r\n```\r\nI discovered this while trying to figure out the words closest to semi-common words.\r\n\r\nFor reference, using your code for 'most similar', the words that seem to be representative of the 'zero vectors' are the following:\r\n\r\n```\r\n['adddresse', 'rudat', 'barhydt', 'weeked', 'inovonics', 'alleppey', 'katten', 'georgievski', 'kopinski', 'waxwing', 'irin_plusnews']\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch/issues/comments/222768736","html_url":"https://github.com/LuminosoInsight/conceptnet-numberbatch/issues/32#issuecomment-222768736","issue_url":"https://api.github.com/repos/LuminosoInsight/conceptnet-numberbatch/issues/32","id":222768736,"user":{"login":"rspeer","id":234227,"avatar_url":"https://avatars.githubusercontent.com/u/234227?v=3","gravatar_id":"","url":"https://api.github.com/users/rspeer","html_url":"https://github.com/rspeer","followers_url":"https://api.github.com/users/rspeer/followers","following_url":"https://api.github.com/users/rspeer/following{/other_user}","gists_url":"https://api.github.com/users/rspeer/gists{/gist_id}","starred_url":"https://api.github.com/users/rspeer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rspeer/subscriptions","organizations_url":"https://api.github.com/users/rspeer/orgs","repos_url":"https://api.github.com/users/rspeer/repos","events_url":"https://api.github.com/users/rspeer/events{/privacy}","received_events_url":"https://api.github.com/users/rspeer/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T17:57:45Z","updated_at":"2016-05-31T17:57:45Z","body":"I just re-downloaded the 600d dataset and, while there are zero-vectors, there are only 5882 of them.\r\n\r\nThis narrows it down: either I made a mistake truncating the 600d vectors to 300d, and you downloaded the 300d version; or you made a mistake in post-processing the data. Can you tell me more specifically what you did?"}}
{"actor":{"display_login":"rspeer"}}