{"action":"created","issue":{"url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition/issues/7","repository_url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition","labels_url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition/issues/7/comments","events_url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition/issues/7/events","html_url":"https://github.com/aatkinson/deep-named-entity-recognition/issues/7","id":340225666,"node_id":"MDU6SXNzdWUzNDAyMjU2NjY=","number":7,"title":"Memory overflow","user":{"login":"ptamas88","id":36891268,"node_id":"MDQ6VXNlcjM2ODkxMjY4","avatar_url":"https://avatars3.githubusercontent.com/u/36891268?v=4","gravatar_id":"","url":"https://api.github.com/users/ptamas88","html_url":"https://github.com/ptamas88","followers_url":"https://api.github.com/users/ptamas88/followers","following_url":"https://api.github.com/users/ptamas88/following{/other_user}","gists_url":"https://api.github.com/users/ptamas88/gists{/gist_id}","starred_url":"https://api.github.com/users/ptamas88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ptamas88/subscriptions","organizations_url":"https://api.github.com/users/ptamas88/orgs","repos_url":"https://api.github.com/users/ptamas88/repos","events_url":"https://api.github.com/users/ptamas88/events{/privacy}","received_events_url":"https://api.github.com/users/ptamas88/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-11T12:44:16Z","updated_at":"2018-07-19T10:51:03Z","closed_at":"2018-07-19T10:51:03Z","author_association":"NONE","body":"Hi!\r\n\r\n1.) I changed your wordvec.txt to a new one (trained with google wordvec, and 2.5 M tokens, 100 features).\r\n2.) Changed the hardcoded 300 values to 100.\r\n3.) Changed the annotated corpus to a new one with 9576 words in it.\r\n4.) Started ner_train.py\r\nI get massive memory overflow during this row in ner_train.py:\r\n``reader = DataUtil(WORDVEC_FILEPATH, TAGGED_NEWS_FILEPATH)``\r\nIn data_util.py this row generates a lot of memory (after 4000 rows in the raw_data it fills 120 GB of RAM (with swap)):\r\n``self.wordvecs = np.vstack((self.wordvecs, new_wv))``\r\nAlso getting this error when limiting the wordvec's number of rows to 100000 or 1000.\r\nCan you help me?"},"comment":{"url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition/issues/comments/406236127","html_url":"https://github.com/aatkinson/deep-named-entity-recognition/issues/7#issuecomment-406236127","issue_url":"https://api.github.com/repos/aatkinson/deep-named-entity-recognition/issues/7","id":406236127,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjIzNjEyNw==","user":{"login":"ptamas88","id":36891268,"node_id":"MDQ6VXNlcjM2ODkxMjY4","avatar_url":"https://avatars3.githubusercontent.com/u/36891268?v=4","gravatar_id":"","url":"https://api.github.com/users/ptamas88","html_url":"https://github.com/ptamas88","followers_url":"https://api.github.com/users/ptamas88/followers","following_url":"https://api.github.com/users/ptamas88/following{/other_user}","gists_url":"https://api.github.com/users/ptamas88/gists{/gist_id}","starred_url":"https://api.github.com/users/ptamas88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ptamas88/subscriptions","organizations_url":"https://api.github.com/users/ptamas88/orgs","repos_url":"https://api.github.com/users/ptamas88/repos","events_url":"https://api.github.com/users/ptamas88/events{/privacy}","received_events_url":"https://api.github.com/users/ptamas88/received_events","type":"User","site_admin":false},"created_at":"2018-07-19T10:51:03Z","updated_at":"2018-07-19T10:51:03Z","author_association":"NONE","body":"I solved this issue by changing `np.vstack` function to a primitive implementation:\r\nreplaced this:\r\n```\r\nself.wordvecs = np.vstack((self.wordvecs, new_wv))\r\n```\r\nwith this:\r\n```\r\nwv_new = []\r\nfor i in xrange(len(self.wordvecs)):\r\n  wv_new.append(self.wordvecs[i])\r\nwv_new.append(new_wv)\r\nself.wordvecs = wv_new\r\n```"}}
{"actor":{"display_login":"ptamas88"}}