{"action":"created","issue":{"url":"https://api.github.com/repos/allenai/allennlp/issues/1823","repository_url":"https://api.github.com/repos/allenai/allennlp","labels_url":"https://api.github.com/repos/allenai/allennlp/issues/1823/labels{/name}","comments_url":"https://api.github.com/repos/allenai/allennlp/issues/1823/comments","events_url":"https://api.github.com/repos/allenai/allennlp/issues/1823/events","html_url":"https://github.com/allenai/allennlp/issues/1823","id":364028723,"node_id":"MDU6SXNzdWUzNjQwMjg3MjM=","number":1823,"title":"How to use ELMo for training in reading comprehension","user":{"login":"robbine","id":16010351,"node_id":"MDQ6VXNlcjE2MDEwMzUx","avatar_url":"https://avatars1.githubusercontent.com/u/16010351?v=4","gravatar_id":"","url":"https://api.github.com/users/robbine","html_url":"https://github.com/robbine","followers_url":"https://api.github.com/users/robbine/followers","following_url":"https://api.github.com/users/robbine/following{/other_user}","gists_url":"https://api.github.com/users/robbine/gists{/gist_id}","starred_url":"https://api.github.com/users/robbine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robbine/subscriptions","organizations_url":"https://api.github.com/users/robbine/orgs","repos_url":"https://api.github.com/users/robbine/repos","events_url":"https://api.github.com/users/robbine/events{/privacy}","received_events_url":"https://api.github.com/users/robbine/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-09-26T13:24:49Z","updated_at":"2018-09-28T17:27:30Z","closed_at":null,"author_association":"NONE","body":"I tried to introduce ELMo in reading comprehension. But a divide-by-zero error occurs. Here is my configuration. Basicly I followed those 3 steps in the ELMo tutorial.\r\n> \r\n      \"elmo\": { \r\n        \"type\": \"elmo_characters\"\r\n      },\r\n> \r\n      \"elmo\": {\r\n        \"type\": \"elmo_token_embedder\",\r\n        \"options_file\": \"https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json\",\r\n        \"weight_file\": \"https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5\",\r\n        \"do_layer_norm\": false,\r\n        \"dropout\": 0.5\r\n      },\r\n\r\n> \r\n    \"elmo_layer\": {\r\n      \"type\": \"alternating_highway_lstm_cuda\",\r\n      \"input_size\": 1224,\r\n      \"hidden_size\": 200,\r\n      \"num_layers\": 4,\r\n      \"recurrent_dropout_probability\": 0.1\r\n    },`\r\n\r\n\r\nI replace \"phrase_layer\" and \"highway_layer\" with \"elmo_layer\" in the following code:\r\n\r\n                 # embedded_question = self._highway_layer(self._text_field_embedder(question))\r\n\t\t# embedded_passage = self._highway_layer(self._text_field_embedder(passage))\r\n\t\tembedded_question = self._text_field_embedder(question).long()\r\n\t\tembedded_passage = self._text_field_embedder(passage).long()\r\n\t\tbatch_size = embedded_question.size(0)\r\n\t\tpassage_length = embedded_passage.size(1)\r\n\t\tquestion_mask = util.get_text_field_mask(question).float()\r\n\t\tpassage_mask = util.get_text_field_mask(passage).float()\r\n\t\tquestion_lstm_mask = question_mask if self._mask_lstms else None\r\n\t\tpassage_lstm_mask = passage_mask if self._mask_lstms else None\r\n\t\t# encoded_question = self._dropout(self._phrase_layer(embedded_question, question_lstm_mask))\r\n\t\t# encoded_passage = self._dropout(self._phrase_layer(embedded_passage, passage_lstm_mask))\r\n\t\tencoded_question = self._dropout(self._elmo_layer(embedded_question, question_lstm_mask))\r\n\t\tencoded_passage = self._dropout(self._elmo_layer(embedded_passage, passage_lstm_mask))\r\n\t\t"},"comment":{"url":"https://api.github.com/repos/allenai/allennlp/issues/comments/425508641","html_url":"https://github.com/allenai/allennlp/issues/1823#issuecomment-425508641","issue_url":"https://api.github.com/repos/allenai/allennlp/issues/1823","id":425508641,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTUwODY0MQ==","user":{"login":"matt-peters","id":619844,"node_id":"MDQ6VXNlcjYxOTg0NA==","avatar_url":"https://avatars1.githubusercontent.com/u/619844?v=4","gravatar_id":"","url":"https://api.github.com/users/matt-peters","html_url":"https://github.com/matt-peters","followers_url":"https://api.github.com/users/matt-peters/followers","following_url":"https://api.github.com/users/matt-peters/following{/other_user}","gists_url":"https://api.github.com/users/matt-peters/gists{/gist_id}","starred_url":"https://api.github.com/users/matt-peters/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matt-peters/subscriptions","organizations_url":"https://api.github.com/users/matt-peters/orgs","repos_url":"https://api.github.com/users/matt-peters/repos","events_url":"https://api.github.com/users/matt-peters/events{/privacy}","received_events_url":"https://api.github.com/users/matt-peters/received_events","type":"User","site_admin":false},"created_at":"2018-09-28T17:27:30Z","updated_at":"2018-09-28T17:27:30Z","author_association":"CONTRIBUTOR","body":"I suspect this is an OOM on the CPU -- usually GPU OOM displays a stack trace.  allennlp by default loads the entire indexed dataset into CPU memory and for larger datasets will require 10's of GB or more.  Try increasing the amount of allocated CPU memory to your container."}}
{"actor":{"display_login":"matt-peters"}}