{"action":"opened","issue":{"url":"https://api.github.com/repos/allenai/allennlp/issues/1411","repository_url":"https://api.github.com/repos/allenai/allennlp","labels_url":"https://api.github.com/repos/allenai/allennlp/issues/1411/labels{/name}","comments_url":"https://api.github.com/repos/allenai/allennlp/issues/1411/comments","events_url":"https://api.github.com/repos/allenai/allennlp/issues/1411/events","html_url":"https://github.com/allenai/allennlp/issues/1411","id":334964353,"node_id":"MDU6SXNzdWUzMzQ5NjQzNTM=","number":1411,"title":"How do add Elmo to BiDAF?","user":{"login":"dhairyadalal","id":22040959,"node_id":"MDQ6VXNlcjIyMDQwOTU5","avatar_url":"https://avatars3.githubusercontent.com/u/22040959?v=4","gravatar_id":"","url":"https://api.github.com/users/dhairyadalal","html_url":"https://github.com/dhairyadalal","followers_url":"https://api.github.com/users/dhairyadalal/followers","following_url":"https://api.github.com/users/dhairyadalal/following{/other_user}","gists_url":"https://api.github.com/users/dhairyadalal/gists{/gist_id}","starred_url":"https://api.github.com/users/dhairyadalal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhairyadalal/subscriptions","organizations_url":"https://api.github.com/users/dhairyadalal/orgs","repos_url":"https://api.github.com/users/dhairyadalal/repos","events_url":"https://api.github.com/users/dhairyadalal/events{/privacy}","received_events_url":"https://api.github.com/users/dhairyadalal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-22T17:46:30Z","updated_at":"2018-06-22T17:46:30Z","closed_at":null,"author_association":"NONE","body":"So I'm trying add ELMO to the allennlp implementation of BiDAF and running into issues getting the dimensions right in the config file. I was following the instructions on adding elmo to Allennlp models (https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md).\r\n\r\nI updated the text_field_embedder and dataset readers as described. Next I updated the various layers impacted by the additional input of the elmo embeddings.\r\n\r\nPhrase Layer\r\n```\r\n\"num_highway_layers\": 2,\r\n    \"phrase_layer\": {\r\n      \"type\": \"lstm\",\r\n      \"bidirectional\": true,\r\n      \"input_size\": 1224,\r\n      \"hidden_size\": 100,\r\n      \"num_layers\": 1,\r\n      \"dropout\": 0.2\r\n    }\r\n```\r\nSimilarity Function\r\n```\r\n    \"similarity_function\": {\r\n      \"type\": \"linear\",\r\n      \"combination\": \"x,y,x*y\",\r\n      \"tensor_1_dim\": 1224,\r\n      \"tensor_2_dim\": 1224\r\n    }\r\n```\r\nSpan End Encoder\r\n```\r\n \"span_end_encoder\": {\r\n      \"type\": \"lstm\",\r\n      \"bidirectional\": true,\r\n      \"input_size\": 8568,\r\n      \"hidden_size\": 100,\r\n      \"num_layers\": 1,\r\n      \"dropout\": 0.2\r\n    },\r\n    \"dropout\": 0.2\r\n  }\r\n```\r\nWhen I try to train the model on the is config, I get \r\n```\r\nallennlp.common.checks.ConfigurationError: 'span end encoder input dim must match 4 * enco│\r\nding dim + 3 * modeling dim, but got 8568 and 1400 instead'         \r\n```\r\n\r\nI'm a bit confused as to where the 1400 is coming from. I  updated the input to reflect that is 4 * (1224) + 3 *(1224) => 8568. Not where that second number is comining from and how to update the config to get it to match up correctly. \r\n\r\nHere's my full config file:\r\n```\r\n{\r\n  \"dataset_reader\": {\r\n    \"type\": \"squad\",\r\n    \"token_indexers\": {\r\n      \"tokens\": {\r\n        \"type\": \"single_id\",\r\n        \"lowercase_tokens\": true\r\n      },\r\n      \"elmo\": {\r\n          \"type\": \"elmo_characters\"\r\n      },\r\n      \"token_characters\": {\r\n        \"type\": \"characters\",\r\n        \"character_tokenizer\": {\r\n          \"byte_encoding\": \"utf-8\",\r\n          \"start_tokens\": [259],\r\n          \"end_tokens\": [260]\r\n        }\r\n      }\r\n    }\r\n  },\r\n  \"train_data_path\": \"lite_train.json\",\r\n  \"validation_data_path\": \"lite_dev.json\",\r\n  \"model\": {\r\n    \"type\": \"bidaf\",\r\n    \"text_field_embedder\": {\r\n      \"tokens\": {\r\n        \"type\": \"embedding\",\r\n        \"pretrained_file\": \"glove.gz\",\r\n        \"embedding_dim\": 100,\r\n        \"trainable\": false\r\n      },\r\n      \"elmo\": {\r\n          \"type\": \"elmo_token_embedder\",\r\n          \"options_file\": \"elmo_options.json\",\r\n          \"weight_file\": \"elmo_weights.hdf5\",\r\n          \"do_layer_norm\": false,\r\n          \"dropout\": 0.5\r\n      },\r\n      \"token_characters\": {\r\n        \"type\": \"character_encoding\",\r\n        \"embedding\": {\r\n          \"num_embeddings\": 262,\r\n          \"embedding_dim\": 16\r\n        },\r\n        \"encoder\": {\r\n          \"type\": \"cnn\",\r\n          \"embedding_dim\": 16,\r\n          \"num_filters\": 100,\r\n          \"ngram_filter_sizes\": [5]\r\n        },\r\n        \"dropout\": 0.2\r\n      }\r\n    },\r\n    \"num_highway_layers\": 2,\r\n    \"phrase_layer\": {\r\n      \"type\": \"lstm\",\r\n      \"bidirectional\": true,\r\n      \"input_size\": 1224,\r\n      \"hidden_size\": 100,\r\n      \"num_layers\": 1,\r\n      \"dropout\": 0.2\r\n    },\r\n    \"similarity_function\": {\r\n      \"type\": \"linear\",\r\n      \"combination\": \"x,y,x*y\",\r\n      \"tensor_1_dim\": 1224,\r\n      \"tensor_2_dim\": 1224\r\n    },\r\n    \"modeling_layer\": {\r\n      \"type\": \"lstm\",\r\n      \"bidirectional\": true,\r\n      \"input_size\": 800,\r\n      \"hidden_size\": 100,\r\n      \"num_layers\": 2,\r\n      \"dropout\": 0.2\r\n    },\r\n    \"span_end_encoder\": {\r\n      \"type\": \"lstm\",\r\n      \"bidirectional\": true,\r\n      \"input_size\": 8568,\r\n      \"hidden_size\": 100,\r\n      \"num_layers\": 1,\r\n      \"dropout\": 0.2\r\n    },\r\n    \"dropout\": 0.2\r\n  },\r\n  \"iterator\": {\r\n    \"type\": \"bucket\",\r\n    \"sorting_keys\": [[\"passage\", \"num_tokens\"], [\"question\", \"num_tokens\"]],\r\n    \"batch_size\": 40\r\n  },\r\n\r\n  \"trainer\": {\r\n    \"num_epochs\": 20,\r\n    \"grad_norm\": 5.0,\r\n    \"patience\": 10,\r\n    \"validation_metric\": \"+em\",\r\n    \"cuda_device\": 0,\r\n    \"learning_rate_scheduler\": {\r\n      \"type\": \"reduce_on_plateau\",\r\n      \"factor\": 0.5,\r\n      \"mode\": \"max\",\r\n      \"patience\": 2\r\n    },\r\n    \"optimizer\": {\r\n      \"type\": \"adam\",\r\n      \"betas\": [0.9, 0.9]\r\n    }\r\n  }\r\n}\r\n```"}}
{"actor":{"display_login":"dhairyadalal"}}