{"action":"created","comment":{"url":"https://api.github.com/repos/allenai/allennlp/pulls/comments/203472530","pull_request_review_id":138369022,"id":203472530,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwMzQ3MjUzMA==","diff_hunk":"@@ -37,7 +37,7 @@ class TextField(SequenceField[Dict[str, torch.Tensor]]):\n     def __init__(self, tokens: List[Token], token_indexers: Dict[str, TokenIndexer]) -> None:\n         self.tokens = tokens\n         self._token_indexers = token_indexers\n-        self._indexed_tokens: Optional[Dict[str, TokenList]] = None\n+        self._indexed_tokens: Optional[Dict[str, Any]] = None","path":"allennlp/data/fields/text_field.py","position":14,"original_position":14,"commit_id":"df054eb6a98f54be98d5d5c84623e6d747e7bcad","original_commit_id":"92bdb37b4a188ed3e715e2402a0533e6e9b5e11f","user":{"login":"DeNeutoy","id":16001974,"node_id":"MDQ6VXNlcjE2MDAxOTc0","avatar_url":"https://avatars1.githubusercontent.com/u/16001974?v=4","gravatar_id":"","url":"https://api.github.com/users/DeNeutoy","html_url":"https://github.com/DeNeutoy","followers_url":"https://api.github.com/users/DeNeutoy/followers","following_url":"https://api.github.com/users/DeNeutoy/following{/other_user}","gists_url":"https://api.github.com/users/DeNeutoy/gists{/gist_id}","starred_url":"https://api.github.com/users/DeNeutoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DeNeutoy/subscriptions","organizations_url":"https://api.github.com/users/DeNeutoy/orgs","repos_url":"https://api.github.com/users/DeNeutoy/repos","events_url":"https://api.github.com/users/DeNeutoy/events{/privacy}","received_events_url":"https://api.github.com/users/DeNeutoy/received_events","type":"User","site_admin":false},"body":"I agree with Joel here - this will work seamlessly with the changes you've already made to the `TextFieldEmbedder` which allow mis-matching keys, whilst reducing the complexity in the indexing code. The case that a `TokenIndexer` generates multiple arrays is rare enough that if you want to do this, you'll just have to use the `indexer_to_embedder_map` argument in the `TextFieldEmbedder` to map the arrays to the embedder. \r\n\r\nThe only downside with creating a flat map is that the 1-1 map between embedders and indexer names is now not enforceable - we should add some documentation that the key passed to `tokens_to_indices` really should be used as a key in isolation, even if it is then used to create auxilary keys for other return arrays.\r\n\r\ne.g if you have a byte_pair_embedder called `bytes`, we really need the returned array to have exactly `bytes` as a key: `{bytes: [....],  bytes-offsets: [....]}`, but we can't actually enforce this if users were to implement their own indexers which return multiple arrays.\r\n\r\nI think this is ok though, because in reality I don't imagine that there are many people implementing new indexers and if they do, they know what they are doing.\r\n","created_at":"2018-07-18T17:53:10Z","updated_at":"2018-07-18T17:53:11Z","html_url":"https://github.com/allenai/allennlp/pull/1506#discussion_r203472530","pull_request_url":"https://api.github.com/repos/allenai/allennlp/pulls/1506","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/allenai/allennlp/pulls/comments/203472530"},"html":{"href":"https://github.com/allenai/allennlp/pull/1506#discussion_r203472530"},"pull_request":{"href":"https://api.github.com/repos/allenai/allennlp/pulls/1506"}},"in_reply_to_id":203432679},"pull_request":{"url":"https://api.github.com/repos/allenai/allennlp/pulls/1506","id":202113999,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAyMTEzOTk5","html_url":"https://github.com/allenai/allennlp/pull/1506","diff_url":"https://github.com/allenai/allennlp/pull/1506.diff","patch_url":"https://github.com/allenai/allennlp/pull/1506.patch","issue_url":"https://api.github.com/repos/allenai/allennlp/issues/1506","number":1506,"state":"open","locked":false,"title":"Text field updates to support multiple arrays in TokenIndexer","user":{"login":"matt-peters","id":619844,"node_id":"MDQ6VXNlcjYxOTg0NA==","avatar_url":"https://avatars1.githubusercontent.com/u/619844?v=4","gravatar_id":"","url":"https://api.github.com/users/matt-peters","html_url":"https://github.com/matt-peters","followers_url":"https://api.github.com/users/matt-peters/followers","following_url":"https://api.github.com/users/matt-peters/following{/other_user}","gists_url":"https://api.github.com/users/matt-peters/gists{/gist_id}","starred_url":"https://api.github.com/users/matt-peters/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matt-peters/subscriptions","organizations_url":"https://api.github.com/users/matt-peters/orgs","repos_url":"https://api.github.com/users/matt-peters/repos","events_url":"https://api.github.com/users/matt-peters/events{/privacy}","received_events_url":"https://api.github.com/users/matt-peters/received_events","type":"User","site_admin":false},"body":"* change the API for `TokenIndexer.pad_token_sequence` to avoid `Union` types\r\n* Handle padding in case where one of the `TokenIndexer` returns multiple arrays (need to pad each sequence separately)\r\n* add an option to `get_text_field_mask` that allows to specify the mask directly instead of computing it.","created_at":"2018-07-18T01:15:59Z","updated_at":"2018-07-18T17:53:11Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d15306f4e2e41f6639154a0ed89817a976bb8d0b","assignee":null,"assignees":[],"requested_reviewers":[{"login":"DeNeutoy","id":16001974,"node_id":"MDQ6VXNlcjE2MDAxOTc0","avatar_url":"https://avatars1.githubusercontent.com/u/16001974?v=4","gravatar_id":"","url":"https://api.github.com/users/DeNeutoy","html_url":"https://github.com/DeNeutoy","followers_url":"https://api.github.com/users/DeNeutoy/followers","following_url":"https://api.github.com/users/DeNeutoy/following{/other_user}","gists_url":"https://api.github.com/users/DeNeutoy/gists{/gist_id}","starred_url":"https://api.github.com/users/DeNeutoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DeNeutoy/subscriptions","organizations_url":"https://api.github.com/users/DeNeutoy/orgs","repos_url":"https://api.github.com/users/DeNeutoy/repos","events_url":"https://api.github.com/users/DeNeutoy/events{/privacy}","received_events_url":"https://api.github.com/users/DeNeutoy/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/allenai/allennlp/pulls/1506/commits","review_comments_url":"https://api.github.com/repos/allenai/allennlp/pulls/1506/comments","review_comment_url":"https://api.github.com/repos/allenai/allennlp/pulls/comments{/number}","comments_url":"https://api.github.com/repos/allenai/allennlp/issues/1506/comments","statuses_url":"https://api.github.com/repos/allenai/allennlp/statuses/df054eb6a98f54be98d5d5c84623e6d747e7bcad","head":{"label":"matt-peters:mp/text_field","ref":"mp/text_field","sha":"df054eb6a98f54be98d5d5c84623e6d747e7bcad","user":{"login":"matt-peters","id":619844,"node_id":"MDQ6VXNlcjYxOTg0NA==","avatar_url":"https://avatars1.githubusercontent.com/u/619844?v=4","gravatar_id":"","url":"https://api.github.com/users/matt-peters","html_url":"https://github.com/matt-peters","followers_url":"https://api.github.com/users/matt-peters/followers","following_url":"https://api.github.com/users/matt-peters/following{/other_user}","gists_url":"https://api.github.com/users/matt-peters/gists{/gist_id}","starred_url":"https://api.github.com/users/matt-peters/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matt-peters/subscriptions","organizations_url":"https://api.github.com/users/matt-peters/orgs","repos_url":"https://api.github.com/users/matt-peters/repos","events_url":"https://api.github.com/users/matt-peters/events{/privacy}","received_events_url":"https://api.github.com/users/matt-peters/received_events","type":"User","site_admin":false},"repo":{"id":117899604,"node_id":"MDEwOlJlcG9zaXRvcnkxMTc4OTk2MDQ=","name":"allennlp","full_name":"matt-peters/allennlp","owner":{"login":"matt-peters","id":619844,"node_id":"MDQ6VXNlcjYxOTg0NA==","avatar_url":"https://avatars1.githubusercontent.com/u/619844?v=4","gravatar_id":"","url":"https://api.github.com/users/matt-peters","html_url":"https://github.com/matt-peters","followers_url":"https://api.github.com/users/matt-peters/followers","following_url":"https://api.github.com/users/matt-peters/following{/other_user}","gists_url":"https://api.github.com/users/matt-peters/gists{/gist_id}","starred_url":"https://api.github.com/users/matt-peters/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matt-peters/subscriptions","organizations_url":"https://api.github.com/users/matt-peters/orgs","repos_url":"https://api.github.com/users/matt-peters/repos","events_url":"https://api.github.com/users/matt-peters/events{/privacy}","received_events_url":"https://api.github.com/users/matt-peters/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/matt-peters/allennlp","description":"An open-source NLP research library, built on PyTorch.","fork":true,"url":"https://api.github.com/repos/matt-peters/allennlp","forks_url":"https://api.github.com/repos/matt-peters/allennlp/forks","keys_url":"https://api.github.com/repos/matt-peters/allennlp/keys{/key_id}","collaborators_url":"https://api.github.com/repos/matt-peters/allennlp/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/matt-peters/allennlp/teams","hooks_url":"https://api.github.com/repos/matt-peters/allennlp/hooks","issue_events_url":"https://api.github.com/repos/matt-peters/allennlp/issues/events{/number}","events_url":"https://api.github.com/repos/matt-peters/allennlp/events","assignees_url":"https://api.github.com/repos/matt-peters/allennlp/assignees{/user}","branches_url":"https://api.github.com/repos/matt-peters/allennlp/branches{/branch}","tags_url":"https://api.github.com/repos/matt-peters/allennlp/tags","blobs_url":"https://api.github.com/repos/matt-peters/allennlp/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/matt-peters/allennlp/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/matt-peters/allennlp/git/refs{/sha}","trees_url":"https://api.github.com/repos/matt-peters/allennlp/git/trees{/sha}","statuses_url":"https://api.github.com/repos/matt-peters/allennlp/statuses/{sha}","languages_url":"https://api.github.com/repos/matt-peters/allennlp/languages","stargazers_url":"https://api.github.com/repos/matt-peters/allennlp/stargazers","contributors_url":"https://api.github.com/repos/matt-peters/allennlp/contributors","subscribers_url":"https://api.github.com/repos/matt-peters/allennlp/subscribers","subscription_url":"https://api.github.com/repos/matt-peters/allennlp/subscription","commits_url":"https://api.github.com/repos/matt-peters/allennlp/commits{/sha}","git_commits_url":"https://api.github.com/repos/matt-peters/allennlp/git/commits{/sha}","comments_url":"https://api.github.com/repos/matt-peters/allennlp/comments{/number}","issue_comment_url":"https://api.github.com/repos/matt-peters/allennlp/issues/comments{/number}","contents_url":"https://api.github.com/repos/matt-peters/allennlp/contents/{+path}","compare_url":"https://api.github.com/repos/matt-peters/allennlp/compare/{base}...{head}","merges_url":"https://api.github.com/repos/matt-peters/allennlp/merges","archive_url":"https://api.github.com/repos/matt-peters/allennlp/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/matt-peters/allennlp/downloads","issues_url":"https://api.github.com/repos/matt-peters/allennlp/issues{/number}","pulls_url":"https://api.github.com/repos/matt-peters/allennlp/pulls{/number}","milestones_url":"https://api.github.com/repos/matt-peters/allennlp/milestones{/number}","notifications_url":"https://api.github.com/repos/matt-peters/allennlp/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/matt-peters/allennlp/labels{/name}","releases_url":"https://api.github.com/repos/matt-peters/allennlp/releases{/id}","deployments_url":"https://api.github.com/repos/matt-peters/allennlp/deployments","created_at":"2018-01-17T22:19:21Z","updated_at":"2018-07-17T21:09:47Z","pushed_at":"2018-07-18T16:07:03Z","git_url":"git://github.com/matt-peters/allennlp.git","ssh_url":"git@github.com:matt-peters/allennlp.git","clone_url":"https://github.com/matt-peters/allennlp.git","svn_url":"https://github.com/matt-peters/allennlp","homepage":"http://www.allennlp.org/","size":13132,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":2,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"allenai:master","ref":"master","sha":"8e5ee656573df13c7bdb59ebc8c57d1288237e27","user":{"login":"allenai","id":5667695,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2Njc2OTU=","avatar_url":"https://avatars3.githubusercontent.com/u/5667695?v=4","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"repo":{"id":91356408,"node_id":"MDEwOlJlcG9zaXRvcnk5MTM1NjQwOA==","name":"allennlp","full_name":"allenai/allennlp","owner":{"login":"allenai","id":5667695,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2Njc2OTU=","avatar_url":"https://avatars3.githubusercontent.com/u/5667695?v=4","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/allenai/allennlp","description":"An open-source NLP research library, built on PyTorch.","fork":false,"url":"https://api.github.com/repos/allenai/allennlp","forks_url":"https://api.github.com/repos/allenai/allennlp/forks","keys_url":"https://api.github.com/repos/allenai/allennlp/keys{/key_id}","collaborators_url":"https://api.github.com/repos/allenai/allennlp/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/allenai/allennlp/teams","hooks_url":"https://api.github.com/repos/allenai/allennlp/hooks","issue_events_url":"https://api.github.com/repos/allenai/allennlp/issues/events{/number}","events_url":"https://api.github.com/repos/allenai/allennlp/events","assignees_url":"https://api.github.com/repos/allenai/allennlp/assignees{/user}","branches_url":"https://api.github.com/repos/allenai/allennlp/branches{/branch}","tags_url":"https://api.github.com/repos/allenai/allennlp/tags","blobs_url":"https://api.github.com/repos/allenai/allennlp/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/allenai/allennlp/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/allenai/allennlp/git/refs{/sha}","trees_url":"https://api.github.com/repos/allenai/allennlp/git/trees{/sha}","statuses_url":"https://api.github.com/repos/allenai/allennlp/statuses/{sha}","languages_url":"https://api.github.com/repos/allenai/allennlp/languages","stargazers_url":"https://api.github.com/repos/allenai/allennlp/stargazers","contributors_url":"https://api.github.com/repos/allenai/allennlp/contributors","subscribers_url":"https://api.github.com/repos/allenai/allennlp/subscribers","subscription_url":"https://api.github.com/repos/allenai/allennlp/subscription","commits_url":"https://api.github.com/repos/allenai/allennlp/commits{/sha}","git_commits_url":"https://api.github.com/repos/allenai/allennlp/git/commits{/sha}","comments_url":"https://api.github.com/repos/allenai/allennlp/comments{/number}","issue_comment_url":"https://api.github.com/repos/allenai/allennlp/issues/comments{/number}","contents_url":"https://api.github.com/repos/allenai/allennlp/contents/{+path}","compare_url":"https://api.github.com/repos/allenai/allennlp/compare/{base}...{head}","merges_url":"https://api.github.com/repos/allenai/allennlp/merges","archive_url":"https://api.github.com/repos/allenai/allennlp/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/allenai/allennlp/downloads","issues_url":"https://api.github.com/repos/allenai/allennlp/issues{/number}","pulls_url":"https://api.github.com/repos/allenai/allennlp/pulls{/number}","milestones_url":"https://api.github.com/repos/allenai/allennlp/milestones{/number}","notifications_url":"https://api.github.com/repos/allenai/allennlp/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/allenai/allennlp/labels{/name}","releases_url":"https://api.github.com/repos/allenai/allennlp/releases{/id}","deployments_url":"https://api.github.com/repos/allenai/allennlp/deployments","created_at":"2017-05-15T15:52:41Z","updated_at":"2018-07-18T16:35:32Z","pushed_at":"2018-07-18T16:43:51Z","git_url":"git://github.com/allenai/allennlp.git","ssh_url":"git@github.com:allenai/allennlp.git","clone_url":"https://github.com/allenai/allennlp.git","svn_url":"https://github.com/allenai/allennlp","homepage":"http://www.allennlp.org/","size":12968,"stargazers_count":2694,"watchers_count":2694,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":451,"mirror_url":null,"archived":false,"open_issues_count":82,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":451,"open_issues":82,"watchers":2694,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/allenai/allennlp/pulls/1506"},"html":{"href":"https://github.com/allenai/allennlp/pull/1506"},"issue":{"href":"https://api.github.com/repos/allenai/allennlp/issues/1506"},"comments":{"href":"https://api.github.com/repos/allenai/allennlp/issues/1506/comments"},"review_comments":{"href":"https://api.github.com/repos/allenai/allennlp/pulls/1506/comments"},"review_comment":{"href":"https://api.github.com/repos/allenai/allennlp/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/allenai/allennlp/pulls/1506/commits"},"statuses":{"href":"https://api.github.com/repos/allenai/allennlp/statuses/df054eb6a98f54be98d5d5c84623e6d747e7bcad"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"DeNeutoy"}}