{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5546","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5546/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5546/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5546/events","html_url":"https://github.com/tensorflow/tensorflow/pull/5546","id":188809263,"number":5546,"title":"Add support for dict Input to DataFeeder, StreamingDataFeeder and allow for different TensorSignature for each of ModeKeys","user":{"login":"abhitopia","id":12864026,"avatar_url":"https://avatars.githubusercontent.com/u/12864026?v=3","gravatar_id":"","url":"https://api.github.com/users/abhitopia","html_url":"https://github.com/abhitopia","followers_url":"https://api.github.com/users/abhitopia/followers","following_url":"https://api.github.com/users/abhitopia/following{/other_user}","gists_url":"https://api.github.com/users/abhitopia/gists{/gist_id}","starred_url":"https://api.github.com/users/abhitopia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhitopia/subscriptions","organizations_url":"https://api.github.com/users/abhitopia/orgs","repos_url":"https://api.github.com/users/abhitopia/repos","events_url":"https://api.github.com/users/abhitopia/events{/privacy}","received_events_url":"https://api.github.com/users/abhitopia/received_events","type":"User","site_admin":false},"labels":[{"id":300136587,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/cla:%20yes","name":"cla: yes","color":"009800","default":false}],"state":"open","locked":false,"assignee":{"login":"martinwicke","id":577277,"avatar_url":"https://avatars.githubusercontent.com/u/577277?v=3","gravatar_id":"","url":"https://api.github.com/users/martinwicke","html_url":"https://github.com/martinwicke","followers_url":"https://api.github.com/users/martinwicke/followers","following_url":"https://api.github.com/users/martinwicke/following{/other_user}","gists_url":"https://api.github.com/users/martinwicke/gists{/gist_id}","starred_url":"https://api.github.com/users/martinwicke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinwicke/subscriptions","organizations_url":"https://api.github.com/users/martinwicke/orgs","repos_url":"https://api.github.com/users/martinwicke/repos","events_url":"https://api.github.com/users/martinwicke/events{/privacy}","received_events_url":"https://api.github.com/users/martinwicke/received_events","type":"User","site_admin":false},"assignees":[{"login":"martinwicke","id":577277,"avatar_url":"https://avatars.githubusercontent.com/u/577277?v=3","gravatar_id":"","url":"https://api.github.com/users/martinwicke","html_url":"https://github.com/martinwicke","followers_url":"https://api.github.com/users/martinwicke/followers","following_url":"https://api.github.com/users/martinwicke/following{/other_user}","gists_url":"https://api.github.com/users/martinwicke/gists{/gist_id}","starred_url":"https://api.github.com/users/martinwicke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinwicke/subscriptions","organizations_url":"https://api.github.com/users/martinwicke/orgs","repos_url":"https://api.github.com/users/martinwicke/repos","events_url":"https://api.github.com/users/martinwicke/events{/privacy}","received_events_url":"https://api.github.com/users/martinwicke/received_events","type":"User","site_admin":false}],"milestone":null,"comments":25,"created_at":"2016-11-11T17:24:05Z","updated_at":"2016-11-23T14:37:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/tensorflow/tensorflow/pulls/5546","html_url":"https://github.com/tensorflow/tensorflow/pull/5546","diff_url":"https://github.com/tensorflow/tensorflow/pull/5546.diff","patch_url":"https://github.com/tensorflow/tensorflow/pull/5546.patch"},"body":"# Problem\r\n\r\nThe `Estimator.fit()` function takes as argument either\r\n- (`x`, `y`, and `batch_size`) where `x` and `y` could be numpy arrays or iterators.\r\n  - **PROS**\r\n    1.   Easy to use.\r\n    2.  Allows feeding data from arbitrary source as long as problem can be decomposed into `x` and `y`.\r\n  - **CONS**\r\n    1.  No provision to provide epoch\r\n    2.  In case, `x` and `y` are arrays, the data aggregate must be available as opposed to reading on the fly ( say from database)\r\n    3. Whether array or iterator, `x` and `y` can't be dictionaries. Most complex problems cannot be reduced to input matrix and output matrix and may require multiple input features matrices.\r\n- `input_fn` - this is callback function which must return `features` and `target` tensors or dictionary of tensors.\r\n  - **PROS**\r\n    1. Allows feeding data from arbitrary source (in theory).\r\n    2. returned features and targets can be dictionary thus allowing to solve complex problems which takes multiple inputs.\r\n  - **CONS**\r\n    1. Only found support for reading files using `read_batch_examples()`, `read_batch_features()`, `read_batch_record_features()`, etc.\r\n    2. No support for passing placeholder and feed_fn to allow for arbitrary source of input data which don't require queue.\r\n# Relevant discussions\r\n1. https://github.com/tensorflow/tensorflow/pull/4696#issuecomment-253632403\r\n2. http://stackoverflow.com/questions/39855375/how-to-use-streamingdatafeeder-as-contrib-learn-estimator-fits-input-fn\r\n# Additional problem\r\n\r\nCurrently the self.feature_info (feature signature) must be same for training as well as evaluation. However, there can be cases where evaluation if done differently than training. For example, DualEncoderLSTM model where training just requires Context and Utterance, while Evaluation (Validation) requires context and multiple utterances.\r\n# Solution\r\n\r\nThis PR basically combines the pros of using (`x`, `y`, and `batch_size`) with pros of input_fn by allowing `x`, `y` to be dictionary of multiple arrays. This also works for `x` and `y` iterators returning dictionary of streaming data. \r\n\r\nAlso, added support for different feature/target signature for training and evaluation.\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/262529173","html_url":"https://github.com/tensorflow/tensorflow/pull/5546#issuecomment-262529173","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5546","id":262529173,"user":{"login":"caisq","id":16824702,"avatar_url":"https://avatars.githubusercontent.com/u/16824702?v=3","gravatar_id":"","url":"https://api.github.com/users/caisq","html_url":"https://github.com/caisq","followers_url":"https://api.github.com/users/caisq/followers","following_url":"https://api.github.com/users/caisq/following{/other_user}","gists_url":"https://api.github.com/users/caisq/gists{/gist_id}","starred_url":"https://api.github.com/users/caisq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caisq/subscriptions","organizations_url":"https://api.github.com/users/caisq/orgs","repos_url":"https://api.github.com/users/caisq/repos","events_url":"https://api.github.com/users/caisq/events{/privacy}","received_events_url":"https://api.github.com/users/caisq/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T14:37:34Z","updated_at":"2016-11-23T14:37:34Z","body":"@abhitopia Please do not worry about testing it yourself. We will run test suites on our Jenkins to test this PR before merging it. Flaky timeouts like the ones you see do occur and in this case they seem to be unrelated to your changes. "}}
{"actor":{"display_login":"caisq"}}