{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/ignite/issues/118","repository_url":"https://api.github.com/repos/pytorch/ignite","labels_url":"https://api.github.com/repos/pytorch/ignite/issues/118/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/ignite/issues/118/comments","events_url":"https://api.github.com/repos/pytorch/ignite/issues/118/events","html_url":"https://github.com/pytorch/ignite/issues/118","id":306328921,"number":118,"title":"Metrics with multiple outputs & standardizing outputs","user":{"login":"veugene","id":8798047,"avatar_url":"https://avatars3.githubusercontent.com/u/8798047?v=4","gravatar_id":"","url":"https://api.github.com/users/veugene","html_url":"https://github.com/veugene","followers_url":"https://api.github.com/users/veugene/followers","following_url":"https://api.github.com/users/veugene/following{/other_user}","gists_url":"https://api.github.com/users/veugene/gists{/gist_id}","starred_url":"https://api.github.com/users/veugene/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/veugene/subscriptions","organizations_url":"https://api.github.com/users/veugene/orgs","repos_url":"https://api.github.com/users/veugene/repos","events_url":"https://api.github.com/users/veugene/events{/privacy}","received_events_url":"https://api.github.com/users/veugene/received_events","type":"User","site_admin":false},"labels":[{"id":867809527,"url":"https://api.github.com/repos/pytorch/ignite/labels/0.1.0","name":"0.1.0","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2018-03-19T04:19:11Z","updated_at":"2018-03-23T10:31:02Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The current implementation of metrics assumes a very specific set of return values from the training/validation function: (y_pred, y), as can be seen for example in https://github.com/pytorch/ignite/blob/master/ignite/metrics/mean_absolute_error.py#L20 .\r\n\r\nThis has a couple inconvenient consequences:\r\n\r\n1. Existing metrics cannot handle a model with multiple outputs.\r\n2. Users cannot pass around additional return values to be picked up by a custom handler via `state.outputs`\r\n\r\nIt would be reasonable to standardize the set of return values expected from training/validation functions if the user would have the option of putting custom values in the calling engine's `state`. This would require exposing state to the training/validation functions. This would solve (2) but metrics would need to be further adjusted to solve (1).\r\n\r\nOne way of standardizing the return values would be to always return only loss, model outputs, and targets. If state is ephemeral but is passed to the training/validation function, this would be of the form:\r\n`return loss, model_outputs, targets, state`\r\n\r\nIf state were to be exposed in an Engine as described in #117, this would be of the form:\r\n`return loss, model_outputs, targets`\r\n\r\nEvery element in `model_outputs` would be paired with each element in the targets. This would allow a Metric to optionally track a single output in a multi-output model, changing\r\n\r\n```python\r\nclass Metric(object):\r\n    def __init__(self):\r\n        self.reset()\r\n\r\ndef iteration_completed(self, engine, state):\r\n        self.update(state.output)\r\n```\r\n\r\nto\r\n\r\n```python\r\nclass Metric(object):\r\n    def __init__(self, output_index=None):\r\n        self.output_index = output_index\r\n        self.reset()\r\n\r\n    def iteration_completed(self, engine, state):\r\n        output = state.output\r\n        target = state.target\r\n        if self.output_index is not None:\r\n            output = output[self.output_index]\r\n            target = target[self.output_index]\r\n        self.update(output, target)\r\n```"},"comment":{"url":"https://api.github.com/repos/pytorch/ignite/issues/comments/375610920","html_url":"https://github.com/pytorch/ignite/issues/118#issuecomment-375610920","issue_url":"https://api.github.com/repos/pytorch/ignite/issues/118","id":375610920,"user":{"login":"alykhantejani","id":687194,"avatar_url":"https://avatars1.githubusercontent.com/u/687194?v=4","gravatar_id":"","url":"https://api.github.com/users/alykhantejani","html_url":"https://github.com/alykhantejani","followers_url":"https://api.github.com/users/alykhantejani/followers","following_url":"https://api.github.com/users/alykhantejani/following{/other_user}","gists_url":"https://api.github.com/users/alykhantejani/gists{/gist_id}","starred_url":"https://api.github.com/users/alykhantejani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alykhantejani/subscriptions","organizations_url":"https://api.github.com/users/alykhantejani/orgs","repos_url":"https://api.github.com/users/alykhantejani/repos","events_url":"https://api.github.com/users/alykhantejani/events{/privacy}","received_events_url":"https://api.github.com/users/alykhantejani/received_events","type":"User","site_admin":false},"created_at":"2018-03-23T10:31:02Z","updated_at":"2018-03-23T10:31:02Z","author_association":"COLLABORATOR","body":"Metrics now support multiple outputs with #121 being merged. "}}
{"actor":{"display_login":"alykhantejani"}}