{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/ignite/issues/296","repository_url":"https://api.github.com/repos/pytorch/ignite","labels_url":"https://api.github.com/repos/pytorch/ignite/issues/296/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/ignite/issues/296/comments","events_url":"https://api.github.com/repos/pytorch/ignite/issues/296/events","html_url":"https://github.com/pytorch/ignite/issues/296","id":372303489,"node_id":"MDU6SXNzdWUzNzIzMDM0ODk=","number":296,"title":"Discussion about ProgressBar and metrics display","user":{"login":"milesial","id":30204471,"node_id":"MDQ6VXNlcjMwMjA0NDcx","avatar_url":"https://avatars0.githubusercontent.com/u/30204471?v=4","gravatar_id":"","url":"https://api.github.com/users/milesial","html_url":"https://github.com/milesial","followers_url":"https://api.github.com/users/milesial/followers","following_url":"https://api.github.com/users/milesial/following{/other_user}","gists_url":"https://api.github.com/users/milesial/gists{/gist_id}","starred_url":"https://api.github.com/users/milesial/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/milesial/subscriptions","organizations_url":"https://api.github.com/users/milesial/orgs","repos_url":"https://api.github.com/users/milesial/repos","events_url":"https://api.github.com/users/milesial/events{/privacy}","received_events_url":"https://api.github.com/users/milesial/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-10-21T11:12:36Z","updated_at":"2018-10-21T19:58:12Z","closed_at":null,"author_association":"NONE","body":"Hello, I would like to discuss the feature in `contrib.handlers.ProgressBar` that displays metrics on each `Events.ITERATION_COMPLETED` as a postfix of the bar. For some context, please read https://github.com/pytorch/ignite/pull/295.\r\n\r\nI think the `ProgressBar` and this feature are essential to the package, as it provides a concise way to have a nice training and evaluation experience (replacing the old way of spamming numbers in the console), and is appealing to beginners.\r\n\r\nHere is some of the issues I found.\r\n\r\n## 1. Non-`RunningAverage` metrics\r\n### Problem\r\nOn the doc, the given example is\r\n```python\r\npbar = ProgressBar()\r\npbar.attach(trainer, ['loss'])\r\n```\r\n\r\nThe `metrics` parameter (here `['loss']`) works with `Metric`s that have been attached to the engine, **AND that rewired `self.computed` (which updates the `state.metrics` dict of the engine) on `Events.ITERATION_COMPLETED` instead of `Events.EPOCH_COMPLETED`**. The second part is unintuitive, and not documented. The current document suggests that this might work with all metrics, but this raises a `KeyError` when the dict has not been updated before the end of the first iteration.\r\nCurrently, this freature only works with `RunningAverage`.\r\n\r\nThe problem has been raised by @vfdev-5 in https://github.com/pytorch/ignite/pull/256, but I did not see the discussion go anywhere, I would like for it to continue.\r\n\r\n### Proposed solutions\r\n\r\n1. The documentation should be updated to reflect the fact that this feature will not work on all of the built-in metrics but `RunningAverage`.\r\n\r\n2. The error message for the `KeyError` is `metrics not found in engine.state.metrics` and could be modified to hint for what the issue might be.\r\n\r\n3. a Maybe the `Metric` class hierarchy should be updated, to distinguish metrics that update the engine state every epoch, and those that update it every iteration. The `ProgressBar` could then only accept the latter one. \r\n\r\n3. b Or maybe all of the current built-in metrics could update the state of the engine at each iteration (on `self.update` instead of on `self.completed`. This will add computation at each epoch. The progress bar would then work on all metrics.\r\n\r\n## 2. Print the loss of a training engine\r\n### Problem\r\nHave a look at the doc example again :\r\n```python\r\npbar = ProgressBar()\r\npbar.attach(trainer, ['loss'])\r\n```\r\nThis suggests that we can print the training loss, that updates at each iteration.\r\n\r\nHowever, we can't. When we are using a trainer created with `engine.create_supervised_trainer`, the `process_function` returns the loss for the processed batch. I think this is the most intuitive way to do it even for custom training engines.\r\nThe `Metric`s that can be attached are computed from the output of this function (`y_pred, y` for an evaluator for example), which may pass through a transform function. You see where I am going: the output of the trainer is already the loss itself, we do not need any further computation.\r\n\r\nIf we want to print the training loss, we have to attach a `Metric` that does nothing but moves the already computed loss into the `state.metrics` dict of the engine at every iteration. See https://github.com/pytorch/ignite/pull/295 for the `Identity` metric. This is not intuitive, we should be able to display the training loss out of the box.\r\n\r\n### Proposed solutions\r\n\r\n1. Built-in Identity metric, see https://github.com/pytorch/ignite/pull/295 for the code.\r\n\r\n2. Modify `ProgressBar` to allow the display in the postfix the value of `engine.state.output`\r\n    The `attach` method could then be \r\n    ```python\r\n    attach(self, engine, metric_names=None, display_output=False)\r\n    ```\r\n    with the output looking like \r\n    `Epoch 1: [59/689]   9%|▊                 , output=3.54e-02 [00:04<00:50]`\r\n3. Modify `ProgressBar` to allow the display in the postfix the value of any custom variables in the `engine.state` object, including the output.\r\n\r\n---\r\nThanks for reading, sorry for the long issue and the english mistakes."},"comment":{"url":"https://api.github.com/repos/pytorch/ignite/issues/comments/431698964","html_url":"https://github.com/pytorch/ignite/issues/296#issuecomment-431698964","issue_url":"https://api.github.com/repos/pytorch/ignite/issues/296","id":431698964,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTY5ODk2NA==","user":{"login":"milesial","id":30204471,"node_id":"MDQ6VXNlcjMwMjA0NDcx","avatar_url":"https://avatars0.githubusercontent.com/u/30204471?v=4","gravatar_id":"","url":"https://api.github.com/users/milesial","html_url":"https://github.com/milesial","followers_url":"https://api.github.com/users/milesial/followers","following_url":"https://api.github.com/users/milesial/following{/other_user}","gists_url":"https://api.github.com/users/milesial/gists{/gist_id}","starred_url":"https://api.github.com/users/milesial/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/milesial/subscriptions","organizations_url":"https://api.github.com/users/milesial/orgs","repos_url":"https://api.github.com/users/milesial/repos","events_url":"https://api.github.com/users/milesial/events{/privacy}","received_events_url":"https://api.github.com/users/milesial/received_events","type":"User","site_admin":false},"created_at":"2018-10-21T19:58:12Z","updated_at":"2018-10-21T19:58:12Z","author_association":"NONE","body":"About the persistency, I was thinking about displaying the training loss for each batch, of course the loss of the last batch will be overwritten by the loss of the new batch. This allows us to look at the bar progression and see the evolution of the loss without getting spammed. At the end of the epoch, a handler can be used to print persistently the total average loss of the epoch."}}
{"actor":{"display_login":"milesial"}}