{"action":"closed","issue":{"url":"https://api.github.com/repos/pytorch/ignite/issues/296","repository_url":"https://api.github.com/repos/pytorch/ignite","labels_url":"https://api.github.com/repos/pytorch/ignite/issues/296/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/ignite/issues/296/comments","events_url":"https://api.github.com/repos/pytorch/ignite/issues/296/events","html_url":"https://github.com/pytorch/ignite/issues/296","id":372303489,"node_id":"MDU6SXNzdWUzNzIzMDM0ODk=","number":296,"title":"Discussion about ProgressBar and metrics display","user":{"login":"milesial","id":30204471,"node_id":"MDQ6VXNlcjMwMjA0NDcx","avatar_url":"https://avatars0.githubusercontent.com/u/30204471?v=4","gravatar_id":"","url":"https://api.github.com/users/milesial","html_url":"https://github.com/milesial","followers_url":"https://api.github.com/users/milesial/followers","following_url":"https://api.github.com/users/milesial/following{/other_user}","gists_url":"https://api.github.com/users/milesial/gists{/gist_id}","starred_url":"https://api.github.com/users/milesial/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/milesial/subscriptions","organizations_url":"https://api.github.com/users/milesial/orgs","repos_url":"https://api.github.com/users/milesial/repos","events_url":"https://api.github.com/users/milesial/events{/privacy}","received_events_url":"https://api.github.com/users/milesial/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-10-21T11:12:36Z","updated_at":"2018-10-29T17:30:10Z","closed_at":"2018-10-29T17:30:10Z","author_association":"NONE","body":"Hello, I would like to discuss the feature in `contrib.handlers.ProgressBar` that displays metrics on each `Events.ITERATION_COMPLETED` as a postfix of the bar. For some context, please read https://github.com/pytorch/ignite/pull/295.\r\n\r\nI think the `ProgressBar` and this feature are essential to the package, as it provides a concise way to have a nice training and evaluation experience (replacing the old way of spamming numbers in the console), and is appealing to beginners.\r\n\r\nHere is some of the issues I found.\r\n\r\n## 1. Non-`RunningAverage` metrics\r\n### Problem\r\nOn the doc, the given example is\r\n```python\r\npbar = ProgressBar()\r\npbar.attach(trainer, ['loss'])\r\n```\r\n\r\nThe `metrics` parameter (here `['loss']`) works with `Metric`s that have been attached to the engine, **AND that rewired `self.computed` (which updates the `state.metrics` dict of the engine) on `Events.ITERATION_COMPLETED` instead of `Events.EPOCH_COMPLETED`**. The second part is unintuitive, and not documented. The current document suggests that this might work with all metrics, but this raises a `KeyError` when the dict has not been updated before the end of the first iteration.\r\nCurrently, this freature only works with `RunningAverage`.\r\n\r\nThe problem has been raised by @vfdev-5 in https://github.com/pytorch/ignite/pull/256, but I did not see the discussion go anywhere, I would like for it to continue.\r\n\r\n### Proposed solutions\r\n\r\n1. The documentation should be updated to reflect the fact that this feature will not work on all of the built-in metrics but `RunningAverage`.\r\n\r\n2. The error message for the `KeyError` is `metrics not found in engine.state.metrics` and could be modified to hint for what the issue might be.\r\n\r\n3. a Maybe the `Metric` class hierarchy should be updated, to distinguish metrics that update the engine state every epoch, and those that update it every iteration. The `ProgressBar` could then only accept the latter one. \r\n\r\n3. b Or maybe all of the current built-in metrics could update the state of the engine at each iteration (on `self.update` instead of on `self.completed`. This will add computation at each epoch. The progress bar would then work on all metrics.\r\n\r\n## 2. Print the loss of a training engine\r\n### Problem\r\nHave a look at the doc example again :\r\n```python\r\npbar = ProgressBar()\r\npbar.attach(trainer, ['loss'])\r\n```\r\nThis suggests that we can print the training loss, that updates at each iteration.\r\n\r\nHowever, we can't. When we are using a trainer created with `engine.create_supervised_trainer`, the `process_function` returns the loss for the processed batch. I think this is the most intuitive way to do it even for custom training engines.\r\nThe `Metric`s that can be attached are computed from the output of this function (`y_pred, y` for an evaluator for example), which may pass through a transform function. You see where I am going: the output of the trainer is already the loss itself, we do not need any further computation.\r\n\r\nIf we want to print the training loss, we have to attach a `Metric` that does nothing but moves the already computed loss into the `state.metrics` dict of the engine at every iteration. See https://github.com/pytorch/ignite/pull/295 for the `Identity` metric. This is not intuitive, we should be able to display the training loss out of the box.\r\n\r\n### Proposed solutions\r\n\r\n1. Built-in Identity metric, see https://github.com/pytorch/ignite/pull/295 for the code.\r\n\r\n2. Modify `ProgressBar` to allow the display in the postfix the value of `engine.state.output`\r\n    The `attach` method could then be \r\n    ```python\r\n    attach(self, engine, metric_names=None, display_output=False)\r\n    ```\r\n    with the output looking like \r\n    `Epoch 1: [59/689]   9%|▊                 , output=3.54e-02 [00:04<00:50]`\r\n3. Modify `ProgressBar` to allow the display in the postfix the value of any custom variables in the `engine.state` object, including the output.\r\n\r\n---\r\nThanks for reading, sorry for the long issue and the english mistakes."}}
{"actor":{"display_login":"jasonkriss"}}