{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/2873","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/2873/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/2873/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/2873/events","html_url":"https://github.com/dmlc/mxnet/issues/2873","id":168279015,"number":2873,"title":"Again about collecting train/test accuracy and loss ","user":{"login":"undertherain","id":1635907,"avatar_url":"https://avatars.githubusercontent.com/u/1635907?v=3","gravatar_id":"","url":"https://api.github.com/users/undertherain","html_url":"https://github.com/undertherain","followers_url":"https://api.github.com/users/undertherain/followers","following_url":"https://api.github.com/users/undertherain/following{/other_user}","gists_url":"https://api.github.com/users/undertherain/gists{/gist_id}","starred_url":"https://api.github.com/users/undertherain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/undertherain/subscriptions","organizations_url":"https://api.github.com/users/undertherain/orgs","repos_url":"https://api.github.com/users/undertherain/repos","events_url":"https://api.github.com/users/undertherain/events{/privacy}","received_events_url":"https://api.github.com/users/undertherain/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-29T08:50:02Z","updated_at":"2016-12-01T00:41:52Z","closed_at":null,"body":"I’m tackling the problem of collecting accuracy and loss data on train and validation sets for every epoch. To plot learning curves etc…\n\nThere seem to be several related issues here, but mostly addressing minute details.\nThere’s also a solution suggesting parsing console output(!) to get some useful data, which to me does not seem to be very elegant. \n\nFrom what I see it seems possible to collect training accuracy and loss (or whatever metrics are specified) from individual training batch callbacks.\nFor example each batch_end_callback adds values to some static/global variable and at epoch_callback it is divided by the total number of training batches. \n\nThere’s variable \"nbatch\" visible among the epoch_callback arguments, do I understand correctly that it is the number of training batches? \n\nThe problem is then with evaluation. First of all, the number of evaluation batches is not explicitly reported. Of course, knowing data size and batch size it is possible to calculate it manually, but if number of train batches is reported, why not report number of eval batches?\n\nSecond, validation happens  after epoch_callback is called. Is there a good reason for this? Can this behavior be changed?  Or is there a good way to detect and of the validation? \n\nStrange in this context is that in the console the validation data is reported after every epoch:\nINFO:root:Epoch[1] Validation-accuracy=0.982595\nINFO:root:Epoch[1] Validation-cross-entropy=0.107147\n\nCan it be [made] simply accessible via the API? (along with same values for training set)\n\nThank you so much!\nAlex\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/264043902","html_url":"https://github.com/dmlc/mxnet/issues/2873#issuecomment-264043902","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/2873","id":264043902,"user":{"login":"CNevd","id":8547394,"avatar_url":"https://avatars.githubusercontent.com/u/8547394?v=3","gravatar_id":"","url":"https://api.github.com/users/CNevd","html_url":"https://github.com/CNevd","followers_url":"https://api.github.com/users/CNevd/followers","following_url":"https://api.github.com/users/CNevd/following{/other_user}","gists_url":"https://api.github.com/users/CNevd/gists{/gist_id}","starred_url":"https://api.github.com/users/CNevd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CNevd/subscriptions","organizations_url":"https://api.github.com/users/CNevd/orgs","repos_url":"https://api.github.com/users/CNevd/repos","events_url":"https://api.github.com/users/CNevd/events{/privacy}","received_events_url":"https://api.github.com/users/CNevd/received_events","type":"User","site_admin":false},"created_at":"2016-12-01T00:41:52Z","updated_at":"2016-12-01T00:41:52Z","body":"u can use mx.metric.CompositeEvalMetric() to get multi-metric"}}
{"actor":{"display_login":"CNevd"}}