{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA-Jetson/redtail/issues/47","repository_url":"https://api.github.com/repos/NVIDIA-Jetson/redtail","labels_url":"https://api.github.com/repos/NVIDIA-Jetson/redtail/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA-Jetson/redtail/issues/47/comments","events_url":"https://api.github.com/repos/NVIDIA-Jetson/redtail/issues/47/events","html_url":"https://github.com/NVIDIA-Jetson/redtail/issues/47","id":306555005,"number":47,"title":"Validation accuracy remains below 60%","user":{"login":"TritonSailor","id":4633834,"avatar_url":"https://avatars1.githubusercontent.com/u/4633834?v=4","gravatar_id":"","url":"https://api.github.com/users/TritonSailor","html_url":"https://github.com/TritonSailor","followers_url":"https://api.github.com/users/TritonSailor/followers","following_url":"https://api.github.com/users/TritonSailor/following{/other_user}","gists_url":"https://api.github.com/users/TritonSailor/gists{/gist_id}","starred_url":"https://api.github.com/users/TritonSailor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TritonSailor/subscriptions","organizations_url":"https://api.github.com/users/TritonSailor/orgs","repos_url":"https://api.github.com/users/TritonSailor/repos","events_url":"https://api.github.com/users/TritonSailor/events{/privacy}","received_events_url":"https://api.github.com/users/TritonSailor/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"Alexey-Kamenev","id":8118464,"avatar_url":"https://avatars0.githubusercontent.com/u/8118464?v=4","gravatar_id":"","url":"https://api.github.com/users/Alexey-Kamenev","html_url":"https://github.com/Alexey-Kamenev","followers_url":"https://api.github.com/users/Alexey-Kamenev/followers","following_url":"https://api.github.com/users/Alexey-Kamenev/following{/other_user}","gists_url":"https://api.github.com/users/Alexey-Kamenev/gists{/gist_id}","starred_url":"https://api.github.com/users/Alexey-Kamenev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alexey-Kamenev/subscriptions","organizations_url":"https://api.github.com/users/Alexey-Kamenev/orgs","repos_url":"https://api.github.com/users/Alexey-Kamenev/repos","events_url":"https://api.github.com/users/Alexey-Kamenev/events{/privacy}","received_events_url":"https://api.github.com/users/Alexey-Kamenev/received_events","type":"User","site_admin":false},"assignees":[{"login":"Alexey-Kamenev","id":8118464,"avatar_url":"https://avatars0.githubusercontent.com/u/8118464?v=4","gravatar_id":"","url":"https://api.github.com/users/Alexey-Kamenev","html_url":"https://github.com/Alexey-Kamenev","followers_url":"https://api.github.com/users/Alexey-Kamenev/followers","following_url":"https://api.github.com/users/Alexey-Kamenev/following{/other_user}","gists_url":"https://api.github.com/users/Alexey-Kamenev/gists{/gist_id}","starred_url":"https://api.github.com/users/Alexey-Kamenev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alexey-Kamenev/subscriptions","organizations_url":"https://api.github.com/users/Alexey-Kamenev/orgs","repos_url":"https://api.github.com/users/Alexey-Kamenev/repos","events_url":"https://api.github.com/users/Alexey-Kamenev/events{/privacy}","received_events_url":"https://api.github.com/users/Alexey-Kamenev/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-03-19T17:17:12Z","updated_at":"2018-03-20T23:36:06Z","closed_at":null,"author_association":"NONE","body":"Hello, I'm trying to verify I get correct results training with the forest trails dataset, using the instructions in the wiki, before I try to fine tune with my own data. However, I haven't been able to get validation accuracy above 60%.  \r\n\r\nI tried running 40 epochs, without much difference.  Here's a few screenshots and a caffe log.  What other info can I provide to help diagnose this issue?\r\n  \r\n\r\n![screenshot from 2018-03-19 12-00-42](https://user-images.githubusercontent.com/4633834/37610526-d2a58d22-2b6d-11e8-93ed-efba16e87655.png)\r\n![screenshot from 2018-03-19 12-00-04](https://user-images.githubusercontent.com/4633834/37610801-9a6776ae-2b6e-11e8-8908-516c241e8431.png)\r\n[caffe_output(5).log](https://github.com/NVIDIA-Jetson/redtail/files/1826060/caffe_output.5.log)\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA-Jetson/redtail/issues/comments/374793460","html_url":"https://github.com/NVIDIA-Jetson/redtail/issues/47#issuecomment-374793460","issue_url":"https://api.github.com/repos/NVIDIA-Jetson/redtail/issues/47","id":374793460,"user":{"login":"Alexey-Kamenev","id":8118464,"avatar_url":"https://avatars0.githubusercontent.com/u/8118464?v=4","gravatar_id":"","url":"https://api.github.com/users/Alexey-Kamenev","html_url":"https://github.com/Alexey-Kamenev","followers_url":"https://api.github.com/users/Alexey-Kamenev/followers","following_url":"https://api.github.com/users/Alexey-Kamenev/following{/other_user}","gists_url":"https://api.github.com/users/Alexey-Kamenev/gists{/gist_id}","starred_url":"https://api.github.com/users/Alexey-Kamenev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alexey-Kamenev/subscriptions","organizations_url":"https://api.github.com/users/Alexey-Kamenev/orgs","repos_url":"https://api.github.com/users/Alexey-Kamenev/repos","events_url":"https://api.github.com/users/Alexey-Kamenev/events{/privacy}","received_events_url":"https://api.github.com/users/Alexey-Kamenev/received_events","type":"User","site_admin":false},"created_at":"2018-03-20T23:36:06Z","updated_at":"2018-03-20T23:36:06Z","author_association":"OWNER","body":"Hello,\r\n60% validation accuracy is certainly low if you are training orientation head on original Forest Trails. With ResNet-18 you should easily get up to 85%. How did you create the dataset? I would guess you used our `idsia_trails_dataset_digits.py` with undersampling? The model that you are using looks good to me, it's the same that we used to train orientation head.\r\nThere are few options you can try:\r\n* try creating larger dataset (oversampling)\r\n* try using ordinary CE (cross entropy) loss instead of our custom loss by setting all parameters like `entScale` and others to 0 in `cee_loss` layer.\r\n* try using a very small subset of your dataset to see if you can get higher accuracy.\r\n* try changing other parameters, most notably minibatch size and learning rate.\r\n\r\nIf everything is setup correctly, here is what you should get (or something like that):\r\n\r\n![resnet_digits](https://user-images.githubusercontent.com/8118464/37688247-4c27cc70-2c5c-11e8-8a51-614aeaeae9fa.png)\r\n\r\nIf nothing of that helps, I will try to re-train our nets again but this might not happen until April as we have GTC 2018 next week and currently a bit busy.\r\n"}}
{"actor":{"display_login":"Alexey-Kamenev"}}