{"action":"created","issue":{"url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch/issues/22","repository_url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch","labels_url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch/issues/22/labels{/name}","comments_url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch/issues/22/comments","events_url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch/issues/22/events","html_url":"https://github.com/JianGoForIt/YellowFin_Pytorch/issues/22","id":297382301,"number":22,"title":"Bad performance on large vision models","user":{"login":"staticfloat","id":130920,"avatar_url":"https://avatars0.githubusercontent.com/u/130920?v=4","gravatar_id":"","url":"https://api.github.com/users/staticfloat","html_url":"https://github.com/staticfloat","followers_url":"https://api.github.com/users/staticfloat/followers","following_url":"https://api.github.com/users/staticfloat/following{/other_user}","gists_url":"https://api.github.com/users/staticfloat/gists{/gist_id}","starred_url":"https://api.github.com/users/staticfloat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/staticfloat/subscriptions","organizations_url":"https://api.github.com/users/staticfloat/orgs","repos_url":"https://api.github.com/users/staticfloat/repos","events_url":"https://api.github.com/users/staticfloat/events{/privacy}","received_events_url":"https://api.github.com/users/staticfloat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-15T09:45:19Z","updated_at":"2018-02-16T17:03:48Z","closed_at":null,"author_association":"NONE","body":"Hello there,\r\n\r\nI am doing my best to learn how to use this optimizer, as I would very much like to have an auto-tuned optimizer where I do not have to spend endless days fiddling with hyperparameters.  I have tried to use YellowFin to learn large vision models such as [MobileNet](https://arxiv.org/abs/1704.04861), but my results are always very disappointing as compared to a traditional optimizer such as `SGD`.  I am not so concerned about convergence time as I am about loss/accuracy; I have found that YellowFin tends to converge to a much worse loss/accuracy than my SGD runs do.\r\n\r\nI am posting here an example of training MobileNet on the ImageNet dataset with a batch size of 64, comparing the training and testing loss (as well as testing accuracy) of a few epochs of training on MobileNet.  In both cases, I have a learning rate schedule applied to set the learning rate factor to `0.3 ^ (epoch // 10)`, which causes the learning rate to fall to 3/10 of its value every 10 epochs.  You can see the effect of this learning rate schedule in the `sgd` plot fairly easily, the `yf` plot shows it less clearly.  In these figures, the training loss (per minibatch) is shown in blue, while the testing loss (per epoch) is shown in red, with the relevant axis shown on the left.  The top-1 and top-5 accuracies on the training dataset are shown in green (per epoch), with their relevant axis given on the right.  Other than the optimizer choice, all other training settings are the same, including minibatch size (64), dataset (ImageNet) and model architecture (MobileNet).\r\n\r\nHere is a plot for an SGD optimizer run (note that I have this model only partially trained, this is because it has trained enough that we can already see it will converge to a significantly better loss than the YF model did, below):\r\n![mobilenet SGD](https://user-images.githubusercontent.com/130920/36285968-71d6325e-1262-11e8-8017-06c00dd0918d.png)\r\n\r\nHere is a plot for a YellowFin optimizer run:\r\n![mobilenet YellowFin](https://user-images.githubusercontent.com/130920/36250128-bf7e10e2-11f1-11e8-887e-125547144027.png)\r\n\r\nIf there are any questions about my methodology I would be happy to explain in greater detail.  There is nothing particularly special going on in my model, I am simply trying to determine why YellowFin seems to converge with such poor results."},"comment":{"url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch/issues/comments/366295758","html_url":"https://github.com/JianGoForIt/YellowFin_Pytorch/issues/22#issuecomment-366295758","issue_url":"https://api.github.com/repos/JianGoForIt/YellowFin_Pytorch/issues/22","id":366295758,"user":{"login":"JianGoForIt","id":6808243,"avatar_url":"https://avatars2.githubusercontent.com/u/6808243?v=4","gravatar_id":"","url":"https://api.github.com/users/JianGoForIt","html_url":"https://github.com/JianGoForIt","followers_url":"https://api.github.com/users/JianGoForIt/followers","following_url":"https://api.github.com/users/JianGoForIt/following{/other_user}","gists_url":"https://api.github.com/users/JianGoForIt/gists{/gist_id}","starred_url":"https://api.github.com/users/JianGoForIt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JianGoForIt/subscriptions","organizations_url":"https://api.github.com/users/JianGoForIt/orgs","repos_url":"https://api.github.com/users/JianGoForIt/repos","events_url":"https://api.github.com/users/JianGoForIt/events{/privacy}","received_events_url":"https://api.github.com/users/JianGoForIt/received_events","type":"User","site_admin":false},"created_at":"2018-02-16T17:03:48Z","updated_at":"2018-02-16T17:03:48Z","author_association":"OWNER","body":"Dear Elliot,\r\n\r\nThanks a lot for the interest in our optimizer and for sharing your observation with us.\r\n\r\nYellowFin aims at the providing good results without much tuning. However, as seen in the SGD curve, the learning rate drop schedule is very important to improve accuracy. If you already know a good hand tuned learning rate / momentum schedule, it can typically help you do even better. Still, we would like to study your specific case, we believe that there is something interesting here.\r\n\r\nIf you want to further fine tune to improve YF performance, I would suggest considering the learning rate factor we provided in the implementation. The learning rate factor can 1) enforce a multiplier to the learning rate given by YF throughout 2) implement learning rate decay / learning rate drop schedule. Here is an example (https://github.com/JianGoForIt/YellowFin_Pytorch/blob/master/pytorch-cifar/main.py#L161) you can refer to. We also have some examples where TF, combined with learning rate drop/decay can further boost the performance in Appendix J4 in our new arxiv manuscript.\r\n\r\nWe would love to study this specific case, please let me know if you can share the code and data with us to play with.\r\n\r\nCheers,\r\n"}}
{"actor":{"display_login":"JianGoForIt"}}