{"action":"opened","issue":{"url":"https://api.github.com/repos/Swall0w/papers/issues/379","repository_url":"https://api.github.com/repos/Swall0w/papers","labels_url":"https://api.github.com/repos/Swall0w/papers/issues/379/labels{/name}","comments_url":"https://api.github.com/repos/Swall0w/papers/issues/379/comments","events_url":"https://api.github.com/repos/Swall0w/papers/issues/379/events","html_url":"https://github.com/Swall0w/papers/issues/379","id":318580265,"number":379,"title":"Revisiting Small Batch Training for Deep Neural Networks","user":{"login":"Swall0w","id":9757156,"avatar_url":"https://avatars2.githubusercontent.com/u/9757156?v=4","gravatar_id":"","url":"https://api.github.com/users/Swall0w","html_url":"https://github.com/Swall0w","followers_url":"https://api.github.com/users/Swall0w/followers","following_url":"https://api.github.com/users/Swall0w/following{/other_user}","gists_url":"https://api.github.com/users/Swall0w/gists{/gist_id}","starred_url":"https://api.github.com/users/Swall0w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Swall0w/subscriptions","organizations_url":"https://api.github.com/users/Swall0w/orgs","repos_url":"https://api.github.com/users/Swall0w/repos","events_url":"https://api.github.com/users/Swall0w/events{/privacy}","received_events_url":"https://api.github.com/users/Swall0w/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-27T23:57:07Z","updated_at":"2018-04-27T23:57:07Z","closed_at":null,"author_association":"OWNER","body":"Masters, Dominic, Luschi, Carlo\n\nModern deep neural network training is typically based on mini-batch\nstochastic gradient optimization. While the use of large mini-batches increases\nthe available computational parallelism, small batch training has been shown to\nprovide improved generalization performance and allows a significantly smaller\nmemory footprint, which might also be exploited to improve machine throughput.\nIn this paper, we review common assumptions on learning rate scaling and\ntraining duration, as a basis for an experimental comparison of test\nperformance for different mini-batch sizes. We adopt a learning rate that\ncorresponds to a constant average weight update per gradient calculation (i.e.,\nper unit cost of computation), and point out that this results in a variance of\nthe weight updates that increases linearly with the mini-batch size $m$.\nThe collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet\ndatasets show that increasing the mini-batch size progressively reduces the\nrange of learning rates that provide stable convergence and acceptable test\nperformance. On the other hand, small mini-batch sizes provide more up-to-date\ngradient calculations, which yields more stable and reliable training. The best\nperformance has been consistently obtained for mini-batch sizes between $m = 2$\nand $m = 32$, which contrasts with recent work advocating the use of mini-batch\nsizes in the thousands.\n\n\nhttps://arxiv.org/abs/1804.07612"}}
{"actor":{"display_login":"Swall0w"}}