{"action":"opened","issue":{"url":"https://api.github.com/repos/uber/pyro/issues/1437","repository_url":"https://api.github.com/repos/uber/pyro","labels_url":"https://api.github.com/repos/uber/pyro/issues/1437/labels{/name}","comments_url":"https://api.github.com/repos/uber/pyro/issues/1437/comments","events_url":"https://api.github.com/repos/uber/pyro/issues/1437/events","html_url":"https://github.com/uber/pyro/issues/1437","id":367905067,"node_id":"MDU6SXNzdWUzNjc5MDUwNjc=","number":1437,"title":"[feature request] manual mini-batching and batch dimension scaling","user":{"login":"mbabadi","id":15305869,"node_id":"MDQ6VXNlcjE1MzA1ODY5","avatar_url":"https://avatars1.githubusercontent.com/u/15305869?v=4","gravatar_id":"","url":"https://api.github.com/users/mbabadi","html_url":"https://github.com/mbabadi","followers_url":"https://api.github.com/users/mbabadi/followers","following_url":"https://api.github.com/users/mbabadi/following{/other_user}","gists_url":"https://api.github.com/users/mbabadi/gists{/gist_id}","starred_url":"https://api.github.com/users/mbabadi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbabadi/subscriptions","organizations_url":"https://api.github.com/users/mbabadi/orgs","repos_url":"https://api.github.com/users/mbabadi/repos","events_url":"https://api.github.com/users/mbabadi/events{/privacy}","received_events_url":"https://api.github.com/users/mbabadi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-08T18:32:06Z","updated_at":"2018-10-08T18:32:06Z","closed_at":null,"author_association":"NONE","body":"In models with mixed levels of nesting (e.g. global_plate > local_plate_1 > local_plate_2 > ...), mibi-batching across different batch dimensions requires introducing proper scale factors for each batch dimension. Pyro handles these scale factors automatically if mini-batching is achieved via `pyro.iarange(..., size=..., subsample_size=...`) or `pyro.iarange(..., size=..., subsample=...)`. The latter construct is flexible and allows arbitrary mibi-batching schemes, including big data situations where the full data tensor can not be loaded all at once.\r\n\r\nMini-batching, however, is often done _manually_ and _externally_ and not via `pyro.iarange`. In such cases, the appropriate scale factors must also be applied manually via `poutine.scale`. We are being consistent here: manual mini-batching? then manual scaling. However, most of the examples (DMM, VAE, ...) have little to no emphasis on this issue and neglect scaling altogether. While convergence is not a big deal while working with adaptive optimizers, neglecting the scale factors leads to wrong ELBO estimates.\r\n\r\n- [ ] Adding a word of caution to the examples about scale factors and/or throwing in `poutine.scale` when mini-batching manually to set a good precedent for the new users?"}}
{"actor":{"display_login":"mbabadi"}}