{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/14054","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/14054/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/14054/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/14054/events","html_url":"https://github.com/pytorch/pytorch/pull/14054","id":381408230,"node_id":"MDExOlB1bGxSZXF1ZXN0MjMxMzg4MjMz","number":14054,"title":"Jaliyae/dataloader api","user":{"login":"jaliyae","id":12703337,"node_id":"MDQ6VXNlcjEyNzAzMzM3","avatar_url":"https://avatars3.githubusercontent.com/u/12703337?v=4","gravatar_id":"","url":"https://api.github.com/users/jaliyae","html_url":"https://github.com/jaliyae","followers_url":"https://api.github.com/users/jaliyae/followers","following_url":"https://api.github.com/users/jaliyae/following{/other_user}","gists_url":"https://api.github.com/users/jaliyae/gists{/gist_id}","starred_url":"https://api.github.com/users/jaliyae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaliyae/subscriptions","organizations_url":"https://api.github.com/users/jaliyae/orgs","repos_url":"https://api.github.com/users/jaliyae/repos","events_url":"https://api.github.com/users/jaliyae/events{/privacy}","received_events_url":"https://api.github.com/users/jaliyae/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-11-16T01:24:36Z","updated_at":"2018-12-20T21:39:44Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/pytorch/pytorch/pulls/14054","html_url":"https://github.com/pytorch/pytorch/pull/14054","diff_url":"https://github.com/pytorch/pytorch/pull/14054.diff","patch_url":"https://github.com/pytorch/pytorch/pull/14054.patch"},"body":"@goldsborough @apaszke \r\nThis PR includes all the changes we need from the dataloader to support chunk loading. I incorporated most of your suggestions in our previous PRs. This still needs to be integrated with the SharedDataSet change once it is merged. In the  meantime, please have a look and let me know your comments. here is a brief description of what we plan to do with chunk loading.\r\n\r\n**Chunk data set** - A dataset that support preloading whole chunks (not a batch) to memory. For this we don't need to know the size of the dataset like in the current data sets. It will be a stateful dataset, so we don't want it to be copied to threads in the dataloader, therefore this PR depends on the SharedDataSet PR. \r\n\r\n**Dataset.reset()** - Similar to dataloader.reset(), this will help us start the preloading and clear Chunk data set at epoch boundaries. \r\n\r\n**Sampler.reset(new_size)** - We need this support, because the size of the dataset or the chunks are unknown at the start, so we need to set the size of the samplers after loading data.\r\n\r\nThese are the minimum changes we need and as you can see, most of the changes going to be in the chunk data set, which will be completed with all the test cases once we conclude the API changes. \r\n\r\nPlease let me know your comments/suggestions.\r\n\r\nPrevious PRs: [Sampler.rest()](https://github.com/pytorch/pytorch/pull/13870/files), [Dataloader](https://github.com/pytorch/pytorch/pull/13585/files), [SharedDataSet](https://github.com/pytorch/pytorch/pull/13800/files)\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/449144049","html_url":"https://github.com/pytorch/pytorch/pull/14054#issuecomment-449144049","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/14054","id":449144049,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0OTE0NDA0OQ==","user":{"login":"ezyang","id":13564,"node_id":"MDQ6VXNlcjEzNTY0","avatar_url":"https://avatars0.githubusercontent.com/u/13564?v=4","gravatar_id":"","url":"https://api.github.com/users/ezyang","html_url":"https://github.com/ezyang","followers_url":"https://api.github.com/users/ezyang/followers","following_url":"https://api.github.com/users/ezyang/following{/other_user}","gists_url":"https://api.github.com/users/ezyang/gists{/gist_id}","starred_url":"https://api.github.com/users/ezyang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ezyang/subscriptions","organizations_url":"https://api.github.com/users/ezyang/orgs","repos_url":"https://api.github.com/users/ezyang/repos","events_url":"https://api.github.com/users/ezyang/events{/privacy}","received_events_url":"https://api.github.com/users/ezyang/received_events","type":"User","site_admin":false},"created_at":"2018-12-20T21:39:44Z","updated_at":"2018-12-20T21:39:44Z","author_association":"CONTRIBUTOR","body":"@apaszke @jaliyae What was the resolution of the meeting?"}}
{"actor":{"display_login":"ezyang"}}