{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/DALI/issues/259","repository_url":"https://api.github.com/repos/NVIDIA/DALI","labels_url":"https://api.github.com/repos/NVIDIA/DALI/issues/259/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/DALI/issues/259/comments","events_url":"https://api.github.com/repos/NVIDIA/DALI/issues/259/events","html_url":"https://github.com/NVIDIA/DALI/issues/259","id":375680193,"node_id":"MDU6SXNzdWUzNzU2ODAxOTM=","number":259,"title":"Pipeline building slow over nfs","user":{"login":"Delaunay","id":4311937,"node_id":"MDQ6VXNlcjQzMTE5Mzc=","avatar_url":"https://avatars2.githubusercontent.com/u/4311937?v=4","gravatar_id":"","url":"https://api.github.com/users/Delaunay","html_url":"https://github.com/Delaunay","followers_url":"https://api.github.com/users/Delaunay/followers","following_url":"https://api.github.com/users/Delaunay/following{/other_user}","gists_url":"https://api.github.com/users/Delaunay/gists{/gist_id}","starred_url":"https://api.github.com/users/Delaunay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Delaunay/subscriptions","organizations_url":"https://api.github.com/users/Delaunay/orgs","repos_url":"https://api.github.com/users/Delaunay/repos","events_url":"https://api.github.com/users/Delaunay/events{/privacy}","received_events_url":"https://api.github.com/users/Delaunay/received_events","type":"User","site_admin":false},"labels":[{"id":951950662,"node_id":"MDU6TGFiZWw5NTE5NTA2NjI=","url":"https://api.github.com/repos/NVIDIA/DALI/labels/question","name":"question","color":"d876e3","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-10-30T20:31:28Z","updated_at":"2018-10-31T08:54:53Z","closed_at":null,"author_association":"NONE","body":"We use nfs to store our datasets. NFS is configured to be cached locally once the files are accessed.\r\nNevertheless, dali pipeline can take up to 300 sec to build.\r\n\r\nWhat is done during the pipeline build time that could explain this huge build time ?\r\n\r\nAfter build is passed it runs fine and the dataset can be successfully cached for subsequent epochs,\r\nbut that initial 300 sec is problematic to us since we do a lot of small quick experiement. "},"comment":{"url":"https://api.github.com/repos/NVIDIA/DALI/issues/comments/434608435","html_url":"https://github.com/NVIDIA/DALI/issues/259#issuecomment-434608435","issue_url":"https://api.github.com/repos/NVIDIA/DALI/issues/259","id":434608435,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDYwODQzNQ==","user":{"login":"JanuszL","id":39967756,"node_id":"MDQ6VXNlcjM5OTY3NzU2","avatar_url":"https://avatars1.githubusercontent.com/u/39967756?v=4","gravatar_id":"","url":"https://api.github.com/users/JanuszL","html_url":"https://github.com/JanuszL","followers_url":"https://api.github.com/users/JanuszL/followers","following_url":"https://api.github.com/users/JanuszL/following{/other_user}","gists_url":"https://api.github.com/users/JanuszL/gists{/gist_id}","starred_url":"https://api.github.com/users/JanuszL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JanuszL/subscriptions","organizations_url":"https://api.github.com/users/JanuszL/orgs","repos_url":"https://api.github.com/users/JanuszL/repos","events_url":"https://api.github.com/users/JanuszL/events{/privacy}","received_events_url":"https://api.github.com/users/JanuszL/received_events","type":"User","site_admin":false},"created_at":"2018-10-31T08:54:53Z","updated_at":"2018-10-31T08:54:53Z","author_association":"CONTRIBUTOR","body":"@nouiz - answering your question yes and no.\r\nYes, technically it is possible to rework current iterator and reader to work in the async way.\r\nNo, because you will get wrong results. Now DALI reads whole data set and shuffles it initially during setup. Thanks to it you wont end up with getting batches with only label 0 data, then label 1 and etc.\r\nI think that PyTorch torchvision.datasets.ImageFolder works in the same way, it discovers classes and then traverses all classes' dirs looking for images - https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L84.\r\nTracked as DALI-322.\r\n"}}
{"actor":{"display_login":"JanuszL"}}