{"action":"created","issue":{"url":"https://api.github.com/repos/nicholas-leonard/dp/issues/194","repository_url":"https://api.github.com/repos/nicholas-leonard/dp","labels_url":"https://api.github.com/repos/nicholas-leonard/dp/issues/194/labels{/name}","comments_url":"https://api.github.com/repos/nicholas-leonard/dp/issues/194/comments","events_url":"https://api.github.com/repos/nicholas-leonard/dp/issues/194/events","html_url":"https://github.com/nicholas-leonard/dp/issues/194","id":148466344,"number":194,"title":"Multi-threaded loading of multi-label dataset","user":{"login":"jrbtaylor","id":15191858,"avatar_url":"https://avatars.githubusercontent.com/u/15191858?v=3","gravatar_id":"","url":"https://api.github.com/users/jrbtaylor","html_url":"https://github.com/jrbtaylor","followers_url":"https://api.github.com/users/jrbtaylor/followers","following_url":"https://api.github.com/users/jrbtaylor/following{/other_user}","gists_url":"https://api.github.com/users/jrbtaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/jrbtaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrbtaylor/subscriptions","organizations_url":"https://api.github.com/users/jrbtaylor/orgs","repos_url":"https://api.github.com/users/jrbtaylor/repos","events_url":"https://api.github.com/users/jrbtaylor/events{/privacy}","received_events_url":"https://api.github.com/users/jrbtaylor/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-04-14T19:59:28Z","updated_at":"2016-07-29T19:08:14Z","closed_at":null,"body":"A flat folder structure is not possible with a multi-label dataset (in this case, video). Is it possible to set up asynchronous loading (and label extraction from a .json file) in dp?"},"comment":{"url":"https://api.github.com/repos/nicholas-leonard/dp/issues/comments/236267670","html_url":"https://github.com/nicholas-leonard/dp/issues/194#issuecomment-236267670","issue_url":"https://api.github.com/repos/nicholas-leonard/dp/issues/194","id":236267670,"user":{"login":"jrbtaylor","id":15191858,"avatar_url":"https://avatars.githubusercontent.com/u/15191858?v=3","gravatar_id":"","url":"https://api.github.com/users/jrbtaylor","html_url":"https://github.com/jrbtaylor","followers_url":"https://api.github.com/users/jrbtaylor/followers","following_url":"https://api.github.com/users/jrbtaylor/following{/other_user}","gists_url":"https://api.github.com/users/jrbtaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/jrbtaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrbtaylor/subscriptions","organizations_url":"https://api.github.com/users/jrbtaylor/orgs","repos_url":"https://api.github.com/users/jrbtaylor/repos","events_url":"https://api.github.com/users/jrbtaylor/events{/privacy}","received_events_url":"https://api.github.com/users/jrbtaylor/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T19:08:14Z","updated_at":"2016-07-29T19:08:14Z","body":"Yes, that is the only way I found to do it. The threads package is pretty easy to use once the basic concept is figured out. You pass each worker two functions: one it runs independent of the other workers and one that is called by the main thread (not a worker) upon completion of the first. Data loading happens in the worker thread (the first function). GPU training has to occur in the latter function, as each worker has no idea what the other is doing, so they might write to overlapping GPU memory."}}
{"actor":{"display_login":"jrbtaylor"}}