{"action":"opened","issue":{"url":"https://api.github.com/repos/SeanNaren/deepspeech.pytorch/issues/310","repository_url":"https://api.github.com/repos/SeanNaren/deepspeech.pytorch","labels_url":"https://api.github.com/repos/SeanNaren/deepspeech.pytorch/issues/310/labels{/name}","comments_url":"https://api.github.com/repos/SeanNaren/deepspeech.pytorch/issues/310/comments","events_url":"https://api.github.com/repos/SeanNaren/deepspeech.pytorch/issues/310/events","html_url":"https://github.com/SeanNaren/deepspeech.pytorch/issues/310","id":336051667,"node_id":"MDU6SXNzdWUzMzYwNTE2Njc=","number":310,"title":"Dataloader get 5x slow down at the 2nd epoch on CPU with augment=True","user":{"login":"xhzhao","id":17486215,"node_id":"MDQ6VXNlcjE3NDg2MjE1","avatar_url":"https://avatars1.githubusercontent.com/u/17486215?v=4","gravatar_id":"","url":"https://api.github.com/users/xhzhao","html_url":"https://github.com/xhzhao","followers_url":"https://api.github.com/users/xhzhao/followers","following_url":"https://api.github.com/users/xhzhao/following{/other_user}","gists_url":"https://api.github.com/users/xhzhao/gists{/gist_id}","starred_url":"https://api.github.com/users/xhzhao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xhzhao/subscriptions","organizations_url":"https://api.github.com/users/xhzhao/orgs","repos_url":"https://api.github.com/users/xhzhao/repos","events_url":"https://api.github.com/users/xhzhao/events{/privacy}","received_events_url":"https://api.github.com/users/xhzhao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-27T02:06:06Z","updated_at":"2018-06-27T02:06:06Z","closed_at":null,"author_association":"NONE","body":"hi i'm trying to run this workload on CPU, and i found that the dataloader takes too much time in the 2nd epoch. The dataset is AN4, and the augment=True, num_worker=1. The hotspot in dataloader is the sox command: https://github.com/SeanNaren/deepspeech.pytorch/blob/master/data/data_loader.py#L294 \r\nI get the average time on this line:\r\n- 1st epoch average: 0.0261 s\r\n- 2nd epoch average: 0.1255 s\r\nI double checked that GPU have no such issue.\r\nI understand that one new dataloader thread will be created in the 2nd epoch, and the augmented file will be saved at the /tmp/ folder. But i really don't know why this sox command takes so much time in the second epoch!!!\r\n"}}
{"actor":{"display_login":"xhzhao"}}