{"action":"opened","issue":{"url":"https://api.github.com/repos/mlperf/reference/issues/47","repository_url":"https://api.github.com/repos/mlperf/reference","labels_url":"https://api.github.com/repos/mlperf/reference/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/mlperf/reference/issues/47/comments","events_url":"https://api.github.com/repos/mlperf/reference/issues/47/events","html_url":"https://github.com/mlperf/reference/issues/47","id":325690434,"number":47,"title":"[speech_recognition] run_and_time error","user":{"login":"linrio","id":16930748,"avatar_url":"https://avatars2.githubusercontent.com/u/16930748?v=4","gravatar_id":"","url":"https://api.github.com/users/linrio","html_url":"https://github.com/linrio","followers_url":"https://api.github.com/users/linrio/followers","following_url":"https://api.github.com/users/linrio/following{/other_user}","gists_url":"https://api.github.com/users/linrio/gists{/gist_id}","starred_url":"https://api.github.com/users/linrio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linrio/subscriptions","organizations_url":"https://api.github.com/users/linrio/orgs","repos_url":"https://api.github.com/users/linrio/repos","events_url":"https://api.github.com/users/linrio/events{/privacy}","received_events_url":"https://api.github.com/users/linrio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-23T12:49:55Z","updated_at":"2018-05-23T12:49:55Z","closed_at":null,"author_association":"NONE","body":"I have built up the docker image and launch the docker instance:\r\n\r\n```\r\ncd pytorch\r\ncd docker\r\nsh build-docker.sh\r\nsh run-dev.sh\r\n```\r\nAnd modifying the run-dev.sh:\r\n\r\n```\r\n#!/bin/bash\r\nnvidia-docker run \\\r\n-v /home/linlf/project/linlf/mlperf/reference/speech_recognition:/speech_r    ecognition:rw \\\r\n-v /etc/passwd:/etc/passwd:ro \\\r\n-it --rm --user $(id -u) ds2-cuda9cudnn7:gpu\r\n```\r\n\r\nLaunch the docker instance:\r\n\r\n```\r\n.......\r\nRemoving intermediate container 450a04f6173f\r\n ---> 86101cb30a55\r\nStep 16/18 : RUN ls ctcdecode/third_party\r\n ---> Running in 1512a8bc9a65\r\nThreadPool\r\nboost_1_63_0.tar.gz\r\nkenlm\r\nopenfst-1.6.7.tar.gz\r\nutf8\r\nRemoving intermediate container 1512a8bc9a65\r\n ---> c482d12d74b2\r\nStep 17/18 : RUN cd ctcdecode; pip install .\r\n ---> Running in c9349e7fe2e2\r\nProcessing /tmp/ctcdecode\r\nRequirement already satisfied: cffi>=1.0.0 in /usr/local/lib/python2.7/dist-packages (from ctcdecode==0.3) (1.11.5)\r\nRequirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.0.0->ctcdecode==0.3) (2.18)\r\nBuilding wheels for collected packages: ctcdecode\r\n  Running setup.py bdist_wheel for ctcdecode: started\r\n  Running setup.py bdist_wheel for ctcdecode: still running...\r\n  Running setup.py bdist_wheel for ctcdecode: finished with status 'done'\r\n  Stored in directory: /tmp/pip-ephem-wheel-cache-dMi3ud/wheels/30/4f/54/e41e2d0b75f971a3f8daad6bb79bf61fef10d0e95c5b28a93d\r\nSuccessfully built ctcdecode\r\nInstalling collected packages: ctcdecode\r\nSuccessfully installed ctcdecode-0.3\r\nRemoving intermediate container c9349e7fe2e2\r\n ---> e3048146fa23\r\nStep 18/18 : ENV SHELL /bin/bash\r\n ---> Running in dae62270553e\r\nRemoving intermediate container dae62270553e\r\n ---> 3d2838335713\r\nSuccessfully built 3d2838335713\r\nSuccessfully tagged ds2-cuda9cudnn7:gpu\r\n```\r\n\r\nNext I ready to execute the `run_and_time.sh`, however it throws an error:\r\n\r\n```\r\nroot@5316ed702b45:/speech_recognition/pytorch# time sh run_and_time.sh | tee speech_ds2.out\r\n=======================================================\r\n***acc                       = 23.0 \r\n***continue_from             =  \r\n***save_folder               = models/ \r\n***start_epoch               = -1 \r\n***checkpoint                = False \r\n***seed                      = 1 \r\n***model_path                = models/deepspeech_t1.pth.tar \r\n=======================================================\r\nDirectory already exists.\r\nDataParallel(\r\n  (module): DeepSpeech(\r\n    (conv): Sequential(\r\n      (0): Conv2d(1, 32, kernel_size=(41, 11), stride=(2, 2))\r\n      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\r\n      (2): Hardtanh(min_val=0, max_val=20, inplace)\r\n      (3): Conv2d(32, 32, kernel_size=(21, 11), stride=(2, 1))\r\n      (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\r\n      (5): Hardtanh(min_val=0, max_val=20, inplace)\r\n    )\r\n    (rnns): Sequential(\r\n      (0): BatchRNN(\r\n        (rnn): GRU(672, 800, bidirectional=True)\r\n      )\r\n      (1): BatchRNN(\r\n        (batch_norm): SequenceWise (\r\n        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))\r\n        (rnn): GRU(800, 800, bidirectional=True)\r\n      )\r\n      (2): BatchRNN(\r\n        (batch_norm): SequenceWise (\r\n        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))\r\n        (rnn): GRU(800, 800, bidirectional=True)\r\n      )\r\n      (3): BatchRNN(\r\n        (batch_norm): SequenceWise (\r\n        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))\r\n        (rnn): GRU(800, 800, bidirectional=True)\r\n      )\r\n      (4): BatchRNN(\r\n        (batch_norm): SequenceWise (\r\n        BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))\r\n        (rnn): GRU(800, 800, bidirectional=True)\r\n      )\r\n    )\r\n    (fc): Sequential(\r\n      (0): SequenceWise (\r\n      Sequential(\r\n        (0): BatchNorm1d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\r\n        (1): Linear(in_features=800, out_features=29, bias=False)\r\n      ))\r\n    )\r\n    (inference_log_softmax): InferenceBatchLogSoftmax()\r\n  )\r\n)\r\nNumber of parameters: 38115968\r\nformats: can't open input file `/tmp/tmp_GBQNr.wav': WAVE: RIFF header not found\r\nERROR: Unexpected segmentation fault encountered in worker.\r\nTraceback (most recent call last):\r\n  File \"train.py\", line 289, in <module>\r\n    main()\r\n  File \"train.py\", line 176, in main\r\n    for i, (data) in enumerate(train_loader, start=start_iter):\r\n  File \"/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py\", line 280, in __next__\r\n    idx, batch = self._get_batch()\r\n  File \"/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py\", line 259, in _get_batch\r\n    return self.data_queue.get()\r\n  File \"/usr/lib/python2.7/multiprocessing/queues.py\", line 378, in get\r\n    return recv()\r\n  File \"/usr/local/lib/python2.7/dist-packages/torch/multiprocessing/queue.py\", line 21, in recv\r\n    buf = self.recv_bytes()\r\n  File \"/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py\", line 178, in handler\r\n    _error_if_any_worker_fails()\r\nRuntimeError: DataLoader worker (pid 81) is killed by signal: Segmentation fault.\r\n\r\nreal\t0m20.221s\r\nuser\t0m5.148s\r\nsys\t0m7.076s\r\n```\r\n\r\nWhat's error mean?\r\n"}}
{"actor":{"display_login":"linrio"}}