{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/4934","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/4934/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/4934/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/4934/events","html_url":"https://github.com/pytorch/pytorch/issues/4934","id":292744990,"number":4934,"title":"[ppc64le] Seg fault with distributed training using multiple data loading workers","user":{"login":"vaisaxena","id":35957509,"avatar_url":"https://avatars0.githubusercontent.com/u/35957509?v=4","gravatar_id":"","url":"https://api.github.com/users/vaisaxena","html_url":"https://github.com/vaisaxena","followers_url":"https://api.github.com/users/vaisaxena/followers","following_url":"https://api.github.com/users/vaisaxena/following{/other_user}","gists_url":"https://api.github.com/users/vaisaxena/gists{/gist_id}","starred_url":"https://api.github.com/users/vaisaxena/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vaisaxena/subscriptions","organizations_url":"https://api.github.com/users/vaisaxena/orgs","repos_url":"https://api.github.com/users/vaisaxena/repos","events_url":"https://api.github.com/users/vaisaxena/events{/privacy}","received_events_url":"https://api.github.com/users/vaisaxena/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-01-30T11:17:26Z","updated_at":"2018-02-12T10:56:02Z","closed_at":null,"author_association":"NONE","body":"## Environment\r\n\r\n    OS: RHEL 7.3\r\n    Architecture: ppc64le\r\n    PyTorch version: 0.4.0a0+f94f572\r\n    How you installed PyTorch (conda, pip, source): source\r\n    Python version: Python 3.6.2 :: Anaconda, Inc.\r\n    CUDA/cuDNN version: 9.1/7.0.3\r\n    GPU models and configuration: Nvidia Tesla P100\r\n    GCC version  (if compiling from source): 4.8.5\r\n    MPI: OpenMPI 2.1.1\r\n\r\n## Problem\r\n\r\n1. I ran the imagenet training example (https://github.com/pytorch/examples/tree/master/imagenet) in distributed mode (with OpenMPI) with non-zero data loading workers (>0) in torch.utils.data.DataLoader. In this case, the application either hangs or seg faults after showing following warning from OpenMPI:\r\n\r\n[Output of mpirun  -np 4 -x LD_LIBRARY_PATH python main.py  -a resnet50 --dist-backend mpi --world-size 4 --epochs 2 -j 4 /gpfs/r92gpfs02/vaibhav/shared/small-imagenet]\r\n\r\ncreating model 'resnet50'\r\ncreating model 'resnet50'\r\ncreating model 'resnet50'\r\ncreating model 'resnet50'\r\n\r\n--------------------------------------------------------------------------\r\nA process has executed an operation involving a call to the\r\n\"fork()\" system call to create a child process.  Open MPI is currently\r\noperating in a condition that could result in memory corruption or\r\nother system errors; your job may hang, crash, or produce silent\r\ndata corruption.  The use of fork() (or system() or other calls that\r\ncreate child processes) is strongly discouraged.\r\n\r\nThe process that invoked fork was:\r\n\r\n  Local host:          [[36650,1],0] (PID 144066)\r\n\r\nIf you are *absolutely sure* that your application will successfully\r\nand correctly survive a call to fork(), you may disable this warning\r\nby setting the mpi_warn_on_fork MCA parameter to 0.\r\n--------------------------------------------------------------------------\"\r\n\r\nThe complete stack trace of the seg fault is provided in the attached file debug_4node_j4_e2.log\r\n[debug_4node_j4_e2.log](https://github.com/pytorch/pytorch/files/1677387/debug_4node_j4_e2.log)\r\n\r\n2. The example works fine in serial mode (no MPI) with multiple data loading workers (running with -j > 0) \r\n\r\n3. The example also works fine in distributed mode (with OpenMPI) with no separate data loading worker (running the application with -j 0). In this case all the data loading happens in the main process itself. There is no \"fork\" related warning from OpenMPI in this case. \r\n\r\nSo there seems to be some problem using OpenMPI with multiple data loading workers. \r\n\r\nI also noticed a warning in https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L24 that DistributedDataParallel only works with \"nccl\" and \"gloo\", so I changed the imagenet/main.py to use torch.nn.DataParallel with a separate gradient average function instead of torch.nn.parallel.DistributedDataParallel in the distributed case (with MPI) but that also didn't help.  \r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/364889335","html_url":"https://github.com/pytorch/pytorch/issues/4934#issuecomment-364889335","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/4934","id":364889335,"user":{"login":"vaisaxena","id":35957509,"avatar_url":"https://avatars0.githubusercontent.com/u/35957509?v=4","gravatar_id":"","url":"https://api.github.com/users/vaisaxena","html_url":"https://github.com/vaisaxena","followers_url":"https://api.github.com/users/vaisaxena/followers","following_url":"https://api.github.com/users/vaisaxena/following{/other_user}","gists_url":"https://api.github.com/users/vaisaxena/gists{/gist_id}","starred_url":"https://api.github.com/users/vaisaxena/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vaisaxena/subscriptions","organizations_url":"https://api.github.com/users/vaisaxena/orgs","repos_url":"https://api.github.com/users/vaisaxena/repos","events_url":"https://api.github.com/users/vaisaxena/events{/privacy}","received_events_url":"https://api.github.com/users/vaisaxena/received_events","type":"User","site_admin":false},"created_at":"2018-02-12T10:56:02Z","updated_at":"2018-02-12T10:56:02Z","author_association":"NONE","body":"The problem gets resolved by changing the multiprocessing start method to 'spawn' by calling \r\n\r\nimport torch.multiprocessing as mp\r\nif  \\_\\_name__ == '\\_\\_main\\_\\_':\r\n    mp.set_start_method('spawn') \r\n\r\nOpenMPI still gives fork warning as before but the code runs fine without any problem."}}
{"actor":{"display_login":"vaisaxena"}}