{"action":"created","issue":{"url":"https://api.github.com/repos/lammps/lammps/issues/250","repository_url":"https://api.github.com/repos/lammps/lammps","labels_url":"https://api.github.com/repos/lammps/lammps/issues/250/labels{/name}","comments_url":"https://api.github.com/repos/lammps/lammps/issues/250/comments","events_url":"https://api.github.com/repos/lammps/lammps/issues/250/events","html_url":"https://github.com/lammps/lammps/issues/250","id":186872332,"number":250,"title":"KOKKOS-CUDA segfault when run with mpirun -n 2","user":{"login":"andeplane","id":16174,"avatar_url":"https://avatars.githubusercontent.com/u/16174?v=3","gravatar_id":"","url":"https://api.github.com/users/andeplane","html_url":"https://github.com/andeplane","followers_url":"https://api.github.com/users/andeplane/followers","following_url":"https://api.github.com/users/andeplane/following{/other_user}","gists_url":"https://api.github.com/users/andeplane/gists{/gist_id}","starred_url":"https://api.github.com/users/andeplane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andeplane/subscriptions","organizations_url":"https://api.github.com/users/andeplane/orgs","repos_url":"https://api.github.com/users/andeplane/repos","events_url":"https://api.github.com/users/andeplane/events{/privacy}","received_events_url":"https://api.github.com/users/andeplane/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-11-02T17:45:57Z","updated_at":"2016-11-02T18:58:24Z","closed_at":null,"body":"Hi,\r\n\r\nI have two GTX Titan on my Ubuntu 16.04 LTS with Cuda 8.0 and OpenMPI 1.10.2. LAMMPS runs perfectly with a single core and a single GPU, but if I run with\r\n\r\n`mpirun -np 2 lmp_kokkos_cuda_openmpi -k on g 2 -sf kk -in in.lj`\r\n\r\n(or any other script), it segfaults every time:\r\n\r\n```\r\nLAMMPS (27 Oct 2016-ICMS)\r\nKOKKOS mode is enabled (../kokkos.cpp:40)\r\n  using 2 GPU(s)\r\n  using 1 OpenMP thread(s) per MPI task\r\nLattice spacing in x,y,z = 1.6796 1.6796 1.6796\r\nCreated orthogonal box = (0 0 0) to (33.5919 33.5919 33.5919)\r\n  1 by 1 by 2 MPI processor grid\r\nCreated 32000 atoms\r\nNeighbor list info ...\r\n  1 neighbor list requests\r\n  update every 20 steps, delay 0 steps, check no\r\n  max neighbors/atom: 2000, page size: 100000\r\n  master list distance cutoff = 2.8\r\n  ghost atom cutoff = 2.8\r\n  binsize = 1.4, bins = 24 24 24\r\nSetting up Verlet run ...\r\n  Unit style    : lj\r\n  Current step  : 0\r\n  Time step     : 0.005\r\n[pinje:16806] *** Process received signal ***\r\n[pinje:16806] Signal: Segmentation fault (11)\r\n[pinje:16806] Signal code: Invalid permissions (2)\r\n[pinje:16806] Failing at address: 0xf06e2fe80\r\n[pinje:16806] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f07c4f463d0]\r\n[pinje:16806] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9ee10)[0x7f07c4147e10]\r\n[pinje:16806] [ 2] /usr/lib/openmpi/lib/openmpi/mca_btl_vader.so(+0x2558)[0x7f07b7ed6558]\r\n[pinje:16806] [ 3] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_rndv+0x1da)[0x7f07b78a7eda]\r\n[pinje:16806] [ 4] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x76a)[0x7f07b789c9ea]\r\n[pinje:16806] [ 5] [pinje:16807] *** Process received signal ***\r\n/usr/lib/libmpi.so.12(PMPI_Send+0x14b)[0x7f07c51bbcdb]\r\n[pinje:16806] [ 6] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0xbcb4be]\r\n[pinje:16806] [ 7] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x521016]\r\n[pinje:16806] [ 8] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x7fa34e]\r\n[pinje:16806] [ 9] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x542989]\r\n[pinje:16806] [10] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x540c5f]\r\n[pinje:16806] [11] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x5417c7]\r\n[pinje:16806] [12] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x4132f6]\r\n[pinje:16806] [13] [pinje:16807] Signal: Segmentation fault (11)\r\n[pinje:16807] Signal code: Invalid permissions (2)\r\n[pinje:16807] Failing at address: 0xf06e2fe80\r\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f07c40c9830]\r\n[pinje:16806] [14] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x41b249]\r\n[pinje:16806] *** End of error message ***\r\n[pinje:16807] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7f9b459813d0]\r\n[pinje:16807] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9ee10)[0x7f9b44b82e10]\r\n[pinje:16807] [ 2] /usr/lib/openmpi/lib/openmpi/mca_btl_vader.so(+0x2558)[0x7f9b388d0558]\r\n[pinje:16807] [ 3] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_rndv+0x1da)[0x7f9b382a1eda]\r\n[pinje:16807] [ 4] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x76a)[0x7f9b382969ea]\r\n[pinje:16807] [ 5] /usr/lib/libmpi.so.12(PMPI_Send+0x14b)[0x7f9b45bf6cdb]\r\n[pinje:16807] [ 6] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0xbcb4be]\r\n[pinje:16807] [ 7] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x521016]\r\n[pinje:16807] [ 8] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x7fa34e]\r\n[pinje:16807] [ 9] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x542989]\r\n[pinje:16807] [10] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x540c5f]\r\n[pinje:16807] [11] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x5417c7]\r\n[pinje:16807] [12] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x4132f6]\r\n[pinje:16807] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f9b44b04830]\r\n[pinje:16807] [14] /repos/lammps/src/lmp_kokkos_cuda_openmpi[0x41b249]\r\n[pinje:16807] *** End of error message ***\r\n--------------------------------------------------------------------------\r\nmpirun noticed that process rank 0 with PID 16806 on node pinje exited on signal 11 (Segmentation fault).\r\n```\r\n\r\nNot the best call stack I've seen. I can recompile and rerun with debug tomorrow. I have had this problem before OS reinstall.\r\n\r\nHave you experienced this before? "},"comment":{"url":"https://api.github.com/repos/lammps/lammps/issues/comments/257965443","html_url":"https://github.com/lammps/lammps/issues/250#issuecomment-257965443","issue_url":"https://api.github.com/repos/lammps/lammps/issues/250","id":257965443,"user":{"login":"andeplane","id":16174,"avatar_url":"https://avatars.githubusercontent.com/u/16174?v=3","gravatar_id":"","url":"https://api.github.com/users/andeplane","html_url":"https://github.com/andeplane","followers_url":"https://api.github.com/users/andeplane/followers","following_url":"https://api.github.com/users/andeplane/following{/other_user}","gists_url":"https://api.github.com/users/andeplane/gists{/gist_id}","starred_url":"https://api.github.com/users/andeplane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andeplane/subscriptions","organizations_url":"https://api.github.com/users/andeplane/orgs","repos_url":"https://api.github.com/users/andeplane/repos","events_url":"https://api.github.com/users/andeplane/events{/privacy}","received_events_url":"https://api.github.com/users/andeplane/received_events","type":"User","site_admin":false},"created_at":"2016-11-02T18:58:24Z","updated_at":"2016-11-02T18:58:24Z","body":"@akohlmey \r\n```\r\nanderhaf@pinje:/projects/sw/sw$ ls -ls /dev/nvidia*\r\n0 crw-rw-rw- 1 root root 195, 254 Nov  1 09:31 /dev/nvidia-modeset\r\n0 crw-rw-rw- 1 root root 242,   0 Nov  2 18:19 /dev/nvidia-uvm\r\n0 crw-rw-rw- 1 root root 242,   1 Nov  2 18:19 /dev/nvidia-uvm-tools\r\n0 crw-rw-rw- 1 root root 195,   0 Nov  1 09:31 /dev/nvidia0\r\n0 crw-rw-rw- 1 root root 195,   1 Nov  1 09:31 /dev/nvidia1\r\n0 crw-rw-rw- 1 root root 195, 255 Nov  1 09:31 /dev/nvidiactl\r\n```\r\n\r\nThe GPU package works fine with both gpu's, also with mpirun -n 2 (tested the SW benchmark). Running the same script with the command i mentioned in original post gives segfault.\r\n\r\nAny ideas?"}}
{"actor":{"display_login":"andeplane"}}