{"action":"opened","issue":{"url":"https://api.github.com/repos/NVIDIA/nccl-tests/issues/7","repository_url":"https://api.github.com/repos/NVIDIA/nccl-tests","labels_url":"https://api.github.com/repos/NVIDIA/nccl-tests/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/nccl-tests/issues/7/comments","events_url":"https://api.github.com/repos/NVIDIA/nccl-tests/issues/7/events","html_url":"https://github.com/NVIDIA/nccl-tests/issues/7","id":317570123,"number":7,"title":"Out of bounds values : 248 FAILED","user":{"login":"wm10240","id":6409840,"avatar_url":"https://avatars0.githubusercontent.com/u/6409840?v=4","gravatar_id":"","url":"https://api.github.com/users/wm10240","html_url":"https://github.com/wm10240","followers_url":"https://api.github.com/users/wm10240/followers","following_url":"https://api.github.com/users/wm10240/following{/other_user}","gists_url":"https://api.github.com/users/wm10240/gists{/gist_id}","starred_url":"https://api.github.com/users/wm10240/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wm10240/subscriptions","organizations_url":"https://api.github.com/users/wm10240/orgs","repos_url":"https://api.github.com/users/wm10240/repos","events_url":"https://api.github.com/users/wm10240/events{/privacy}","received_events_url":"https://api.github.com/users/wm10240/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-25T10:33:03Z","updated_at":"2018-04-25T10:33:03Z","closed_at":null,"author_association":"NONE","body":"hi,   when I use command \"mpirun -v --allow-run-as-root -x NCCL_SOCKET_IFNAME=ib0  -x NCCL_IB_CUDA_SUPPORT=1 -mca btl_tcp_if_include ib0 -np 2 -host 192.168.1.14,192.168.1.13 \r\n-pernode ./build/all_reduce_perf -b 9 -e 32M -g 8 -c 1 -z 0\" , the result show \"Out of bounds values : 248 FAILED\",  I have no idea about this error, So It‘s Very grateful that you have any message to me about this error。\r\n\r\nThe result：\r\n\r\n--------------------------------------------------------------------------\r\nWARNING: There are more than one active ports on host 'seedsmed-f02hpc02', but the\r\ndefault subnet GID prefix was detected on more than one of these\r\nports.  If these ports are connected to different physical IB\r\nnetworks, this configuration will fail in Open MPI.  This version of\r\nOpen MPI requires that every physically separate IB subnet that is\r\nused between connected MPI processes must have different subnet ID\r\nvalues.\r\n\r\nPlease see this FAQ entry for more details:\r\n\r\n  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid\r\n\r\nNOTE: You can turn off this warning by setting the MCA parameter\r\n      btl_openib_warn_default_gid_prefix to 0.\r\n--------------------------------------------------------------------------\r\nnThread 1 nGpus 8 minBytes 9 maxBytes 33554432 step: 1048576(bytes) warmup iters: 5 iters: 20 validation: 1\r\n--------------------------------------------------------------------------\r\nA process has executed an operation involving a call to the\r\n\"fork()\" system call to create a child process.  Open MPI is currently\r\noperating in a condition that could result in memory corruption or\r\nother system errors; your job may hang, crash, or produce silent\r\ndata corruption.  The use of fork() (or system() or other calls that\r\ncreate child processes) is strongly discouraged.\r\n\r\nThe process that invoked fork was:\r\n\r\n  Local host:          [[62619,1],1] (PID 3535)\r\n\r\nIf you are *absolutely sure* that your application will successfully\r\nand correctly survive a call to fork(), you may disable this warning\r\nby setting the mpi_warn_on_fork MCA parameter to 0.\r\n--------------------------------------------------------------------------\r\n[seedsmed-f02hpc01:03622] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix\r\n[seedsmed-f02hpc01:03622] Set MCA parameter \"orte_base_help_aggregate\" to 0 to see all help / error messages\r\n[seedsmed-f02hpc01:03622] 1 more process has sent help message help-opal-runtime.txt / opal_init:warn-fork\r\n# NCCL Tests compiled with NCCL 2.1\r\n# Using devices\r\n#   Rank  0 on seedsmed-f02hpc01 device  0 [0x04] Tesla V100-PCIE-16GB\r\n#   Rank  1 on seedsmed-f02hpc01 device  1 [0x06] Tesla V100-PCIE-16GB\r\n#   Rank  2 on seedsmed-f02hpc01 device  2 [0x07] Tesla V100-PCIE-16GB\r\n#   Rank  3 on seedsmed-f02hpc01 device  3 [0x08] Tesla V100-PCIE-16GB\r\n#   Rank  4 on seedsmed-f02hpc01 device  4 [0x0c] Tesla V100-PCIE-16GB\r\n#   Rank  5 on seedsmed-f02hpc01 device  5 [0x0d] Tesla V100-PCIE-16GB\r\n#   Rank  6 on seedsmed-f02hpc01 device  6 [0x0e] Tesla V100-PCIE-16GB\r\n#   Rank  7 on seedsmed-f02hpc01 device  7 [0x0f] Tesla V100-PCIE-16GB\r\n#   Rank  8 on seedsmed-f02hpc02 device  0 [0x04] Tesla V100-PCIE-16GB\r\n#   Rank  9 on seedsmed-f02hpc02 device  1 [0x06] Tesla V100-PCIE-16GB\r\n#   Rank 10 on seedsmed-f02hpc02 device  2 [0x07] Tesla V100-PCIE-16GB\r\n#   Rank 11 on seedsmed-f02hpc02 device  3 [0x08] Tesla V100-PCIE-16GB\r\n#   Rank 12 on seedsmed-f02hpc02 device  4 [0x0c] Tesla V100-PCIE-16GB\r\n#   Rank 13 on seedsmed-f02hpc02 device  5 [0x0d] Tesla V100-PCIE-16GB\r\n#   Rank 14 on seedsmed-f02hpc02 device  6 [0x0e] Tesla V100-PCIE-16GB\r\n\r\n#                                                 out-of-place                    in-place\r\n#      bytes             N    type      op     time  algbw  busbw      res     time  algbw  busbw      res\r\n#   Rank 15 on seedsmed-f02hpc02 device  7 [0x0f] Tesla V100-PCIE-16GB\r\n           8             2   float     sum    0.445   0.00   0.00    1e-06    0.106   0.00   0.00    0e+00\r\n     1048584        262146   float     sum    0.624   1.68   3.15    1e+01    0.624   1.68   3.15    1e+01\r\n     2097160        524290   float     sum    1.026   2.04   3.83    1e+01    1.019   2.06   3.86    1e+01\r\n     3145736        786434   float     sum    1.359   2.31   4.34    1e+01    1.358   2.32   4.34    1e+01\r\n     4194312       1048578   float     sum    1.732   2.42   4.54    1e+01    1.728   2.43   4.55    1e+01\r\n     5242888       1310722   float     sum    2.070   2.53   4.75    1e+01    2.078   2.52   4.73    1e+01\r\n     6291464       1572866   float     sum    2.447   2.57   4.82    1e+01    2.445   2.57   4.82    1e+01\r\n     7340040       1835010   float     sum    2.794   2.63   4.93    1e+01    2.798   2.62   4.92    1e+01\r\n     8388616       2097154   float     sum    3.165   2.65   4.97    1e+01    3.168   2.65   4.97    1e+01\r\n     9437192       2359298   float     sum    3.529   2.67   5.01    1e+01    3.526   2.68   5.02    1e+01\r\n    10485768       2621442   float     sum    3.918   2.68   5.02    1e+01    3.938   2.66   4.99    1e+01\r\n    11534344       2883586   float     sum    4.308   2.68   5.02    1e+01    4.308   2.68   5.02    1e+01\r\n    12582920       3145730   float     sum    4.682   2.69   5.04    1e+01    4.718   2.67   5.00    1e+01\r\n    13631496       3407874   float     sum    5.051   2.70   5.06    1e+01    5.065   2.69   5.05    1e+01\r\n    14680072       3670018   float     sum    5.455   2.69   5.05    1e+01    5.462   2.69   5.04    1e+01\r\n    15728648       3932162   float     sum    5.864   2.68   5.03    1e+01    5.841   2.69   5.05    1e+01\r\n    16777224       4194306   float     sum    6.239   2.69   5.04    1e+01    6.228   2.69   5.05    1e+01\r\n    17825800       4456450   float     sum    6.612   2.70   5.06    1e+01    6.631   2.69   5.04    1e+01\r\n    18874376       4718594   float     sum    7.052   2.68   5.02    1e+01    7.022   2.69   5.04    1e+01\r\n    19922952       4980738   float     sum    7.419   2.69   5.03    1e+01    7.420   2.69   5.03    1e+01\r\n    20971528       5242882   float     sum    7.826   2.68   5.02    1e+01    7.811   2.68   5.03    1e+01\r\n    22020104       5505026   float     sum    8.214   2.68   5.03    1e+01    8.182   2.69   5.05    1e+01\r\n    23068680       5767170   float     sum    8.629   2.67   5.01    1e+01    8.590   2.69   5.04    1e+01\r\n    24117256       6029314   float     sum    9.007   2.68   5.02    1e+01    8.979   2.69   5.04    1e+01\r\n    25165832       6291458   float     sum    9.368   2.69   5.04    1e+01    9.441   2.67   5.00    1e+01\r\n    26214408       6553602   float     sum    9.750   2.69   5.04    1e+01    9.749   2.69   5.04    1e+01\r\n    27262984       6815746   float     sum   10.181   2.68   5.02    1e+01   10.150   2.69   5.04    1e+01\r\n    28311560       7077890   float     sum   10.546   2.68   5.03    1e+01   10.535   2.69   5.04    1e+01\r\n    29360136       7340034   float     sum   10.940   2.68   5.03    1e+01   10.944   2.68   5.03    1e+01\r\n    30408712       7602178   float     sum   11.317   2.69   5.04    1e+01   11.296   2.69   5.05    1e+01\r\n    31457288       7864322   float     sum   11.695   2.69   5.04    1e+01   11.688   2.69   5.05    1e+01\r\n    32505864       8126466   float     sum   12.052   2.70   5.06    1e+01   12.092   2.69   5.04    1e+01\r\n Out of bounds values : 248 FAILED\r\n Avg bus bandwidth    : 4.72189\r\n\r\n-------------------------------------------------------\r\nPrimary job  terminated normally, but 1 process returned\r\na non-zero exit code. Per user-direction, the job has been aborted.\r\n-------------------------------------------------------\r\n--------------------------------------------------------------------------\r\nmpirun detected that one or more processes exited with non-zero status, thus causing\r\nthe job to be terminated. The first process to do so was:\r\n\r\n  Process name: [[62619,1],1]\r\n  Exit code:    1\r\n--------------------------------------------------------------------------"}}
{"actor":{"display_login":"wm10240"}}