{"action":"closed","issue":{"url":"https://api.github.com/repos/openucx/ucx/issues/2365","repository_url":"https://api.github.com/repos/openucx/ucx","labels_url":"https://api.github.com/repos/openucx/ucx/issues/2365/labels{/name}","comments_url":"https://api.github.com/repos/openucx/ucx/issues/2365/comments","events_url":"https://api.github.com/repos/openucx/ucx/issues/2365/events","html_url":"https://github.com/openucx/ucx/issues/2365","id":302090888,"number":2365,"title":"[mtt] hang in mpi-small-tests/collmeas with ROCE","user":{"login":"amaslenn","id":1412944,"avatar_url":"https://avatars2.githubusercontent.com/u/1412944?v=4","gravatar_id":"","url":"https://api.github.com/users/amaslenn","html_url":"https://github.com/amaslenn","followers_url":"https://api.github.com/users/amaslenn/followers","following_url":"https://api.github.com/users/amaslenn/following{/other_user}","gists_url":"https://api.github.com/users/amaslenn/gists{/gist_id}","starred_url":"https://api.github.com/users/amaslenn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaslenn/subscriptions","organizations_url":"https://api.github.com/users/amaslenn/orgs","repos_url":"https://api.github.com/users/amaslenn/repos","events_url":"https://api.github.com/users/amaslenn/events{/privacy}","received_events_url":"https://api.github.com/users/amaslenn/received_events","type":"User","site_admin":false},"labels":[{"id":138949041,"url":"https://api.github.com/repos/openucx/ucx/labels/bug","name":"bug","color":"fc2929","default":true},{"id":710246243,"url":"https://api.github.com/repos/openucx/ucx/labels/mtt","name":"mtt","color":"e976ff","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-04T14:40:42Z","updated_at":"2018-03-15T12:49:00Z","closed_at":"2018-03-15T12:49:00Z","author_association":"CONTRIBUTOR","body":"Configuration\r\n```\r\nOMPI: 3.1.0rc2\r\nJazz x3 (ppn=28(x3), nodelist=jazz[14,30-31])\r\n```\r\n\r\nMTT log: http://hpcweb.lab.mtl.com//hpc/mtr_scrap/users/mtt/scratch/ucx_ompi/20180303_030837_117790_143217_jazz14/html/test_stdout_tG3wLg.txt\r\n\r\nReproduction:\r\n`module load hpcx-gcc`\r\n`mpirun -np 2 -mca btl self --tag-output --timestamp-output -mca pml ucx -mca coll '^hcoll' --bind-to core -x UCX_NET_DEVICES=mlx5_1:1 -x UCX_IB_GID_INDEX=0 -x UCX_IB_REG_METHODS=rcache,direct -x UCX_IB_ETH_PAUSE_ON=y -x UCX_TLS=rc_x,sm -mca pmix_base_async_modex 0 -mca mpi_add_procs_cutoff 100000 -mca pmix_base_collect_data 1 --map-by node /hpc/mtr_scrap/users/mtt/scratch/ucx_ompi/20180303_030837_117790_143217_jazz14/installs/f5Lk/tests/mpi-small-tests/hpc_tests.git/mpi/misc/collmeas`\r\n\r\nIt hangs on:\r\n```sh\r\n...\r\nSun Mar  4 16:34:43 2018[1,0]<stderr>:  25165824 &    9161.20 &    9841.45 & Sun Mar  4 16:34:43 2018[1,0]<stderr>:  10521.70 \\\\\r\nSun Mar  4 16:34:43 2018[1,0]<stderr>:  33554424 &   14173.09 &   14377.74 &   14582.39 \\\\\r\nSun Mar  4 16:34:43 2018[1,0]<stderr>:  33554432 &   14180.65 &   14383.46 &   14586.26 \\\\\r\nSun Mar  4 16:34:43 2018[1,0]<stderr>:\r\nSun Mar  4 16:34:43 2018[1,0]<stderr>:Collective=Allreduce, Communicator=WORLD\r\nSun Mar  4 16:34:43 2018[1,0]<stderr>:========================================\r\n```\r\n\r\nThis cmd works well with `ud`.\r\n\r\nOriginal hang with `NP=84`:\r\n```sh\r\nSat Mar  3 10:37:39 2018[1,0]<stderr>:  33554424 &  191698.57 &  194328.56 &  195209.38 \\\\\r\nSat Mar  3 10:37:39 2018[1,0]<stderr>:  33554432 &  190561.24 & Sat Mar  3 10:37:39 2018[1,0]<stderr>: 193695.18 &  194912.92 \\\\\r\nSat Mar  3 10:37:39 2018[1,0]<stderr>:\r\nSat Mar  3 10:37:39 2018[1,0]<stderr>:Collective=Scan, Communicator=WORLD\r\nSat Mar  3 10:37:39 2018[1,0]<stderr>:===================================\r\n```\r\n\r\nThis cmd hangs with `ud` on `Sun Mar  4 16:38:00 2018[1,0]<stderr>:Collective=Reduce, Communicator=WORLD`."}}
{"actor":{"display_login":"yosefe"}}