{"action":"opened","issue":{"url":"https://api.github.com/repos/openucx/ucx/issues/2323","repository_url":"https://api.github.com/repos/openucx/ucx","labels_url":"https://api.github.com/repos/openucx/ucx/issues/2323/labels{/name}","comments_url":"https://api.github.com/repos/openucx/ucx/issues/2323/comments","events_url":"https://api.github.com/repos/openucx/ucx/issues/2323/events","html_url":"https://github.com/openucx/ucx/issues/2323","id":298071888,"number":2323,"title":"hang on osu_ialltoallw","user":{"login":"alinask","id":2405183,"avatar_url":"https://avatars1.githubusercontent.com/u/2405183?v=4","gravatar_id":"","url":"https://api.github.com/users/alinask","html_url":"https://github.com/alinask","followers_url":"https://api.github.com/users/alinask/followers","following_url":"https://api.github.com/users/alinask/following{/other_user}","gists_url":"https://api.github.com/users/alinask/gists{/gist_id}","starred_url":"https://api.github.com/users/alinask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alinask/subscriptions","organizations_url":"https://api.github.com/users/alinask/orgs","repos_url":"https://api.github.com/users/alinask/repos","events_url":"https://api.github.com/users/alinask/events{/privacy}","received_events_url":"https://api.github.com/users/alinask/received_events","type":"User","site_admin":false},"labels":[{"id":138949041,"url":"https://api.github.com/repos/openucx/ucx/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-18T09:44:01Z","updated_at":"2018-02-18T09:44:01Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"There is a hang that happens on the osu_ialltoallw test and can be reproduced on every run.\r\nThis is ucx-roce.\r\n\r\nSetup:\r\n\r\nNine orion hosts: clx-orion-[040-048]. ppn=3 or 4\r\n\r\n```\r\nalinas@mngx-orion-01 ~\r\n$module load hpcx-gcc\r\n\r\nalinas@mngx-orion-01 ~\r\n$which mpirun\r\n/hpc/local/benchmarks/hpcx_install_2018-02-16/hpcx-gcc-redhat7.4/ompi-b6e825f/bin/mpirun\r\n\r\nalinas@mngx-orion-01 ~\r\n$ucx_info -v\r\n+ ucx_info -v\r\n# UCT version=1.3.0 revision 822e820\r\n\r\nalinas@mngx-orion-01 ~\r\n$pdsh -w  clx-orion-[040-048] ofed_info -s |dshbak -c\r\n----------------\r\nclx-orion-[040-048]\r\n----------------\r\nMLNX_OFED_LINUX-4.2-1.2.0.0:\r\n\r\nalinas@mngx-orion-01 ~\r\n$pdsh -w  clx-orion-[040-048] ibv_devinfo -d mlx5_2 | grep fw_ver |dshbak -c\r\n----------------\r\nclx-orion-[040-048]\r\n----------------\r\n        fw_ver:                         12.21.2010\r\n\r\nalinas@mngx-orion-01 ~\r\n$pdsh -w  clx-orion-[040-048] ibv_devinfo -d mlx5_2 | grep link_layer |dshbak -c\r\n----------------\r\nclx-orion-[040-048]\r\n----------------\r\n                        link_layer:             Ethernet\r\n\r\nalinas@mngx-orion-01 ~\r\n$pdsh -w  clx-orion-[040-048] ibv_devinfo -d mlx5_2 | grep vendor_part_id |dshbak -c\r\n----------------\r\nclx-orion-[040-048]\r\n----------------\r\n        vendor_part_id:                 4115\r\n```\r\n\r\nCommand line:\r\n\r\nmpirun -np 32 --display-map -mca btl self --tag-output --timestamp-output -mca pml ucx -mca coll '^hcoll' --bind-to core -x UCX_NET_DEVICES=mlx5_2:1 -x UCX_IB_GID_INDEX=3 -x UCX_IB_ETH_PAUSE_ON=y -x UCX_TLS=rc,sm --map-by node /mnt/lustre/users/mtt/scratch/ucx_ompi/20180217_224605_15847_59996_clx-orion-040/installs/wyzU/tests/osu_micro_benchmark/osu-micro-benchmarks-5.3.2/mpi/collective/osu_ialltoallw\r\n```\r\n Data for JOB [60732,1] offset 0 Total slots allocated 252\r\n\r\n ========================   JOB MAP   ========================\r\n\r\n Data for node: clx-orion-040   Num slots: 28   Max slots: 0    Num procs: 4\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 0 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 9 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 18 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 27 Bound: N/A\r\n\r\n Data for node: clx-orion-041   Num slots: 28   Max slots: 0    Num procs: 4\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 1 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 10 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 19 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 28 Bound: N/A\r\n\r\n Data for node: clx-orion-042   Num slots: 28   Max slots: 0    Num procs: 4\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 2 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 11 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 20 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 29 Bound: N/A\r\n\r\n Data for node: clx-orion-043   Num slots: 28   Max slots: 0    Num procs: 4\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 3 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 12 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 21 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 30 Bound: N/A\r\n\r\n Data for node: clx-orion-044   Num slots: 28   Max slots: 0    Num procs: 4\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 4 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 13 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 22 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 31 Bound: N/A\r\n\r\n Data for node: clx-orion-045   Num slots: 28   Max slots: 0    Num procs: 3\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 5 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 14 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 23 Bound: N/A\r\n\r\n Data for node: clx-orion-046   Num slots: 28   Max slots: 0    Num procs: 3\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 6 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 15 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 24 Bound: N/A\r\n\r\n Data for node: clx-orion-047   Num slots: 28   Max slots: 0    Num procs: 3\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 7 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 16 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 25 Bound: N/A\r\n\r\n Data for node: clx-orion-048   Num slots: 28   Max slots: 0    Num procs: 3\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 8 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 17 Bound: N/A\r\n        Process OMPI jobid: [60732,1] App: 0 Process rank: 26 Bound: N/A\r\n\r\n =============================================================\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:# OSU MPI Non-blocking All-to-Allv Personalized Exchange Latency Test v5.3.2\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:# Overall = Coll. Init + Compute + MPI_Test + MPI_Wait\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:# Size           Overall(us)       Compute(us)    Pure Comm.(us)        Overlap(%)\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:1                      39.24             21.13             19.76              8.38\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:2                      37.29             19.25             17.90              0.00\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:4                      37.26             19.19             17.85              0.00\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:8                      37.32             19.27             17.87              0.00\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:16                     37.33             19.26             17.85              0.00\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:32                     37.71             19.34             18.06              0.00\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:64                     39.48             20.25             18.99              0.00\r\nSun Feb 18 11:41:44 2018[1,0]<stdout>:128                    45.85             23.75             22.24              0.65\r\nSun Feb 18 11:41:45 2018[1,0]<stdout>:256                    47.24             24.20             22.73              0.00\r\nSun Feb 18 11:41:45 2018[1,0]<stdout>:512                    51.07             26.42             24.93              1.14\r\nSun Feb 18 11:41:45 2018[1,0]<stdout>:1024                   63.97             35.90             33.94             17.30\r\nSun Feb 18 11:41:45 2018[1,0]<stdout>:2048                  103.18             68.86             65.56             47.65\r\nSun Feb 18 11:41:46 2018[1,0]<stdout>:4096                  180.24            137.83            131.84             67.83\r\nSun Feb 18 11:41:46 2018[1,0]<stdout>:8192                  356.75            286.02            274.23             74.21\r\nSun Feb 18 11:41:46 2018[1,0]<stdout>:16384                 600.46            487.29            467.47             75.79\r\n... Hang ...\r\n```\r\n\r\nMTT links:\r\n\r\nhttp://e2e-gw.mellanox.com:4080//mnt/lustre/users/mtt/scratch/ucx_ompi/20180217_224605_15847_59996_clx-orion-040/html/test_stdout_Cngisl.txt\r\n\r\nhttp://e2e-gw.mellanox.com:4080//mnt/lustre/users/mtt/scratch/ucx_ompi/20180217_224605_15847_59996_clx-orion-040/html/test_stdout_C7R3b_.txt"}}
{"actor":{"display_login":"alinask"}}