{"action":"created","issue":{"url":"https://api.github.com/repos/openucx/ucx/issues/2934","repository_url":"https://api.github.com/repos/openucx/ucx","labels_url":"https://api.github.com/repos/openucx/ucx/issues/2934/labels{/name}","comments_url":"https://api.github.com/repos/openucx/ucx/issues/2934/comments","events_url":"https://api.github.com/repos/openucx/ucx/issues/2934/events","html_url":"https://github.com/openucx/ucx/issues/2934","id":368202355,"node_id":"MDU6SXNzdWUzNjgyMDIzNTU=","number":2934,"title":"[mtt] timeout exceeded in mpi-small-tests","user":{"login":"amaslenn","id":1412944,"node_id":"MDQ6VXNlcjE0MTI5NDQ=","avatar_url":"https://avatars2.githubusercontent.com/u/1412944?v=4","gravatar_id":"","url":"https://api.github.com/users/amaslenn","html_url":"https://github.com/amaslenn","followers_url":"https://api.github.com/users/amaslenn/followers","following_url":"https://api.github.com/users/amaslenn/following{/other_user}","gists_url":"https://api.github.com/users/amaslenn/gists{/gist_id}","starred_url":"https://api.github.com/users/amaslenn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaslenn/subscriptions","organizations_url":"https://api.github.com/users/amaslenn/orgs","repos_url":"https://api.github.com/users/amaslenn/repos","events_url":"https://api.github.com/users/amaslenn/events{/privacy}","received_events_url":"https://api.github.com/users/amaslenn/received_events","type":"User","site_admin":false},"labels":[{"id":138949041,"node_id":"MDU6TGFiZWwxMzg5NDkwNDE=","url":"https://api.github.com/repos/openucx/ucx/labels/Bug","name":"Bug","color":"fc2929","default":false},{"id":448762857,"node_id":"MDU6TGFiZWw0NDg3NjI4NTc=","url":"https://api.github.com/repos/openucx/ucx/labels/External","name":"External","color":"fbca04","default":false},{"id":710246243,"node_id":"MDU6TGFiZWw3MTAyNDYyNDM=","url":"https://api.github.com/repos/openucx/ucx/labels/MTT","name":"MTT","color":"e976ff","default":false}],"state":"open","locked":false,"assignee":{"login":"brminich","id":14221180,"node_id":"MDQ6VXNlcjE0MjIxMTgw","avatar_url":"https://avatars1.githubusercontent.com/u/14221180?v=4","gravatar_id":"","url":"https://api.github.com/users/brminich","html_url":"https://github.com/brminich","followers_url":"https://api.github.com/users/brminich/followers","following_url":"https://api.github.com/users/brminich/following{/other_user}","gists_url":"https://api.github.com/users/brminich/gists{/gist_id}","starred_url":"https://api.github.com/users/brminich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brminich/subscriptions","organizations_url":"https://api.github.com/users/brminich/orgs","repos_url":"https://api.github.com/users/brminich/repos","events_url":"https://api.github.com/users/brminich/events{/privacy}","received_events_url":"https://api.github.com/users/brminich/received_events","type":"User","site_admin":false},"assignees":[{"login":"brminich","id":14221180,"node_id":"MDQ6VXNlcjE0MjIxMTgw","avatar_url":"https://avatars1.githubusercontent.com/u/14221180?v=4","gravatar_id":"","url":"https://api.github.com/users/brminich","html_url":"https://github.com/brminich","followers_url":"https://api.github.com/users/brminich/followers","following_url":"https://api.github.com/users/brminich/following{/other_user}","gists_url":"https://api.github.com/users/brminich/gists{/gist_id}","starred_url":"https://api.github.com/users/brminich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brminich/subscriptions","organizations_url":"https://api.github.com/users/brminich/orgs","repos_url":"https://api.github.com/users/brminich/repos","events_url":"https://api.github.com/users/brminich/events{/privacy}","received_events_url":"https://api.github.com/users/brminich/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/openucx/ucx/milestones/10","html_url":"https://github.com/openucx/ucx/milestone/10","labels_url":"https://api.github.com/repos/openucx/ucx/milestones/10/labels","id":3739008,"node_id":"MDk6TWlsZXN0b25lMzczOTAwOA==","number":10,"title":"v1.5.0","description":"","creator":{"login":"yosefe","id":2255631,"node_id":"MDQ6VXNlcjIyNTU2MzE=","avatar_url":"https://avatars0.githubusercontent.com/u/2255631?v=4","gravatar_id":"","url":"https://api.github.com/users/yosefe","html_url":"https://github.com/yosefe","followers_url":"https://api.github.com/users/yosefe/followers","following_url":"https://api.github.com/users/yosefe/following{/other_user}","gists_url":"https://api.github.com/users/yosefe/gists{/gist_id}","starred_url":"https://api.github.com/users/yosefe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosefe/subscriptions","organizations_url":"https://api.github.com/users/yosefe/orgs","repos_url":"https://api.github.com/users/yosefe/repos","events_url":"https://api.github.com/users/yosefe/events{/privacy}","received_events_url":"https://api.github.com/users/yosefe/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":16,"state":"open","created_at":"2018-10-15T11:10:06Z","updated_at":"2018-11-05T20:35:16Z","due_on":"2018-11-30T08:00:00Z","closed_at":null},"comments":6,"created_at":"2018-10-09T13:09:54Z","updated_at":"2018-11-06T07:38:19Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Configuration\r\n```\r\nMOFED: MLNX_OFED_LINUX-4.4-2.0.7.0\r\nOMPI: OMPI: 4.0.0rc4\r\nNodes: hercules x55 (ppn=32(x55), nodelist=clx-hercules-[036,043-096])\r\nJob: ucx-hwtm-dc\r\n```\r\n\r\nMTT log: http://e2e-gw.mellanox.com:4080/mnt/lustre/users/mtt/scratch/ucx_ompi/20181005_205511_9823_201836_clx-hercules-036/html/test_stdout_sgzRIm.txt\r\n\r\nIt doesn't always reproduce.\r\n\r\nCmd:\r\n`mpirun -np 1760 --display-map -mca btl self --tag-output --timestamp-output -mca pml ucx -mca coll ^hcoll --bind-to core -x UCX_NET_DEVICES=mlx5_2:1 -mca osc ucx -x UCX_IB_REG_METHODS=rcache,direct -x UCX_TLS=dc,sm -x UCX_IB_SL=1 -x UCX_DC_VERBS_TM_ENABLE=y -x UCX_DC_VERBS_TM_MAX_BCOPY=8k -mca pmix_base_async_modex 1 -mca mpi_add_procs_cutoff 0 -mca pmix_base_collect_data 0  --map-by node  /mnt/lustre/users/mtt/scratch/ucx_ompi/20181005_205511_9823_201836_clx-hercules-036/installs/N8mV/tests/mpi-small-tests/hpc_tests.git/mpi/misc/all2all_ibm`\r\n\r\nOutput:\r\n```sh\r\nnode=clx-hercules-096, pid=15531:\r\nThread 4 (Thread 0x7ffff3e8c700 (LWP 15571)):\r\n#0  0x00007ffff72fd113 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007ffff6cb28f3 in epoll_dispatch (base=0x669030, tv=<optimized out>) at epoll.c:407\r\n#2  0x00007ffff6cb6340 in opal_libevent2022_event_base_loop (base=0x669030, flags=flags@entry=1) at event.c:1630\r\n#3  0x00007ffff6c72c6e in progress_engine (obj=<optimized out>) at runtime/opal_progress_threads.c:105\r\n#4  0x00007ffff75d2dd5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#5  0x00007ffff72fcb3d in clone () from /usr/lib64/libc.so.6\r\nThread 3 (Thread 0x7ffff10af700 (LWP 15613)):\r\n#0  0x00007ffff72fd113 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007ffff6cb28f3 in epoll_dispatch (base=0x6a8320, tv=<optimized out>) at epoll.c:407\r\n#2  0x00007ffff6cb6340 in opal_libevent2022_event_base_loop (base=0x6a8320, flags=flags@entry=1) at event.c:1630\r\n#3  0x00007ffff2ff35fe in progress_engine (obj=<optimized out>) at runtime/pmix_progress_threads.c:109\r\n#4  0x00007ffff75d2dd5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#5  0x00007ffff72fcb3d in clone () from /usr/lib64/libc.so.6\r\nThread 2 (Thread 0x7fffe34ca700 (LWP 15759)):\r\n#0  0x00007ffff72fd113 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007fffe8f33294 in ucs_async_thread_func (arg=0x855320) at async/thread.c:93\r\n#2  0x00007ffff75d2dd5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#3  0x00007ffff72fcb3d in clone () from /usr/lib64/libc.so.6\r\nThread 1 (Thread 0x7ffff7fc9740 (LWP 15531)):\r\n#0  0x00007ffff75d7483 in pthread_spin_lock () from /usr/lib64/libpthread.so.0\r\n#1  0x00007fffe9684064 in mlx5_poll_cq_1 () from /usr/lib64/libmlx5.so.1\r\n#2  0x00007fffe9b169f6 in ibv_poll_cq (wc=0x7fffffffae10, num_entries=<optimized out>, cq=<optimized out>) at /usr/include/infiniband/verbs.h:1458\r\n#3  uct_ib_poll_cq (wcs=0x7fffffffae10, count=<synthetic pointer>, cq=<optimized out>) at /hpc/local/benchmarks/hpcx_install_2018-10-05/src/hpcx-gcc-redhat7.4/ucx-master/src/uct/ib/base/ib_device.h:311\r\n#4  uct_dc_verbs_poll_tx (iface=0x7f3ff0) at ib/dc/verbs/dc_verbs.c:720\r\n#5  uct_dc_verbs_iface_progress_tm (arg=0x7f3ff0) at ib/dc/verbs/dc_verbs.c:960\r\n#6  0x00007fffe9d6ea72 in ucs_callbackq_dispatch (cbq=<optimized out>) at /hpc/local/benchmarks/hpcx_install_2018-10-05/src/hpcx-gcc-redhat7.4/ucx-master/src/ucs/datastruct/callbackq.h:208\r\n#7  uct_worker_progress (worker=<optimized out>) at /hpc/local/benchmarks/hpcx_install_2018-10-05/src/hpcx-gcc-redhat7.4/ucx-master/src/uct/api/uct.h:1675\r\n#8  ucp_worker_progress (worker=0x79b510) at core/ucp_worker.c:1390\r\n#9  0x00007fffea1a5457 in mca_pml_ucx_progress () at pml_ucx.c:412\r\n#10 0x00007ffff6c6cdec in opal_progress () at runtime/opal_progress.c:231\r\n#11 0x00007ffff7832865 in ompi_request_wait_completion (req=0x8a2668) at ../ompi/request/request.h:415\r\n#12 ompi_request_default_wait (req_ptr=0x7fffffffb320, status=0x7fffffffb330) at request/req_wait.c:42\r\n#13 0x00007ffff786e0e9 in ompi_coll_base_sendrecv_actual (sendbuf=0xc5a71c, scount=58, sdatatype=sdatatype@entry=0x6020e0 <ompi_mpi_int>, dest=1392, stag=stag@entry=-14, recvbuf=<optimized out>, rcount=1, rdatatype=rdatatype@entry=0x6020e0 <ompi_mpi_int>, source=source@entry=476, rtag=rtag@entry=-14, comm=comm@entry=0x6022e0 <ompi_mpi_comm_world>, status=status@entry=0x0) at base/coll_base_util.c:59\r\n#14 0x00007ffff7873623 in ompi_coll_base_sendrecv (stag=-14, rtag=-14, status=0x0, myid=54, comm=0x6022e0 <ompi_mpi_comm_world>, source=476, rdatatype=0x6020e0 <ompi_mpi_int>, rcount=<optimized out>, recvbuf=<optimized out>, dest=1392, sdatatype=0x6020e0 <ompi_mpi_int>, scount=<optimized out>, sendbuf=<optimized out>) at base/coll_base_util.h:67\r\n#15 ompi_coll_base_alltoallv_intra_pairwise (sbuf=0xc3c470, scounts=0x8bcec0, sdisps=0x8c05e0, sdtype=0x6020e0 <ompi_mpi_int>, rbuf=0xcff2c0, rcounts=0x8bea50, rdisps=0x8b6be0, rdtype=0x6020e0 <ompi_mpi_int>, comm=0x6022e0 <ompi_mpi_comm_world>, module=0x891020) at base/coll_base_alltoallv.c:162\r\n#16 0x00007ffff78461e1 in PMPI_Alltoallv (sendbuf=<optimized out>, sendcounts=<optimized out>, sdispls=<optimized out>, sendtype=<optimized out>, recvbuf=<optimized out>, recvcounts=<optimized out>, rdispls=0x8b6be0, recvtype=0x6020e0 <ompi_mpi_int>, comm=0x6022e0 <ompi_mpi_comm_world>) at palltoallv.c:129\r\n#17 0x000000000040149d in a2av (iter=10761) at all2all_ibm.c:170\r\n#18 0x00000000004016de in main () at all2all_ibm.c:218\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/openucx/ucx/issues/comments/436158151","html_url":"https://github.com/openucx/ucx/issues/2934#issuecomment-436158151","issue_url":"https://api.github.com/repos/openucx/ucx/issues/2934","id":436158151,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjE1ODE1MQ==","user":{"login":"amaslenn","id":1412944,"node_id":"MDQ6VXNlcjE0MTI5NDQ=","avatar_url":"https://avatars2.githubusercontent.com/u/1412944?v=4","gravatar_id":"","url":"https://api.github.com/users/amaslenn","html_url":"https://github.com/amaslenn","followers_url":"https://api.github.com/users/amaslenn/followers","following_url":"https://api.github.com/users/amaslenn/following{/other_user}","gists_url":"https://api.github.com/users/amaslenn/gists{/gist_id}","starred_url":"https://api.github.com/users/amaslenn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaslenn/subscriptions","organizations_url":"https://api.github.com/users/amaslenn/orgs","repos_url":"https://api.github.com/users/amaslenn/repos","events_url":"https://api.github.com/users/amaslenn/events{/privacy}","received_events_url":"https://api.github.com/users/amaslenn/received_events","type":"User","site_admin":false},"created_at":"2018-11-06T07:38:19Z","updated_at":"2018-11-06T07:38:19Z","author_association":"CONTRIBUTOR","body":"Yes, all nodes in my allocation (reproduced manually) had lossless configured."}}
{"actor":{"display_login":"amaslenn"}}