{"action":"created","issue":{"url":"https://api.github.com/repos/openucx/ucx/issues/2354","repository_url":"https://api.github.com/repos/openucx/ucx","labels_url":"https://api.github.com/repos/openucx/ucx/issues/2354/labels{/name}","comments_url":"https://api.github.com/repos/openucx/ucx/issues/2354/comments","events_url":"https://api.github.com/repos/openucx/ucx/issues/2354/events","html_url":"https://github.com/openucx/ucx/issues/2354","id":300944775,"number":2354,"title":"crash on dcx/test_ucp_perf.envelope/0","user":{"login":"hoopoepg","id":2535990,"avatar_url":"https://avatars0.githubusercontent.com/u/2535990?v=4","gravatar_id":"","url":"https://api.github.com/users/hoopoepg","html_url":"https://github.com/hoopoepg","followers_url":"https://api.github.com/users/hoopoepg/followers","following_url":"https://api.github.com/users/hoopoepg/following{/other_user}","gists_url":"https://api.github.com/users/hoopoepg/gists{/gist_id}","starred_url":"https://api.github.com/users/hoopoepg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hoopoepg/subscriptions","organizations_url":"https://api.github.com/users/hoopoepg/orgs","repos_url":"https://api.github.com/users/hoopoepg/repos","events_url":"https://api.github.com/users/hoopoepg/events{/privacy}","received_events_url":"https://api.github.com/users/hoopoepg/received_events","type":"User","site_admin":false},"labels":[{"id":138949041,"url":"https://api.github.com/repos/openucx/ucx/labels/bug","name":"bug","color":"fc2929","default":true},{"id":710246309,"url":"https://api.github.com/repos/openucx/ucx/labels/jenkins","name":"jenkins","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"hoopoepg","id":2535990,"avatar_url":"https://avatars0.githubusercontent.com/u/2535990?v=4","gravatar_id":"","url":"https://api.github.com/users/hoopoepg","html_url":"https://github.com/hoopoepg","followers_url":"https://api.github.com/users/hoopoepg/followers","following_url":"https://api.github.com/users/hoopoepg/following{/other_user}","gists_url":"https://api.github.com/users/hoopoepg/gists{/gist_id}","starred_url":"https://api.github.com/users/hoopoepg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hoopoepg/subscriptions","organizations_url":"https://api.github.com/users/hoopoepg/orgs","repos_url":"https://api.github.com/users/hoopoepg/repos","events_url":"https://api.github.com/users/hoopoepg/events{/privacy}","received_events_url":"https://api.github.com/users/hoopoepg/received_events","type":"User","site_admin":false},"assignees":[{"login":"hoopoepg","id":2535990,"avatar_url":"https://avatars0.githubusercontent.com/u/2535990?v=4","gravatar_id":"","url":"https://api.github.com/users/hoopoepg","html_url":"https://github.com/hoopoepg","followers_url":"https://api.github.com/users/hoopoepg/followers","following_url":"https://api.github.com/users/hoopoepg/following{/other_user}","gists_url":"https://api.github.com/users/hoopoepg/gists{/gist_id}","starred_url":"https://api.github.com/users/hoopoepg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hoopoepg/subscriptions","organizations_url":"https://api.github.com/users/hoopoepg/orgs","repos_url":"https://api.github.com/users/hoopoepg/repos","events_url":"https://api.github.com/users/hoopoepg/events{/privacy}","received_events_url":"https://api.github.com/users/hoopoepg/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-02-28T09:13:36Z","updated_at":"2018-04-26T15:12:20Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"reproduced on hpc-test-node:\r\n```\r\n09:10:48 [----------] 1 test from dcx/test_ucp_perf\r\n09:10:48 [ RUN      ] dcx/test_ucp_perf.envelope/0\r\n09:10:50 [     INFO ]                tag latency : 26561.876 usec (performance not checked)\r\n09:10:52 [     INFO ]            tag iov latency : 30666.613 usec (performance not checked)\r\n09:10:53 [     INFO ]                     tag mr : 0.019 Mpps (performance not checked)\r\n09:10:56 [     INFO ]                tag sync mr : 0.000 Mpps (performance not checked)\r\n09:10:57 [     INFO ]                tag wild mr : 0.025 Mpps (performance not checked)\r\n09:10:58 [     INFO ]                     tag bw : 0.409 MB/sec (performance not checked)\r\n09:10:59 [     INFO ]         tag bw_zcopy_multi : 0.407 MB/sec (performance not checked)\r\n09:11:00 [     INFO ]                put latency : 4616.281 usec (performance not checked)\r\n09:11:01 [     INFO ]                   put rate : 0.008 Mpps (performance not checked)\r\n09:11:02 [     INFO ]                     put bw : 0.757 MB/sec (performance not checked)\r\n09:11:03 [     INFO ]                get latency : 2750.234 usec (performance not checked)\r\n09:11:04 [     INFO ]                     get bw : 168.698 MB/sec (performance not checked)\r\n09:11:06 [     INFO ]             stream latency : 29347.669 usec (performance not checked)\r\n09:11:07 [     INFO ]                  stream bw : 55.842 MB/sec (performance not checked)\r\n09:11:10 [     INFO ]   stream recv-data latency : 27920.142 usec (performance not checked)\r\n09:11:11 [     INFO ]        stream recv-data bw : 73.786 MB/sec (performance not checked)\r\n09:11:12 [     INFO ]            atomic add rate : 0.000 Mpps (performance not checked)\r\n09:11:12 [hpc-test-node:8460 :0:8715] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))\r\n\r\n11:05:04 Build was aborted\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/openucx/ucx/issues/comments/384677404","html_url":"https://github.com/openucx/ucx/issues/2354#issuecomment-384677404","issue_url":"https://api.github.com/repos/openucx/ucx/issues/2354","id":384677404,"user":{"login":"yosefe","id":2255631,"avatar_url":"https://avatars0.githubusercontent.com/u/2255631?v=4","gravatar_id":"","url":"https://api.github.com/users/yosefe","html_url":"https://github.com/yosefe","followers_url":"https://api.github.com/users/yosefe/followers","following_url":"https://api.github.com/users/yosefe/following{/other_user}","gists_url":"https://api.github.com/users/yosefe/gists{/gist_id}","starred_url":"https://api.github.com/users/yosefe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosefe/subscriptions","organizations_url":"https://api.github.com/users/yosefe/orgs","repos_url":"https://api.github.com/users/yosefe/repos","events_url":"https://api.github.com/users/yosefe/events{/privacy}","received_events_url":"https://api.github.com/users/yosefe/received_events","type":"User","site_admin":false},"created_at":"2018-04-26T15:12:20Z","updated_at":"2018-04-26T15:12:20Z","author_association":"CONTRIBUTOR","body":"similar failure:\r\n```\r\n16:43:33 [----------] 1 test from mm/test_uct_perf\r\n16:43:33 [ RUN      ] mm/test_uct_perf.envelope/0\r\n16:43:37 [     INFO ] sysv                am latency : 93729.314 usec (performance not checked)\r\n16:43:38 [     INFO ] sysv                   am rate : 0.000 Mpps (performance not checked)\r\n16:43:38 [     INFO ] sysv                 am rate64 : 0.000 Mpps (performance not checked)\r\n16:43:41 [     INFO ] sysv          am bcopy latency : 64358.401 usec (performance not checked)\r\n16:43:42 [     INFO ] sysv               am bcopy bw : 0.168 MB/sec (performance not checked)\r\n16:43:45 [     INFO ] sysv               put latency : 63728.748 usec (performance not checked)\r\n16:43:46 [     INFO ] sysv                  put rate : 0.004 Mpps (performance not checked)\r\n16:43:47 [     INFO ] sysv              put bcopy bw : 8.616 MB/sec (performance not checked)\r\n16:43:52 [     INFO ] sysv        atomic add latency : 98361.201 usec (performance not checked)\r\n16:43:52 ==10010== Thread 6:\r\n16:43:52 ==10010== Use of uninitialised value of size 8\r\n16:43:52 ==10010==    at 0x508DB1B: ucs_queue_push (queue.h:57)\r\n16:43:52 ==10010==    by 0x508DB1B: ucs_rcache_unmapped_callback (rcache.c:299)\r\n16:43:52 ==10010==    by 0x568224D: ucm_event_dispatch (event.c:128)\r\n16:43:52 ==10010==    by 0x568224D: ucm_dispatch_vm_munmap (event.c:176)\r\n16:43:52 ==10010==    by 0x568224D: ucm_munmap (event.c:215)\r\n16:43:52 ==10010==    by 0x9ABC657: ??? (in /usr/lib64/mlnx_ofed/valgrind/libmlx5.so.1.0.0)\r\n16:43:52 ==10010==    by 0x9AFC8D6: ??? (in /usr/lib64/mlnx_ofed/valgrind/libmlx5.so.1.0.0)\r\n16:43:52 ==10010==    by 0x53C96B0: uct_ib_md_umr_qp_destroy (ib_md.c:290)\r\n16:43:52 ==10010==    by 0x53C96B0: uct_ib_md_close (ib_md.c:1432)\r\n16:43:52 ==10010==    by 0x790737: uct_perf_create_md (libperf.c:1174)\r\n16:43:52 ==10010==    by 0x790737: uct_perf_setup (libperf.c:1212)\r\n16:43:52 ==10010==    by 0x792024: ucx_perf_run (libperf.c:1386)\r\n16:43:52 ==10010==    by 0x47A0D8: test_perf::thread_func(void*) (test_perf.cc:165)\r\n16:43:52 ==10010==    by 0x7572E24: start_thread (in /usr/lib64/libpthread-2.17.so)\r\n16:43:52 ==10010==    by 0x88E734C: clone (in /usr/lib64/libc-2.17.so)\r\n16:43:52 ==10010==  Uninitialised value was created by a heap allocation\r\n16:43:52 ==10010==    at 0x568647B: ucm_malloc_allocated (malloc_hook.c:197)\r\n16:43:52 ==10010==    by 0x5687681: ucm_malloc_impl (malloc_hook.c:233)\r\n16:43:52 ==10010==    by 0x5687681: ucm_malloc (malloc_hook.c:300)\r\n16:43:52 ==10010==    by 0x508A85E: ucs_malloc (memtrack.c:176)\r\n16:43:52 ==10010==    by 0x508F8F8: ucs_rcache_create (rcache.c:683)\r\n16:43:52 ==10010==    by 0x53CCAFB: uct_ib_md_parse_reg_methods (ib_md.c:1120)\r\n16:43:52 ==10010==    by 0x53CCAFB: uct_ib_md_open (ib_md.c:1383)\r\n16:43:52 ==10010==    by 0x53BB3CF: uct_md_open (uct_md.c:124)\r\n16:43:52 ==10010==    by 0x790682: uct_perf_create_md (libperf.c:1151)\r\n16:43:52 ==10010==    by 0x790682: uct_perf_setup (libperf.c:1212)\r\n16:43:52 ==10010==    by 0x792024: ucx_perf_run (libperf.c:1386)\r\n16:43:52 ==10010==    by 0x47A0D8: test_perf::thread_func(void*) (test_perf.cc:165)\r\n16:43:52 ==10010==    by 0x7572E24: start_thread (in /usr/lib64/libpthread-2.17.so)\r\n16:43:52 ==10010==    by 0x88E734C: clone (in /usr/lib64/libc-2.17.so)\r\n16:43:52 ==10010== \r\n16:43:52 {\r\n16:43:52    <insert_a_suppression_name_here>\r\n16:43:52    Memcheck:Value8\r\n16:43:52    fun:ucs_queue_push\r\n16:43:52    fun:ucs_rcache_unmapped_callback\r\n16:43:52    fun:ucm_event_dispatch\r\n16:43:52    fun:ucm_dispatch_vm_munmap\r\n16:43:52    fun:ucm_munmap\r\n16:43:52    obj:/usr/lib64/mlnx_ofed/valgrind/libmlx5.so.1.0.0\r\n16:43:52    obj:/usr/lib64/mlnx_ofed/valgrind/libmlx5.so.1.0.0\r\n16:43:52    fun:uct_ib_md_umr_qp_destroy\r\n16:43:52    fun:uct_ib_md_close\r\n16:43:52    fun:uct_perf_create_md\r\n16:43:52    fun:uct_perf_setup\r\n16:43:52    fun:ucx_perf_run\r\n16:43:52    fun:_ZN9test_perf11thread_funcEPv\r\n16:43:52    fun:start_thread\r\n16:43:52    fun:clone\r\n16:43:52 }\r\n16:43:52 ==10010== Thread 4:\r\n16:43:52 ==10010== Invalid read of size 8\r\n16:43:52 ==10010==    at 0x5081C1D: ucs_mpool_chunk_leak_check (mpool.c:36)\r\n16:43:52 ==10010==    by 0x5081C1D: ucs_mpool_cleanup (mpool.c:121)\r\n16:43:52 ==10010==    by 0x508E944: ucs_rcache_t_cleanup (rcache.c:674)\r\n16:43:52 ==10010==    by 0x50935C5: ucs_class_call_cleanup_chain (class.c:50)\r\n16:43:52 ==10010==    by 0x508FC4F: ucs_rcache_destroy (rcache.c:686)\r\n16:43:52 ==10010==    by 0x53C9653: uct_ib_md_release_reg_method (ib_md.c:1247)\r\n16:43:52 ==10010==    by 0x53C9653: uct_ib_md_close (ib_md.c:1431)\r\n16:43:52 ==10010==    by 0x790737: uct_perf_create_md (libperf.c:1174)\r\n16:43:52 ==10010==    by 0x790737: uct_perf_setup (libperf.c:1212)\r\n16:43:52 ==10010==    by 0x792024: ucx_perf_run (libperf.c:1386)\r\n16:43:52 ==10010==    by 0x47A0D8: test_perf::thread_func(void*) (test_perf.cc:165)\r\n16:43:52 ==10010==    by 0x7572E24: start_thread (in /usr/lib64/libpthread-2.17.so)\r\n16:43:52 ==10010==    by 0x88E734C: clone (in /usr/lib64/libc-2.17.so)\r\n16:43:52 ==10010==  Address 0xc0d0fe0 is in a rw- anonymous segment\r\n16:43:52 ==10010== \r\n16:43:52 {\r\n16:43:52    <insert_a_suppression_name_here>\r\n16:43:52    Memcheck:Addr8\r\n16:43:52    fun:ucs_mpool_chunk_leak_check\r\n16:43:52    fun:ucs_mpool_cleanup\r\n16:43:52    fun:ucs_rcache_t_cleanup\r\n16:43:52    fun:ucs_class_call_cleanup_chain\r\n16:43:52    fun:ucs_rcache_destroy\r\n16:43:52    fun:uct_ib_md_release_reg_method\r\n16:43:52    fun:uct_ib_md_close\r\n16:43:52    fun:uct_perf_create_md\r\n16:43:52    fun:uct_perf_setup\r\n16:43:52    fun:ucx_perf_run\r\n16:43:52    fun:_ZN9test_perf11thread_funcEPv\r\n16:43:52    fun:start_thread\r\n16:43:52    fun:clone\r\n16:43:52 }\r\n16:43:52 **10010** [1524750232.706960]          mpool.c:38   UCX  WARN  object 0xc0d0fe8 was not returned to mpool rcache_inv_mp\r\n16:43:53 [     INFO ] sysv           atomic add rate : 0.003 Mpps (performance not checked)\r\n```\r\n\r\nprobably also related to #2089"}}
{"actor":{"display_login":"yosefe"}}