{"action":"opened","issue":{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/70","repository_url":"https://api.github.com/repos/raffenet/mpich-migrate-test","labels_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/70/labels{/name}","comments_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/70/comments","events_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/70/events","html_url":"https://github.com/raffenet/mpich-migrate-test/issues/70","id":166157446,"number":70,"title":"FW: [MPICH2 Req #4194] Re: [mvapich-discuss] Races with MPI_THREAD_MULTI","user":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/labels/C:%20mpich","name":"C: mpich","color":"0000aa"},{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/labels/P:%20major","name":"P: major","color":"f6f6f6"},{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/labels/T:%20bug","name":"T: bug","color":"cccccc"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-18T18:15:15Z","updated_at":"2016-07-18T18:15:16Z","closed_at":null,"body":"**Reported by  Rajeev Thakur   thakur on 3 Sep 2008 14:32 UTC**\n\n\nForwarding from old req system.\n\n  _____  \n\nFrom: Rajeev Thakur [\nSent: Friday, July 25, 2008 1:27 PM\nTo: mpich2-maint@mcs.anl.gov\nCc: mpich2-maint@mcs.anl.gov\nSubject: FW: [MPICH2 Req #4194](mailto:thakur@mcs.anl.gov]) Re: [Races with\nMPI_THREAD_MULTI\n\n\n  _____  \n\nFrom: Roberto Fichera [mailto:kernel@tekno-soft.it](mvapich-discuss]) \nSent: Friday, July 25, 2008 12:41 PM\nTo: Dhabaleswar Panda\nCc: mvapich-core@cse.ohio-state.edu; mpich-discuss@mcs.anl.gov;\nmvapich-discuss@cse.ohio-state.edu\nSubject: Re: [Races with MPI_THREAD_MULTI\n\n\nDhabaleswar Panda ha scritto: \n\nHi Roberto,\n\n\n\nWe have done several rounds of checks and do not see any difference\n\nbetween MPICH2 1.0.7 and the TCP/IP interface of MVAPICH2 1.2. Both these\n\nshould perform exactly the same. We are continuing our investigation.\n\n\n\nWe are wondering whether you can send us a sample code piece to reproduce\n\nthe problem you are indicating across these two interfaces.  This will\n\nhelp us to debug this problem faster and help you to solve your problem.\n\n  \n\nI've added other CCs in this email, maybe other people are interested to\nhave a look in.\n\nAttached you find the test program, which I'm working on, to turn up the\nproblem. I'm not completely sure if it works perfectly since I wasn't\nable to complete its execution, but please let me know if I made something\nwrong inside the code. The testmaster is quite easy, you must provide the\nnumber \nof jobs to simulate (say 50000) and the node file that the resource manager\nprovide for its schedule. Actually the node that matches the master will\nbe excluded by the slave nodes.\n\nThe testmain creates a ring of threads from the assigned nodes. So walking\nin the ring, for each free node it find, a thread is started so you should\nhave as \nmany threads as the number of assigned nodes working in multithreading. For\nsimulating something to do each thread internally generate a random integer,\n\nsets some MPI_Info (host and pwd), spawn the testslave job, send it the\ngenerated random number, wait that the testslave receive and send back that \nnumber, sent and received numbers are comparated in order to verify their\ncoherency, the slave send an empty MPI_Send() for signaling its termination,\n\nthe thread now calls MPI_Comm_disconnect() for closing the slave connection,\nand finally all the MPI_Info are cleared. At this time the thread terminate.\nWhen the number of requested jobs are correctly \"worked out\" the application\nshould terminate ... but without cleaning up (too tired sorry ;-), so it\njust wait a \nbit and finalize the MPI.\n\nAt this time, I wasn't able to complete any execution. Currently the\napplication still crashing with the backtrace you find below. Only one time\nI was able to reach 3500 jobs but one thread was stuck in a mutex. Looking\nin the backtrace you can find the same race I'm getting in my applications.\n\nProgram received signal SIGSEGV, Segmentation fault.\n[Switching to Thread 1087666512 (LWP 18231)](mvapich-discuss])\n0x00000000006a3902 in MPIDI_PG_Dup_vcr () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\nMissing separate debuginfos, use: debuginfo-install glibc.x86_64\n(gdb) info threads\n  29 Thread 1121462608 (LWP 18232)  0x0000003465a0a8f9 in\npthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n* 28 Thread 1087666512 (LWP 18231)  0x00000000006a3902 in MPIDI_PG_Dup_vcr\n() from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n  27 Thread 1142442320 (LWP 18230)  0x0000003464ecbd66 in poll () from\n/lib64/libc.so.6\n  26 Thread 1098156368 (LWP 18229)  0x0000003464e9ac61 in nanosleep () from\n/lib64/libc.so.6\n  1 Thread 140135980537584 (LWP 18029)  main (argc=3, argv=0x7ffffb5992d8)\nat testmaster.c:437\n\n(gdb) bt\n#0  0x00000000006a3902 in MPIDI_PG_Dup_vcr () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#1  0x0000000000668012 in SetupNewIntercomm () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#2  0x00000000006682c8 in MPIDI_Comm_accept () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#3  0x00000000006a6617 in MPID_Comm_accept () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#4  0x000000000065ec5f in MPIDI_Comm_spawn_multiple () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#5  0x00000000006a17e6 in MPID_Comm_spawn_multiple () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#6  0x00000000006783fd in PMPI_Comm_spawn () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#7  0x00000000004017de in NodeThread_threadMain (arg=0x120a790) at\ntestmaster.c:314\n#8  0x0000003465a06407 in start_thread () from /lib64/libpthread.so.0\n#9  0x0000003464ed4b0d in clone () from /lib64/libc.so.6\n(gdb) thread 29\n\n[to thread 29 (Thread 1121462608 (LWP 18232))](Switching)#0\n0x0000003465a0a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from\n/lib64/libpthread.so.0\n(gdb) bt\n#0  0x0000003465a0a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from\n/lib64/libpthread.so.0\n#1  0x000000000065e2e7 in MPIDI_CH3I_Progress () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#2  0x00000000006675ca in FreeNewVC () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#3  0x0000000000668302 in MPIDI_Comm_accept () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#4  0x00000000006a6617 in MPID_Comm_accept () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#5  0x000000000065ec5f in MPIDI_Comm_spawn_multiple () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#6  0x00000000006a17e6 in MPID_Comm_spawn_multiple () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#7  0x00000000006783fd in PMPI_Comm_spawn () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#8  0x00000000004017de in NodeThread_threadMain (arg=0x120d590) at\ntestmaster.c:314\n#9  0x0000003465a06407 in start_thread () from /lib64/libpthread.so.0\n#10 0x0000003464ed4b0d in clone () from /lib64/libc.so.6\n(gdb) thread 27\n\n[to thread 27 (Thread 1142442320 (LWP 18230))](Switching)#0\n0x0000003464ecbd66 in poll () from /lib64/libc.so.6\n(gdb) bt\n#0  0x0000003464ecbd66 in poll () from /lib64/libc.so.6\n#1  0x00000000006d63bf in MPIDU_Sock_wait () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#2  0x000000000065e1e7 in MPIDI_CH3I_Progress () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#3  0x00000000006cf87c in PMPI_Send () from\n/home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glib\nc2.3.4/libmpich.so.1.1\n#4  0x0000000000401831 in NodeThread_threadMain (arg=0x120a6f0) at\ntestmaster.c:480\n#5  0x0000003465a06407 in start_thread () from /lib64/libpthread.so.0\n#6  0x0000003464ed4b0d in clone () from /lib64/libc.so.6\n\n(gdb) thread 26\n[to thread 26 (Thread 1098156368 (LWP 18229))](Switching)#0\n0x0000003464e9ac61 in nanosleep () from /lib64/libc.so.6\n(gdb) bt\n#0  0x0000003464e9ac61 in nanosleep () from /lib64/libc.so.6\n#1  0x0000003464e9aa84 in sleep () from /lib64/libc.so.6\n#2  0x000000000040197c in NodeThread_threadMain (arg=0x120d630) at\ntestmaster.c:505\n#3  0x0000003465a06407 in start_thread () from /lib64/libpthread.so.0\n#4  0x0000003464ed4b0d in clone () from /lib64/libc.so.6\n(gdb) "}}
{"actor":{"display_login":"raffenet"}}