{"action":"opened","issue":{"url":"https://api.github.com/repos/pmix/pmix/issues/877","repository_url":"https://api.github.com/repos/pmix/pmix","labels_url":"https://api.github.com/repos/pmix/pmix/issues/877/labels{/name}","comments_url":"https://api.github.com/repos/pmix/pmix/issues/877/comments","events_url":"https://api.github.com/repos/pmix/pmix/issues/877/events","html_url":"https://github.com/pmix/pmix/issues/877","id":369197907,"node_id":"MDU6SXNzdWUzNjkxOTc5MDc=","number":877,"title":"PMIx cross version issues","user":{"login":"nysal","id":8509287,"node_id":"MDQ6VXNlcjg1MDkyODc=","avatar_url":"https://avatars1.githubusercontent.com/u/8509287?v=4","gravatar_id":"","url":"https://api.github.com/users/nysal","html_url":"https://github.com/nysal","followers_url":"https://api.github.com/users/nysal/followers","following_url":"https://api.github.com/users/nysal/following{/other_user}","gists_url":"https://api.github.com/users/nysal/gists{/gist_id}","starred_url":"https://api.github.com/users/nysal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nysal/subscriptions","organizations_url":"https://api.github.com/users/nysal/orgs","repos_url":"https://api.github.com/users/nysal/repos","events_url":"https://api.github.com/users/nysal/events{/privacy}","received_events_url":"https://api.github.com/users/nysal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-11T16:14:17Z","updated_at":"2018-10-11T16:14:17Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Thank you for taking the time to submit an issue!\r\n\r\n## Background information\r\nInitial findings from PMIx cross version compatibility testing\r\n\r\n### What version of the PMIx Reference Library are you using? (e.g., v1.0, v2.1, git master @ hash, etc.)\r\nv1.2 through master\r\n\r\n### Describe how PMIx was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)\r\ngit clone\r\n\r\n### Please describe the system on which you are running\r\n\r\n* Operating system/version: RHEL 7.5\r\n* Computer hardware: POWER8 \r\n* Network type: Single node\r\n\r\n-----------------------------\r\n\r\n## Details of the problem\r\n\r\nThe detail logs of cross version testing are available here - https://gist.github.com/nysal/666098bb955df2f896ad0f01bf73a092\r\n\r\nI've used the simptest test case and ran PMIx servers based on master v3.0, v2.1 and v2.0. The clients (simpclient) tested were from  master, v3.1, v3.0, v2.2, v2.1, v2.0, v1.2, karasevb:2.x_dstore/lock and karasevb:3.0_dstore/lock branches.\r\n\r\nWith a v2.1 server and v2.2 client, the client crashes with a SEGV. Here's the stack:\r\n```\r\nProgram terminated with signal 11, Segmentation fault.\r\n#0  0x00000000100ef28c in _dstore_fetch ()\r\nMissing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.ppc64le zlib-1.2.7-17.el7.ppc64le\r\n(gdb) thread apply all bt\r\n\r\nThread 2 (Thread 0x3fff97415560 (LWP 143149)):\r\n#0  0x00003fff9716e72c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0\r\n#1  0x00000000100200ac in PMIx_Get ()\r\n#2  0x0000000010016ffc in PMIx_Init ()\r\n#3  0x0000000010003a24 in main ()\r\n\r\nThread 1 (Thread 0x3fff96f5f1b0 (LWP 143151)):\r\n#0  0x00000000100ef28c in _dstore_fetch ()\r\n#1  0x00000000100f1b54 in dstore_fetch ()\r\n#2  0x000000001001dc28 in _getnbfn ()\r\n#3  0x00003fff97322960 in event_process_active_single_queue (activeq=0x1002ed0dcb0, base=0x1002ed1f800) at event.c:1368\r\n#4  event_process_active (base=<optimized out>) at event.c:1438\r\n#5  event_base_loop (base=0x1002ed1f800, flags=<optimized out>) at event.c:1639\r\n#6  0x0000000010049bcc in pmix_progress_thread_init ()\r\n#7  0x00003fff97168af4 in start_thread () from /lib64/libpthread.so.0\r\n#8  0x00003fff97098814 in clone () from /lib64/libc.so.6\r\n(gdb) \r\n```\r\nIf I use the client from karasevb:2.x_dstore/lock and the v2.1 server, the client gets a SEGV, but the failure is at a slightly different spot:\r\n```\r\nProgram terminated with signal 11, Segmentation fault.\r\n#0  0x0000000010102544 in pmix_usock_send_handler ()\r\nMissing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.ppc64le zlib-1.2.7-17.el7.ppc64le\r\n(gdb) thread apply all bt\r\n\r\nThread 2 (Thread 0x3fff8e5b5560 (LWP 139244)):\r\n#0  0x00003fff8e30e72c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0\r\n#1  0x000000001002010c in PMIx_Get ()\r\n#2  0x000000001001705c in PMIx_Init ()\r\n#3  0x0000000010003a84 in main ()\r\n\r\nThread 1 (Thread 0x3fff8e0ff1b0 (LWP 139246)):\r\n#0  0x0000000010102544 in pmix_usock_send_handler ()\r\n#1  0x0000000010106a20 in pmix_util_keyval_yylex ()\r\n#2  0x00000000100e5290 in _get_track_elem_for_namespace ()\r\n#3  0x000000001001dc88 in _getnbfn ()\r\n#4  0x00003fff8e4c2960 in event_process_active_single_queue (activeq=0x1003c9cfdf0, base=0x1003c9d0070) at event.c:1368\r\n#5  event_process_active (base=<optimized out>) at event.c:1438\r\n#6  event_base_loop (base=0x1003c9d0070, flags=<optimized out>) at event.c:1639\r\n#7  0x0000000010049c2c in pmix_progress_thread_init ()\r\n#8  0x00003fff8e308af4 in start_thread () from /lib64/libpthread.so.0\r\n#9  0x00003fff8e238814 in clone () from /lib64/libc.so.6\r\n(gdb) \r\n```\r\n\r\nIn certain cases the server and client pass the tests, but the server prints the following message\r\n```\r\n[c712f6n06:131681] PMIX ERROR: NOT-FOUND in file server/pmix_server_ops.c at line 1865\r\n[c712f6n06:131681] PMIX ERROR: NOT-FOUND in file server/pmix_server_ops.c at line 1865\r\n```"}}
{"actor":{"display_login":"nysal"}}