{"action":"created","comment":{"url":"https://api.github.com/repos/pmodels/mpich/pulls/comments/193100849","pull_request_review_id":126005376,"id":193100849,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5MzEwMDg0OQ==","diff_hunk":"@@ -726,7 +726,9 @@ static inline int MPIDI_CH4R_win_finalize(MPIR_Win ** win_ptr)\n     do {\n         int all_local_completed = 0, all_remote_completed = 0;\n \n-        MPIDI_CH4R_PROGRESS();","path":"src/mpid/ch4/src/ch4r_win.h","position":43,"original_position":4,"commit_id":"8deede6a2589367cc001a9f82f1a9b0e1ce9b8c5","original_commit_id":"9601d634ca0d16ff43c5256381df62d36ca676d7","user":{"login":"hajimefu","id":21089164,"node_id":"MDQ6VXNlcjIxMDg5MTY0","avatar_url":"https://avatars1.githubusercontent.com/u/21089164?v=4","gravatar_id":"","url":"https://api.github.com/users/hajimefu","html_url":"https://github.com/hajimefu","followers_url":"https://api.github.com/users/hajimefu/followers","following_url":"https://api.github.com/users/hajimefu/following{/other_user}","gists_url":"https://api.github.com/users/hajimefu/gists{/gist_id}","starred_url":"https://api.github.com/users/hajimefu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hajimefu/subscriptions","organizations_url":"https://api.github.com/users/hajimefu/orgs","repos_url":"https://api.github.com/users/hajimefu/repos","events_url":"https://api.github.com/users/hajimefu/events{/privacy}","received_events_url":"https://api.github.com/users/hajimefu/received_events","type":"User","site_admin":false},"body":"Makes sense. Will do.","created_at":"2018-06-05T14:46:34Z","updated_at":"2018-06-05T14:46:34Z","html_url":"https://github.com/pmodels/mpich/pull/3117#discussion_r193100849","pull_request_url":"https://api.github.com/repos/pmodels/mpich/pulls/3117","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/pmodels/mpich/pulls/comments/193100849"},"html":{"href":"https://github.com/pmodels/mpich/pull/3117#discussion_r193100849"},"pull_request":{"href":"https://api.github.com/repos/pmodels/mpich/pulls/3117"}},"in_reply_to_id":192470036},"pull_request":{"url":"https://api.github.com/repos/pmodels/mpich/pulls/3117","id":185284828,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg1Mjg0ODI4","html_url":"https://github.com/pmodels/mpich/pull/3117","diff_url":"https://github.com/pmodels/mpich/pull/3117.diff","patch_url":"https://github.com/pmodels/mpich/pull/3117.patch","issue_url":"https://api.github.com/repos/pmodels/mpich/issues/3117","number":3117,"state":"open","locked":false,"title":"ch4: Introduce work queue for better multi-threading","user":{"login":"hajimefu","id":21089164,"node_id":"MDQ6VXNlcjIxMDg5MTY0","avatar_url":"https://avatars1.githubusercontent.com/u/21089164?v=4","gravatar_id":"","url":"https://api.github.com/users/hajimefu","html_url":"https://github.com/hajimefu","followers_url":"https://api.github.com/users/hajimefu/followers","following_url":"https://api.github.com/users/hajimefu/following{/other_user}","gists_url":"https://api.github.com/users/hajimefu/gists{/gist_id}","starred_url":"https://api.github.com/users/hajimefu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hajimefu/subscriptions","organizations_url":"https://api.github.com/users/hajimefu/orgs","repos_url":"https://api.github.com/users/hajimefu/repos","events_url":"https://api.github.com/users/hajimefu/events{/privacy}","received_events_url":"https://api.github.com/users/hajimefu/received_events","type":"User","site_admin":false},"body":"Summary\r\n--\r\nThis PR introduces a work queue infrastructure for better multi-threading.\r\nA new critical section granularity, per-VNI CS is introduced. Then a work queue infrastructure using izem is introduced. On top of them, three different CH4-level threading models, `direct`, `handoff`, and `trylock` are introduced. This PR also includes a few changes to OFI netmod to work under `handoff` and `trylock` models. Once we are confident with this approach we should also modify other netmods as well.\r\n\r\nThis PR supports single VNI case only. So far we have confirmed the `direct` and `handoff` modes work for most basic pt2pt/collective/rma test cases (with `--enable-ch4-pobj-workqueues=no`). Currently it does not support CH4 shared memory (`--enable-ch4-shm=exclusive`).\r\n\r\nThis PR includes the prereq patches from #3116.\r\n\r\nImplements #2391\r\n\r\nDiscussion\r\n--\r\nThis PR introduces a new netmod API, `MPIDI_NM_win_fence_flush`. This is essentially `MPI_Win_fence` minus barrier. We think this is necessary, otherwise the `MPI_Win_fence` implementation for per-VNI CS would look like this:\r\n```C\r\nMPID_Win_fence(assert, win) {\r\n  lock(VNI);\r\n  MPIDI_NM_mpi_win_fence(assert, win);\r\n  unlock(VNI);\r\n}\r\n```\r\nand causes deadlock when multiple threads are calling `MPI_Win_fence` for different windows at the same time (e.g. `test/mpi/threads/rma/multifence`), because the lock is held during the barrier inside `MPIDI_NM_win_fence`. With `MPIDI_NM_win_fence_flush`, the implementation would become:\r\n```C\r\nMPID_Win_fence(assert, win) {\r\n  lock(VNI);\r\n  MPIDI_NM_win_fence_flush(assert, win);\r\n  unlock(VNI);\r\n  MPIR_Barrier(win->comm_ptr);\r\n}\r\n```\r\nSo far we think `MPI_Win_fence` is the only one among other window synchronization functions that requires this kind of special treatment.\r\n\r\nUnsupported Features\r\n--\r\n\r\n* Shared memory `--enable-ch4-shm=exclusive`\r\n* `MPI_Mrecv`, `MPI_Imrecv`\r\n* `MPI_Accumulate`, `MPI_Get_accumulate`, `MPI_Fetch_and_op`, `MPI_Compare_and_swap`\r\n\r\nSample Configuration\r\n--\r\nHere is my configure line for testing this branch:\r\n```\r\nConfiguring MPICH version 3.3b1 with  '-C' '--prefix=/home/hajimefu/opt/mpich-yes-tvni-gnu-debug' '--disable-perftest' '--with-libfabric=/home/hajimefu' '--disable-ft-tests' '--with-fwrapname=mpigf' '--with-filesystem=ufs+nfs' '--enable-timer-type=linux86_cycle' '--enable-romio' '--with-mpe=no' '--with-smpcoll=yes' '--with-assert-level=0' '--enable-shared' '--enable-static' '--enable-error-messages=yes' '--enable-visibility' '--enable-large-tests' '--enable-strict' '--enable-g=all' '--enable-timing=runtime' '--enable-error-checking=all' '--enable-fast=none' '--enable-debuginfo' '--with-device=ch4:ofi' '--enable-handle-allocation=default' '--enable-threads=multiple' '--enable-ch4-shm=yes' '--with-ch4-netmod-ofi-args=' '--enable-thread-cs=per-vni' '--enable-izem=queue' '--with-zm-prefix=yes' '--enable-ch4-mt=runtime' '--enable-ch4-pobj-workqueue=runtime' 'MPICHLIB_CFLAGS=-O0 -Wall -ggdb -DMPIDI_USE_GLQUEUE -ggdb -Wall -mtune=generic -std=gnu99' 'MPICHLIB_CXXFLAGS=-O0 -Wall -ggdb  -ggdb -Wall -mtune=generic' 'MPICHLIB_FCFLAGS=-O0 -Wall -ggdb  -ggdb -Wall -mtune=generic' 'MPICHLIB_F77FLAGS=-O0 -Wall -ggdb  -ggdb -Wall -mtune=generic' 'MPICHLIB_LDFLAGS=-O0 -L/usr/lib64  -mtune=generic' 'CC=gcc' 'LDFLAGS= -Wl,-z,muldefs -Wl,-z,now' 'CXX=g++' 'FC=gfortran' 'F77=gfortran'\r\n```\r\nNotable change from the master branch: `'--enable-thread-cs=per-vni' '--enable-izem=queue' '--with-zm-prefix=yes' '--enable-ch4-mt=runtime' '--enable-ch4-pobj-workqueue=runtime' -DMPIDI_USE_GLQUEUE`","created_at":"2018-05-01T21:46:25Z","updated_at":"2018-06-05T14:46:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"07d781db0196d0b2e4bd40664d45313bb2870f3e","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":{"url":"https://api.github.com/repos/pmodels/mpich/milestones/33","html_url":"https://github.com/pmodels/mpich/milestone/33","labels_url":"https://api.github.com/repos/pmodels/mpich/milestones/33/labels","id":3087464,"node_id":"MDk6TWlsZXN0b25lMzA4NzQ2NA==","number":33,"title":"mpich-3.3b3","description":"","creator":{"login":"pavanbalaji","id":4279602,"node_id":"MDQ6VXNlcjQyNzk2MDI=","avatar_url":"https://avatars0.githubusercontent.com/u/4279602?v=4","gravatar_id":"","url":"https://api.github.com/users/pavanbalaji","html_url":"https://github.com/pavanbalaji","followers_url":"https://api.github.com/users/pavanbalaji/followers","following_url":"https://api.github.com/users/pavanbalaji/following{/other_user}","gists_url":"https://api.github.com/users/pavanbalaji/gists{/gist_id}","starred_url":"https://api.github.com/users/pavanbalaji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pavanbalaji/subscriptions","organizations_url":"https://api.github.com/users/pavanbalaji/orgs","repos_url":"https://api.github.com/users/pavanbalaji/repos","events_url":"https://api.github.com/users/pavanbalaji/events{/privacy}","received_events_url":"https://api.github.com/users/pavanbalaji/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":23,"state":"open","created_at":"2018-02-04T18:28:38Z","updated_at":"2018-06-04T16:11:02Z","due_on":"2018-05-31T07:00:00Z","closed_at":null},"commits_url":"https://api.github.com/repos/pmodels/mpich/pulls/3117/commits","review_comments_url":"https://api.github.com/repos/pmodels/mpich/pulls/3117/comments","review_comment_url":"https://api.github.com/repos/pmodels/mpich/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pmodels/mpich/issues/3117/comments","statuses_url":"https://api.github.com/repos/pmodels/mpich/statuses/8deede6a2589367cc001a9f82f1a9b0e1ce9b8c5","head":{"label":"hajimefu:ch4-vni-pr1","ref":"ch4-vni-pr1","sha":"8deede6a2589367cc001a9f82f1a9b0e1ce9b8c5","user":{"login":"hajimefu","id":21089164,"node_id":"MDQ6VXNlcjIxMDg5MTY0","avatar_url":"https://avatars1.githubusercontent.com/u/21089164?v=4","gravatar_id":"","url":"https://api.github.com/users/hajimefu","html_url":"https://github.com/hajimefu","followers_url":"https://api.github.com/users/hajimefu/followers","following_url":"https://api.github.com/users/hajimefu/following{/other_user}","gists_url":"https://api.github.com/users/hajimefu/gists{/gist_id}","starred_url":"https://api.github.com/users/hajimefu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hajimefu/subscriptions","organizations_url":"https://api.github.com/users/hajimefu/orgs","repos_url":"https://api.github.com/users/hajimefu/repos","events_url":"https://api.github.com/users/hajimefu/events{/privacy}","received_events_url":"https://api.github.com/users/hajimefu/received_events","type":"User","site_admin":false},"repo":{"id":73229221,"node_id":"MDEwOlJlcG9zaXRvcnk3MzIyOTIyMQ==","name":"mpich","full_name":"hajimefu/mpich","owner":{"login":"hajimefu","id":21089164,"node_id":"MDQ6VXNlcjIxMDg5MTY0","avatar_url":"https://avatars1.githubusercontent.com/u/21089164?v=4","gravatar_id":"","url":"https://api.github.com/users/hajimefu","html_url":"https://github.com/hajimefu","followers_url":"https://api.github.com/users/hajimefu/followers","following_url":"https://api.github.com/users/hajimefu/following{/other_user}","gists_url":"https://api.github.com/users/hajimefu/gists{/gist_id}","starred_url":"https://api.github.com/users/hajimefu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hajimefu/subscriptions","organizations_url":"https://api.github.com/users/hajimefu/orgs","repos_url":"https://api.github.com/users/hajimefu/repos","events_url":"https://api.github.com/users/hajimefu/events{/privacy}","received_events_url":"https://api.github.com/users/hajimefu/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/hajimefu/mpich","description":"Official MPICH Repository","fork":true,"url":"https://api.github.com/repos/hajimefu/mpich","forks_url":"https://api.github.com/repos/hajimefu/mpich/forks","keys_url":"https://api.github.com/repos/hajimefu/mpich/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hajimefu/mpich/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hajimefu/mpich/teams","hooks_url":"https://api.github.com/repos/hajimefu/mpich/hooks","issue_events_url":"https://api.github.com/repos/hajimefu/mpich/issues/events{/number}","events_url":"https://api.github.com/repos/hajimefu/mpich/events","assignees_url":"https://api.github.com/repos/hajimefu/mpich/assignees{/user}","branches_url":"https://api.github.com/repos/hajimefu/mpich/branches{/branch}","tags_url":"https://api.github.com/repos/hajimefu/mpich/tags","blobs_url":"https://api.github.com/repos/hajimefu/mpich/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hajimefu/mpich/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hajimefu/mpich/git/refs{/sha}","trees_url":"https://api.github.com/repos/hajimefu/mpich/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hajimefu/mpich/statuses/{sha}","languages_url":"https://api.github.com/repos/hajimefu/mpich/languages","stargazers_url":"https://api.github.com/repos/hajimefu/mpich/stargazers","contributors_url":"https://api.github.com/repos/hajimefu/mpich/contributors","subscribers_url":"https://api.github.com/repos/hajimefu/mpich/subscribers","subscription_url":"https://api.github.com/repos/hajimefu/mpich/subscription","commits_url":"https://api.github.com/repos/hajimefu/mpich/commits{/sha}","git_commits_url":"https://api.github.com/repos/hajimefu/mpich/git/commits{/sha}","comments_url":"https://api.github.com/repos/hajimefu/mpich/comments{/number}","issue_comment_url":"https://api.github.com/repos/hajimefu/mpich/issues/comments{/number}","contents_url":"https://api.github.com/repos/hajimefu/mpich/contents/{+path}","compare_url":"https://api.github.com/repos/hajimefu/mpich/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hajimefu/mpich/merges","archive_url":"https://api.github.com/repos/hajimefu/mpich/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hajimefu/mpich/downloads","issues_url":"https://api.github.com/repos/hajimefu/mpich/issues{/number}","pulls_url":"https://api.github.com/repos/hajimefu/mpich/pulls{/number}","milestones_url":"https://api.github.com/repos/hajimefu/mpich/milestones{/number}","notifications_url":"https://api.github.com/repos/hajimefu/mpich/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hajimefu/mpich/labels{/name}","releases_url":"https://api.github.com/repos/hajimefu/mpich/releases{/id}","deployments_url":"https://api.github.com/repos/hajimefu/mpich/deployments","created_at":"2016-11-08T21:37:29Z","updated_at":"2016-11-08T21:37:42Z","pushed_at":"2018-05-24T15:52:07Z","git_url":"git://github.com/hajimefu/mpich.git","ssh_url":"git@github.com:hajimefu/mpich.git","clone_url":"https://github.com/hajimefu/mpich.git","svn_url":"https://github.com/hajimefu/mpich","homepage":"http://www.mpich.org","size":58626,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"pmodels:master","ref":"master","sha":"d810cb0742a450a777d7bf231fc9e0cec59a6796","user":{"login":"pmodels","id":15700567,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1NzAwNTY3","avatar_url":"https://avatars2.githubusercontent.com/u/15700567?v=4","gravatar_id":"","url":"https://api.github.com/users/pmodels","html_url":"https://github.com/pmodels","followers_url":"https://api.github.com/users/pmodels/followers","following_url":"https://api.github.com/users/pmodels/following{/other_user}","gists_url":"https://api.github.com/users/pmodels/gists{/gist_id}","starred_url":"https://api.github.com/users/pmodels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmodels/subscriptions","organizations_url":"https://api.github.com/users/pmodels/orgs","repos_url":"https://api.github.com/users/pmodels/repos","events_url":"https://api.github.com/users/pmodels/events{/privacy}","received_events_url":"https://api.github.com/users/pmodels/received_events","type":"Organization","site_admin":false},"repo":{"id":70918679,"node_id":"MDEwOlJlcG9zaXRvcnk3MDkxODY3OQ==","name":"mpich","full_name":"pmodels/mpich","owner":{"login":"pmodels","id":15700567,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1NzAwNTY3","avatar_url":"https://avatars2.githubusercontent.com/u/15700567?v=4","gravatar_id":"","url":"https://api.github.com/users/pmodels","html_url":"https://github.com/pmodels","followers_url":"https://api.github.com/users/pmodels/followers","following_url":"https://api.github.com/users/pmodels/following{/other_user}","gists_url":"https://api.github.com/users/pmodels/gists{/gist_id}","starred_url":"https://api.github.com/users/pmodels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmodels/subscriptions","organizations_url":"https://api.github.com/users/pmodels/orgs","repos_url":"https://api.github.com/users/pmodels/repos","events_url":"https://api.github.com/users/pmodels/events{/privacy}","received_events_url":"https://api.github.com/users/pmodels/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pmodels/mpich","description":"Official MPICH Repository","fork":false,"url":"https://api.github.com/repos/pmodels/mpich","forks_url":"https://api.github.com/repos/pmodels/mpich/forks","keys_url":"https://api.github.com/repos/pmodels/mpich/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pmodels/mpich/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pmodels/mpich/teams","hooks_url":"https://api.github.com/repos/pmodels/mpich/hooks","issue_events_url":"https://api.github.com/repos/pmodels/mpich/issues/events{/number}","events_url":"https://api.github.com/repos/pmodels/mpich/events","assignees_url":"https://api.github.com/repos/pmodels/mpich/assignees{/user}","branches_url":"https://api.github.com/repos/pmodels/mpich/branches{/branch}","tags_url":"https://api.github.com/repos/pmodels/mpich/tags","blobs_url":"https://api.github.com/repos/pmodels/mpich/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pmodels/mpich/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pmodels/mpich/git/refs{/sha}","trees_url":"https://api.github.com/repos/pmodels/mpich/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pmodels/mpich/statuses/{sha}","languages_url":"https://api.github.com/repos/pmodels/mpich/languages","stargazers_url":"https://api.github.com/repos/pmodels/mpich/stargazers","contributors_url":"https://api.github.com/repos/pmodels/mpich/contributors","subscribers_url":"https://api.github.com/repos/pmodels/mpich/subscribers","subscription_url":"https://api.github.com/repos/pmodels/mpich/subscription","commits_url":"https://api.github.com/repos/pmodels/mpich/commits{/sha}","git_commits_url":"https://api.github.com/repos/pmodels/mpich/git/commits{/sha}","comments_url":"https://api.github.com/repos/pmodels/mpich/comments{/number}","issue_comment_url":"https://api.github.com/repos/pmodels/mpich/issues/comments{/number}","contents_url":"https://api.github.com/repos/pmodels/mpich/contents/{+path}","compare_url":"https://api.github.com/repos/pmodels/mpich/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pmodels/mpich/merges","archive_url":"https://api.github.com/repos/pmodels/mpich/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pmodels/mpich/downloads","issues_url":"https://api.github.com/repos/pmodels/mpich/issues{/number}","pulls_url":"https://api.github.com/repos/pmodels/mpich/pulls{/number}","milestones_url":"https://api.github.com/repos/pmodels/mpich/milestones{/number}","notifications_url":"https://api.github.com/repos/pmodels/mpich/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pmodels/mpich/labels{/name}","releases_url":"https://api.github.com/repos/pmodels/mpich/releases{/id}","deployments_url":"https://api.github.com/repos/pmodels/mpich/deployments","created_at":"2016-10-14T14:39:42Z","updated_at":"2018-06-05T12:55:25Z","pushed_at":"2018-06-05T13:55:53Z","git_url":"git://github.com/pmodels/mpich.git","ssh_url":"git@github.com:pmodels/mpich.git","clone_url":"https://github.com/pmodels/mpich.git","svn_url":"https://github.com/pmodels/mpich","homepage":"http://www.mpich.org","size":56242,"stargazers_count":91,"watchers_count":91,"language":"C","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":91,"mirror_url":null,"archived":false,"open_issues_count":393,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":91,"open_issues":393,"watchers":91,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pmodels/mpich/pulls/3117"},"html":{"href":"https://github.com/pmodels/mpich/pull/3117"},"issue":{"href":"https://api.github.com/repos/pmodels/mpich/issues/3117"},"comments":{"href":"https://api.github.com/repos/pmodels/mpich/issues/3117/comments"},"review_comments":{"href":"https://api.github.com/repos/pmodels/mpich/pulls/3117/comments"},"review_comment":{"href":"https://api.github.com/repos/pmodels/mpich/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pmodels/mpich/pulls/3117/commits"},"statuses":{"href":"https://api.github.com/repos/pmodels/mpich/statuses/8deede6a2589367cc001a9f82f1a9b0e1ce9b8c5"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"hajimefu"}}