{"action":"created","issue":{"url":"https://api.github.com/repos/mercury-hpc/mercury/issues/111","repository_url":"https://api.github.com/repos/mercury-hpc/mercury","labels_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/111/labels{/name}","comments_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/111/comments","events_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/111/events","html_url":"https://github.com/mercury-hpc/mercury/issues/111","id":161345591,"number":111,"title":"NA CCI: race in connection","user":{"login":"soumagne","id":5613371,"avatar_url":"https://avatars.githubusercontent.com/u/5613371?v=3","gravatar_id":"","url":"https://api.github.com/users/soumagne","html_url":"https://github.com/soumagne","followers_url":"https://api.github.com/users/soumagne/followers","following_url":"https://api.github.com/users/soumagne/following{/other_user}","gists_url":"https://api.github.com/users/soumagne/gists{/gist_id}","starred_url":"https://api.github.com/users/soumagne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumagne/subscriptions","organizations_url":"https://api.github.com/users/soumagne/orgs","repos_url":"https://api.github.com/users/soumagne/repos","events_url":"https://api.github.com/users/soumagne/events{/privacy}","received_events_url":"https://api.github.com/users/soumagne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mercury-hpc/mercury/labels/bug","name":"bug","color":"000000"},{"url":"https://api.github.com/repos/mercury-hpc/mercury/labels/major","name":"major","color":"fbca04"},{"url":"https://api.github.com/repos/mercury-hpc/mercury/labels/na","name":"na","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/mercury-hpc/mercury/milestones/8","html_url":"https://github.com/mercury-hpc/mercury/milestones/mercury-1.0.0","labels_url":"https://api.github.com/repos/mercury-hpc/mercury/milestones/8/labels","id":1576694,"number":8,"title":"mercury-1.0.0","description":"### Features:\r\n * Full cancellation support","creator":{"login":"soumagne","id":5613371,"avatar_url":"https://avatars.githubusercontent.com/u/5613371?v=3","gravatar_id":"","url":"https://api.github.com/users/soumagne","html_url":"https://github.com/soumagne","followers_url":"https://api.github.com/users/soumagne/followers","following_url":"https://api.github.com/users/soumagne/following{/other_user}","gists_url":"https://api.github.com/users/soumagne/gists{/gist_id}","starred_url":"https://api.github.com/users/soumagne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumagne/subscriptions","organizations_url":"https://api.github.com/users/soumagne/orgs","repos_url":"https://api.github.com/users/soumagne/repos","events_url":"https://api.github.com/users/soumagne/events{/privacy}","received_events_url":"https://api.github.com/users/soumagne/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":0,"state":"open","created_at":"2016-02-10T23:23:55Z","updated_at":"2016-06-29T22:41:39Z","due_on":"2016-11-15T06:00:00Z","closed_at":null},"comments":19,"created_at":"2016-06-21T04:08:38Z","updated_at":"2016-07-01T13:48:02Z","closed_at":null,"body":"Xuezhao reported the following issue (must investigate):\r\n\r\n> One problem we observed is that sometimes mercury reports such error and cause the RPC broken:\r\nNA: Error in /scratch/DAOS_OPT/src/mercury_github/src/na/na_cci.c:1678             \r\n # handle_recv_unexpected(): host boro-47.boro.hpdd.intel.com, na_cci_addr->cci_addr: (nil), NA_CCI_PRIVATE_DATA(na_class)->uri: verbs://192.168.1.47:21090,  peer verbs://192.168.1.33:51218 refcnt 1\r\n (the above msg is a little different with mercury master as I added some info)\r\n\r\n> It can be triggered when there are multiple client-side processes send RPC request to the server concurrently.\r\n \r\n> After some investigations, I think it is a race condition in the connection establish process:\r\nThe server receive connect request from client, and do “cci_accept” in which will send reply to client. The client will get a “CCI_EVENT_CONNECT” event to notify the connecting done, and server will get a “CCI_EVENT_ACCEPT” event to notify the connection is established.\r\n \r\n> The problem is in some race condition, the client gets the “CCI_EVENT_CONNECT” first and then send RPC request (unexpected msg), but when the server got the unexpected msg it has not polled out the “CCI_EVENT_ACCEPT” event and firstly got another “CCI_EVENT_RECV” notification of the new unexpected msg arriving. Then when processing the unexpected msg the server treats it is invalid as the connection is not valid.\r\n \r\n> The attached simple patch is my fixing to that problem, I think that it is possibly not the best fix, but I cannot find out a better solution right now. And our testing can pass after that patch.\r\n\r\n[0001-handle-the-possible-race-in-connection-establish.txt](https://github.com/mercury-hpc/mercury/files/324771/0001-handle-the-possible-race-in-connection-establish.txt)\r\n"},"comment":{"url":"https://api.github.com/repos/mercury-hpc/mercury/issues/comments/229951226","html_url":"https://github.com/mercury-hpc/mercury/issues/111#issuecomment-229951226","issue_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/111","id":229951226,"user":{"login":"scottatchley","id":612032,"avatar_url":"https://avatars.githubusercontent.com/u/612032?v=3","gravatar_id":"","url":"https://api.github.com/users/scottatchley","html_url":"https://github.com/scottatchley","followers_url":"https://api.github.com/users/scottatchley/followers","following_url":"https://api.github.com/users/scottatchley/following{/other_user}","gists_url":"https://api.github.com/users/scottatchley/gists{/gist_id}","starred_url":"https://api.github.com/users/scottatchley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scottatchley/subscriptions","organizations_url":"https://api.github.com/users/scottatchley/orgs","repos_url":"https://api.github.com/users/scottatchley/repos","events_url":"https://api.github.com/users/scottatchley/events{/privacy}","received_events_url":"https://api.github.com/users/scottatchley/received_events","type":"User","site_admin":false},"created_at":"2016-07-01T13:48:02Z","updated_at":"2016-07-01T13:48:02Z","body":"Ok.\r\n\r\nCan you export CCI_DEBUG=conn,msg,info and run again?\r\n\r\nDo you think CCI is losing a message and/or event?\r\n\r\nScott\r\n\r\n> On Jul 1, 2016, at 3:40 AM, Liu Xuezhao <notifications@github.com> wrote:\r\n> \r\n> \"We mark the vep->rdma_channel as non-blocking, so this should not happen\"\r\n> In the above back trace, i think it is not blocking in the write(), it is just because client repeatly call the NA_progress so show that trace.\r\n> \r\n> —\r\n> You are receiving this because you commented.\r\n> Reply to this email directly, view it on GitHub, or mute the thread.\r\n> \r\n\r\n"}}
{"actor":{"display_login":"scottatchley"}}