{"action":"created","issue":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/2291","repository_url":"https://api.github.com/repos/ofiwg/libfabric","labels_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2291/labels{/name}","comments_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2291/comments","events_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2291/events","html_url":"https://github.com/ofiwg/libfabric/issues/2291","id":173315441,"number":2291,"title":"prov/sockets: Under heavy load provider can mis-direct messages","user":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ofiwg/libfabric/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"shantonu","id":1229120,"avatar_url":"https://avatars.githubusercontent.com/u/1229120?v=3","gravatar_id":"","url":"https://api.github.com/users/shantonu","html_url":"https://github.com/shantonu","followers_url":"https://api.github.com/users/shantonu/followers","following_url":"https://api.github.com/users/shantonu/following{/other_user}","gists_url":"https://api.github.com/users/shantonu/gists{/gist_id}","starred_url":"https://api.github.com/users/shantonu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shantonu/subscriptions","organizations_url":"https://api.github.com/users/shantonu/orgs","repos_url":"https://api.github.com/users/shantonu/repos","events_url":"https://api.github.com/users/shantonu/events{/privacy}","received_events_url":"https://api.github.com/users/shantonu/received_events","type":"User","site_admin":false},"assignees":[{"login":"shantonu","id":1229120,"avatar_url":"https://avatars.githubusercontent.com/u/1229120?v=3","gravatar_id":"","url":"https://api.github.com/users/shantonu","html_url":"https://github.com/shantonu","followers_url":"https://api.github.com/users/shantonu/followers","following_url":"https://api.github.com/users/shantonu/following{/other_user}","gists_url":"https://api.github.com/users/shantonu/gists{/gist_id}","starred_url":"https://api.github.com/users/shantonu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shantonu/subscriptions","organizations_url":"https://api.github.com/users/shantonu/orgs","repos_url":"https://api.github.com/users/shantonu/repos","events_url":"https://api.github.com/users/shantonu/events{/privacy}","received_events_url":"https://api.github.com/users/shantonu/received_events","type":"User","site_admin":false},{"login":"jithinjosepkl","id":6690627,"avatar_url":"https://avatars.githubusercontent.com/u/6690627?v=3","gravatar_id":"","url":"https://api.github.com/users/jithinjosepkl","html_url":"https://github.com/jithinjosepkl","followers_url":"https://api.github.com/users/jithinjosepkl/followers","following_url":"https://api.github.com/users/jithinjosepkl/following{/other_user}","gists_url":"https://api.github.com/users/jithinjosepkl/gists{/gist_id}","starred_url":"https://api.github.com/users/jithinjosepkl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jithinjosepkl/subscriptions","organizations_url":"https://api.github.com/users/jithinjosepkl/orgs","repos_url":"https://api.github.com/users/jithinjosepkl/repos","events_url":"https://api.github.com/users/jithinjosepkl/events{/privacy}","received_events_url":"https://api.github.com/users/jithinjosepkl/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/ofiwg/libfabric/milestones/13","html_url":"https://github.com/ofiwg/libfabric/milestones/release%201.4.0","labels_url":"https://api.github.com/repos/ofiwg/libfabric/milestones/13/labels","id":1671739,"number":13,"title":"release 1.4.0","description":"","creator":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":112,"state":"open","created_at":"2016-03-29T12:35:33Z","updated_at":"2016-09-14T12:23:45Z","due_on":"2016-09-30T04:00:00Z","closed_at":null},"comments":23,"created_at":"2016-08-25T21:32:29Z","updated_at":"2016-09-14T12:28:57Z","closed_at":null,"body":"# Overview\r\n\r\nWhen the sockets provider is under heavy load, messages destined for peer A sometimes end up at peer B.\r\n\r\nThis problem is seen at the libfabric HEAD as of today (25 Aug 2016): 986d5f47f9435938bb459e8b2569be9288a5a1ce (a.k.a. v1.3.0-662-g986d5f4).\r\n\r\n# Reproducer\r\n\r\nI have a reproducer that sends and receives messages between multiple \"servers\" and \"clients\" using libfabric API calls, specifically forcing the use of the sockets provider:\r\n\r\nhttps://github.com/jsquyres/random-libfabric-tests/tree/master/abuse\r\n\r\n## Reproducer behavior\r\n\r\nThis reproducer is mimicking the behavior of a larger, more complex internal application that exhibits a similar message passing traffic pattern.  Specifically:\r\n\r\n* There are N server processes (where N is a \"small\" number -- e.g., single digits)\r\n* There are M client processes (where `M>=N`; M can be anywhere from `(N+1)` to `(very large)`)\r\n* All communication uses EP_RDM endpoints\r\n\r\n### Clients\r\n\r\nAfter receiving all the server addresses via MPI, clients loop forever doing this:\r\n* 5% of the time:\r\n   * There is code to destroy the client's EP and re-initialize all of the client's libfabric data structures.\r\n   * However, I was running into problems with this code, so it is currently commented out -- so 5% of the time, the loop iteration is a no-op.\r\n* 25% of the time:\r\n  * Randomly select a server.\r\n  * If this client is \"connected\" to this server, \"disconnect\" from that server.  Meaning:\r\n    * Send a \"DISCONNECT\" message to the server\r\n    * Remove this server from internal data structures\r\n    * Remove this server from the AV\r\n* The rest of the time:\r\n  * Randomly select a server.\r\n  * If this client is not already \"connected\" to this server:\r\n     * Send a \"CONNECT\" message to the server with some meta information about this client\r\n     * Wait for \"CONNECT_ACK\" message back from the server\r\n  * Send a \"SOLICIT_RDMA\" message to the server\r\n  * Server will RDMA a message back to me\r\n  * Server will then send a \"RDMA_SENT\" message back to me\r\n\r\n### Servers\r\n\r\nAfter sending their addressing data to all clients (via MPI), servers loop forever waiting for completions:\r\n\r\n* If it's a send completion, just make sure the send completed ok\r\n* If it's an RDMA write completion, just make sure the RDMA write completed ok\r\n* If it's a receive completion, it will be one of three incoming messages:\r\n   * CONNECT: from a client, sending a bunch of meta data about itself.  Send back a CONNECT_ACK message.\r\n   * DISCONNECT: from a client, telling us to forget everything about that client (and remove it from the AV)\r\n   * SOLICIT_RDMA: server sends an RDMA message back to the client, and then send a RDMA_SENT message back to the client indicating that the RDMA completed.  **NOTE:** The ordering of messages should guarantee that the RDMA_SENT message does not arrive until the RDMA write has actually arrived at the client.\r\n\r\n## Failure behavior\r\n\r\nIn all cases, both the client and the server add some meta data in the beginning of all messages sent indicating to whom the message is intended.\r\n\r\nThe failure mode that I see is that server A sometimes receives a message that contains meta data indicating that the message was actually sent to server B.\r\n\r\n## Reproducer code\r\n\r\nSome helpful tips about the reproducer code:\r\n\r\n1. It's C++, but really, it's mostly C code -- it's only C++ so that I could use the STL.\r\n1. The use of MPI is really only for bootstrapping:\r\n   * You run X processes per server, where `X>1`.\r\n   * The first MPI process on each server is designated the \"server\" -- all other MPI processes on that server are designated \"clients\".\r\n   * Once the servers open up EP_RDM endpoints, they send their fi_getname() addresses to all the clients via MPI API calls.\r\n1. Additionally, the MPI_Abort() function is used to kill the whole job (i.e., all clients and servers) when an inconsistency is found in the message data (i.e., anyone receives a message that was supposed to be sent to anyone else).\r\n\r\n## Local customizations\r\n\r\nThe reproducer has two things that you might/will need to edit:\r\n\r\n1. If you're not using Open MPI, you'll need to edit how the clients figure out that they are the first MPI process on a given node (other MPI's have implementation-specific ways to do this, but I don't know what they are).  See lines 130-139 for the details.\r\n2. The `Makefile` is hard-coded to my own paths; it's not a generalized `configure`-based setup or anything like that.\r\n\r\n# Running the test\r\n\r\nI have been able to exhibit the bad behavior:\r\n\r\n* On networks as slow as 1Gbs (i.e., it doesn't require 10, 40, or 100Gbps networks).\r\n* With 6 nodes and 18 MPI processes (i.e., 1 server and 2 clients per node).\r\n   * YMMV\r\n* With Open MPI:\r\n  * I ensure that Open MPI is not using libfabric, just to be fully sure that it isn't interfering with the test.\r\n  * I further limited MPI's message passing to a single TCP network, just to limit the variables under test.\r\n  * For example:\r\n\r\n```\r\n$ mpirun --host a:3,b:3,c:3 -np 18 --map-by node \\\r\n    --mca btl tcp,vader,self \\\r\n    --mca btl_tcp_if_include 10.0.0.0/16 \\\r\n    ./abuse\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/comments/246997284","html_url":"https://github.com/ofiwg/libfabric/issues/2291#issuecomment-246997284","issue_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2291","id":246997284,"user":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"created_at":"2016-09-14T12:28:57Z","updated_at":"2016-09-14T12:28:57Z","body":"@shefty Are you referring to `FI_ORDER_SAS`?  That seems to be the only `FI_ORDER_*` flag that refers to send semantics (that are unrelated to RMA semantics)."}}
{"actor":{"display_login":"jsquyres"}}