{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/2067","repository_url":"https://api.github.com/repos/open-mpi/ompi","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/2067/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/2067/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/2067/events","html_url":"https://github.com/open-mpi/ompi/issues/2067","id":175863388,"number":2067,"title":"Out-of-sequence (OOS) messages","user":{"login":"bosilca","id":642701,"avatar_url":"https://avatars.githubusercontent.com/u/642701?v=3","gravatar_id":"","url":"https://api.github.com/users/bosilca","html_url":"https://github.com/bosilca","followers_url":"https://api.github.com/users/bosilca/followers","following_url":"https://api.github.com/users/bosilca/following{/other_user}","gists_url":"https://api.github.com/users/bosilca/gists{/gist_id}","starred_url":"https://api.github.com/users/bosilca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bosilca/subscriptions","organizations_url":"https://api.github.com/users/bosilca/orgs","repos_url":"https://api.github.com/users/bosilca/repos","events_url":"https://api.github.com/users/bosilca/events{/privacy}","received_events_url":"https://api.github.com/users/bosilca/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/open-mpi/ompi/labels/blocker","name":"blocker","color":"e11d21"},{"url":"https://api.github.com/repos/open-mpi/ompi/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/open-mpi/ompi/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/open-mpi/ompi/milestones/15","html_url":"https://github.com/open-mpi/ompi/milestones/v2.1.0","labels_url":"https://api.github.com/repos/open-mpi/ompi/milestones/15/labels","id":1435864,"number":15,"title":"v2.1.0","description":"","creator":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":23,"state":"open","created_at":"2015-11-30T19:03:32Z","updated_at":"2016-09-08T21:01:06Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-09-08T21:00:55Z","updated_at":"2016-09-09T03:14:49Z","closed_at":null,"body":"**Short version** Out-of-sequence messages exists in a single link scenario\r\n\r\n**Cause** Intermediary buffering at different layers in the software stack allows the delivery of message out-of-sequence\r\n\r\n**Target** Most of the BTLs, with a particular emphasis on vader and IB.\r\n\r\n**Long version** This issue was raised during the discussion about the performance degradation seem between 1.8 and what will eventually become 3.x. While we identified the builtin atomics as being the main culprit, it turns out that enabling multi-threading raised a set of additional issues, not necessarily visible outside this particular usage.\r\n\r\nHaving multiple threads injecting messages into the PML in the context of a single communicator, lead to a significant number of out-of-sequence messages. The reason is that the per peer sequence number is taken very early in the software stack (optimization that makes sense for single threaded scenarios). Thus, between the moment when a thread acquires the sequence number and the moment when it's message is pushed into the network, there are many opportunities for another thread to bypass and reach the network first. From the receiver perspective this is seen as an out-of-sequence message, and it will be kept in linear structures and copied multiple time before it becomes in-sequence and can be delivered to the matching logic. There are multiple ways to mitigate this, but this discussion is outside the scope of this particular issue.\r\n\r\nMore worrisome is the fact that we observe out-of-sequence messages, using a single link and supposedly ordered BTLs, and this even when each thread is using a unique communicator. Logically, in this case no out-of-sequence message should be seen. At this point we assume that the immediate send optimization without a proper implementation in the BTLs is allowing message to bypass other messages waiting in the PML/BTL queues."},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/245807446","html_url":"https://github.com/open-mpi/ompi/issues/2067#issuecomment-245807446","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/2067","id":245807446,"user":{"login":"bosilca","id":642701,"avatar_url":"https://avatars.githubusercontent.com/u/642701?v=3","gravatar_id":"","url":"https://api.github.com/users/bosilca","html_url":"https://github.com/bosilca","followers_url":"https://api.github.com/users/bosilca/followers","following_url":"https://api.github.com/users/bosilca/following{/other_user}","gists_url":"https://api.github.com/users/bosilca/gists{/gist_id}","starred_url":"https://api.github.com/users/bosilca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bosilca/subscriptions","organizations_url":"https://api.github.com/users/bosilca/orgs","repos_url":"https://api.github.com/users/bosilca/repos","events_url":"https://api.github.com/users/bosilca/events{/privacy}","received_events_url":"https://api.github.com/users/bosilca/received_events","type":"User","site_admin":false},"created_at":"2016-09-09T03:14:49Z","updated_at":"2016-09-09T03:14:49Z","body":"Yes, @thananon has confirmed that a small number of OOS messages exists with vader.\r\n\r\nGoing back to your example, this is indeed how things unfold in the single threaded scenario. For the multi-threaded case it doesn't have to be MPI_Isend. The OOS only kicks in for the matching fragments, but if you push small messages from multiple threads we have this issue for every send below the eager limit, and for up to the number of sending threads for larger messages."}}
{"actor":{"display_login":"bosilca"}}