{"action":"created","issue":{"url":"https://api.github.com/repos/real-logic/Aeron/issues/281","repository_url":"https://api.github.com/repos/real-logic/Aeron","labels_url":"https://api.github.com/repos/real-logic/Aeron/issues/281/labels{/name}","comments_url":"https://api.github.com/repos/real-logic/Aeron/issues/281/comments","events_url":"https://api.github.com/repos/real-logic/Aeron/issues/281/events","html_url":"https://github.com/real-logic/Aeron/issues/281","id":179961122,"number":281,"title":"Stalled Publication/Subscription","user":{"login":"patriknw","id":336161,"avatar_url":"https://avatars.githubusercontent.com/u/336161?v=3","gravatar_id":"","url":"https://api.github.com/users/patriknw","html_url":"https://github.com/patriknw","followers_url":"https://api.github.com/users/patriknw/followers","following_url":"https://api.github.com/users/patriknw/following{/other_user}","gists_url":"https://api.github.com/users/patriknw/gists{/gist_id}","starred_url":"https://api.github.com/users/patriknw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patriknw/subscriptions","organizations_url":"https://api.github.com/users/patriknw/orgs","repos_url":"https://api.github.com/users/patriknw/repos","events_url":"https://api.github.com/users/patriknw/events{/privacy}","received_events_url":"https://api.github.com/users/patriknw/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-09-29T06:12:36Z","updated_at":"2016-09-29T21:26:49Z","closed_at":null,"body":"Version 1.0.1\r\n\r\nI'm debugging a case where `sub.poll` suddenly doesn't receive any more messages, and it stays that way. On the sending side `pub.offer` is successful (positive return value and connected=true closed=false).\r\n\r\nOnly two nodes.\r\n* node1: 172-31-10-77\r\n* node2: 172-31-8-204\r\n\r\nnode1 tries to send to node2, but node2 is not started yet. ~30 seconds later node2 is started and they successfully exchange a few messages (I have application level logs for that), and then node2 stops receiving more messages in `sub.poll`. `onFragment` in the FragmentAssembler is not invoked even though I repeatedly call `poll`. Those messages are below mtu size.\r\n\r\nThe systems are rather loaded in this scenario but not overloaded, and the load is stopped after a while.\r\n\r\nAeronStat from node1 172-31-10-77:\r\n\r\n```\r\n 23:                    1 - recv-channel: aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 24:        1,627,656,896 - snd-pos: 8 1287205435 1 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 25:                    1 - send-channel: aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 26:        1,636,045,504 - pub-lmt: 8 1287205435 1 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 27:              170,432 - sub-pos: 1 434696915 1 aeron:udp?endpoint=ip-172-31-10-77:25520 @0\r\n 28:              170,432 - rcv-hwm: 7 434696915 1 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 29:           19,750,176 - sub-pos: 2 434696916 2 aeron:udp?endpoint=ip-172-31-10-77:25520 @0\r\n 30:           19,750,176 - rcv-hwm: 9 434696916 2 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 31:           19,750,176 - rcv-pos: 9 434696916 2 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 32:           28,953,216 - pub-lmt: 10 1287205436 2 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 33:           20,564,608 - snd-pos: 10 1287205436 2 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n```\r\n\r\nAeronStat from node2 172-31-8-204:\r\n\r\n```\r\n 23:                    1 - recv-channel: aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 24:                    1 - send-channel: aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 25:            8,559,040 - pub-lmt: 3 434696915 1 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 26:              170,432 - snd-pos: 3 434696915 1 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 27:           28,138,784 - pub-lmt: 4 434696916 2 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 28:           19,750,176 - snd-pos: 4 434696916 2 aeron:udp?endpoint=ip-172-31-10-77:25520\r\n 29:                2,176 - sub-pos: 1 1287205435 1 aeron:udp?endpoint=ip-172-31-8-204:25520 @0\r\n 30:        1,627,656,896 - rcv-hwm: 5 1287205435 1 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 31:                2,176 - rcv-pos: 5 1287205435 1 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 32:           20,500,992 - sub-pos: 2 1287205436 2 aeron:udp?endpoint=ip-172-31-8-204:25520 @0\r\n 33:           20,500,992 - rcv-hwm: 6 1287205436 2 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n 34:           20,500,992 - rcv-pos: 6 1287205436 2 aeron:udp?endpoint=ip-172-31-8-204:25520\r\n```\r\n\r\nThe problematic session is 1287205435. The other streams seem to progress. I kept it running for minutes after the stall.\r\n\r\nStream 1 is our control stream and it's low traffic, a few messages per second on this stream.\r\n\r\nI have all files, if you need more information."},"comment":{"url":"https://api.github.com/repos/real-logic/Aeron/issues/comments/250596327","html_url":"https://github.com/real-logic/Aeron/issues/281#issuecomment-250596327","issue_url":"https://api.github.com/repos/real-logic/Aeron/issues/281","id":250596327,"user":{"login":"tmontgomery","id":1979352,"avatar_url":"https://avatars.githubusercontent.com/u/1979352?v=3","gravatar_id":"","url":"https://api.github.com/users/tmontgomery","html_url":"https://github.com/tmontgomery","followers_url":"https://api.github.com/users/tmontgomery/followers","following_url":"https://api.github.com/users/tmontgomery/following{/other_user}","gists_url":"https://api.github.com/users/tmontgomery/gists{/gist_id}","starred_url":"https://api.github.com/users/tmontgomery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmontgomery/subscriptions","organizations_url":"https://api.github.com/users/tmontgomery/orgs","repos_url":"https://api.github.com/users/tmontgomery/repos","events_url":"https://api.github.com/users/tmontgomery/events{/privacy}","received_events_url":"https://api.github.com/users/tmontgomery/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T21:26:49Z","updated_at":"2016-09-29T21:26:49Z","body":"@mjpt777 BTW, the rcv-hwm is incremented on a heartbeat irregardless of overrun. So, it would be possible to have a heartbeat like message come in that sets the rcv-hwm up to some value."}}
{"actor":{"display_login":"tmontgomery"}}