{"action":"created","issue":{"url":"https://api.github.com/repos/nats-io/gnatsd/issues/632","repository_url":"https://api.github.com/repos/nats-io/gnatsd","labels_url":"https://api.github.com/repos/nats-io/gnatsd/issues/632/labels{/name}","comments_url":"https://api.github.com/repos/nats-io/gnatsd/issues/632/comments","events_url":"https://api.github.com/repos/nats-io/gnatsd/issues/632/events","html_url":"https://github.com/nats-io/gnatsd/issues/632","id":302369622,"number":632,"title":"Defect: messages lost with non-zero max_msgs in 2-node cluster (bi-directional routes)","user":{"login":"vkhorozvs","id":22294408,"avatar_url":"https://avatars0.githubusercontent.com/u/22294408?v=4","gravatar_id":"","url":"https://api.github.com/users/vkhorozvs","html_url":"https://github.com/vkhorozvs","followers_url":"https://api.github.com/users/vkhorozvs/followers","following_url":"https://api.github.com/users/vkhorozvs/following{/other_user}","gists_url":"https://api.github.com/users/vkhorozvs/gists{/gist_id}","starred_url":"https://api.github.com/users/vkhorozvs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkhorozvs/subscriptions","organizations_url":"https://api.github.com/users/vkhorozvs/orgs","repos_url":"https://api.github.com/users/vkhorozvs/repos","events_url":"https://api.github.com/users/vkhorozvs/events{/privacy}","received_events_url":"https://api.github.com/users/vkhorozvs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-05T16:32:23Z","updated_at":"2018-03-05T16:35:33Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Versions of `gnatsd` affected:**\r\nFound in v0.9.4, reproduced in v1.0.4.\r\nMost probably affects all versions.\r\n\r\n**OS/Container environment:**\r\nLinux CentOS 7\r\n\r\n**Steps or code to reproduce the issue:**\r\n1. Setup a NATS cluster of 2 (or more) nodes with bi-directional routes.\r\n2. Connect 2 (or more) queue subscribers with max_msgs set to non-zero value to different NATS server nodes.\r\n3. Have 2 (or more) publishers connect to different NATS server nodes.\r\n4. Let publishers publish as many messages as needed to feed all subscribers.\r\n    E.g. with 2 nodes, 10 subscribers, and max_msgs set to 1 need to publish 20 messages.\r\n \r\n**Expected result:**\r\nAll messages are delivered\r\n \r\n**Actual result:**\r\nSome messages are lost.\r\nAn exact number differs from test to test, sometimes all messages are lost.\r\n\r\n**Test Code:**\r\nI've created the following bash script to reproduce this issue:\r\n```\r\n./gnatsd -p 4001 --cluster nats-route://localhost:4011/ --routes nats-route://localhost:4012/ &\r\n./gnatsd -p 4002 --cluster nats-route://localhost:4012/ --routes nats-route://localhost:4011/ &\r\nsleep 1\r\nfor i in `seq 100`; do\r\n    ( echo \"sub foo bar $i\"; echo \"unsub $i 1\"; sleep 5; ) | nc localhost 400$(($i % 2 + 1)) -C &\r\ndone\r\nsleep 1\r\nfor i in `seq 0 9`; do\r\n    ( sleep 1; for n in `seq 0 9`; do echo 'pub foo 6'; echo \"test$i$n\"; done ) | nc localhost 400$(($i % 2 + 1)) -C &\r\ndone\r\n\r\nsleep 5\r\n\r\nkill %1 %2\r\n```\r\n\r\nA test script is sending 100 messages from 10 publisher to 100 subscribers."},"comment":{"url":"https://api.github.com/repos/nats-io/gnatsd/issues/comments/370479422","html_url":"https://github.com/nats-io/gnatsd/issues/632#issuecomment-370479422","issue_url":"https://api.github.com/repos/nats-io/gnatsd/issues/632","id":370479422,"user":{"login":"vkhorozvs","id":22294408,"avatar_url":"https://avatars0.githubusercontent.com/u/22294408?v=4","gravatar_id":"","url":"https://api.github.com/users/vkhorozvs","html_url":"https://github.com/vkhorozvs","followers_url":"https://api.github.com/users/vkhorozvs/followers","following_url":"https://api.github.com/users/vkhorozvs/following{/other_user}","gists_url":"https://api.github.com/users/vkhorozvs/gists{/gist_id}","starred_url":"https://api.github.com/users/vkhorozvs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkhorozvs/subscriptions","organizations_url":"https://api.github.com/users/vkhorozvs/orgs","repos_url":"https://api.github.com/users/vkhorozvs/repos","events_url":"https://api.github.com/users/vkhorozvs/events{/privacy}","received_events_url":"https://api.github.com/users/vkhorozvs/received_events","type":"User","site_admin":false},"created_at":"2018-03-05T16:35:33Z","updated_at":"2018-03-05T16:35:33Z","author_association":"CONTRIBUTOR","body":"A problem is a race condition inside server/client.go in processMsg related to how unsubscribe is broadcasted between routes.  I've created a patch to fix this issue (will create a pull request in short).\r\n\r\nWith patched NATS server no messages were lost:\r\n```\r\n[vkhoroz@localhost gnatsd]$ for t in `seq 20`; do bash ./test.sh 2>&1 | grep test | wc -l && sleep 1; done\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n100\r\n```\r\n\r\nHowever, I'd like to ask somebody from NATS core developers to assess if all use cases are covered.\r\nI would appreciate if a patch is included in some of upcoming releases."}}
{"actor":{"display_login":"vkhorozvs"}}