{"action":"created","issue":{"url":"https://api.github.com/repos/emqx/emqx/issues/1880","repository_url":"https://api.github.com/repos/emqx/emqx","labels_url":"https://api.github.com/repos/emqx/emqx/issues/1880/labels{/name}","comments_url":"https://api.github.com/repos/emqx/emqx/issues/1880/comments","events_url":"https://api.github.com/repos/emqx/emqx/issues/1880/events","html_url":"https://github.com/emqx/emqx/issues/1880","id":366271746,"node_id":"MDU6SXNzdWUzNjYyNzE3NDY=","number":1880,"title":"Not receiving messages after a load test","user":{"login":"manas-chaudhari","id":4344796,"node_id":"MDQ6VXNlcjQzNDQ3OTY=","avatar_url":"https://avatars2.githubusercontent.com/u/4344796?v=4","gravatar_id":"","url":"https://api.github.com/users/manas-chaudhari","html_url":"https://github.com/manas-chaudhari","followers_url":"https://api.github.com/users/manas-chaudhari/followers","following_url":"https://api.github.com/users/manas-chaudhari/following{/other_user}","gists_url":"https://api.github.com/users/manas-chaudhari/gists{/gist_id}","starred_url":"https://api.github.com/users/manas-chaudhari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manas-chaudhari/subscriptions","organizations_url":"https://api.github.com/users/manas-chaudhari/orgs","repos_url":"https://api.github.com/users/manas-chaudhari/repos","events_url":"https://api.github.com/users/manas-chaudhari/events{/privacy}","received_events_url":"https://api.github.com/users/manas-chaudhari/received_events","type":"User","site_admin":false},"labels":[{"id":267457867,"node_id":"MDU6TGFiZWwyNjc0NTc4Njc=","url":"https://api.github.com/repos/emqx/emqx/labels/Benchmark","name":"Benchmark","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"Gilbert-Wong","id":20479699,"node_id":"MDQ6VXNlcjIwNDc5Njk5","avatar_url":"https://avatars0.githubusercontent.com/u/20479699?v=4","gravatar_id":"","url":"https://api.github.com/users/Gilbert-Wong","html_url":"https://github.com/Gilbert-Wong","followers_url":"https://api.github.com/users/Gilbert-Wong/followers","following_url":"https://api.github.com/users/Gilbert-Wong/following{/other_user}","gists_url":"https://api.github.com/users/Gilbert-Wong/gists{/gist_id}","starred_url":"https://api.github.com/users/Gilbert-Wong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Gilbert-Wong/subscriptions","organizations_url":"https://api.github.com/users/Gilbert-Wong/orgs","repos_url":"https://api.github.com/users/Gilbert-Wong/repos","events_url":"https://api.github.com/users/Gilbert-Wong/events{/privacy}","received_events_url":"https://api.github.com/users/Gilbert-Wong/received_events","type":"User","site_admin":false},"assignees":[{"login":"Gilbert-Wong","id":20479699,"node_id":"MDQ6VXNlcjIwNDc5Njk5","avatar_url":"https://avatars0.githubusercontent.com/u/20479699?v=4","gravatar_id":"","url":"https://api.github.com/users/Gilbert-Wong","html_url":"https://github.com/Gilbert-Wong","followers_url":"https://api.github.com/users/Gilbert-Wong/followers","following_url":"https://api.github.com/users/Gilbert-Wong/following{/other_user}","gists_url":"https://api.github.com/users/Gilbert-Wong/gists{/gist_id}","starred_url":"https://api.github.com/users/Gilbert-Wong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Gilbert-Wong/subscriptions","organizations_url":"https://api.github.com/users/Gilbert-Wong/orgs","repos_url":"https://api.github.com/users/Gilbert-Wong/repos","events_url":"https://api.github.com/users/Gilbert-Wong/events{/privacy}","received_events_url":"https://api.github.com/users/Gilbert-Wong/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/emqx/emqx/milestones/117","html_url":"https://github.com/emqx/emqx/milestone/117","labels_url":"https://api.github.com/repos/emqx/emqx/milestones/117/labels","id":2998268,"node_id":"MDk6TWlsZXN0b25lMjk5ODI2OA==","number":117,"title":"3.0-rc.1","description":"","creator":{"login":"emqplus","id":11869,"node_id":"MDQ6VXNlcjExODY5","avatar_url":"https://avatars0.githubusercontent.com/u/11869?v=4","gravatar_id":"","url":"https://api.github.com/users/emqplus","html_url":"https://github.com/emqplus","followers_url":"https://api.github.com/users/emqplus/followers","following_url":"https://api.github.com/users/emqplus/following{/other_user}","gists_url":"https://api.github.com/users/emqplus/gists{/gist_id}","starred_url":"https://api.github.com/users/emqplus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emqplus/subscriptions","organizations_url":"https://api.github.com/users/emqplus/orgs","repos_url":"https://api.github.com/users/emqplus/repos","events_url":"https://api.github.com/users/emqplus/events{/privacy}","received_events_url":"https://api.github.com/users/emqplus/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":3,"state":"open","created_at":"2017-12-26T14:25:24Z","updated_at":"2018-10-12T06:22:03Z","due_on":"2018-09-30T07:00:00Z","closed_at":null},"comments":3,"created_at":"2018-10-03T10:28:38Z","updated_at":"2018-10-16T01:18:23Z","closed_at":null,"author_association":"NONE","body":"#### Environment\r\n\r\n- OS: Ubuntu\r\n- Erlang/OTP: R20/9.0 \r\n- EMQ: 2.3.11\r\n\r\n#### Description\r\n\r\nI am observing that after a fan out load test, message routing is getting stuck for the topic used in load test. \r\nI am using a cluster of 5 c5.4xlarge machines, and the load test sets up 1M subscribers on a single topic and 1 publisher that publishes twice per minute for 5 minutes after all subscribers are set up.\r\n\r\nThe test runs well for the first time. I see 10M messages consumed. I saw mean latency of 6s. However, when I run the same test again, no messages are getting consumed. I confirmed this by doing a manual sub & pub on the same topic on one of the nodes using mosquitto_pub/mosquitto_sub. Pubsub on other topic works though. \r\n\r\nOn adding a trace on the topic, I can see `PUBLISH` logs. So, messages are reaching the node.\r\nOn adding a trace on the subscriber client, I can see `RECV SUBSCRIBE`, `SEND SUBACK`. But, no `SEND PUBLISH`. \r\n\r\nRunning a load test on a different topic works successfully (once). I have observed this issue on a single node also. On restarting the service, it had got resolved.\r\n\r\nWhat might be happening here? How can I debug this further?"},"comment":{"url":"https://api.github.com/repos/emqx/emqx/issues/comments/430067313","html_url":"https://github.com/emqx/emqx/issues/1880#issuecomment-430067313","issue_url":"https://api.github.com/repos/emqx/emqx/issues/1880","id":430067313,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMDA2NzMxMw==","user":{"login":"jinfahua","id":8246540,"node_id":"MDQ6VXNlcjgyNDY1NDA=","avatar_url":"https://avatars1.githubusercontent.com/u/8246540?v=4","gravatar_id":"","url":"https://api.github.com/users/jinfahua","html_url":"https://github.com/jinfahua","followers_url":"https://api.github.com/users/jinfahua/followers","following_url":"https://api.github.com/users/jinfahua/following{/other_user}","gists_url":"https://api.github.com/users/jinfahua/gists{/gist_id}","starred_url":"https://api.github.com/users/jinfahua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinfahua/subscriptions","organizations_url":"https://api.github.com/users/jinfahua/orgs","repos_url":"https://api.github.com/users/jinfahua/repos","events_url":"https://api.github.com/users/jinfahua/events{/privacy}","received_events_url":"https://api.github.com/users/jinfahua/received_events","type":"User","site_admin":false},"created_at":"2018-10-16T01:18:23Z","updated_at":"2018-10-16T01:18:23Z","author_association":"CONTRIBUTOR","body":"> Hi, can you provide your hardware information which would help us reproduce the problem you mentioned?\r\n\r\nI supplied you the hardware info..."}}
{"actor":{"display_login":"jinfahua"}}