{"action":"created","issue":{"url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/issues/35","repository_url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq","labels_url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/issues/35/comments","events_url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/issues/35/events","html_url":"https://github.com/Particular/NServiceBus.Distributor.Msmq/pull/35","id":131722431,"number":35,"title":"[WIP] Hotfix 5.0.4","user":{"login":"ramonsmits","id":152998,"avatar_url":"https://avatars.githubusercontent.com/u/152998?v=3","gravatar_id":"","url":"https://api.github.com/users/ramonsmits","html_url":"https://github.com/ramonsmits","followers_url":"https://api.github.com/users/ramonsmits/followers","following_url":"https://api.github.com/users/ramonsmits/following{/other_user}","gists_url":"https://api.github.com/users/ramonsmits/gists{/gist_id}","starred_url":"https://api.github.com/users/ramonsmits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ramonsmits/subscriptions","organizations_url":"https://api.github.com/users/ramonsmits/orgs","repos_url":"https://api.github.com/users/ramonsmits/repos","events_url":"https://api.github.com/users/ramonsmits/events{/privacy}","received_events_url":"https://api.github.com/users/ramonsmits/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/labels/State:%20In%20Progress","name":"State: In Progress","color":"ededed"},{"url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/labels/Type:%20Bug","name":"Type: Bug","color":"e10c02"}],"state":"open","locked":false,"assignee":{"login":"janovesk","id":396480,"avatar_url":"https://avatars.githubusercontent.com/u/396480?v=3","gravatar_id":"","url":"https://api.github.com/users/janovesk","html_url":"https://github.com/janovesk","followers_url":"https://api.github.com/users/janovesk/followers","following_url":"https://api.github.com/users/janovesk/following{/other_user}","gists_url":"https://api.github.com/users/janovesk/gists{/gist_id}","starred_url":"https://api.github.com/users/janovesk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janovesk/subscriptions","organizations_url":"https://api.github.com/users/janovesk/orgs","repos_url":"https://api.github.com/users/janovesk/repos","events_url":"https://api.github.com/users/janovesk/events{/privacy}","received_events_url":"https://api.github.com/users/janovesk/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/milestones/10","html_url":"https://github.com/Particular/NServiceBus.Distributor.Msmq/milestones/5.0.4","labels_url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/milestones/10/labels","id":1566367,"number":10,"title":"5.0.4","description":null,"creator":{"login":"ramonsmits","id":152998,"avatar_url":"https://avatars.githubusercontent.com/u/152998?v=3","gravatar_id":"","url":"https://api.github.com/users/ramonsmits","html_url":"https://github.com/ramonsmits","followers_url":"https://api.github.com/users/ramonsmits/followers","following_url":"https://api.github.com/users/ramonsmits/following{/other_user}","gists_url":"https://api.github.com/users/ramonsmits/gists{/gist_id}","starred_url":"https://api.github.com/users/ramonsmits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ramonsmits/subscriptions","organizations_url":"https://api.github.com/users/ramonsmits/orgs","repos_url":"https://api.github.com/users/ramonsmits/repos","events_url":"https://api.github.com/users/ramonsmits/events{/privacy}","received_events_url":"https://api.github.com/users/ramonsmits/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":0,"state":"open","created_at":"2016-02-05T18:18:08Z","updated_at":"2016-02-05T18:44:27Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-02-05T18:44:27Z","updated_at":"2016-02-18T16:02:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/pulls/35","html_url":"https://github.com/Particular/NServiceBus.Distributor.Msmq/pull/35","diff_url":"https://github.com/Particular/NServiceBus.Distributor.Msmq/pull/35.diff","patch_url":"https://github.com/Particular/NServiceBus.Distributor.Msmq/pull/35.patch"},"body":"This fixes #34\r\n\r\n## Behavior\r\n\r\n### Distributor restart when worker is under load\r\n\r\nAssuming worker(s) are setup correctly.\r\n\r\n- After a restart the distributor receives RDY message and registers this in the storage queue with the session ID from the RDY message.\r\n- Incoming business message will be forwarded to worker with session ID from storage message.\r\n- Worker processes message\r\n- Worker sends RDY message to distributor if it has matching session ID\r\n\r\n### Worker crashes when under full load\r\n\r\n- Worker queue has # message equal to concurrency limit\r\n- Worker sends WorkerCapacityAvailable with concurrency limit + session ID\r\n- Distributor writes RDY message to storage queues based on this limit\r\n- Distributor sends new messages to worker as it thinks it is not processing any messages\r\n- Worker has now also a backlog of messages equal to the concurrency limit\r\n- After worker finished processing messages from its backlog it only send RDY messages with a matching session ID back to the distributor.\r\n\r\n## Drawback\r\n\r\nIf the workers continiously crashes, its queue will be filled with message as it is deemed available by the distributor.\r\n\r\nThe only way to solve this is:\r\n\r\n- When distributor receives WorkerCapacityAvailable, it sends an ACK+SessionID to the worker\r\n- When worker receives an ACK it sends RDY messages if the SessionID matches\r\n\r\nThis works, as the backlog of existing messages will now have been processed. I do assume here that messages are processed chronologycally.\r\n\r\nWe could even filter 'old' items from the storage queue before sending the ACK. This makes sure that no new work will be send to the worker until it send all its RDY messages based on the ACK.\r\n"},"comment":{"url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/issues/comments/185791459","html_url":"https://github.com/Particular/NServiceBus.Distributor.Msmq/pull/35#issuecomment-185791459","issue_url":"https://api.github.com/repos/Particular/NServiceBus.Distributor.Msmq/issues/35","id":185791459,"user":{"login":"ramonsmits","id":152998,"avatar_url":"https://avatars.githubusercontent.com/u/152998?v=3","gravatar_id":"","url":"https://api.github.com/users/ramonsmits","html_url":"https://github.com/ramonsmits","followers_url":"https://api.github.com/users/ramonsmits/followers","following_url":"https://api.github.com/users/ramonsmits/following{/other_user}","gists_url":"https://api.github.com/users/ramonsmits/gists{/gist_id}","starred_url":"https://api.github.com/users/ramonsmits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ramonsmits/subscriptions","organizations_url":"https://api.github.com/users/ramonsmits/orgs","repos_url":"https://api.github.com/users/ramonsmits/repos","events_url":"https://api.github.com/users/ramonsmits/events{/privacy}","received_events_url":"https://api.github.com/users/ramonsmits/received_events","type":"User","site_admin":false},"created_at":"2016-02-18T16:02:30Z","updated_at":"2016-02-18T16:02:30Z","body":"I tested this locally and in a real (virtualized) cluster. Doing graceful shutdowns of both worker and distributor processes.\r\n\r\nWhen the worker does a graceful shutdown it will always handle the current messages in the queue which is equal to the number of threads.\r\n\r\nQuickly restarting the worker will never result in the backlog to be larger than the thread count.\r\n\r\nKilling the worker and then starting it again (recovery) resulting in the backlog to grow but if the worker is running stable when it is back to the thread count it stays that way.\r\n\r\nCluster roll-over works while workers are busy. Workers are not disconnected. Here distributor process is gracefully shutting down.\r\n\r\nCluster fail-over works while workers are busy. Workers are not disconnected. Here the node is just switched off, waiting for windows cluster service to promote the secondary node.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n"}}