{"action":"created","issue":{"url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS/issues/244","repository_url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS","labels_url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS/issues/244/labels{/name}","comments_url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS/issues/244/comments","events_url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS/issues/244/events","html_url":"https://github.com/Particular/NServiceBus.AmazonSQS/issues/244","id":364354911,"node_id":"MDU6SXNzdWUzNjQzNTQ5MTE=","number":244,"title":"Use CloudWatch Events to manage delayed messages","user":{"login":"mauroservienti","id":1325611,"node_id":"MDQ6VXNlcjEzMjU2MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/1325611?v=4","gravatar_id":"","url":"https://api.github.com/users/mauroservienti","html_url":"https://github.com/mauroservienti","followers_url":"https://api.github.com/users/mauroservienti/followers","following_url":"https://api.github.com/users/mauroservienti/following{/other_user}","gists_url":"https://api.github.com/users/mauroservienti/gists{/gist_id}","starred_url":"https://api.github.com/users/mauroservienti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mauroservienti/subscriptions","organizations_url":"https://api.github.com/users/mauroservienti/orgs","repos_url":"https://api.github.com/users/mauroservienti/repos","events_url":"https://api.github.com/users/mauroservienti/events{/privacy}","received_events_url":"https://api.github.com/users/mauroservienti/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-27T08:15:15Z","updated_at":"2018-10-09T10:44:30Z","closed_at":null,"author_association":"MEMBER","body":"_spun off from #191_\r\n\r\nQuoted from https://github.com/Particular/NServiceBus.AmazonSQS/issues/191#issuecomment-424955637\r\n\r\nThe idea is to use CloudWatch Events to trigger the timeout - this allows an AWS native service to take ownership of the timing of the trigger without having a complex algorithm of repeatedly sending messages, nor requiring FIFO queues.\r\n\r\nA simple algorithm could be:\r\n\r\n- On Outgoing: When a timeout request is detected:\r\n   1. serialise the message and store to an S3 object (could use the existing large messages bucket)\r\n   1. Create a CloudWatch Event Rule with a cron expression for the requested Timeout time (see https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html) - the event rule should trigger an SQS message for the appropriate queue, and the parameter should be the message ID (or the s3 object uri)\r\n\r\n- On Incoming: When a Cloudwatch event message is detected on the queue:\r\n   1. delete the Cloudwatch Event Rule\r\n   1. Retrieve the message from S3 and use it as the timeout message that is passed through the rest of the NServiceBus pipeline\r\n\r\nThis would permit indefinite timeouts without FIFO queues, without satellite queues and without special algorithms to check and resend the message every 15 minutes."},"comment":{"url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS/issues/comments/428145774","html_url":"https://github.com/Particular/NServiceBus.AmazonSQS/issues/244#issuecomment-428145774","issue_url":"https://api.github.com/repos/Particular/NServiceBus.AmazonSQS/issues/244","id":428145774,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODE0NTc3NA==","user":{"login":"chrissimon-au","id":11002331,"node_id":"MDQ6VXNlcjExMDAyMzMx","avatar_url":"https://avatars1.githubusercontent.com/u/11002331?v=4","gravatar_id":"","url":"https://api.github.com/users/chrissimon-au","html_url":"https://github.com/chrissimon-au","followers_url":"https://api.github.com/users/chrissimon-au/followers","following_url":"https://api.github.com/users/chrissimon-au/following{/other_user}","gists_url":"https://api.github.com/users/chrissimon-au/gists{/gist_id}","starred_url":"https://api.github.com/users/chrissimon-au/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrissimon-au/subscriptions","organizations_url":"https://api.github.com/users/chrissimon-au/orgs","repos_url":"https://api.github.com/users/chrissimon-au/repos","events_url":"https://api.github.com/users/chrissimon-au/events{/privacy}","received_events_url":"https://api.github.com/users/chrissimon-au/received_events","type":"User","site_admin":false},"created_at":"2018-10-09T10:44:30Z","updated_at":"2018-10-09T10:44:30Z","author_association":"CONTRIBUTOR","body":"Hi @danielmarbach - thanks for posting, that's a great analysis!\r\n\r\nI can add some info that may or may not adjust your thinking:\r\n\r\n### 100-rule limit\r\n\r\nYou can request an increase on the rule limit - I have checked and in our region (ap-southeast-2) the hard limit is 2000 rules per account.  This still may be prohibitive for some use cases.\r\n\r\n### 1-minute resolution\r\n\r\nIn our case, this would be acceptable - as long as the SQS deferral (with more precise resolution I think?) was used for timeouts < 15 minutes.  Once the timeout is > 15 minutes we tend not to have second-resolution requirements.  So if the implementation could use native deferral if the timeout is < 15 minutes, and a cloudwatch event rule if > 15 minutes, that would be fine for us.\r\n\r\n### Rule management\r\n\r\nI agree - it would not be trivial, however I don't think _too_ complex either.  I think the key would be to consider each rule to be associated with a specific datetime (minute) rather than a specific timeout event.  Each rule can have up to 5 triggers associated with it, and each trigger can carry a payload with potentially multiple message Ids.  \r\n\r\nSo, as timeouts are raised, the algorithm could be:\r\n\r\n#### Outbound\r\n\r\n* Store the message to S3 keyed by the message id\r\n\r\n* If an existing rule exists for the requested time, add the message Id to that rule\r\n* if not, create a rule with the message id in the payload\r\n\r\nWhen adding a message id to a rule:\r\n* If an existing trigger is for the target queue, add to the payload (the `input` of the rule)\r\n* If not, create a new trigger for the target queue with the message id in the payload\r\n\r\nWhen adding a message id to a payload:\r\n* If the payload size would not exceed the limit, update the rule with the payload (I'm struggling to find documentation for the limit - I'll run some tests later and let you know - I suspect 256kB).\r\n* If the payload size would exceed the limit, add a new trigger with the message id\r\n\r\n(There may also need to be some concurrency handling to avoid losing message ids in a payload)\r\n\r\n#### Inbound\r\n\r\n* Retrieve the message from S3\r\n* Delete the rule\r\n\r\n### Variable resolution\r\n\r\nTo accommodate the rule limit, it may be acceptable as a compromise to scale the resolution - e.g. within 24 hours support 1 minute resolution, after 24 hours support 5 minute resolution, and after 48 hours, support 1 hour resolution - so only 24 rules would be required to support any given hour on a given day.\r\n\r\n### Why bother?\r\n\r\nOur main reason for being interested in this is that we can't use FIFO queues which are a requirement for the alternative algorithm which has already been implemented.  We are looking at hangfire for recurring activity, but we still have some use cases where nservicebus timeouts are a better fit, for non-recurring timeouts that exceed 15 minutes. \r\n\r\nI appreciate if our use case is not too common that there may not be enough appetite for this :)"}}
{"actor":{"display_login":"chrissimon-au"}}