{"action":"created","issue":{"url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis/issues/86","repository_url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis","labels_url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis/issues/86/labels{/name}","comments_url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis/issues/86/comments","events_url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis/issues/86/events","html_url":"https://github.com/Terracotta-OSS/terracotta-apis/issues/86","id":155046641,"number":86,"title":"Experiment:  Deferred message acknowledgement","user":{"login":"jd0-sag","id":16048428,"avatar_url":"https://avatars.githubusercontent.com/u/16048428?v=3","gravatar_id":"","url":"https://api.github.com/users/jd0-sag","html_url":"https://github.com/jd0-sag","followers_url":"https://api.github.com/users/jd0-sag/followers","following_url":"https://api.github.com/users/jd0-sag/following{/other_user}","gists_url":"https://api.github.com/users/jd0-sag/gists{/gist_id}","starred_url":"https://api.github.com/users/jd0-sag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jd0-sag/subscriptions","organizations_url":"https://api.github.com/users/jd0-sag/orgs","repos_url":"https://api.github.com/users/jd0-sag/repos","events_url":"https://api.github.com/users/jd0-sag/events{/privacy}","received_events_url":"https://api.github.com/users/jd0-sag/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"jd0-sag","id":16048428,"avatar_url":"https://avatars.githubusercontent.com/u/16048428?v=3","gravatar_id":"","url":"https://api.github.com/users/jd0-sag","html_url":"https://github.com/jd0-sag","followers_url":"https://api.github.com/users/jd0-sag/followers","following_url":"https://api.github.com/users/jd0-sag/following{/other_user}","gists_url":"https://api.github.com/users/jd0-sag/gists{/gist_id}","starred_url":"https://api.github.com/users/jd0-sag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jd0-sag/subscriptions","organizations_url":"https://api.github.com/users/jd0-sag/orgs","repos_url":"https://api.github.com/users/jd0-sag/repos","events_url":"https://api.github.com/users/jd0-sag/events{/privacy}","received_events_url":"https://api.github.com/users/jd0-sag/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2016-05-16T14:53:57Z","updated_at":"2016-05-16T20:07:31Z","closed_at":null,"body":"While the experiment in #82 did a good job of describing how an active entity could re-interpret client-originating messages for its passive counterpart(s), it did add complexity to the passive entities, in the form of the write-ahead-log, and added substantial complexity to the server, in the form of re-sending internally-generated messages (required for the multiple passive case).\r\n\r\nUltimately, both of these problems were to address the issue that the client would not re-send the original logical operation, because it had already returned.  We can't block execution, within the server's request processing threads, so these approaches were used in order to \"fake up\" a kind of re-send behavior.\r\n\r\nThis experiment describes a different approach:  **deferred message acknowledgement**.\r\n\r\nThe general idea is that the existing client-server re-send support is invoked when a message from the client doesn't receive its COMPLETE acknowledgement (and return value) from the server before a reconnect event occurs (either restart or fail-over).  We already wait for the passive to ack to the active before the active will ack to the client so it should be possible to expand this delay, further.\r\n\r\nIdeally, this means requesting that the active server doesn't ack the logical operation to the client until all passives have acked the final sub-operation back to the active.  This still doesn't require blocking in the critical path, just changing the trigger for the original ack.\r\n\r\nThere are a few ways of exposing this:\r\n\r\n1. Create a new service which allows the active entity to directly handle the ack for the message (asks it to become manual - which gives the entity an ack token it can activate, in the future - then activates the ack token once the logical operation is complete)\r\n1. Extend the existing EntityMessenger service interface so that it is possible to tell it that the message it is sending should actually *inherit* the ack state of the original message\r\n\r\n(1) is very powerful (it would be possible to implement server-side synchronization primitives without any ClientCommunicator messaging or worrying about cross-network races) but has some concerns which make it a non-starter:\r\n\r\n- How would the ack token activation decision be made?  The sub-operation still completes without any way for the active to know this.  Additionally, it would need to be running at that time, but we want it to already have returned\r\n- This opens the door for creating cluster-wide deadlocks and other problems as a result of entity-level software bugs.  If an ack token is dropped, that client will never recover.\r\n\r\n(2) is more tightly constrained and describes something more literally in line with our specific intent, in this situation.  Explicitly stating that the ack to the client should happen in response to the ack for a specific active-generated message, instead of the currently executing message, means that all of the ack logic can still be handled by the platform, with no additional tracking in the entity.  The fact that the description of how the acks must be handled is still kept largely abstract means that the platform has a high degree of freedom in terms of how it interprets this requirement, allowing for freedom in future improvements.\r\n\r\nIn terms of the other requirements mentioned by #82:\r\n\r\n- Write-ahead-log:  This is no longer required.  It existed purely to \"fake up\" the data which couldn't be re-sent, on fail-over.  Now that the ack is being extended, the client will re-send the original message, directly.  This is superior to tracking the data in the write-ahead-log (same content but we no know who sent the message)\r\n- Re-send active-generated message for multiple passive case:  The re-send from the client will address this concern since all the passives will receive the original message, again.  Active-generated messages go back to their previous delivery rules where there is no re-send"},"comment":{"url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis/issues/comments/219532039","html_url":"https://github.com/Terracotta-OSS/terracotta-apis/issues/86#issuecomment-219532039","issue_url":"https://api.github.com/repos/Terracotta-OSS/terracotta-apis/issues/86","id":219532039,"user":{"login":"jd0-sag","id":16048428,"avatar_url":"https://avatars.githubusercontent.com/u/16048428?v=3","gravatar_id":"","url":"https://api.github.com/users/jd0-sag","html_url":"https://github.com/jd0-sag","followers_url":"https://api.github.com/users/jd0-sag/followers","following_url":"https://api.github.com/users/jd0-sag/following{/other_user}","gists_url":"https://api.github.com/users/jd0-sag/gists{/gist_id}","starred_url":"https://api.github.com/users/jd0-sag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jd0-sag/subscriptions","organizations_url":"https://api.github.com/users/jd0-sag/orgs","repos_url":"https://api.github.com/users/jd0-sag/repos","events_url":"https://api.github.com/users/jd0-sag/events{/privacy}","received_events_url":"https://api.github.com/users/jd0-sag/received_events","type":"User","site_admin":false},"created_at":"2016-05-16T20:07:31Z","updated_at":"2016-05-16T20:07:31Z","body":"Another concern which may come about from this approach is regarding the order of write-back to the client, on COMPLETE.\r\n\r\nCurrently, the COMPLETE for a sequence of invokes on the same concurrency key are all issued in the order they were executed.  Now that this final ack might be completed by a different invoke, that pushes it out of the strict order.  This could cause problems on re-send, where a sequence of re-sent invokes may now contain gaps.  This would violate our current statement that, although re-sends may cause the same invoke to be run more than once, it will always be run in the same relative order.  These gaps in the order would constitute a violation of this statement.\r\n\r\nForcing the messages awaiting COMPLETE into a sort of \"re-order buffer\" would address this problem but slow down the system as no client could receive the COMPLETE message until any of these deferred acknowledgement messages before it had completed.\r\n\r\nProposal (2), from above, may alleviate this latency in the relevant parts of this problem.  In this case, the COMPLETE messages could be returned, as soon as possible, but only the RETIRE acks would be put into the re-order buffer.  Most client-side messages which either don't return a value or wouldn't return a different value, on re-send, could only wait on the COMPLETE message.  This way, the common-case of the client-side calls wouldn't see this additional latency but it would allow the complete picture to be re-sent, on fail-over or restart."}}