{"action":"closed","issue":{"url":"https://api.github.com/repos/openwhisk/openwhisk/issues/993","repository_url":"https://api.github.com/repos/openwhisk/openwhisk","labels_url":"https://api.github.com/repos/openwhisk/openwhisk/issues/993/labels{/name}","comments_url":"https://api.github.com/repos/openwhisk/openwhisk/issues/993/comments","events_url":"https://api.github.com/repos/openwhisk/openwhisk/issues/993/events","html_url":"https://github.com/openwhisk/openwhisk/issues/993","id":168961465,"number":993,"title":"Move concurrent activation information from consul to controller","user":{"login":"perryibm","id":14339749,"avatar_url":"https://avatars.githubusercontent.com/u/14339749?v=3","gravatar_id":"","url":"https://api.github.com/users/perryibm","html_url":"https://github.com/perryibm","followers_url":"https://api.github.com/users/perryibm/followers","following_url":"https://api.github.com/users/perryibm/following{/other_user}","gists_url":"https://api.github.com/users/perryibm/gists{/gist_id}","starred_url":"https://api.github.com/users/perryibm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perryibm/subscriptions","organizations_url":"https://api.github.com/users/perryibm/orgs","repos_url":"https://api.github.com/users/perryibm/repos","events_url":"https://api.github.com/users/perryibm/events{/privacy}","received_events_url":"https://api.github.com/users/perryibm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openwhisk/openwhisk/labels/controller","name":"controller","color":"e99695"},{"url":"https://api.github.com/repos/openwhisk/openwhisk/labels/invoker","name":"invoker","color":"0e8a16"},{"url":"https://api.github.com/repos/openwhisk/openwhisk/labels/monitoring","name":"monitoring","color":"e99695"}],"state":"closed","locked":false,"assignee":{"login":"perryibm","id":14339749,"avatar_url":"https://avatars.githubusercontent.com/u/14339749?v=3","gravatar_id":"","url":"https://api.github.com/users/perryibm","html_url":"https://github.com/perryibm","followers_url":"https://api.github.com/users/perryibm/followers","following_url":"https://api.github.com/users/perryibm/following{/other_user}","gists_url":"https://api.github.com/users/perryibm/gists{/gist_id}","starred_url":"https://api.github.com/users/perryibm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perryibm/subscriptions","organizations_url":"https://api.github.com/users/perryibm/orgs","repos_url":"https://api.github.com/users/perryibm/repos","events_url":"https://api.github.com/users/perryibm/events{/privacy}","received_events_url":"https://api.github.com/users/perryibm/received_events","type":"User","site_admin":false},"assignees":[{"login":"perryibm","id":14339749,"avatar_url":"https://avatars.githubusercontent.com/u/14339749?v=3","gravatar_id":"","url":"https://api.github.com/users/perryibm","html_url":"https://github.com/perryibm","followers_url":"https://api.github.com/users/perryibm/followers","following_url":"https://api.github.com/users/perryibm/following{/other_user}","gists_url":"https://api.github.com/users/perryibm/gists{/gist_id}","starred_url":"https://api.github.com/users/perryibm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perryibm/subscriptions","organizations_url":"https://api.github.com/users/perryibm/orgs","repos_url":"https://api.github.com/users/perryibm/repos","events_url":"https://api.github.com/users/perryibm/events{/privacy}","received_events_url":"https://api.github.com/users/perryibm/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-08-02T18:57:30Z","updated_at":"2016-09-27T22:28:27Z","closed_at":"2016-09-27T22:28:27Z","body":"Currently, the controller writes out the number of completed activations (per use) and so does every invoker.  This creates a lot of continuous traffic on consul and I believe can cause latency issues.  Now that active ack is in place, the invoker can stream back completion as they occur instead of writing a large piece of information for the state.  This has the benefit of reducing traffic, making consul healthier, and centralizing information in the load balancer.\r\n\r\n@jeremiaswerner As discussed before, we want to reduce consul traffic.  The proposal that @rabbah and I arrived at is the following.   @ChrBi @markusthoemmes  Any thoughts?\r\n\r\n1. The controller extends the outstanding activation map to include non-blocking as well as blocking calls.  Both types are already part of the active ack from the invoker so no work needed there.\r\n2. A secondary map based on user id can then be maintained in the controller giving the number of concurrent activations.  This also optimizes the responsiveness of the throttle because the load balancer immediately has information to this rather than enduring consul latency.\r\n3. The N invokers no longer writes any such information to consul - thus greatly decreasing the load.  The controller only writes the digested information (queued activations per user) to consul.  This also improves monitoring.\r\n4. As a future option, centralizing the notion in the controller also allows the creation of an admin endpoint that would allow manual un-wedging of a user.\r\n5. RAS:  This is the hard part.  What if the invoker's active ack fails?  From the controller's perspective, an activation was issued but it did not hear back.  This will be rare but we must be robust against this.  Also we must consider the case when the invoker is restarted.\r\n\r\n- Initial idea.  Use the global action timeout (or some factor thereof) as a proxy for when the invocation has already been completed.   This is no worse than the existing system in that we could fail to write to consul and have the same issue.  However, repeated lost acks can cause unbounded queue growth so we don't like this.\r\n\r\n- New idea.  Use a revised version of the original protocol.  We extend the active ack with a count for actions completed for that user as well as an indication of invoker restart by adding the timestamp for when the invoker started.  Since there is a count reported, a few active ack will not affect the cumulative count.  If the timestamp varies indicating a reboot, we reset the count for that user+invoker pair.\r\n\r\n"}}
{"actor":{"display_login":"perryibm"}}