{"action":"created","issue":{"url":"https://api.github.com/repos/twitter/summingbird/issues/651","labels_url":"https://api.github.com/repos/twitter/summingbird/issues/651/labels{/name}","comments_url":"https://api.github.com/repos/twitter/summingbird/issues/651/comments","events_url":"https://api.github.com/repos/twitter/summingbird/issues/651/events","html_url":"https://github.com/twitter/summingbird/issues/651","id":129625779,"number":651,"title":"Consuming the output of sumByKey","user":{"login":"johnynek","id":67958,"avatar_url":"https://avatars.githubusercontent.com/u/67958?v=3","gravatar_id":"","url":"https://api.github.com/users/johnynek","html_url":"https://github.com/johnynek","followers_url":"https://api.github.com/users/johnynek/followers","following_url":"https://api.github.com/users/johnynek/following{/other_user}","gists_url":"https://api.github.com/users/johnynek/gists{/gist_id}","starred_url":"https://api.github.com/users/johnynek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnynek/subscriptions","organizations_url":"https://api.github.com/users/johnynek/orgs","repos_url":"https://api.github.com/users/johnynek/repos","events_url":"https://api.github.com/users/johnynek/events{/privacy}","received_events_url":"https://api.github.com/users/johnynek/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-01-29T00:22:01Z","updated_at":"2016-01-29T01:08:27Z","closed_at":null,"body":"Now away from Twitter, where insane data rate might not be the problem, I'd like the ability to have stricter semantics on sumByKey.\r\n\r\nRight now, we don't guarantee that each value into a summer emits one output, indeed offline each key per batch emits at most one key value event.\r\n\r\nIt would be nice for that not to be the case. I'd like to see the full stream of outputs from a sumByKey, especially when used with a join.\r\n\r\nI guess that:\r\n\r\n1. Most users don't consume the output of sumByKey.\r\n2. Most users want the behavior I mention when they do consume the event stream.\r\n3. For the few users that don't want all updates, and want a cache-enabled view, we give an option to keep current behavior.\r\n\r\nIn the case that there is in downstream consumer (common case) we continue to do the map-side aggregation and the batched aggregation.\r\n\r\n*Impacts*: Twitter summingbird hadoop jobs would need to add an Option or may take linger to run in some cases.\r\n\r\nThoughts? @ianoc @isnotinvain @rubanm \r\n"},"comment":{"url":"https://api.github.com/repos/twitter/summingbird/issues/comments/176506823","html_url":"https://github.com/twitter/summingbird/issues/651#issuecomment-176506823","issue_url":"https://api.github.com/repos/twitter/summingbird/issues/651","id":176506823,"user":{"login":"johnynek","id":67958,"avatar_url":"https://avatars.githubusercontent.com/u/67958?v=3","gravatar_id":"","url":"https://api.github.com/users/johnynek","html_url":"https://github.com/johnynek","followers_url":"https://api.github.com/users/johnynek/followers","following_url":"https://api.github.com/users/johnynek/following{/other_user}","gists_url":"https://api.github.com/users/johnynek/gists{/gist_id}","starred_url":"https://api.github.com/users/johnynek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnynek/subscriptions","organizations_url":"https://api.github.com/users/johnynek/orgs","repos_url":"https://api.github.com/users/johnynek/repos","events_url":"https://api.github.com/users/johnynek/events{/privacy}","received_events_url":"https://api.github.com/users/johnynek/received_events","type":"User","site_admin":false},"created_at":"2016-01-29T01:08:27Z","updated_at":"2016-01-29T01:08:27Z","body":"Another idea is to put a node in the graph for local summing:\r\n```scala\r\n// maybe reduces the number of events by combining events into the later event.\r\n.sumByLocalKeys: Producer[P, (K, V)] \r\n```"}}