{"action":"created","issue":{"url":"https://api.github.com/repos/CommBank/coppersmith/issues/54","repository_url":"https://api.github.com/repos/CommBank/coppersmith","labels_url":"https://api.github.com/repos/CommBank/coppersmith/issues/54/labels{/name}","comments_url":"https://api.github.com/repos/CommBank/coppersmith/issues/54/comments","events_url":"https://api.github.com/repos/CommBank/coppersmith/issues/54/events","html_url":"https://github.com/CommBank/coppersmith/pull/54","id":166726654,"number":54,"title":"Diagnostic counters","user":{"login":"toddmowen","id":370691,"avatar_url":"https://avatars.githubusercontent.com/u/370691?v=3","gravatar_id":"","url":"https://api.github.com/users/toddmowen","html_url":"https://github.com/toddmowen","followers_url":"https://api.github.com/users/toddmowen/followers","following_url":"https://api.github.com/users/toddmowen/following{/other_user}","gists_url":"https://api.github.com/users/toddmowen/gists{/gist_id}","starred_url":"https://api.github.com/users/toddmowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/toddmowen/subscriptions","organizations_url":"https://api.github.com/users/toddmowen/orgs","repos_url":"https://api.github.com/users/toddmowen/repos","events_url":"https://api.github.com/users/toddmowen/events{/privacy}","received_events_url":"https://api.github.com/users/toddmowen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-21T02:53:59Z","updated_at":"2016-08-25T00:52:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/CommBank/coppersmith/pulls/54","html_url":"https://github.com/CommBank/coppersmith/pull/54","diff_url":"https://github.com/CommBank/coppersmith/pull/54.diff","patch_url":"https://github.com/CommBank/coppersmith/pull/54.patch"},"body":"~~Just creating this PR to share with @SamRoberts.~~\r\n\r\nRead for review now.\r\n\r\nThis PR provides a new method `withCounter(name: String)` on any TypedPipe (available after importing the implicit conversion `CoppersmithStats.fromTypedPipe`). Code in the scalding subproject should call this whenever the number of records could change, e.g. after filter, join, etc. See the TROUBLESHOOTING file for an example of what the log output looks like.\r\n\r\nThe following workarounds should possibly be pushed upstream:\r\n- [ ] slf4j-test: loggers are not serializable\r\n- [ ] uniform: putting hive-exec last on the classpath helps avoid surprises"},"comment":{"url":"https://api.github.com/repos/CommBank/coppersmith/issues/comments/242252191","html_url":"https://github.com/CommBank/coppersmith/pull/54#issuecomment-242252191","issue_url":"https://api.github.com/repos/CommBank/coppersmith/issues/54","id":242252191,"user":{"login":"dkristian","id":551256,"avatar_url":"https://avatars.githubusercontent.com/u/551256?v=3","gravatar_id":"","url":"https://api.github.com/users/dkristian","html_url":"https://github.com/dkristian","followers_url":"https://api.github.com/users/dkristian/followers","following_url":"https://api.github.com/users/dkristian/following{/other_user}","gists_url":"https://api.github.com/users/dkristian/gists{/gist_id}","starred_url":"https://api.github.com/users/dkristian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dkristian/subscriptions","organizations_url":"https://api.github.com/users/dkristian/orgs","repos_url":"https://api.github.com/users/dkristian/repos","events_url":"https://api.github.com/users/dkristian/events{/privacy}","received_events_url":"https://api.github.com/users/dkristian/received_events","type":"User","site_admin":false},"created_at":"2016-08-25T00:52:59Z","updated_at":"2016-08-25T00:52:59Z","body":"Ok, I've merged this with the latest master, and got some counters being logged on DS. The load counts are exactly as expected, but the write count is 25% short (806811 actual, 615750 reported). This seems a little high to be within the apparent known shortcomings with scaldings counters to me. Happy to go through my methodology when you have time.\r\n\r\nBefore giving a thumbs up on the code, I'd be keen to see it merged with the latest master branch, as there have been some changes in the same areas. My attempt at rebasing master in order to test is at https://github.com/CommBank/coppersmith/compare/counters-latest-kd?expand=1"}}
{"actor":{"display_login":"dkristian"}}