{"action":"opened","issue":{"url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/issues/35","repository_url":"https://api.github.com/repos/mozilla/fxa-activity-metrics","labels_url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/issues/35/comments","events_url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/issues/35/events","html_url":"https://github.com/mozilla/fxa-activity-metrics/issues/35","id":191705776,"number":35,"title":"Duplicate events in the raw data","user":{"login":"philbooth","id":64367,"avatar_url":"https://avatars.githubusercontent.com/u/64367?v=3","gravatar_id":"","url":"https://api.github.com/users/philbooth","html_url":"https://github.com/philbooth","followers_url":"https://api.github.com/users/philbooth/followers","following_url":"https://api.github.com/users/philbooth/following{/other_user}","gists_url":"https://api.github.com/users/philbooth/gists{/gist_id}","starred_url":"https://api.github.com/users/philbooth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philbooth/subscriptions","organizations_url":"https://api.github.com/users/philbooth/orgs","repos_url":"https://api.github.com/users/philbooth/repos","events_url":"https://api.github.com/users/philbooth/events{/privacy}","received_events_url":"https://api.github.com/users/philbooth/received_events","type":"User","site_admin":false},"labels":[{"id":403748818,"url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/labels/waffle:active","name":"waffle:active","color":"ededed","default":false}],"state":"open","locked":false,"assignee":{"login":"philbooth","id":64367,"avatar_url":"https://avatars.githubusercontent.com/u/64367?v=3","gravatar_id":"","url":"https://api.github.com/users/philbooth","html_url":"https://github.com/philbooth","followers_url":"https://api.github.com/users/philbooth/followers","following_url":"https://api.github.com/users/philbooth/following{/other_user}","gists_url":"https://api.github.com/users/philbooth/gists{/gist_id}","starred_url":"https://api.github.com/users/philbooth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philbooth/subscriptions","organizations_url":"https://api.github.com/users/philbooth/orgs","repos_url":"https://api.github.com/users/philbooth/repos","events_url":"https://api.github.com/users/philbooth/events{/privacy}","received_events_url":"https://api.github.com/users/philbooth/received_events","type":"User","site_admin":false},"assignees":[{"login":"philbooth","id":64367,"avatar_url":"https://avatars.githubusercontent.com/u/64367?v=3","gravatar_id":"","url":"https://api.github.com/users/philbooth","html_url":"https://github.com/philbooth","followers_url":"https://api.github.com/users/philbooth/followers","following_url":"https://api.github.com/users/philbooth/following{/other_user}","gists_url":"https://api.github.com/users/philbooth/gists{/gist_id}","starred_url":"https://api.github.com/users/philbooth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philbooth/subscriptions","organizations_url":"https://api.github.com/users/philbooth/orgs","repos_url":"https://api.github.com/users/philbooth/repos","events_url":"https://api.github.com/users/philbooth/events{/privacy}","received_events_url":"https://api.github.com/users/philbooth/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/milestones/17","html_url":"https://github.com/mozilla/fxa-activity-metrics/milestones/FxA-41:signin%20funnel%20metrics","labels_url":"https://api.github.com/repos/mozilla/fxa-activity-metrics/milestones/17/labels","id":1933142,"number":17,"title":"FxA-41:signin funnel metrics","description":"","creator":{"login":"vladikoff","id":128755,"avatar_url":"https://avatars.githubusercontent.com/u/128755?v=3","gravatar_id":"","url":"https://api.github.com/users/vladikoff","html_url":"https://github.com/vladikoff","followers_url":"https://api.github.com/users/vladikoff/followers","following_url":"https://api.github.com/users/vladikoff/following{/other_user}","gists_url":"https://api.github.com/users/vladikoff/gists{/gist_id}","starred_url":"https://api.github.com/users/vladikoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vladikoff/subscriptions","organizations_url":"https://api.github.com/users/vladikoff/orgs","repos_url":"https://api.github.com/users/vladikoff/repos","events_url":"https://api.github.com/users/vladikoff/events{/privacy}","received_events_url":"https://api.github.com/users/vladikoff/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":6,"state":"open","created_at":"2016-08-09T17:52:53Z","updated_at":"2016-11-25T13:47:34Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2016-11-25T13:47:34Z","updated_at":"2016-11-25T13:47:34Z","closed_at":null,"body":"Related to but not the same as #33.\r\n\r\nThat issue concerns duplicates that don't exist in the raw data but are generated from the way we process events around the boundaries of each day. In debugging it, I've noticed a large number of duplicates that exist in the source data.\r\n\r\nThe [duplicate flow event query](https://sql.telemetry.mozilla.org/queries/1758/source) makes it easy to identify such cases. Some examples taken from the most recent dump:\r\n\r\n```\r\n$ grep 1480031993,email.verify_code.clicked,33461309c3023f800956f938b70e7393675b2d1d4296d9aeb0d355bfb19a459f flow-2016-11-24.csv\r\n1480031993,email.verify_code.clicked,33461309c3023f800956f938b70e7393675b2d1d4296d9aeb0d355bfb19a459f,3000060,Firefox,50,Android,fx_desktop_v3,uitour,,sync,page+referral+-+not+part+of+a+campaign,/firefox/sync/,,,\r\n1480031993,email.verify_code.clicked,33461309c3023f800956f938b70e7393675b2d1d4296d9aeb0d355bfb19a459f,3000054,Firefox,50,Android,fx_desktop_v3,uitour,,sync,page+referral+-+not+part+of+a+campaign,/firefox/sync/,,,\r\n```\r\n\r\n```\r\n$ grep flow.signup.begin,4fba07e2129b273c0be1d14eaeadd226811ad70c674c22495e26fef507771035 flow-2016-11-24.csv\r\n1480031891,flow.signup.begin,4fba07e2129b273c0be1d14eaeadd226811ad70c674c22495e26fef507771035,0,Firefox,28,FirefoxOS,oauth,,,0a42ac8a73be8762,,,,,\r\n1480031891,flow.signup.begin,4fba07e2129b273c0be1d14eaeadd226811ad70c674c22495e26fef507771035,0,Firefox,28,FirefoxOS,oauth,,,0a42ac8a73be8762,,,,,\r\n```\r\n\r\n```\r\n$ grep account.signed,cd66e207645f0b840866f74e5a71f221bef4c1a3372e615bb28a5274bb7e0095 flow-2016-11-24.csv\r\n1479989408,account.signed,cd66e207645f0b840866f74e5a71f221bef4c1a3372e615bb28a5274bb7e0095,61103,Firefox,50,Windows 10,fx_desktop_v3,preferences,,sync,,,,,\r\n1479989408,account.signed,cd66e207645f0b840866f74e5a71f221bef4c1a3372e615bb28a5274bb7e0095,61105,Firefox,50,Windows 10,fx_desktop_v3,preferences,,sync,,,,,\r\n```\r\n\r\nSo that is both content and auth server events, and it occurs throughout the day regardless of time."}}
{"actor":{"display_login":"philbooth"}}