{"action":"created","comment":{"url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/comments/228343252","pull_request_review_id":168597330,"id":228343252,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIyODM0MzI1Mg==","diff_hunk":"@@ -0,0 +1,144 @@\n+# GCP Ingestion Design\n+\n+This document specifies the design for GCP Ingestion as a whole.\n+\n+<!-- START doctoc generated TOC please keep comment here to allow auto update -->\n+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->\n+\n+\n+- [Architecture Diagram](#architecture-diagram)\n+- [Data Flow](#data-flow)\n+  - [Notes](#notes)\n+- [Design Decisions](#design-decisions)\n+  - [Kubernetes and PubSub](#kubernetes-and-pubsub)\n+  - [Different topics for \"raw\" and \"validated\"](#different-topics-for-raw-and-validated)\n+  - [BigQuery](#bigquery)\n+  - [Saving as JSON](#saving-as-json)\n+  - [Use destination tables?](#use-destination-tables)\n+  - [Use views for user-facing data?](#use-views-for-user-facing-data)\n+- [Known Issues](#known-issues)\n+\n+<!-- END doctoc generated TOC please keep comment here to allow auto update -->\n+\n+## Architecture Diagram\n+\n+![alt text](architecture.svg \"Architecture Diagram\")\n+\n+## Data Flow\n+\n+- The Kubernetes Edge service passes data from Producers (e.g. Firefox) to a\n+  PubSub Raw Topic.\n+   - The service will store data on disk when PubSub is unavailable.\n+      - New messages always attempt delivery to PubSub first.\n+      - Data on disk is retried periodically.\n+      - The service cannot be autoscaled down when there is data on disk.\n+   - The service will respond 500 if PubSub and disk space are both unavailable.\n+      - This is to be avoided because it has numerous negative side-effects,\n+        including delayed `submission_date` and usually some amount of data\n+loss.\n+   - The PubSub Topic for data is configurable based on `uri`.\n+   - Expected topics are Structured Ingestion, Telemetry, and Pioneer.\n+- The Dataflow Landfill Sink job copies data from a PubSub Raw Topic to Cloud\n+  Storage.\n+   - Data can be read from Cloud Storage for backfill, recovery, or testing.\n+- The Dataflow Decoder job decodes data from a PubSub Raw Topic to a PubSub\n+  Decoded Topic.\n+   - Decoding data applies these transforms in order:\n+      - Parse `uri` attribute into multiple attributes.\n+      - Gzip decompress `payload` if gzip compressed.\n+      - Validate `payload` using a JSON Schema determined by metadata fields.\n+      - Resolve GeoIP from `remote_addr` or `x_forwarded_for` attribute into\n+        `geo_*` attributes.\n+      - Parse `agent` attribute into `user_agent_*` attributes.\n+      - Copy attributes to `metadata` top level key in `payload`.\n+      - Remove duplicates based on `document_id` using Cloud Memorystore\n+        (Redis).\n+   - The job sends invalid data, including duplicates, to a configurable error\n+     output.\n+      - The error output can be a Cloud Storage path or a PubSub Topic.\n+- The Dataflow BigQuery Sink job copies data from a PubSub Decoded Topic to\n+  BigQuery.\n+   - The job also sends rejected data to a configurable error output.\n+      - The error output can be a Cloud Storage path or a PubSub Topic.\n+- The Dataflow Dataset Sink job copies data from a PubSub Decoded Topic to Cloud\n+  Storage.\n+   - BigQuery can load data to backfill columns previously unspecified in the\n+     schema.\n+   - BigQuery, Spark, and Dataflow can load data to execute ad-hoc queries.\n+- The Dataflow `DocType` Splitter job splits data from a PubSub Decoded Topic to\n+  multiple PubSub `DocType` Topics.\n+   - This job is only configured for requested `DocType`s.\n+\n+### Notes\n+\n+PubSub stores unacknowledged messages for 7 days. Any PubSub subscription more\n+than 7 days behind requires a backfill.\n+\n+Dataflow will extend ack deadlines indefinitely when consuming messages, and\n+will not ack messages until they are processed by an output or GroupByKey\n+transform.\n+\n+Dataflow jobs achieve at least once delivery by *not* using GroupByKey\n+transforms and *not* falling more than 7 days behind in processing.\n+\n+## Design Decisions\n+\n+### Kubernetes and PubSub\n+\n+They are simple, scalable, managed services. Using them as opposed to a compute","path":"docs/design.md","position":null,"original_position":88,"commit_id":"27c092d02d9cfad72812d071aa2b31b34c93c3b0","original_commit_id":"b68d029f4dedfa809bfd8b692a1b48e398411386","user":{"login":"relud","id":433717,"node_id":"MDQ6VXNlcjQzMzcxNw==","avatar_url":"https://avatars0.githubusercontent.com/u/433717?v=4","gravatar_id":"","url":"https://api.github.com/users/relud","html_url":"https://github.com/relud","followers_url":"https://api.github.com/users/relud/followers","following_url":"https://api.github.com/users/relud/following{/other_user}","gists_url":"https://api.github.com/users/relud/gists{/gist_id}","starred_url":"https://api.github.com/users/relud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/relud/subscriptions","organizations_url":"https://api.github.com/users/relud/orgs","repos_url":"https://api.github.com/users/relud/repos","events_url":"https://api.github.com/users/relud/events{/privacy}","received_events_url":"https://api.github.com/users/relud/received_events","type":"User","site_admin":false},"body":"I updated the wording here to describe PubSub and Kubernetes separately","created_at":"2018-10-25T21:37:11Z","updated_at":"2018-10-25T21:37:11Z","html_url":"https://github.com/mozilla/gcp-ingestion/pull/114#discussion_r228343252","pull_request_url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/comments/228343252"},"html":{"href":"https://github.com/mozilla/gcp-ingestion/pull/114#discussion_r228343252"},"pull_request":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114"}},"in_reply_to_id":227784853},"pull_request":{"url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114","id":224433503,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI0NDMzNTAz","html_url":"https://github.com/mozilla/gcp-ingestion/pull/114","diff_url":"https://github.com/mozilla/gcp-ingestion/pull/114.diff","patch_url":"https://github.com/mozilla/gcp-ingestion/pull/114.patch","issue_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/114","number":114,"state":"open","locked":false,"title":"Add design documentation for whole pipeline","user":{"login":"relud","id":433717,"node_id":"MDQ6VXNlcjQzMzcxNw==","avatar_url":"https://avatars0.githubusercontent.com/u/433717?v=4","gravatar_id":"","url":"https://api.github.com/users/relud","html_url":"https://github.com/relud","followers_url":"https://api.github.com/users/relud/followers","following_url":"https://api.github.com/users/relud/following{/other_user}","gists_url":"https://api.github.com/users/relud/gists{/gist_id}","starred_url":"https://api.github.com/users/relud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/relud/subscriptions","organizations_url":"https://api.github.com/users/relud/orgs","repos_url":"https://api.github.com/users/relud/repos","events_url":"https://api.github.com/users/relud/events{/privacy}","received_events_url":"https://api.github.com/users/relud/received_events","type":"User","site_admin":false},"body":"Mostly just copied information from the [GCP Architecture Proposal](https://docs.google.com/document/d/18CaLmI7lUYUC2pHsfCFM3TFBdzn-PPj4kpLPxhwon3A)\r\n\r\nI left out the `Workflows` section, because it was incomplete, but I have it in another branch.","created_at":"2018-10-19T23:25:45Z","updated_at":"2018-10-25T21:37:11Z","closed_at":null,"merged_at":null,"merge_commit_sha":"610a91cd3fe791a54a9c22279271224862a75b00","assignee":null,"assignees":[],"requested_reviewers":[{"login":"mreid-moz","id":969479,"node_id":"MDQ6VXNlcjk2OTQ3OQ==","avatar_url":"https://avatars2.githubusercontent.com/u/969479?v=4","gravatar_id":"","url":"https://api.github.com/users/mreid-moz","html_url":"https://github.com/mreid-moz","followers_url":"https://api.github.com/users/mreid-moz/followers","following_url":"https://api.github.com/users/mreid-moz/following{/other_user}","gists_url":"https://api.github.com/users/mreid-moz/gists{/gist_id}","starred_url":"https://api.github.com/users/mreid-moz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreid-moz/subscriptions","organizations_url":"https://api.github.com/users/mreid-moz/orgs","repos_url":"https://api.github.com/users/mreid-moz/repos","events_url":"https://api.github.com/users/mreid-moz/events{/privacy}","received_events_url":"https://api.github.com/users/mreid-moz/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114/commits","review_comments_url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114/comments","review_comment_url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/114/comments","statuses_url":"https://api.github.com/repos/mozilla/gcp-ingestion/statuses/27c092d02d9cfad72812d071aa2b31b34c93c3b0","head":{"label":"mozilla:design","ref":"design","sha":"27c092d02d9cfad72812d071aa2b31b34c93c3b0","user":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":143201223,"node_id":"MDEwOlJlcG9zaXRvcnkxNDMyMDEyMjM=","name":"gcp-ingestion","full_name":"mozilla/gcp-ingestion","private":false,"owner":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/mozilla/gcp-ingestion","description":"Documentation and implementation for google cloud ingestion pipeline","fork":false,"url":"https://api.github.com/repos/mozilla/gcp-ingestion","forks_url":"https://api.github.com/repos/mozilla/gcp-ingestion/forks","keys_url":"https://api.github.com/repos/mozilla/gcp-ingestion/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/gcp-ingestion/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/gcp-ingestion/teams","hooks_url":"https://api.github.com/repos/mozilla/gcp-ingestion/hooks","issue_events_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/gcp-ingestion/events","assignees_url":"https://api.github.com/repos/mozilla/gcp-ingestion/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/gcp-ingestion/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/gcp-ingestion/tags","blobs_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/gcp-ingestion/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/gcp-ingestion/languages","stargazers_url":"https://api.github.com/repos/mozilla/gcp-ingestion/stargazers","contributors_url":"https://api.github.com/repos/mozilla/gcp-ingestion/contributors","subscribers_url":"https://api.github.com/repos/mozilla/gcp-ingestion/subscribers","subscription_url":"https://api.github.com/repos/mozilla/gcp-ingestion/subscription","commits_url":"https://api.github.com/repos/mozilla/gcp-ingestion/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/gcp-ingestion/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/gcp-ingestion/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/gcp-ingestion/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/gcp-ingestion/merges","archive_url":"https://api.github.com/repos/mozilla/gcp-ingestion/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/gcp-ingestion/downloads","issues_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/gcp-ingestion/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/gcp-ingestion/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/gcp-ingestion/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/gcp-ingestion/releases{/id}","deployments_url":"https://api.github.com/repos/mozilla/gcp-ingestion/deployments","created_at":"2018-08-01T19:42:48Z","updated_at":"2018-10-25T18:04:48Z","pushed_at":"2018-10-25T21:37:09Z","git_url":"git://github.com/mozilla/gcp-ingestion.git","ssh_url":"git@github.com:mozilla/gcp-ingestion.git","clone_url":"https://github.com/mozilla/gcp-ingestion.git","svn_url":"https://github.com/mozilla/gcp-ingestion","homepage":null,"size":250,"stargazers_count":1,"watchers_count":1,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":25,"license":{"key":"mpl-2.0","name":"Mozilla Public License 2.0","spdx_id":"MPL-2.0","url":"https://api.github.com/licenses/mpl-2.0","node_id":"MDc6TGljZW5zZTE0"},"forks":1,"open_issues":25,"watchers":1,"default_branch":"master"}},"base":{"label":"mozilla:master","ref":"master","sha":"9d3a3dd782c1b5d9a81423f8ff563f4c4461f03a","user":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":143201223,"node_id":"MDEwOlJlcG9zaXRvcnkxNDMyMDEyMjM=","name":"gcp-ingestion","full_name":"mozilla/gcp-ingestion","private":false,"owner":{"login":"mozilla","id":131524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzMTUyNA==","avatar_url":"https://avatars2.githubusercontent.com/u/131524?v=4","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/mozilla/gcp-ingestion","description":"Documentation and implementation for google cloud ingestion pipeline","fork":false,"url":"https://api.github.com/repos/mozilla/gcp-ingestion","forks_url":"https://api.github.com/repos/mozilla/gcp-ingestion/forks","keys_url":"https://api.github.com/repos/mozilla/gcp-ingestion/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/gcp-ingestion/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/gcp-ingestion/teams","hooks_url":"https://api.github.com/repos/mozilla/gcp-ingestion/hooks","issue_events_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/gcp-ingestion/events","assignees_url":"https://api.github.com/repos/mozilla/gcp-ingestion/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/gcp-ingestion/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/gcp-ingestion/tags","blobs_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/gcp-ingestion/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/gcp-ingestion/languages","stargazers_url":"https://api.github.com/repos/mozilla/gcp-ingestion/stargazers","contributors_url":"https://api.github.com/repos/mozilla/gcp-ingestion/contributors","subscribers_url":"https://api.github.com/repos/mozilla/gcp-ingestion/subscribers","subscription_url":"https://api.github.com/repos/mozilla/gcp-ingestion/subscription","commits_url":"https://api.github.com/repos/mozilla/gcp-ingestion/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/gcp-ingestion/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/gcp-ingestion/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/gcp-ingestion/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/gcp-ingestion/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/gcp-ingestion/merges","archive_url":"https://api.github.com/repos/mozilla/gcp-ingestion/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/gcp-ingestion/downloads","issues_url":"https://api.github.com/repos/mozilla/gcp-ingestion/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/gcp-ingestion/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/gcp-ingestion/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/gcp-ingestion/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/gcp-ingestion/releases{/id}","deployments_url":"https://api.github.com/repos/mozilla/gcp-ingestion/deployments","created_at":"2018-08-01T19:42:48Z","updated_at":"2018-10-25T18:04:48Z","pushed_at":"2018-10-25T21:37:09Z","git_url":"git://github.com/mozilla/gcp-ingestion.git","ssh_url":"git@github.com:mozilla/gcp-ingestion.git","clone_url":"https://github.com/mozilla/gcp-ingestion.git","svn_url":"https://github.com/mozilla/gcp-ingestion","homepage":null,"size":250,"stargazers_count":1,"watchers_count":1,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":25,"license":{"key":"mpl-2.0","name":"Mozilla Public License 2.0","spdx_id":"MPL-2.0","url":"https://api.github.com/licenses/mpl-2.0","node_id":"MDc6TGljZW5zZTE0"},"forks":1,"open_issues":25,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114"},"html":{"href":"https://github.com/mozilla/gcp-ingestion/pull/114"},"issue":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/114"},"comments":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/issues/114/comments"},"review_comments":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114/comments"},"review_comment":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/pulls/114/commits"},"statuses":{"href":"https://api.github.com/repos/mozilla/gcp-ingestion/statuses/27c092d02d9cfad72812d071aa2b31b34c93c3b0"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"relud"}}