{"action":"created","comment":{"url":"https://api.github.com/repos/improbable-eng/thanos/pulls/comments/242255152","pull_request_review_id":185713213,"id":242255152,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MjI1NTE1Mg==","diff_hunk":"@@ -0,0 +1,174 @@\n+## Thanos Remote Write\n+\n+Status: draft | *in-review* | rejected | accepted | complete\n+\n+Implementation Owner: @brancz\n+\n+## Summary\n+\n+This document describes the motivation and design of the Thanos receiver component, as well as how it fits into the rest of the Thanos ecosystem and components.\n+\n+## Motivation\n+\n+The Thanos receiver is the missing piece within Thanos in order to use it to build Prometheus as a Service offering, either as an internal service to the rest of an organization or as an actual pay-as-you-go off the shelf service. It builds on top of existing Prometheus servers and retains their usefulness, while extending their functionality with long-term-storage, horizontal scalability and downsampling. The normal Thanos sidecar is not sufficient for this, as the system would always lack the block length behind (typically 2 hours), which would prevent the most common query pattern of Prometheus: real time queries of very recent data. With the Thanos receiver on the service provider side users can simply deploy vanilla Prometheus servers and configure Prometheus with built-in functionality to send its data to the service provider.\n+\n+This component is only recommended for uses, where pushing is the only viable solution, for example analytics use cases or cases where the data ingestion must be client initiated, for example a software as a service type environments.\n+\n+This component is not recommended in order to achieve a global view of data of a single tenant, for those cases the sidecar based approach with layered Thanos queriers is recommended. Also users are asked to note the [various pros and cons of pushing metrics](https://docs.google.com/document/d/1H47v7WfyKkSLMrR8_iku6u9VB73WrVzBHb2SB6dL9_g/edit#heading=h.2v27snv0lsur).\n+\n+## Technical summary of Thanos receiver component\n+\n+To solve the above mentioned problems, a new Thanos component is proposed: the Thanos receiver.\n+\n+Prometheus has the remote write API to send samples collected by a Prometheus server to a remote location. This is essentially replicating the Prometheus write-ahead-log to that remote location. The Thanos receiver represents the remote location that accepts the Prometheus remote write API. This brings some interesting characteristics with it. It retains the metadata aspects that has contributed to a large part to Prometheus' success, while the remote location can add additional features to the stack that are not present (and arguably shouldn't be) in Prometheus, such as long term storage and horizontal scalability, while keeping the local Prometheus useful for reliable alerting on short term data.\n+\n+## Architecture\n+\n+The Thanos receiver component seamlessly integrates into the rest of the Thanos components. It acts similarly to what is referred to in Thanos as a \"source\", in the current set of components, this is typically represented by the Thanos sidecar that is put next to Prometheus to ship tsdb blocks into object storage and reply to store API requests, however in the case of the Thanos receiver, the Thanos sidecar is not necessary anymore, as the data is replicated from the original Prometheus server to the Thanos receiver, and the Thanos receiver participates in the Thanos gossip mesh. The Prometheus server on a tenant’s infrastructure can therefore be completely vanilla and is just configured to replicate its time-series to the Thanos receiver.\n+\n+Instead of directly scraping metrics, however, the Thanos receiver accepts Prometheus remote write requests, and writes these into a local instance of the Prometheus tsdb, to prevent data leaking at the database level, each tenant has an individual tsdb instance, meaning a single Thanos receiver may manage multiple tsdb instances. When the receiver is queried, the respective tsdb is used to retrieve data. Furthermore, like the Thanos sidecar, the receiver answers Thanos store API requests and uploads built blocks of the Prometheus tsdb. Implementation-wise, this just requires wiring up existing components.","path":"docs/proposals/201812_thanos-remote-receive.md","position":29,"original_position":29,"commit_id":"dc3b7823dce4a2fa2875913a1944f3289d2a7da9","original_commit_id":"dc3b7823dce4a2fa2875913a1944f3289d2a7da9","user":{"login":"bwplotka","id":6950331,"node_id":"MDQ6VXNlcjY5NTAzMzE=","avatar_url":"https://avatars2.githubusercontent.com/u/6950331?v=4","gravatar_id":"","url":"https://api.github.com/users/bwplotka","html_url":"https://github.com/bwplotka","followers_url":"https://api.github.com/users/bwplotka/followers","following_url":"https://api.github.com/users/bwplotka/following{/other_user}","gists_url":"https://api.github.com/users/bwplotka/gists{/gist_id}","starred_url":"https://api.github.com/users/bwplotka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bwplotka/subscriptions","organizations_url":"https://api.github.com/users/bwplotka/orgs","repos_url":"https://api.github.com/users/bwplotka/repos","events_url":"https://api.github.com/users/bwplotka/events{/privacy}","received_events_url":"https://api.github.com/users/bwplotka/received_events","type":"User","site_admin":false},"body":"> When the receiver is queried, the respective tsdb is used to retrieve data. Furthermore, like the Thanos sidecar, the receiver answers Thanos store API requests and uploads built blocks of the Prometheus tsdb. Implementation-wise, this just requires wiring up existing components.\r\n\r\nI think the \" Furthermore, like the Thanos sidecar, the receiver answers Thanos store API requests\" overlaps with  \"When the receiver is queried, the respective tsdb is used to retrieve data.\" bit?","created_at":"2018-12-17T18:04:25Z","updated_at":"2018-12-17T18:20:51Z","html_url":"https://github.com/improbable-eng/thanos/pull/659#discussion_r242255152","pull_request_url":"https://api.github.com/repos/improbable-eng/thanos/pulls/659","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/improbable-eng/thanos/pulls/comments/242255152"},"html":{"href":"https://github.com/improbable-eng/thanos/pull/659#discussion_r242255152"},"pull_request":{"href":"https://api.github.com/repos/improbable-eng/thanos/pulls/659"}},"in_reply_to_id":242246581},"pull_request":{"url":"https://api.github.com/repos/improbable-eng/thanos/pulls/659","id":235817275,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM1ODE3Mjc1","html_url":"https://github.com/improbable-eng/thanos/pull/659","diff_url":"https://github.com/improbable-eng/thanos/pull/659.diff","patch_url":"https://github.com/improbable-eng/thanos/pull/659.patch","issue_url":"https://api.github.com/repos/improbable-eng/thanos/issues/659","number":659,"state":"open","locked":false,"title":"proposals: Add proposal for a component for remote-write support","user":{"login":"brancz","id":4546722,"node_id":"MDQ6VXNlcjQ1NDY3MjI=","avatar_url":"https://avatars3.githubusercontent.com/u/4546722?v=4","gravatar_id":"","url":"https://api.github.com/users/brancz","html_url":"https://github.com/brancz","followers_url":"https://api.github.com/users/brancz/followers","following_url":"https://api.github.com/users/brancz/following{/other_user}","gists_url":"https://api.github.com/users/brancz/gists{/gist_id}","starred_url":"https://api.github.com/users/brancz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brancz/subscriptions","organizations_url":"https://api.github.com/users/brancz/orgs","repos_url":"https://api.github.com/users/brancz/repos","events_url":"https://api.github.com/users/brancz/events{/privacy}","received_events_url":"https://api.github.com/users/brancz/received_events","type":"User","site_admin":false},"body":"This adds a proposal for a new component to be added to Thanos to enable Prometheus servers replicate their time-series data via Prometheus' remote-write support. Take the motivation for this from the proposal.\r\n\r\nOur concrete case is for analytics where customers have tight insight over what data is sent and it must be initiated by them, instead of us using something like [PushProx](https://github.com/RobustPerception/PushProx).\r\n\r\nThe RedHat Prometheus/monitoring team intends to do maintainership of this component as well as it's initial implementation. A working, but proof of concept state version of the proposal can be found [here](https://github.com/Go-SIP/thanos-receiver). Note this proof-of-concept, only has a minimal feature set working, to demonstrate the concept. Also the proposal mentions remote write via the write-ahead-log a number of times, which has not been merged [yet](https://github.com/prometheus/prometheus/pull/4588).\r\n\r\nThis proposal is the result of a few weeks of discussions with various Prometheus team members as well as members of the [Cortex](https://github.com/cortexproject/cortex) team. As this particular proposal has a lot in common with Cortex, we will look into re-using Cortex code as we see fit.\r\n\r\nWhen added we will want to make sure, that the documentation is clear on whether users need this or are fine with the current way of using Thanos.\r\n\r\n@bwplotka @domgreen @fabxc @tomwilkie @gouthamve @bbrazil @s-urbaniak @mxinden @squat @metalmatze @krasi-georgiev @simonpasquier ","created_at":"2018-12-04T13:54:30Z","updated_at":"2018-12-17T18:20:51Z","closed_at":null,"merged_at":null,"merge_commit_sha":"caa6c4425ff63f114b1c69d234e4c359a9cca720","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/improbable-eng/thanos/pulls/659/commits","review_comments_url":"https://api.github.com/repos/improbable-eng/thanos/pulls/659/comments","review_comment_url":"https://api.github.com/repos/improbable-eng/thanos/pulls/comments{/number}","comments_url":"https://api.github.com/repos/improbable-eng/thanos/issues/659/comments","statuses_url":"https://api.github.com/repos/improbable-eng/thanos/statuses/dc3b7823dce4a2fa2875913a1944f3289d2a7da9","head":{"label":"brancz:remote-write-proposal","ref":"remote-write-proposal","sha":"dc3b7823dce4a2fa2875913a1944f3289d2a7da9","user":{"login":"brancz","id":4546722,"node_id":"MDQ6VXNlcjQ1NDY3MjI=","avatar_url":"https://avatars3.githubusercontent.com/u/4546722?v=4","gravatar_id":"","url":"https://api.github.com/users/brancz","html_url":"https://github.com/brancz","followers_url":"https://api.github.com/users/brancz/followers","following_url":"https://api.github.com/users/brancz/following{/other_user}","gists_url":"https://api.github.com/users/brancz/gists{/gist_id}","starred_url":"https://api.github.com/users/brancz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brancz/subscriptions","organizations_url":"https://api.github.com/users/brancz/orgs","repos_url":"https://api.github.com/users/brancz/repos","events_url":"https://api.github.com/users/brancz/events{/privacy}","received_events_url":"https://api.github.com/users/brancz/received_events","type":"User","site_admin":false},"repo":{"id":130072249,"node_id":"MDEwOlJlcG9zaXRvcnkxMzAwNzIyNDk=","name":"thanos","full_name":"brancz/thanos","private":false,"owner":{"login":"brancz","id":4546722,"node_id":"MDQ6VXNlcjQ1NDY3MjI=","avatar_url":"https://avatars3.githubusercontent.com/u/4546722?v=4","gravatar_id":"","url":"https://api.github.com/users/brancz","html_url":"https://github.com/brancz","followers_url":"https://api.github.com/users/brancz/followers","following_url":"https://api.github.com/users/brancz/following{/other_user}","gists_url":"https://api.github.com/users/brancz/gists{/gist_id}","starred_url":"https://api.github.com/users/brancz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brancz/subscriptions","organizations_url":"https://api.github.com/users/brancz/orgs","repos_url":"https://api.github.com/users/brancz/repos","events_url":"https://api.github.com/users/brancz/events{/privacy}","received_events_url":"https://api.github.com/users/brancz/received_events","type":"User","site_admin":false},"html_url":"https://github.com/brancz/thanos","description":"Highly available Prometheus setup with long term storage capabilities.","fork":true,"url":"https://api.github.com/repos/brancz/thanos","forks_url":"https://api.github.com/repos/brancz/thanos/forks","keys_url":"https://api.github.com/repos/brancz/thanos/keys{/key_id}","collaborators_url":"https://api.github.com/repos/brancz/thanos/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/brancz/thanos/teams","hooks_url":"https://api.github.com/repos/brancz/thanos/hooks","issue_events_url":"https://api.github.com/repos/brancz/thanos/issues/events{/number}","events_url":"https://api.github.com/repos/brancz/thanos/events","assignees_url":"https://api.github.com/repos/brancz/thanos/assignees{/user}","branches_url":"https://api.github.com/repos/brancz/thanos/branches{/branch}","tags_url":"https://api.github.com/repos/brancz/thanos/tags","blobs_url":"https://api.github.com/repos/brancz/thanos/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/brancz/thanos/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/brancz/thanos/git/refs{/sha}","trees_url":"https://api.github.com/repos/brancz/thanos/git/trees{/sha}","statuses_url":"https://api.github.com/repos/brancz/thanos/statuses/{sha}","languages_url":"https://api.github.com/repos/brancz/thanos/languages","stargazers_url":"https://api.github.com/repos/brancz/thanos/stargazers","contributors_url":"https://api.github.com/repos/brancz/thanos/contributors","subscribers_url":"https://api.github.com/repos/brancz/thanos/subscribers","subscription_url":"https://api.github.com/repos/brancz/thanos/subscription","commits_url":"https://api.github.com/repos/brancz/thanos/commits{/sha}","git_commits_url":"https://api.github.com/repos/brancz/thanos/git/commits{/sha}","comments_url":"https://api.github.com/repos/brancz/thanos/comments{/number}","issue_comment_url":"https://api.github.com/repos/brancz/thanos/issues/comments{/number}","contents_url":"https://api.github.com/repos/brancz/thanos/contents/{+path}","compare_url":"https://api.github.com/repos/brancz/thanos/compare/{base}...{head}","merges_url":"https://api.github.com/repos/brancz/thanos/merges","archive_url":"https://api.github.com/repos/brancz/thanos/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/brancz/thanos/downloads","issues_url":"https://api.github.com/repos/brancz/thanos/issues{/number}","pulls_url":"https://api.github.com/repos/brancz/thanos/pulls{/number}","milestones_url":"https://api.github.com/repos/brancz/thanos/milestones{/number}","notifications_url":"https://api.github.com/repos/brancz/thanos/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/brancz/thanos/labels{/name}","releases_url":"https://api.github.com/repos/brancz/thanos/releases{/id}","deployments_url":"https://api.github.com/repos/brancz/thanos/deployments","created_at":"2018-04-18T14:08:52Z","updated_at":"2018-04-18T14:08:55Z","pushed_at":"2018-12-15T03:36:29Z","git_url":"git://github.com/brancz/thanos.git","ssh_url":"git@github.com:brancz/thanos.git","clone_url":"https://github.com/brancz/thanos.git","svn_url":"https://github.com/brancz/thanos","homepage":"","size":46125,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"improbable-eng:master","ref":"master","sha":"ee025a7d438de8da44401c19b1603ffdd5ba870d","user":{"login":"improbable-eng","id":26084283,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MDg0Mjgz","avatar_url":"https://avatars2.githubusercontent.com/u/26084283?v=4","gravatar_id":"","url":"https://api.github.com/users/improbable-eng","html_url":"https://github.com/improbable-eng","followers_url":"https://api.github.com/users/improbable-eng/followers","following_url":"https://api.github.com/users/improbable-eng/following{/other_user}","gists_url":"https://api.github.com/users/improbable-eng/gists{/gist_id}","starred_url":"https://api.github.com/users/improbable-eng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/improbable-eng/subscriptions","organizations_url":"https://api.github.com/users/improbable-eng/orgs","repos_url":"https://api.github.com/users/improbable-eng/repos","events_url":"https://api.github.com/users/improbable-eng/events{/privacy}","received_events_url":"https://api.github.com/users/improbable-eng/received_events","type":"Organization","site_admin":false},"repo":{"id":109162639,"node_id":"MDEwOlJlcG9zaXRvcnkxMDkxNjI2Mzk=","name":"thanos","full_name":"improbable-eng/thanos","private":false,"owner":{"login":"improbable-eng","id":26084283,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MDg0Mjgz","avatar_url":"https://avatars2.githubusercontent.com/u/26084283?v=4","gravatar_id":"","url":"https://api.github.com/users/improbable-eng","html_url":"https://github.com/improbable-eng","followers_url":"https://api.github.com/users/improbable-eng/followers","following_url":"https://api.github.com/users/improbable-eng/following{/other_user}","gists_url":"https://api.github.com/users/improbable-eng/gists{/gist_id}","starred_url":"https://api.github.com/users/improbable-eng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/improbable-eng/subscriptions","organizations_url":"https://api.github.com/users/improbable-eng/orgs","repos_url":"https://api.github.com/users/improbable-eng/repos","events_url":"https://api.github.com/users/improbable-eng/events{/privacy}","received_events_url":"https://api.github.com/users/improbable-eng/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/improbable-eng/thanos","description":"Highly available Prometheus setup with long term storage capabilities.","fork":false,"url":"https://api.github.com/repos/improbable-eng/thanos","forks_url":"https://api.github.com/repos/improbable-eng/thanos/forks","keys_url":"https://api.github.com/repos/improbable-eng/thanos/keys{/key_id}","collaborators_url":"https://api.github.com/repos/improbable-eng/thanos/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/improbable-eng/thanos/teams","hooks_url":"https://api.github.com/repos/improbable-eng/thanos/hooks","issue_events_url":"https://api.github.com/repos/improbable-eng/thanos/issues/events{/number}","events_url":"https://api.github.com/repos/improbable-eng/thanos/events","assignees_url":"https://api.github.com/repos/improbable-eng/thanos/assignees{/user}","branches_url":"https://api.github.com/repos/improbable-eng/thanos/branches{/branch}","tags_url":"https://api.github.com/repos/improbable-eng/thanos/tags","blobs_url":"https://api.github.com/repos/improbable-eng/thanos/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/improbable-eng/thanos/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/improbable-eng/thanos/git/refs{/sha}","trees_url":"https://api.github.com/repos/improbable-eng/thanos/git/trees{/sha}","statuses_url":"https://api.github.com/repos/improbable-eng/thanos/statuses/{sha}","languages_url":"https://api.github.com/repos/improbable-eng/thanos/languages","stargazers_url":"https://api.github.com/repos/improbable-eng/thanos/stargazers","contributors_url":"https://api.github.com/repos/improbable-eng/thanos/contributors","subscribers_url":"https://api.github.com/repos/improbable-eng/thanos/subscribers","subscription_url":"https://api.github.com/repos/improbable-eng/thanos/subscription","commits_url":"https://api.github.com/repos/improbable-eng/thanos/commits{/sha}","git_commits_url":"https://api.github.com/repos/improbable-eng/thanos/git/commits{/sha}","comments_url":"https://api.github.com/repos/improbable-eng/thanos/comments{/number}","issue_comment_url":"https://api.github.com/repos/improbable-eng/thanos/issues/comments{/number}","contents_url":"https://api.github.com/repos/improbable-eng/thanos/contents/{+path}","compare_url":"https://api.github.com/repos/improbable-eng/thanos/compare/{base}...{head}","merges_url":"https://api.github.com/repos/improbable-eng/thanos/merges","archive_url":"https://api.github.com/repos/improbable-eng/thanos/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/improbable-eng/thanos/downloads","issues_url":"https://api.github.com/repos/improbable-eng/thanos/issues{/number}","pulls_url":"https://api.github.com/repos/improbable-eng/thanos/pulls{/number}","milestones_url":"https://api.github.com/repos/improbable-eng/thanos/milestones{/number}","notifications_url":"https://api.github.com/repos/improbable-eng/thanos/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/improbable-eng/thanos/labels{/name}","releases_url":"https://api.github.com/repos/improbable-eng/thanos/releases{/id}","deployments_url":"https://api.github.com/repos/improbable-eng/thanos/deployments","created_at":"2017-11-01T17:38:03Z","updated_at":"2018-12-17T17:53:26Z","pushed_at":"2018-12-17T16:59:18Z","git_url":"git://github.com/improbable-eng/thanos.git","ssh_url":"git@github.com:improbable-eng/thanos.git","clone_url":"https://github.com/improbable-eng/thanos.git","svn_url":"https://github.com/improbable-eng/thanos","homepage":"","size":55554,"stargazers_count":2477,"watchers_count":2477,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":207,"mirror_url":null,"archived":false,"open_issues_count":116,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":207,"open_issues":116,"watchers":2477,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/improbable-eng/thanos/pulls/659"},"html":{"href":"https://github.com/improbable-eng/thanos/pull/659"},"issue":{"href":"https://api.github.com/repos/improbable-eng/thanos/issues/659"},"comments":{"href":"https://api.github.com/repos/improbable-eng/thanos/issues/659/comments"},"review_comments":{"href":"https://api.github.com/repos/improbable-eng/thanos/pulls/659/comments"},"review_comment":{"href":"https://api.github.com/repos/improbable-eng/thanos/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/improbable-eng/thanos/pulls/659/commits"},"statuses":{"href":"https://api.github.com/repos/improbable-eng/thanos/statuses/dc3b7823dce4a2fa2875913a1944f3289d2a7da9"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"bwplotka"}}