{"action":"created","comment":{"url":"https://api.github.com/repos/prometheus/prometheus/pulls/comments/56486684","id":56486684,"diff_hunk":"@@ -306,6 +312,21 @@ func rangeValues(it chunkIterator, in metric.Interval) ([]model.SamplePair, erro\n \treturn result, it.err()\n }\n \n+// addToOverflowChunk is a utility function that creates a new chunk as overflow\n+// chunk, addse the provided sample to it, and returns a chunk slice containing","path":"storage/local/chunk.go","position":null,"original_position":39,"commit_id":"fa54dea3e2ca6f4db70bfc1d1a09f5bc2a28cf5a","original_commit_id":"c0a8b598f3d53098891d7b78890884a8684314cc","user":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"body":"Fixed.","created_at":"2016-03-17T10:59:11Z","updated_at":"2016-03-17T10:59:11Z","html_url":"https://github.com/prometheus/prometheus/pull/1491#discussion_r56486684","pull_request_url":"https://api.github.com/repos/prometheus/prometheus/pulls/1491","_links":{"self":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/comments/56486684"},"html":{"href":"https://github.com/prometheus/prometheus/pull/1491#discussion_r56486684"},"pull_request":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1491"}}},"pull_request":{"url":"https://api.github.com/repos/prometheus/prometheus/pulls/1491","id":63153985,"html_url":"https://github.com/prometheus/prometheus/pull/1491","diff_url":"https://github.com/prometheus/prometheus/pull/1491.diff","patch_url":"https://github.com/prometheus/prometheus/pull/1491.patch","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1491","number":1491,"state":"open","locked":false,"title":"Gorilla-style chunks","user":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"body":"The culmination of all my preparations of the last weeks: Chunk encoding inspired by Facebook's Gorilla paper, joined with ideas from and adjustments to the Prometheus world.\r\n\r\nOn a real-world beefy Prometheus server, I have measured 1.77 bytes / sample (measured as bytes taken within the 1kB chunks). This is an upper bound as there were many uncompleted chunks. I'll keep reporting results as the server runs longer.\r\n\r\nOn disk (including all the indices, snapshot and everything), the same server currently uses 3.0 bytes / sample with 500k time series and 520M samples ingested so far. I expect this value to drop, too, as the server runs longer.\r\n\r\nA self-monitoring Prometheus server needs less than 0.3 bytes / sample, which illustrates how far down we can go for special data set.\r\n\r\n@juliusv @fabxc @brian-brazil Please have a look. Note that the existing tests (depsite not being very low-level) cover the code completely with the exception of error branches (like \"unknown encoding\"). The fuzz test I wrote ages ago was very useful in uncovering weird cornercases and bugs. Interestingly, almost none of the bugs were in the weird bit-twiddling code. It was mostly more high level (like reading an unsigned int where a signed had to be read or getting the iterator behavior wrong etc.).\r\n\r\n@juliusv Please note that this PR depends on https://github.com/prometheus/prometheus/pull/1477 , which you have already 70% reviewed. I need your thumb for the final version. Another PR this depends upon is https://github.com/prometheus/prometheus/pull/1453 , too, which should be fairly easy to review. I (and be now the whole world) would be grateful if you could get to it. That PR has been created 13d ago.\r\n\r\n@fabxc Please note that this PR depends on https://github.com/prometheus/prometheus/pull/1466 , which you have already 70% reviewed. I need your thumb for the final version. Another PR this depends upon is https://github.com/prometheus/prometheus/pull/1453 , too, which should be fairly easy to review. I (and be now the whole world) would be grateful if you could get to it. That PR has been created 13d ago.\r\n","created_at":"2016-03-16T23:47:36Z","updated_at":"2016-03-17T10:59:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"12a2ae204ece7ca9be25585d1b3eb264e1f5b78f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/prometheus/prometheus/pulls/1491/commits","review_comments_url":"https://api.github.com/repos/prometheus/prometheus/pulls/1491/comments","review_comment_url":"https://api.github.com/repos/prometheus/prometheus/pulls/comments{/number}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1491/comments","statuses_url":"https://api.github.com/repos/prometheus/prometheus/statuses/fa54dea3e2ca6f4db70bfc1d1a09f5bc2a28cf5a","head":{"label":"prometheus:beorn7/storage8","ref":"beorn7/storage8","sha":"fa54dea3e2ca6f4db70bfc1d1a09f5bc2a28cf5a","user":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"repo":{"id":6838921,"name":"prometheus","full_name":"prometheus/prometheus","owner":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prometheus/prometheus","description":"The Prometheus monitoring system and time series database.","fork":false,"url":"https://api.github.com/repos/prometheus/prometheus","forks_url":"https://api.github.com/repos/prometheus/prometheus/forks","keys_url":"https://api.github.com/repos/prometheus/prometheus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prometheus/prometheus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prometheus/prometheus/teams","hooks_url":"https://api.github.com/repos/prometheus/prometheus/hooks","issue_events_url":"https://api.github.com/repos/prometheus/prometheus/issues/events{/number}","events_url":"https://api.github.com/repos/prometheus/prometheus/events","assignees_url":"https://api.github.com/repos/prometheus/prometheus/assignees{/user}","branches_url":"https://api.github.com/repos/prometheus/prometheus/branches{/branch}","tags_url":"https://api.github.com/repos/prometheus/prometheus/tags","blobs_url":"https://api.github.com/repos/prometheus/prometheus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prometheus/prometheus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prometheus/prometheus/git/refs{/sha}","trees_url":"https://api.github.com/repos/prometheus/prometheus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prometheus/prometheus/statuses/{sha}","languages_url":"https://api.github.com/repos/prometheus/prometheus/languages","stargazers_url":"https://api.github.com/repos/prometheus/prometheus/stargazers","contributors_url":"https://api.github.com/repos/prometheus/prometheus/contributors","subscribers_url":"https://api.github.com/repos/prometheus/prometheus/subscribers","subscription_url":"https://api.github.com/repos/prometheus/prometheus/subscription","commits_url":"https://api.github.com/repos/prometheus/prometheus/commits{/sha}","git_commits_url":"https://api.github.com/repos/prometheus/prometheus/git/commits{/sha}","comments_url":"https://api.github.com/repos/prometheus/prometheus/comments{/number}","issue_comment_url":"https://api.github.com/repos/prometheus/prometheus/issues/comments{/number}","contents_url":"https://api.github.com/repos/prometheus/prometheus/contents/{+path}","compare_url":"https://api.github.com/repos/prometheus/prometheus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prometheus/prometheus/merges","archive_url":"https://api.github.com/repos/prometheus/prometheus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prometheus/prometheus/downloads","issues_url":"https://api.github.com/repos/prometheus/prometheus/issues{/number}","pulls_url":"https://api.github.com/repos/prometheus/prometheus/pulls{/number}","milestones_url":"https://api.github.com/repos/prometheus/prometheus/milestones{/number}","notifications_url":"https://api.github.com/repos/prometheus/prometheus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prometheus/prometheus/labels{/name}","releases_url":"https://api.github.com/repos/prometheus/prometheus/releases{/id}","deployments_url":"https://api.github.com/repos/prometheus/prometheus/deployments","created_at":"2012-11-24T11:14:12Z","updated_at":"2016-03-17T09:16:09Z","pushed_at":"2016-03-17T10:59:11Z","git_url":"git://github.com/prometheus/prometheus.git","ssh_url":"git@github.com:prometheus/prometheus.git","clone_url":"https://github.com/prometheus/prometheus.git","svn_url":"https://github.com/prometheus/prometheus","homepage":"https://prometheus.io/","size":11480,"stargazers_count":4118,"watchers_count":4118,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":331,"mirror_url":null,"open_issues_count":128,"forks":331,"open_issues":128,"watchers":4118,"default_branch":"master"}},"base":{"label":"prometheus:beorn7/storage7","ref":"beorn7/storage7","sha":"168333d6625206ddc8656793ffd3f6c70b841c0b","user":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"repo":{"id":6838921,"name":"prometheus","full_name":"prometheus/prometheus","owner":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prometheus/prometheus","description":"The Prometheus monitoring system and time series database.","fork":false,"url":"https://api.github.com/repos/prometheus/prometheus","forks_url":"https://api.github.com/repos/prometheus/prometheus/forks","keys_url":"https://api.github.com/repos/prometheus/prometheus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prometheus/prometheus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prometheus/prometheus/teams","hooks_url":"https://api.github.com/repos/prometheus/prometheus/hooks","issue_events_url":"https://api.github.com/repos/prometheus/prometheus/issues/events{/number}","events_url":"https://api.github.com/repos/prometheus/prometheus/events","assignees_url":"https://api.github.com/repos/prometheus/prometheus/assignees{/user}","branches_url":"https://api.github.com/repos/prometheus/prometheus/branches{/branch}","tags_url":"https://api.github.com/repos/prometheus/prometheus/tags","blobs_url":"https://api.github.com/repos/prometheus/prometheus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prometheus/prometheus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prometheus/prometheus/git/refs{/sha}","trees_url":"https://api.github.com/repos/prometheus/prometheus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prometheus/prometheus/statuses/{sha}","languages_url":"https://api.github.com/repos/prometheus/prometheus/languages","stargazers_url":"https://api.github.com/repos/prometheus/prometheus/stargazers","contributors_url":"https://api.github.com/repos/prometheus/prometheus/contributors","subscribers_url":"https://api.github.com/repos/prometheus/prometheus/subscribers","subscription_url":"https://api.github.com/repos/prometheus/prometheus/subscription","commits_url":"https://api.github.com/repos/prometheus/prometheus/commits{/sha}","git_commits_url":"https://api.github.com/repos/prometheus/prometheus/git/commits{/sha}","comments_url":"https://api.github.com/repos/prometheus/prometheus/comments{/number}","issue_comment_url":"https://api.github.com/repos/prometheus/prometheus/issues/comments{/number}","contents_url":"https://api.github.com/repos/prometheus/prometheus/contents/{+path}","compare_url":"https://api.github.com/repos/prometheus/prometheus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prometheus/prometheus/merges","archive_url":"https://api.github.com/repos/prometheus/prometheus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prometheus/prometheus/downloads","issues_url":"https://api.github.com/repos/prometheus/prometheus/issues{/number}","pulls_url":"https://api.github.com/repos/prometheus/prometheus/pulls{/number}","milestones_url":"https://api.github.com/repos/prometheus/prometheus/milestones{/number}","notifications_url":"https://api.github.com/repos/prometheus/prometheus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prometheus/prometheus/labels{/name}","releases_url":"https://api.github.com/repos/prometheus/prometheus/releases{/id}","deployments_url":"https://api.github.com/repos/prometheus/prometheus/deployments","created_at":"2012-11-24T11:14:12Z","updated_at":"2016-03-17T09:16:09Z","pushed_at":"2016-03-17T10:59:11Z","git_url":"git://github.com/prometheus/prometheus.git","ssh_url":"git@github.com:prometheus/prometheus.git","clone_url":"https://github.com/prometheus/prometheus.git","svn_url":"https://github.com/prometheus/prometheus","homepage":"https://prometheus.io/","size":11480,"stargazers_count":4118,"watchers_count":4118,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":331,"mirror_url":null,"open_issues_count":128,"forks":331,"open_issues":128,"watchers":4118,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1491"},"html":{"href":"https://github.com/prometheus/prometheus/pull/1491"},"issue":{"href":"https://api.github.com/repos/prometheus/prometheus/issues/1491"},"comments":{"href":"https://api.github.com/repos/prometheus/prometheus/issues/1491/comments"},"review_comments":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1491/comments"},"review_comment":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1491/commits"},"statuses":{"href":"https://api.github.com/repos/prometheus/prometheus/statuses/fa54dea3e2ca6f4db70bfc1d1a09f5bc2a28cf5a"}}}}