{"action":"created","comment":{"url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/comments/93118252","pull_request_review_id":13640319,"id":93118252,"diff_hunk":"@@ -0,0 +1,93 @@\n+= Hawkular-Metrics and the new compression algorithm\n+Michael Burman\n+2016-12-19\n+:jbake-type: post\n+:jbake-status: published\n+:jbake-tags: blog, metrics, compression, gorilla, storage, performance\n+\n+In Hawkular-Metrics release 0.20.0 we introduced a new compression capability that reduces the required storage space. This blog post will describe details about the implementation in the processing pipelines and what implications there are for the user. Should you want to skip the technical details, head over to the <<benchmark,benchmark section>> below.\n+\n+Some of the details mentioned here were improved or introduced in the 0.21.0, but the compression ratio is equal between 0.20.0 and 0.21.0. Results are published against 0.22.0 which is the newest released version. \n+\n+=== Compression algorithm\n+\n+In this release, we support the compression from the Facebook's great link:www.vldb.org/pvldb/vol8/p1816-teller.pdf[Gorilla paper] with small changes. For complete overview on how the compression method works I recommend reading the original paper. For short overview, the timestamps are compressed by calculating the delta of delta and writing them with variable length encoding while the data values are compressed by taking a XOR (eXclusive OR) between values (using the previous value as predictor for the next one) and storing that with variable length encoding also. We also slightly change the implementation details, as we need to use millisecond precision instead of the second precision used by Facebook.\n+\n+As we store values in a millisecond precision, it was important for us to increase the size of the block's maximum delta value. In Facebook's implementation they reserved 14 bits to represent the first delta from the block start (giving them 4 hour blocks with 1 second precision) where as we use 27 bits to get a maximum of 24 hour blocks with a millisecond precision. In 0.20.0/0.21.0 we however use 2 hour block sizes but with millisecond precision. We will revisit the block size requirements later.\n+\n+=== Insert pipeline\n+\n+We didn't change anything. When you issue a write to the Cassandra, we store it just like before to the `data` table. No restrictions on out-of-order times or changes to the persistence. Instead, we opted to do everything as an ETL (Extract Transform Load) job on the write path as described in the next section and merge results from both tables in the read path. \n+\n+=== Compression job\n+\n+A while ago, we introduced a job scheduler to the Hawkular-Metrics and it's been improved on 0.20.0 and 0.21.0 based on our requirements to do reliable repeated compression jobs. When the server launches it schedules a repeated job called `CompressionData` that is ran every 2 hours at the start of odd hours. Should you for some reason miss one execution (because the server down for example), the scheduler will immediately start the missing jobs in order with only one being active at a time (cluster wide, state is maintained in Cassandra). The scheduled job `CompressionData` will take care of our ETL process for the whole cluster (this is something we will also improve later so that one can scale the amount of compression jobs, but for now we deemed it to be enough fast).\n+\n+When the job is started, the job itself takes a look at the intended runtime (irrelevant of the actual running time as that could be influenced by server downtime or previous job) and starts to fetch all the metrics that were created during the last 2 even hour block. For example, assuming the job should start at 03:00, we will compress the datapoints that were sent between 00:00 and 02:00. This allows up to one hour of out-of-order writes to be still compressed. If an out-of-order writes happens after this, it is kept uncompressed in the Cassandra and merged during the read phase.\n+\n+We have limited the compression job to only process a single time serie at once, to reduce the memory pressure on the Cassandra node as it might have to load existing data to memory from the disk. To reduce the I/O effect you can allocate more memory to the Cassandra as that will make reading from the disk less likely and can also reduce the amount of compaction jobs that are required as already compressed data doesn't have to be written to the disk. Should any Cassandra request fail during the compression job, we will retry the operation with a linear wait period between the requests. \n+\n+All the datapoints that were read are then streamed to the compression method of choice out of heap to reduce GC pressure. We place a header as the first byte in the compressed array, with 4 first bits reserved for the compression type and last 4 bits as metadata for the compression type. \n+\n+=== Reading pipeline\n+\n+Since we store the data now potentially in two different places, `data` for uncompressed metrics and `data_compressed` for compressed metrics, we need to read from both if necessary. While usually the `data` table holds newer datapoints than `data_compressed` this is not guaranteed as we support writing out-of-order writes without any limitations. \n+\n+As such, we need to push the read requests to both tables, with exact timestamps to the `data` table and with modified request timestamp to the `data_compressed` table to account for block sizes. For `data` reads this often means that nothing is there for older timestamps and reads there could be a waste of time. Luckily Cassandra does use a bloom filter in the read path that should get rid of these extra reads quite efficiently.\n+\n+At this point we now have two distinct streams, one with `DataPoint<T>` items and one with `ByteBuffers`. The first job is obviously to transform the `Observable<ByteBuffer>` to `Observable<DataPoint<T>>` in the given sorting order. For this, we have a transformer that takes the input ByteBuffer from the Cassandra driver, reads the first byte as our header and checks then what to do. Given that we only support a single compression method in this release, the straightforward approach is to then decompress the byte array and transform the compressed block back to uncompressed DataPoints. As we already know the type of the metric that's going to be read from the compressed block, we can correctly read either `long` or a `double` and as such avoid any casting that would reduce precision.\n+\n+Next, we must read from these two streams in the given order as the datapoints could be interleaved. Luckily, for this to function we don't need to do in-memory sorting of everything, but instead we read one point from each of the streams and then with a comparator select which one to forward and which one will be hold in the memory for next comparison. And then we request the next one from the Cassandra / uncompressed pool and continue. This allows us to reduce the memory pressure by keeping only one active datapoint and one uncompressed block in the memory for read operations. If either stream ends, then the other stream is allowed to push data without comparing it.\n+\n+This way, all the internal calls to getting datapoints (which go through a single method, link:github[findDataPoints()]) are unaware if the datapoints are coming from compressed or uncompressed streams. This allows easier refactoring in the future where we could read from several different streams if we so decided and no changes to elsewhere in the application is required.\n+\n+anchor:benchmark[]\n+\n+=== Benchmarks\n+\n+Disclaimer: the following performance numbers are an indication of results with the given test data. Your real world results can vary.\n+\n+The results are created against Cassandra 3.7, using LZ4 table compression settings.\n+\n+The dataset consist of real world data coming from a Windows host. A total of 19166 different time series are stored in the dataset, with each serie having a 10910 measurements with a millisecond precision. That means the total is 209110368 (209M) datapoints. All of them were stored as gauge type with 64 bit floating point precision.\n+\n+[cols=\"2*\", options=\"header\"]\n+.Storage usage, data set #1\n+|===\n+|0.19.0\n+|0.22.0\n+\n+| 3.2GB\n+| 158MB\n+|===\n+","path":"src/main/jbake/content/blog/2016/12/19/hawkular-metrics-compression.adoc","position":63,"original_position":63,"commit_id":"99e9be8987b321e1ae1c549aeb2b79848b1698fa","original_commit_id":"99e9be8987b321e1ae1c549aeb2b79848b1698fa","user":{"login":"burmanm","id":562574,"avatar_url":"https://avatars.githubusercontent.com/u/562574?v=3","gravatar_id":"","url":"https://api.github.com/users/burmanm","html_url":"https://github.com/burmanm","followers_url":"https://api.github.com/users/burmanm/followers","following_url":"https://api.github.com/users/burmanm/following{/other_user}","gists_url":"https://api.github.com/users/burmanm/gists{/gist_id}","starred_url":"https://api.github.com/users/burmanm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burmanm/subscriptions","organizations_url":"https://api.github.com/users/burmanm/orgs","repos_url":"https://api.github.com/users/burmanm/repos","events_url":"https://api.github.com/users/burmanm/events{/privacy}","received_events_url":"https://api.github.com/users/burmanm/received_events","type":"User","site_admin":false},"body":"Well, actually this was referring to the block cache as we have less to cache there.","created_at":"2016-12-19T20:59:23Z","updated_at":"2016-12-19T20:59:23Z","html_url":"https://github.com/hawkular/hawkular.github.io/pull/237#discussion_r93118252","pull_request_url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237","_links":{"self":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/comments/93118252"},"html":{"href":"https://github.com/hawkular/hawkular.github.io/pull/237#discussion_r93118252"},"pull_request":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237"}}},"pull_request":{"url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237","id":90013057,"html_url":"https://github.com/hawkular/hawkular.github.io/pull/237","diff_url":"https://github.com/hawkular/hawkular.github.io/pull/237.diff","patch_url":"https://github.com/hawkular/hawkular.github.io/pull/237.patch","issue_url":"https://api.github.com/repos/hawkular/hawkular.github.io/issues/237","number":237,"state":"open","locked":false,"title":"Compression details in new HWKMETRICS","user":{"login":"burmanm","id":562574,"avatar_url":"https://avatars.githubusercontent.com/u/562574?v=3","gravatar_id":"","url":"https://api.github.com/users/burmanm","html_url":"https://github.com/burmanm","followers_url":"https://api.github.com/users/burmanm/followers","following_url":"https://api.github.com/users/burmanm/following{/other_user}","gists_url":"https://api.github.com/users/burmanm/gists{/gist_id}","starred_url":"https://api.github.com/users/burmanm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burmanm/subscriptions","organizations_url":"https://api.github.com/users/burmanm/orgs","repos_url":"https://api.github.com/users/burmanm/repos","events_url":"https://api.github.com/users/burmanm/events{/privacy}","received_events_url":"https://api.github.com/users/burmanm/received_events","type":"User","site_admin":false},"body":"This is a draft of the compression blog post, but still missing the actual benchmarks (I want to rerun them with more data)\n","created_at":"2016-10-19T13:49:10Z","updated_at":"2016-12-19T20:59:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0366ea6ed70f1203d05d2573b5962844eb46e2bd","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237/commits","review_comments_url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237/comments","review_comment_url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/comments{/number}","comments_url":"https://api.github.com/repos/hawkular/hawkular.github.io/issues/237/comments","statuses_url":"https://api.github.com/repos/hawkular/hawkular.github.io/statuses/99e9be8987b321e1ae1c549aeb2b79848b1698fa","head":{"label":"burmanm:compression","ref":"compression","sha":"99e9be8987b321e1ae1c549aeb2b79848b1698fa","user":{"login":"burmanm","id":562574,"avatar_url":"https://avatars.githubusercontent.com/u/562574?v=3","gravatar_id":"","url":"https://api.github.com/users/burmanm","html_url":"https://github.com/burmanm","followers_url":"https://api.github.com/users/burmanm/followers","following_url":"https://api.github.com/users/burmanm/following{/other_user}","gists_url":"https://api.github.com/users/burmanm/gists{/gist_id}","starred_url":"https://api.github.com/users/burmanm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burmanm/subscriptions","organizations_url":"https://api.github.com/users/burmanm/orgs","repos_url":"https://api.github.com/users/burmanm/repos","events_url":"https://api.github.com/users/burmanm/events{/privacy}","received_events_url":"https://api.github.com/users/burmanm/received_events","type":"User","site_admin":false},"repo":{"id":38246162,"name":"hawkular.github.io","full_name":"burmanm/hawkular.github.io","owner":{"login":"burmanm","id":562574,"avatar_url":"https://avatars.githubusercontent.com/u/562574?v=3","gravatar_id":"","url":"https://api.github.com/users/burmanm","html_url":"https://github.com/burmanm","followers_url":"https://api.github.com/users/burmanm/followers","following_url":"https://api.github.com/users/burmanm/following{/other_user}","gists_url":"https://api.github.com/users/burmanm/gists{/gist_id}","starred_url":"https://api.github.com/users/burmanm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burmanm/subscriptions","organizations_url":"https://api.github.com/users/burmanm/orgs","repos_url":"https://api.github.com/users/burmanm/repos","events_url":"https://api.github.com/users/burmanm/events{/privacy}","received_events_url":"https://api.github.com/users/burmanm/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/burmanm/hawkular.github.io","description":"Web presence of Hawkular","fork":true,"url":"https://api.github.com/repos/burmanm/hawkular.github.io","forks_url":"https://api.github.com/repos/burmanm/hawkular.github.io/forks","keys_url":"https://api.github.com/repos/burmanm/hawkular.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/burmanm/hawkular.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/burmanm/hawkular.github.io/teams","hooks_url":"https://api.github.com/repos/burmanm/hawkular.github.io/hooks","issue_events_url":"https://api.github.com/repos/burmanm/hawkular.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/burmanm/hawkular.github.io/events","assignees_url":"https://api.github.com/repos/burmanm/hawkular.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/burmanm/hawkular.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/burmanm/hawkular.github.io/tags","blobs_url":"https://api.github.com/repos/burmanm/hawkular.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/burmanm/hawkular.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/burmanm/hawkular.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/burmanm/hawkular.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/burmanm/hawkular.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/burmanm/hawkular.github.io/languages","stargazers_url":"https://api.github.com/repos/burmanm/hawkular.github.io/stargazers","contributors_url":"https://api.github.com/repos/burmanm/hawkular.github.io/contributors","subscribers_url":"https://api.github.com/repos/burmanm/hawkular.github.io/subscribers","subscription_url":"https://api.github.com/repos/burmanm/hawkular.github.io/subscription","commits_url":"https://api.github.com/repos/burmanm/hawkular.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/burmanm/hawkular.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/burmanm/hawkular.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/burmanm/hawkular.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/burmanm/hawkular.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/burmanm/hawkular.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/burmanm/hawkular.github.io/merges","archive_url":"https://api.github.com/repos/burmanm/hawkular.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/burmanm/hawkular.github.io/downloads","issues_url":"https://api.github.com/repos/burmanm/hawkular.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/burmanm/hawkular.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/burmanm/hawkular.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/burmanm/hawkular.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/burmanm/hawkular.github.io/labels{/name}","releases_url":"https://api.github.com/repos/burmanm/hawkular.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/burmanm/hawkular.github.io/deployments","created_at":"2015-06-29T12:20:57Z","updated_at":"2016-10-19T12:55:45Z","pushed_at":"2016-12-16T09:40:16Z","git_url":"git://github.com/burmanm/hawkular.github.io.git","ssh_url":"git@github.com:burmanm/hawkular.github.io.git","clone_url":"https://github.com/burmanm/hawkular.github.io.git","svn_url":"https://github.com/burmanm/hawkular.github.io","homepage":null,"size":10591,"stargazers_count":0,"watchers_count":0,"language":"CSS","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"pages"}},"base":{"label":"hawkular:pages","ref":"pages","sha":"b46859a4cf2ce448a042e52afd813d5dee43e324","user":{"login":"hawkular","id":10179627,"avatar_url":"https://avatars.githubusercontent.com/u/10179627?v=3","gravatar_id":"","url":"https://api.github.com/users/hawkular","html_url":"https://github.com/hawkular","followers_url":"https://api.github.com/users/hawkular/followers","following_url":"https://api.github.com/users/hawkular/following{/other_user}","gists_url":"https://api.github.com/users/hawkular/gists{/gist_id}","starred_url":"https://api.github.com/users/hawkular/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hawkular/subscriptions","organizations_url":"https://api.github.com/users/hawkular/orgs","repos_url":"https://api.github.com/users/hawkular/repos","events_url":"https://api.github.com/users/hawkular/events{/privacy}","received_events_url":"https://api.github.com/users/hawkular/received_events","type":"Organization","site_admin":false},"repo":{"id":29196628,"name":"hawkular.github.io","full_name":"hawkular/hawkular.github.io","owner":{"login":"hawkular","id":10179627,"avatar_url":"https://avatars.githubusercontent.com/u/10179627?v=3","gravatar_id":"","url":"https://api.github.com/users/hawkular","html_url":"https://github.com/hawkular","followers_url":"https://api.github.com/users/hawkular/followers","following_url":"https://api.github.com/users/hawkular/following{/other_user}","gists_url":"https://api.github.com/users/hawkular/gists{/gist_id}","starred_url":"https://api.github.com/users/hawkular/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hawkular/subscriptions","organizations_url":"https://api.github.com/users/hawkular/orgs","repos_url":"https://api.github.com/users/hawkular/repos","events_url":"https://api.github.com/users/hawkular/events{/privacy}","received_events_url":"https://api.github.com/users/hawkular/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hawkular/hawkular.github.io","description":"Web presence of Hawkular","fork":false,"url":"https://api.github.com/repos/hawkular/hawkular.github.io","forks_url":"https://api.github.com/repos/hawkular/hawkular.github.io/forks","keys_url":"https://api.github.com/repos/hawkular/hawkular.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hawkular/hawkular.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hawkular/hawkular.github.io/teams","hooks_url":"https://api.github.com/repos/hawkular/hawkular.github.io/hooks","issue_events_url":"https://api.github.com/repos/hawkular/hawkular.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/hawkular/hawkular.github.io/events","assignees_url":"https://api.github.com/repos/hawkular/hawkular.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/hawkular/hawkular.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/hawkular/hawkular.github.io/tags","blobs_url":"https://api.github.com/repos/hawkular/hawkular.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hawkular/hawkular.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hawkular/hawkular.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/hawkular/hawkular.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hawkular/hawkular.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/hawkular/hawkular.github.io/languages","stargazers_url":"https://api.github.com/repos/hawkular/hawkular.github.io/stargazers","contributors_url":"https://api.github.com/repos/hawkular/hawkular.github.io/contributors","subscribers_url":"https://api.github.com/repos/hawkular/hawkular.github.io/subscribers","subscription_url":"https://api.github.com/repos/hawkular/hawkular.github.io/subscription","commits_url":"https://api.github.com/repos/hawkular/hawkular.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/hawkular/hawkular.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/hawkular/hawkular.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/hawkular/hawkular.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/hawkular/hawkular.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/hawkular/hawkular.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hawkular/hawkular.github.io/merges","archive_url":"https://api.github.com/repos/hawkular/hawkular.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hawkular/hawkular.github.io/downloads","issues_url":"https://api.github.com/repos/hawkular/hawkular.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/hawkular/hawkular.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/hawkular/hawkular.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hawkular/hawkular.github.io/labels{/name}","releases_url":"https://api.github.com/repos/hawkular/hawkular.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/hawkular/hawkular.github.io/deployments","created_at":"2015-01-13T15:33:36Z","updated_at":"2016-10-24T13:27:00Z","pushed_at":"2016-12-19T14:45:27Z","git_url":"git://github.com/hawkular/hawkular.github.io.git","ssh_url":"git@github.com:hawkular/hawkular.github.io.git","clone_url":"https://github.com/hawkular/hawkular.github.io.git","svn_url":"https://github.com/hawkular/hawkular.github.io","homepage":null,"size":15958,"stargazers_count":6,"watchers_count":6,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":30,"mirror_url":null,"open_issues_count":4,"forks":30,"open_issues":4,"watchers":6,"default_branch":"pages"}},"_links":{"self":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237"},"html":{"href":"https://github.com/hawkular/hawkular.github.io/pull/237"},"issue":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/issues/237"},"comments":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/issues/237/comments"},"review_comments":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237/comments"},"review_comment":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/pulls/237/commits"},"statuses":{"href":"https://api.github.com/repos/hawkular/hawkular.github.io/statuses/99e9be8987b321e1ae1c549aeb2b79848b1698fa"}}}}
{"actor":{"display_login":"burmanm"}}