{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-quickstep/issues/109","repository_url":"https://api.github.com/repos/apache/incubator-quickstep","labels_url":"https://api.github.com/repos/apache/incubator-quickstep/issues/109/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-quickstep/issues/109/comments","events_url":"https://api.github.com/repos/apache/incubator-quickstep/issues/109/events","html_url":"https://github.com/apache/incubator-quickstep/pull/109","id":181275561,"number":109,"title":"Refactored SplitRowStore bulk insertion","user":{"login":"cramja","id":5708822,"avatar_url":"https://avatars.githubusercontent.com/u/5708822?v=3","gravatar_id":"","url":"https://api.github.com/users/cramja","html_url":"https://github.com/cramja","followers_url":"https://api.github.com/users/cramja/followers","following_url":"https://api.github.com/users/cramja/following{/other_user}","gists_url":"https://api.github.com/users/cramja/gists{/gist_id}","starred_url":"https://api.github.com/users/cramja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cramja/subscriptions","organizations_url":"https://api.github.com/users/cramja/orgs","repos_url":"https://api.github.com/users/cramja/repos","events_url":"https://api.github.com/users/cramja/events{/privacy}","received_events_url":"https://api.github.com/users/cramja/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2016-10-05T21:52:47Z","updated_at":"2016-10-18T23:54:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-quickstep/pulls/109","html_url":"https://github.com/apache/incubator-quickstep/pull/109","diff_url":"https://github.com/apache/incubator-quickstep/pull/109.diff","patch_url":"https://github.com/apache/incubator-quickstep/pull/109.patch"},"body":"The inner loop of the insert algorithm has been changed to reduce function calls to only those that are absolutely necessary. Also, we merge copies which come from other rowstore source, speeding up insertion time. \r\n\r\nAlso adds support for the idea of 'partial inserts'. Partial inserts are when you are only inserting a subset of the columns at a time. Partial inserts will be used in a later commit.\r\n\r\n*Testing*\r\nUnit tests have been updated. The old bulkInsert tests needed to be modified because now we have situations where a block will not be filled up completely- only to a threshold value. This reduces the runtime of the costly inner loop at the cost of a few tuples.\r\n\r\n*Performance*\r\nI had a [similar PR-100 open](https://github.com/apache/incubator-quickstep/pull/100) last week. I ran TPCH SF100 queries 1-17 with this branch and with the branch from PR-100. They performed within a 1% margin of each other so it is safe to say that this branch is as fast as the last branch (which was 2x the base).\r\n\r\n**Best times for contrived 1gb relation, 50% selectivity selection test**\r\n(See PR100)\r\nmaster: 1602.358 ms\r\nsplitrow_insert_refactor: 469.664 ms\r\n\r\n**TPCH runtime Queries 1-16 (sum of the average of middle 3 runs of 5 runs total per query**\r\nmaster: 10.61 minutes\r\nsplitrow_insert_refactor: 10.48 minutes\r\n\r\n**TLDR** this is an incremental improvement.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-quickstep/issues/comments/254672628","html_url":"https://github.com/apache/incubator-quickstep/pull/109#issuecomment-254672628","issue_url":"https://api.github.com/repos/apache/incubator-quickstep/issues/109","id":254672628,"user":{"login":"cramja","id":5708822,"avatar_url":"https://avatars.githubusercontent.com/u/5708822?v=3","gravatar_id":"","url":"https://api.github.com/users/cramja","html_url":"https://github.com/cramja","followers_url":"https://api.github.com/users/cramja/followers","following_url":"https://api.github.com/users/cramja/following{/other_user}","gists_url":"https://api.github.com/users/cramja/gists{/gist_id}","starred_url":"https://api.github.com/users/cramja/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cramja/subscriptions","organizations_url":"https://api.github.com/users/cramja/orgs","repos_url":"https://api.github.com/users/cramja/repos","events_url":"https://api.github.com/users/cramja/events{/privacy}","received_events_url":"https://api.github.com/users/cramja/received_events","type":"User","site_admin":false},"created_at":"2016-10-18T23:54:57Z","updated_at":"2016-10-18T23:54:57Z","body":"I ran TPCH 10 on a cloudlab instance. The update is marginally better. Note that *master* is using splitrows, not columnstores. Also this is queries `01 04 06 09 10 11 12 13 14 15 16 19 22`\r\n\r\n|master|splitrow refactor| master - splitrow refactor|\r\n|---|---|---|\r\n|   2244.56|   2227.01|      17.5|\r\n|    482.29|    475.22|       7.1|\r\n|    298.32|    295.05|       3.3|\r\n|    823.28|    822.34|       0.9|\r\n|   1655.41|   1680.74|     -25.3|\r\n|    343.62|    360.24|     -16.6|\r\n|    566.85|    568.88|      -2.0|\r\n|   3189.44|   3104.58|      84.9|\r\n|    315.47|    312.61|       2.9|\r\n|    594.03|    555.04|      39.0|\r\n|   1076.28|   1132.68|     -56.4|\r\n|    448.39|    441.66|       6.7|\r\n|    776.47|    777.97|      -1.5|\r\n|  12814   |12754|60.39|\r\n**Note the last row is a total**\r\n\r\n\r\nComparing it against *Master* on which uses compressed columnstores, I found:\r\n\r\n|master_cs|splitrow refactor| master_cs - splitrow refactor|\r\n|---|---|---|\r\n|   1676.65|   2227.01|    -550.4|\r\n|    324.30|    475.22|    -150.9|\r\n|     44.78|    295.05|    -250.3|\r\n|    692.21|    822.34|    -130.1|\r\n|   1522.79|   1680.74|    -158.0|\r\n|    325.94|    360.24|     -34.3|\r\n|    210.29|    568.88|    -358.6|\r\n|   3278.79|   3104.58|     174.2|\r\n|     98.74|    312.61|    -213.9|\r\n|    430.47|    555.04|    -124.6|\r\n|    977.81|   1132.68|    -154.9|\r\n|    113.18|    441.66|    -328.5|\r\n|    779.42|    777.97|       1.5|\r\n|10475.37|12754.02|-2278.64|\r\n\r\n**Note the last row is a total**\r\n\r\nSo apparently compression is the way to go here. Though, this is of course still an improvement on the old splitrow."}}
{"actor":{"display_login":"cramja"}}