{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-druid/issues/6107","repository_url":"https://api.github.com/repos/apache/incubator-druid","labels_url":"https://api.github.com/repos/apache/incubator-druid/issues/6107/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-druid/issues/6107/comments","events_url":"https://api.github.com/repos/apache/incubator-druid/issues/6107/events","html_url":"https://github.com/apache/incubator-druid/pull/6107","id":347588607,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA2MTc2NTAz","number":6107,"title":"Order rows during incremental index persist when rollup is disabled.","user":{"login":"clintropolis","id":1577461,"node_id":"MDQ6VXNlcjE1Nzc0NjE=","avatar_url":"https://avatars1.githubusercontent.com/u/1577461?v=4","gravatar_id":"","url":"https://api.github.com/users/clintropolis","html_url":"https://github.com/clintropolis","followers_url":"https://api.github.com/users/clintropolis/followers","following_url":"https://api.github.com/users/clintropolis/following{/other_user}","gists_url":"https://api.github.com/users/clintropolis/gists{/gist_id}","starred_url":"https://api.github.com/users/clintropolis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintropolis/subscriptions","organizations_url":"https://api.github.com/users/clintropolis/orgs","repos_url":"https://api.github.com/users/clintropolis/repos","events_url":"https://api.github.com/users/clintropolis/events{/privacy}","received_events_url":"https://api.github.com/users/clintropolis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-08-04T05:12:33Z","updated_at":"2018-08-06T21:17:45Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/incubator-druid/pulls/6107","html_url":"https://github.com/apache/incubator-druid/pull/6107","diff_url":"https://github.com/apache/incubator-druid/pull/6107.diff","patch_url":"https://github.com/apache/incubator-druid/pull/6107.patch"},"body":"Resolves #6066 by modifying the `FactsHolder` interface to include a new method `Iterable<IncrementalIndexRow> getPersistIterable()` and using this when persisting incremental indexes. Added an additional benchmark generator schema with 4 low cardinality dimensions to enable testing this scenario.\r\n\r\nBefore this patch:\r\n```\r\nBenchmark                        (rollup)  (rollupOpportunity)  (rowsPerSegment)  (schema)  Mode  Cnt       Score       Error  Units\r\nIndexPersistBenchmark.persistV9      true                 none             75000     rollo  avgt   25  429409.821 ± 17771.526  us/op\r\nIndexPersistBenchmark.persistV9      true             moderate             75000     rollo  avgt   25   57578.929 ±  2650.596  us/op\r\nIndexPersistBenchmark.persistV9      true                 high             75000     rollo  avgt   25   11023.976 ±   461.142  us/op\r\nIndexPersistBenchmark.persistV9     false                 none             75000     rollo  avgt   25  414289.365 ± 16384.902  us/op\r\nIndexPersistBenchmark.persistV9     false             moderate             75000     rollo  avgt   25  407060.720 ± 16965.695  us/op\r\nIndexPersistBenchmark.persistV9     false                 high             75000     rollo  avgt   25  400008.825 ± 19613.728  us/op\r\n\r\nsize [2262258] bytes.\r\nsize [276631] bytes.\r\nsize [47597] bytes.\r\nsize [2280590] bytes.\r\nsize [2095354] bytes.\r\nsize [2094972] bytes.\r\n```\r\n\r\nAfter:\r\n```\r\n\r\nBenchmark                        (rollup)  (rollupOpportunity)  (rowsPerSegment)  (schema)  Mode  Cnt       Score       Error  Units\r\nIndexPersistBenchmark.persistV9      true                 none             75000     rollo  avgt   25  436966.463 ± 45936.358  us/op\r\nIndexPersistBenchmark.persistV9      true             moderate             75000     rollo  avgt   25   54724.237 ±  7500.566  us/op\r\nIndexPersistBenchmark.persistV9      true                 high             75000     rollo  avgt   25   11010.033 ±   718.345  us/op\r\nIndexPersistBenchmark.persistV9     false                 none             75000     rollo  avgt   25  464730.668 ± 30413.613  us/op\r\nIndexPersistBenchmark.persistV9     false             moderate             75000     rollo  avgt   25  523597.179 ± 43443.648  us/op\r\nIndexPersistBenchmark.persistV9     false                 high             75000     rollo  avgt   25  535282.839 ± 46529.297  us/op\r\n\r\nsize [2262258] bytes.\r\nsize [276631] bytes.\r\nsize [47597] bytes.\r\nsize [2269144] bytes. \r\nsize [1475402] bytes.\r\nsize [1357298] bytes.\r\n```\r\n\r\nActual difference in segment size will vary quite a lot from this contrived scenario, but should generally be smaller, at the cost of slower index persist time.\r\n\r\nQuery performance should be unaffected. See #6066 for additional benchmarks and discussion."},"comment":{"url":"https://api.github.com/repos/apache/incubator-druid/issues/comments/410855961","html_url":"https://github.com/apache/incubator-druid/pull/6107#issuecomment-410855961","issue_url":"https://api.github.com/repos/apache/incubator-druid/issues/6107","id":410855961,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMDg1NTk2MQ==","user":{"login":"gianm","id":1214075,"node_id":"MDQ6VXNlcjEyMTQwNzU=","avatar_url":"https://avatars3.githubusercontent.com/u/1214075?v=4","gravatar_id":"","url":"https://api.github.com/users/gianm","html_url":"https://github.com/gianm","followers_url":"https://api.github.com/users/gianm/followers","following_url":"https://api.github.com/users/gianm/following{/other_user}","gists_url":"https://api.github.com/users/gianm/gists{/gist_id}","starred_url":"https://api.github.com/users/gianm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gianm/subscriptions","organizations_url":"https://api.github.com/users/gianm/orgs","repos_url":"https://api.github.com/users/gianm/repos","events_url":"https://api.github.com/users/gianm/events{/privacy}","received_events_url":"https://api.github.com/users/gianm/received_events","type":"User","site_admin":false},"created_at":"2018-08-06T21:17:45Z","updated_at":"2018-08-06T21:17:45Z","author_association":"CONTRIBUTOR","body":"Tests look good now, so I'll merge it. Thanks @clintropolis!"}}
{"actor":{"display_login":"gianm"}}