{"action":"created","issue":{"url":"https://api.github.com/repos/apache/carbondata/issues/2816","repository_url":"https://api.github.com/repos/apache/carbondata","labels_url":"https://api.github.com/repos/apache/carbondata/issues/2816/labels{/name}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/2816/comments","events_url":"https://api.github.com/repos/apache/carbondata/issues/2816/events","html_url":"https://github.com/apache/carbondata/pull/2816","id":370422823,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIzMTAyNzEz","number":2816,"title":" [CARBONDATA-3003] Suppor read batch row in CSDK","user":{"login":"xubo245","id":8759816,"node_id":"MDQ6VXNlcjg3NTk4MTY=","avatar_url":"https://avatars1.githubusercontent.com/u/8759816?v=4","gravatar_id":"","url":"https://api.github.com/users/xubo245","html_url":"https://github.com/xubo245","followers_url":"https://api.github.com/users/xubo245/followers","following_url":"https://api.github.com/users/xubo245/following{/other_user}","gists_url":"https://api.github.com/users/xubo245/gists{/gist_id}","starred_url":"https://api.github.com/users/xubo245/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xubo245/subscriptions","organizations_url":"https://api.github.com/users/xubo245/orgs","repos_url":"https://api.github.com/users/xubo245/repos","events_url":"https://api.github.com/users/xubo245/events{/privacy}","received_events_url":"https://api.github.com/users/xubo245/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":29,"created_at":"2018-10-16T03:16:33Z","updated_at":"2018-10-26T13:44:10Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/2816","html_url":"https://github.com/apache/carbondata/pull/2816","diff_url":"https://github.com/apache/carbondata/pull/2816.diff","patch_url":"https://github.com/apache/carbondata/pull/2816.patch"},"body":" [CARBONDATA-3003] Suppor read batch row in CSDK\r\n    1. support read batch row in SDK\r\n    2. support read batch row in CSDK\r\n    3. add SDKReaderBenchmark IN SDK and testNextBatchRowPerformance in CSDK\r\n    4. improve CSDK read performance\r\n\r\nThis PR based on https://github.com/apache/carbondata/pull/2792 and cherry pick its commits. After PR2792 merged , this PR will remove its commits.\r\n\r\nFor SDK batch read:\r\nreadNextBatchRow:\r\n```\r\ntotal lines is 200100000, build time is 10.434133262 s, \ttotal read time is 167.567157044 s, \taverage speed is 1194148.0868321797records/s.\r\n```\r\nreadNextCarbonRow:\r\n```\r\ntotal lines is 200100000, build time is 15.775965656 s, \ttotal read time is 183.312544655 s, \taverage speed is 1091578.322567037records/s.\r\n```\r\nread batch row is faster 9.4% than readCarbonRow( one by one)\r\n\r\nFor CSDK:\r\n\r\nTest next Row Performance:\r\n\r\n\r\nbuild time is: 2.749129 s\r\n\r\n```\r\n100000: time is 0.147732 s, speed is 676901.416078 records/s  from_email10144phillip.allen@enron.com\tto_email69149buck.buckner@honeywell.com\tfrom_to\t<5164240.1075855667637.JavaMail.evans@thyme>\t1538015558000000\t971703720000000\t\r\n200000: time is 0.320773 s, speed is 311746.936307 records/s  from_email10324veronica.espinoza@enron.com\tto_email44094cynthia.franklin@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n300000: time is 0.138412 s, speed is 722480.709765 records/s  from_email10545store-news@amazon.com\tto_email59755pallen@enron.com\tfrom_to\t<5977904.1075858636257.JavaMail.evans@thyme>\t1538015558000000\t1004057196000000\t\r\n400000: time is 0.381501 s, speed is 262122.510819 records/s  from_email1075sarah.novosel@enron.com\tto_email16354shelley.corman@enron.com\tfrom_to\t<23732985.1075855665438.JavaMail.evans@thyme>\t1538015558000000\t976725540000000\t\r\n500000: time is 0.124684 s, speed is 802027.525585 records/s  from_email10951ray.alvarez@enron.com\tto_email63518alan.comnes@enron.com\tfrom_to\t<31706076.1075858632278.JavaMail.evans@thyme>\t1538015558000000\t1003441879000000\t\r\n600000: time is 1.260054 s, speed is 79361.678150 records/s  from_email11168veronica.espinoza@enron.com\tto_email62073clinton.anderson@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n700000: time is 0.120333 s, speed is 831027.232762 records/s  from_email11347ryan.o'rourke@enron.com\tto_email33299m..scott@enron.com\tfrom_to\t<2047280.1075858635378.JavaMail.evans@thyme>\t1538015558000000\t1003974318000000\t\r\n800000: time is 0.424332 s, speed is 235664.526833 records/s  from_email11540ryan.o'rourke@enron.com\tto_email60688keith.holst@enron.com\tfrom_to\t<2047280.1075858635378.JavaMail.evans@thyme>\t1538015558000000\t1003974318000000\t\r\n900000: time is 0.127125 s, speed is 786627.335300 records/s  from_email11756veronica.espinoza@enron.com\tto_email40317p..south@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n1000000: time is 0.135605 s, speed is 737435.935253 records/s  from_email11965veronica.espinoza@enron.com\tto_email28764r..conner@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n1100000: time is 0.653121 s, speed is 153110.985560 records/s  from_email12200phillip.allen@enron.com\tto_email53803anne.bike@enron.com\tfrom_to\t<12338129.1075855667248.JavaMail.evans@thyme>\t1538015558000000\t972994320000000\t\r\n\r\n```\r\nreadNextBatchRow\r\n```\r\nTest next Batch Row Performance:\r\n\r\n\r\nbuild time is: 10.018114 s\r\n\r\n100000: hasNext time is 0.000005 s\r\n100000: readNextBatchRow time is 0.008313 s\r\n100000: time is 0.013860 s, speed is 7215007.215007 records/s  from_email10144phillip.allen@enron.com\tto_email69149buck.buckner@honeywell.com\tfrom_to\t<5164240.1075855667637.JavaMail.evans@thyme>\t1538015558000000\t971703720000000\t\r\n\r\n200000: hasNext time is 0.072777 s\r\n200000: readNextBatchRow time is 0.005466 s\r\n200000: time is 0.084476 s, speed is 1183768.170841 records/s  from_email10324veronica.espinoza@enron.com\tto_email44094cynthia.franklin@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n\r\n300000: hasNext time is 0.343702 s\r\n300000: readNextBatchRow time is 0.000469 s\r\n300000: time is 0.350707 s, speed is 285138.306335 records/s  from_email10545store-news@amazon.com\tto_email59755pallen@enron.com\tfrom_to\t<5977904.1075858636257.JavaMail.evans@thyme>\t1538015558000000\t1004057196000000\t\r\n\r\n400000: hasNext time is 0.066913 s\r\n400000: readNextBatchRow time is 0.000252 s\r\n400000: time is 0.072537 s, speed is 1378606.779988 records/s  from_email1075sarah.novosel@enron.com\tto_email16354shelley.corman@enron.com\tfrom_to\t<23732985.1075855665438.JavaMail.evans@thyme>\t1538015558000000\t976725540000000\t\r\n\r\n500000: hasNext time is 0.074619 s\r\n500000: readNextBatchRow time is 0.000388 s\r\n500000: time is 0.082063 s, speed is 1218575.972119 records/s  from_email10951ray.alvarez@enron.com\tto_email63518alan.comnes@enron.com\tfrom_to\t<31706076.1075858632278.JavaMail.evans@thyme>\t1538015558000000\t1003441879000000\t\r\n\r\n600000: hasNext time is 0.076493 s\r\n600000: readNextBatchRow time is 0.000286 s\r\n600000: time is 0.083565 s, speed is 1196673.248370 records/s  from_email11168veronica.espinoza@enron.com\tto_email62073clinton.anderson@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n\r\n700000: hasNext time is 2.201641 s\r\n700000: readNextBatchRow time is 0.000288 s\r\n700000: time is 2.207710 s, speed is 45295.804250 records/s  from_email11347ryan.o'rourke@enron.com\tto_email33299m..scott@enron.com\tfrom_to\t<2047280.1075858635378.JavaMail.evans@thyme>\t1538015558000000\t1003974318000000\t\r\n\r\n800000: hasNext time is 0.072724 s\r\n800000: readNextBatchRow time is 0.000298 s\r\n800000: time is 0.081025 s, speed is 1234186.979327 records/s  from_email11540ryan.o'rourke@enron.com\tto_email60688keith.holst@enron.com\tfrom_to\t<2047280.1075858635378.JavaMail.evans@thyme>\t1538015558000000\t1003974318000000\t\r\n\r\n900000: hasNext time is 0.069825 s\r\n900000: readNextBatchRow time is 0.000277 s\r\n900000: time is 0.075498 s, speed is 1324538.398368 records/s  from_email11756veronica.espinoza@enron.com\tto_email40317p..south@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n\r\n1000000: hasNext time is 2.799625 s\r\n1000000: readNextBatchRow time is 0.000307 s\r\n1000000: time is 2.805359 s, speed is 35646.061698 records/s  from_email11965veronica.espinoza@enron.com\tto_email28764r..conner@enron.com\tfrom_to\t<14154714.1075858633174.JavaMail.evans@thyme>\t1538015558000000\t1003768608000000\t\r\n\r\n1100000: hasNext time is 0.074910 s\r\n1100000: readNextBatchRow time is 0.000379 s\r\n1100000: time is 0.083387 s, speed is 1199227.697363 records/s  from_email12200phillip.allen@enron.com\tto_email53803anne.bike@enron.com\tfrom_to\t<12338129.1075855667248.JavaMail.evans@thyme>\t1538015558000000\t972994320000000\t\r\n\r\n```\r\nBe sure to do all of the following checklist to help us incorporate \r\nyour contribution quickly and easily:\r\n\r\n - [x] Any interfaces changed?\r\n add interface\r\n - [x] Any backward compatibility impacted?\r\n No\r\n - [ ] Document update required?\r\nyes\r\n - [x] Testing done\r\n  add test case in main.cpp and SDKReaderBenchmark\r\n - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. \r\nhttps://issues.apache.org/jira/browse/CARBONDATA-2951"},"comment":{"url":"https://api.github.com/repos/apache/carbondata/issues/comments/433412321","html_url":"https://github.com/apache/carbondata/pull/2816#issuecomment-433412321","issue_url":"https://api.github.com/repos/apache/carbondata/issues/2816","id":433412321,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzQxMjMyMQ==","user":{"login":"CarbonDataQA","id":24402133,"node_id":"MDQ6VXNlcjI0NDAyMTMz","avatar_url":"https://avatars0.githubusercontent.com/u/24402133?v=4","gravatar_id":"","url":"https://api.github.com/users/CarbonDataQA","html_url":"https://github.com/CarbonDataQA","followers_url":"https://api.github.com/users/CarbonDataQA/followers","following_url":"https://api.github.com/users/CarbonDataQA/following{/other_user}","gists_url":"https://api.github.com/users/CarbonDataQA/gists{/gist_id}","starred_url":"https://api.github.com/users/CarbonDataQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CarbonDataQA/subscriptions","organizations_url":"https://api.github.com/users/CarbonDataQA/orgs","repos_url":"https://api.github.com/users/CarbonDataQA/repos","events_url":"https://api.github.com/users/CarbonDataQA/events{/privacy}","received_events_url":"https://api.github.com/users/CarbonDataQA/received_events","type":"User","site_admin":false},"created_at":"2018-10-26T13:44:10Z","updated_at":"2018-10-26T13:44:10Z","author_association":"NONE","body":"Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9314/\n"}}
{"actor":{"display_login":"CarbonDataQA"}}