{"action":"created","comment":{"url":"https://api.github.com/repos/apache/carbondata/pulls/comments/199316329","pull_request_review_id":133419060,"id":199316329,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5OTMxNjMyOQ==","diff_hunk":"@@ -207,6 +209,8 @@ public CarbonReaderBuilder setEndPoint(String value) {\n           format.getSplits(new JobContextImpl(job.getConfiguration(), new JobID()));\n \n       List<RecordReader<Void, T>> readers = new ArrayList<>(splits.size());\n+      CarbonProperties.getInstance()\n+          .addProperty(CarbonCommonConstants.ENABLE_SDK_QUERY_EXECUTOR, \"true\");","path":"store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReaderBuilder.java","position":18,"original_position":18,"commit_id":"69210f8ac7e64ed8a5c6a0c0a586e0cf8fc95812","original_commit_id":"69210f8ac7e64ed8a5c6a0c0a586e0cf8fc95812","user":{"login":"jackylk","id":6500698,"node_id":"MDQ6VXNlcjY1MDA2OTg=","avatar_url":"https://avatars3.githubusercontent.com/u/6500698?v=4","gravatar_id":"","url":"https://api.github.com/users/jackylk","html_url":"https://github.com/jackylk","followers_url":"https://api.github.com/users/jackylk/followers","following_url":"https://api.github.com/users/jackylk/following{/other_user}","gists_url":"https://api.github.com/users/jackylk/gists{/gist_id}","starred_url":"https://api.github.com/users/jackylk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jackylk/subscriptions","organizations_url":"https://api.github.com/users/jackylk/orgs","repos_url":"https://api.github.com/users/jackylk/repos","events_url":"https://api.github.com/users/jackylk/events{/privacy}","received_events_url":"https://api.github.com/users/jackylk/received_events","type":"User","site_admin":false},"body":"If it is always setting to true, then no need to add this configuration","created_at":"2018-06-30T07:50:43Z","updated_at":"2018-06-30T07:50:43Z","html_url":"https://github.com/apache/carbondata/pull/2391#discussion_r199316329","pull_request_url":"https://api.github.com/repos/apache/carbondata/pulls/2391","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/carbondata/pulls/comments/199316329"},"html":{"href":"https://github.com/apache/carbondata/pull/2391#discussion_r199316329"},"pull_request":{"href":"https://api.github.com/repos/apache/carbondata/pulls/2391"}}},"pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/2391","id":196333091,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk2MzMzMDkx","html_url":"https://github.com/apache/carbondata/pull/2391","diff_url":"https://github.com/apache/carbondata/pull/2391.diff","patch_url":"https://github.com/apache/carbondata/pull/2391.patch","issue_url":"https://api.github.com/repos/apache/carbondata/issues/2391","number":2391,"state":"open","locked":false,"title":"[CARBONDATA-2625] Optimize the performance of CarbonReader read many files","user":{"login":"xubo245","id":8759816,"node_id":"MDQ6VXNlcjg3NTk4MTY=","avatar_url":"https://avatars1.githubusercontent.com/u/8759816?v=4","gravatar_id":"","url":"https://api.github.com/users/xubo245","html_url":"https://github.com/xubo245","followers_url":"https://api.github.com/users/xubo245/followers","following_url":"https://api.github.com/users/xubo245/following{/other_user}","gists_url":"https://api.github.com/users/xubo245/gists{/gist_id}","starred_url":"https://api.github.com/users/xubo245/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xubo245/subscriptions","organizations_url":"https://api.github.com/users/xubo245/orgs","repos_url":"https://api.github.com/users/xubo245/repos","events_url":"https://api.github.com/users/xubo245/events{/privacy}","received_events_url":"https://api.github.com/users/xubo245/received_events","type":"User","site_admin":false},"body":"About the issue:  it's timeout and no result in 8 minutes  when read more than 10 million data with 140 files,  Even though increase 200000 rows for each carbon Writer and it can reduce the index files and data files when the number of rows is 13000000, but when there are more than  1 billion or more, the number of files still still many.  I check the code and find read more 140 files can be optimize:\r\n\r\nIn the cache.getAll, the IO is more than 140  if there are 140 carbon files,  in fact, the IO are more than 70 * 140 times, it's slow and can be optimized\r\n\r\nSecondly, there are some duplicate operate in getDataMaps and can be optimized \r\n\r\nThirdly, SDK need much time to create multiple carbonRecorderReader, it need more than 8 minutes  by testing 150 files and 15million rows data when create more than 16 carbonReorederReader if the machine has 8 cores .   It can be optimized\r\n\r\nBy optimizing the three points,including cache.getAll, getDatamaps and create carbonRecordReader,  now SDK can work for reading 150 files and 15million rows data in 8 minutes,  it need about 340 seconds by testing.\r\n\r\nOne case: 150 files , each file has 200000 rows, total rows is 15000000\r\nFinished write data time: 449.102 s\r\nFinished build reader time:192.596 s\r\nRead first row time: 192.597 s, including build reader\r\nRead time:341.556 s,  including build reader\r\n\r\nAnother case: 15 files , each file has 2000000 rows, total rows is 15000000\r\nFinished write data time: 286.907 s\r\nFinished build reader time: 134.665 s\r\nRead first row time: 134.666 s,  including build reader\r\nFinished read, the count of rows is:15000000\r\nRead time:156.427 s,  including build reader\r\n\r\nBe sure to do all of the following checklist to help us incorporate \r\nyour contribution quickly and easily:\r\n\r\n - [x] Any interfaces changed?\r\n Yes, add new one for optimizing performance\r\n - [x] Any backward compatibility impacted?\r\n NA\r\n - [x] Document update required?\r\nNO\r\n - [x] Testing done\r\n    add example for it\r\n       \r\n - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. \r\nNO\r\n","created_at":"2018-06-21T04:29:15Z","updated_at":"2018-06-30T07:50:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"090cbf3c77dd99b3fa68fdaa0da3deecc1373bde","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/carbondata/pulls/2391/commits","review_comments_url":"https://api.github.com/repos/apache/carbondata/pulls/2391/comments","review_comment_url":"https://api.github.com/repos/apache/carbondata/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/2391/comments","statuses_url":"https://api.github.com/repos/apache/carbondata/statuses/69210f8ac7e64ed8a5c6a0c0a586e0cf8fc95812","head":{"label":"xubo245:CARBONDATA-2625-SDKReaderFiled","ref":"CARBONDATA-2625-SDKReaderFiled","sha":"69210f8ac7e64ed8a5c6a0c0a586e0cf8fc95812","user":{"login":"xubo245","id":8759816,"node_id":"MDQ6VXNlcjg3NTk4MTY=","avatar_url":"https://avatars1.githubusercontent.com/u/8759816?v=4","gravatar_id":"","url":"https://api.github.com/users/xubo245","html_url":"https://github.com/xubo245","followers_url":"https://api.github.com/users/xubo245/followers","following_url":"https://api.github.com/users/xubo245/following{/other_user}","gists_url":"https://api.github.com/users/xubo245/gists{/gist_id}","starred_url":"https://api.github.com/users/xubo245/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xubo245/subscriptions","organizations_url":"https://api.github.com/users/xubo245/orgs","repos_url":"https://api.github.com/users/xubo245/repos","events_url":"https://api.github.com/users/xubo245/events{/privacy}","received_events_url":"https://api.github.com/users/xubo245/received_events","type":"User","site_admin":false},"repo":{"id":99896462,"node_id":"MDEwOlJlcG9zaXRvcnk5OTg5NjQ2Mg==","name":"carbondata","full_name":"xubo245/carbondata","owner":{"login":"xubo245","id":8759816,"node_id":"MDQ6VXNlcjg3NTk4MTY=","avatar_url":"https://avatars1.githubusercontent.com/u/8759816?v=4","gravatar_id":"","url":"https://api.github.com/users/xubo245","html_url":"https://github.com/xubo245","followers_url":"https://api.github.com/users/xubo245/followers","following_url":"https://api.github.com/users/xubo245/following{/other_user}","gists_url":"https://api.github.com/users/xubo245/gists{/gist_id}","starred_url":"https://api.github.com/users/xubo245/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xubo245/subscriptions","organizations_url":"https://api.github.com/users/xubo245/orgs","repos_url":"https://api.github.com/users/xubo245/repos","events_url":"https://api.github.com/users/xubo245/events{/privacy}","received_events_url":"https://api.github.com/users/xubo245/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/xubo245/carbondata","description":"Mirror of Apache CarbonData","fork":true,"url":"https://api.github.com/repos/xubo245/carbondata","forks_url":"https://api.github.com/repos/xubo245/carbondata/forks","keys_url":"https://api.github.com/repos/xubo245/carbondata/keys{/key_id}","collaborators_url":"https://api.github.com/repos/xubo245/carbondata/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/xubo245/carbondata/teams","hooks_url":"https://api.github.com/repos/xubo245/carbondata/hooks","issue_events_url":"https://api.github.com/repos/xubo245/carbondata/issues/events{/number}","events_url":"https://api.github.com/repos/xubo245/carbondata/events","assignees_url":"https://api.github.com/repos/xubo245/carbondata/assignees{/user}","branches_url":"https://api.github.com/repos/xubo245/carbondata/branches{/branch}","tags_url":"https://api.github.com/repos/xubo245/carbondata/tags","blobs_url":"https://api.github.com/repos/xubo245/carbondata/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/xubo245/carbondata/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/xubo245/carbondata/git/refs{/sha}","trees_url":"https://api.github.com/repos/xubo245/carbondata/git/trees{/sha}","statuses_url":"https://api.github.com/repos/xubo245/carbondata/statuses/{sha}","languages_url":"https://api.github.com/repos/xubo245/carbondata/languages","stargazers_url":"https://api.github.com/repos/xubo245/carbondata/stargazers","contributors_url":"https://api.github.com/repos/xubo245/carbondata/contributors","subscribers_url":"https://api.github.com/repos/xubo245/carbondata/subscribers","subscription_url":"https://api.github.com/repos/xubo245/carbondata/subscription","commits_url":"https://api.github.com/repos/xubo245/carbondata/commits{/sha}","git_commits_url":"https://api.github.com/repos/xubo245/carbondata/git/commits{/sha}","comments_url":"https://api.github.com/repos/xubo245/carbondata/comments{/number}","issue_comment_url":"https://api.github.com/repos/xubo245/carbondata/issues/comments{/number}","contents_url":"https://api.github.com/repos/xubo245/carbondata/contents/{+path}","compare_url":"https://api.github.com/repos/xubo245/carbondata/compare/{base}...{head}","merges_url":"https://api.github.com/repos/xubo245/carbondata/merges","archive_url":"https://api.github.com/repos/xubo245/carbondata/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/xubo245/carbondata/downloads","issues_url":"https://api.github.com/repos/xubo245/carbondata/issues{/number}","pulls_url":"https://api.github.com/repos/xubo245/carbondata/pulls{/number}","milestones_url":"https://api.github.com/repos/xubo245/carbondata/milestones{/number}","notifications_url":"https://api.github.com/repos/xubo245/carbondata/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/xubo245/carbondata/labels{/name}","releases_url":"https://api.github.com/repos/xubo245/carbondata/releases{/id}","deployments_url":"https://api.github.com/repos/xubo245/carbondata/deployments","created_at":"2017-08-10T07:50:07Z","updated_at":"2018-04-18T12:40:01Z","pushed_at":"2018-06-30T02:53:53Z","git_url":"git://github.com/xubo245/carbondata.git","ssh_url":"git@github.com:xubo245/carbondata.git","clone_url":"https://github.com/xubo245/carbondata.git","svn_url":"https://github.com/xubo245/carbondata","homepage":"","size":59896,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"589fe18834b75c797aeda590debad6f619be7a1d","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":62117818,"node_id":"MDEwOlJlcG9zaXRvcnk2MjExNzgxOA==","name":"carbondata","full_name":"apache/carbondata","owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/carbondata","description":"Mirror of Apache CarbonData","fork":false,"url":"https://api.github.com/repos/apache/carbondata","forks_url":"https://api.github.com/repos/apache/carbondata/forks","keys_url":"https://api.github.com/repos/apache/carbondata/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/carbondata/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/carbondata/teams","hooks_url":"https://api.github.com/repos/apache/carbondata/hooks","issue_events_url":"https://api.github.com/repos/apache/carbondata/issues/events{/number}","events_url":"https://api.github.com/repos/apache/carbondata/events","assignees_url":"https://api.github.com/repos/apache/carbondata/assignees{/user}","branches_url":"https://api.github.com/repos/apache/carbondata/branches{/branch}","tags_url":"https://api.github.com/repos/apache/carbondata/tags","blobs_url":"https://api.github.com/repos/apache/carbondata/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/carbondata/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/carbondata/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/carbondata/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/carbondata/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/carbondata/languages","stargazers_url":"https://api.github.com/repos/apache/carbondata/stargazers","contributors_url":"https://api.github.com/repos/apache/carbondata/contributors","subscribers_url":"https://api.github.com/repos/apache/carbondata/subscribers","subscription_url":"https://api.github.com/repos/apache/carbondata/subscription","commits_url":"https://api.github.com/repos/apache/carbondata/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/carbondata/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/carbondata/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/carbondata/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/carbondata/contents/{+path}","compare_url":"https://api.github.com/repos/apache/carbondata/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/carbondata/merges","archive_url":"https://api.github.com/repos/apache/carbondata/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/carbondata/downloads","issues_url":"https://api.github.com/repos/apache/carbondata/issues{/number}","pulls_url":"https://api.github.com/repos/apache/carbondata/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/carbondata/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/carbondata/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/carbondata/labels{/name}","releases_url":"https://api.github.com/repos/apache/carbondata/releases{/id}","deployments_url":"https://api.github.com/repos/apache/carbondata/deployments","created_at":"2016-06-28T07:00:06Z","updated_at":"2018-06-30T07:45:43Z","pushed_at":"2018-06-30T07:45:27Z","git_url":"git://github.com/apache/carbondata.git","ssh_url":"git@github.com:apache/carbondata.git","clone_url":"https://github.com/apache/carbondata.git","svn_url":"https://github.com/apache/carbondata","homepage":"","size":59895,"stargazers_count":702,"watchers_count":702,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":431,"mirror_url":null,"archived":false,"open_issues_count":93,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":431,"open_issues":93,"watchers":702,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/carbondata/pulls/2391"},"html":{"href":"https://github.com/apache/carbondata/pull/2391"},"issue":{"href":"https://api.github.com/repos/apache/carbondata/issues/2391"},"comments":{"href":"https://api.github.com/repos/apache/carbondata/issues/2391/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/carbondata/pulls/2391/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/carbondata/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/carbondata/pulls/2391/commits"},"statuses":{"href":"https://api.github.com/repos/apache/carbondata/statuses/69210f8ac7e64ed8a5c6a0c0a586e0cf8fc95812"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jackylk"}}