{"action":"created","issue":{"url":"https://api.github.com/repos/apache/carbondata/issues/1808","repository_url":"https://api.github.com/repos/apache/carbondata","labels_url":"https://api.github.com/repos/apache/carbondata/issues/1808/labels{/name}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/1808/comments","events_url":"https://api.github.com/repos/apache/carbondata/issues/1808/events","html_url":"https://github.com/apache/carbondata/pull/1808","id":288776437,"number":1808,"title":"[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading","user":{"login":"xuchuanyin","id":10445758,"avatar_url":"https://avatars1.githubusercontent.com/u/10445758?v=4","gravatar_id":"","url":"https://api.github.com/users/xuchuanyin","html_url":"https://github.com/xuchuanyin","followers_url":"https://api.github.com/users/xuchuanyin/followers","following_url":"https://api.github.com/users/xuchuanyin/following{/other_user}","gists_url":"https://api.github.com/users/xuchuanyin/gists{/gist_id}","starred_url":"https://api.github.com/users/xuchuanyin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuchuanyin/subscriptions","organizations_url":"https://api.github.com/users/xuchuanyin/orgs","repos_url":"https://api.github.com/users/xuchuanyin/repos","events_url":"https://api.github.com/users/xuchuanyin/events{/privacy}","received_events_url":"https://api.github.com/users/xuchuanyin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":65,"created_at":"2018-01-16T03:37:39Z","updated_at":"2018-02-09T06:04:53Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/1808","html_url":"https://github.com/apache/carbondata/pull/1808","diff_url":"https://github.com/apache/carbondata/pull/1808.diff","patch_url":"https://github.com/apache/carbondata/pull/1808.patch"},"body":"Carbondata assign blocks to nodes at the beginning of data loading.\r\nPrevious block allocation strategy is block number based and it will\r\nsuffer skewed data problem if the size of input files differs a lot.\r\n\r\nWe introduced a size based block allocation strategy to optimize data\r\nloading performance in skewed data scenario.\r\n\r\nBe sure to do all of the following checklist to help us incorporate \r\nyour contribution quickly and easily:\r\n\r\n - [x] Any interfaces changed?\r\n `Only changed the internal interfaces`\r\n - [x] Any backward compatibility impacted?\r\n `No`\r\n - [x] Document update required?\r\n`Updated the document`\r\n - [x] Testing done\r\n        Please provide details on \r\n        - Whether new unit test cases have been added or why no new tests are required?\r\n`Added tests to verify the block-allocation correctness`\r\n        - How it is tested? Please attach test report.\r\n`Tested in local 3-node cluster`\r\n        - Is it a performance related change? Please attach the performance test report.\r\n```\r\nIn my scenario, the size of input data file varies from 1KB to about 5GB. \r\nBefore enabling this feature, each executor processed the same number of blocks\r\n and the processed data size had a 5X gap.  --(block number based allocation)\r\nAfter enabling this feature, each executor processed almost the same size of data\r\n and the processed number of data blocks had 6X gap. -- (block size based allocation)\r\n\r\nThe data loading performance had been promoted from 41MB/s/Node to 61MB/s/Node, \r\nabout 50% performance enhancement gained.\r\n```\r\n\r\n        - Any additional information to help reviewers in testing this change.\r\n       `I refactored the code to make it more readable. The core code mainly lies in CarbonLoaderUtil`\r\n - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. \r\n`Not related`\r\n"},"comment":{"url":"https://api.github.com/repos/apache/carbondata/issues/comments/364343923","html_url":"https://github.com/apache/carbondata/pull/1808#issuecomment-364343923","issue_url":"https://api.github.com/repos/apache/carbondata/issues/1808","id":364343923,"user":{"login":"xuchuanyin","id":10445758,"avatar_url":"https://avatars1.githubusercontent.com/u/10445758?v=4","gravatar_id":"","url":"https://api.github.com/users/xuchuanyin","html_url":"https://github.com/xuchuanyin","followers_url":"https://api.github.com/users/xuchuanyin/followers","following_url":"https://api.github.com/users/xuchuanyin/following{/other_user}","gists_url":"https://api.github.com/users/xuchuanyin/gists{/gist_id}","starred_url":"https://api.github.com/users/xuchuanyin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuchuanyin/subscriptions","organizations_url":"https://api.github.com/users/xuchuanyin/orgs","repos_url":"https://api.github.com/users/xuchuanyin/repos","events_url":"https://api.github.com/users/xuchuanyin/events{/privacy}","received_events_url":"https://api.github.com/users/xuchuanyin/received_events","type":"User","site_admin":false},"created_at":"2018-02-09T06:04:53Z","updated_at":"2018-02-09T06:04:53Z","author_association":"CONTRIBUTOR","body":"retest this please"}}
{"actor":{"display_login":"xuchuanyin"}}