{"action":"created","issue":{"url":"https://api.github.com/repos/apache/carbondata/issues/2706","repository_url":"https://api.github.com/repos/apache/carbondata","labels_url":"https://api.github.com/repos/apache/carbondata/issues/2706/labels{/name}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/2706/comments","events_url":"https://api.github.com/repos/apache/carbondata/issues/2706/events","html_url":"https://github.com/apache/carbondata/pull/2706","id":358733150,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE0Mzk3NTY5","number":2706,"title":"[CARBONDATA-2927] multiple issue fixes for varchar column and complex columns, row that grows more than 2MB ","user":{"login":"ajantha-bhat","id":5889404,"node_id":"MDQ6VXNlcjU4ODk0MDQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5889404?v=4","gravatar_id":"","url":"https://api.github.com/users/ajantha-bhat","html_url":"https://github.com/ajantha-bhat","followers_url":"https://api.github.com/users/ajantha-bhat/followers","following_url":"https://api.github.com/users/ajantha-bhat/following{/other_user}","gists_url":"https://api.github.com/users/ajantha-bhat/gists{/gist_id}","starred_url":"https://api.github.com/users/ajantha-bhat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajantha-bhat/subscriptions","organizations_url":"https://api.github.com/users/ajantha-bhat/orgs","repos_url":"https://api.github.com/users/ajantha-bhat/repos","events_url":"https://api.github.com/users/ajantha-bhat/events{/privacy}","received_events_url":"https://api.github.com/users/ajantha-bhat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-09-10T18:07:25Z","updated_at":"2018-09-12T03:26:17Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/2706","html_url":"https://github.com/apache/carbondata/pull/2706","diff_url":"https://github.com/apache/carbondata/pull/2706.diff","patch_url":"https://github.com/apache/carbondata/pull/2706.patch"},"body":"**1. varchar data length is more than 2MB, buffer overflow exception (thread local row buffer)**\r\nroot casue*: thread* loaclbuffer was hardcoded with 2MB.\r\nsolution: grow dynamically based on the row size.\r\n\r\n**2. read data from carbon file having one row of varchar data with 150 MB length is very slow.**\r\nroot casue:  At UnsafeDMStore, ensure memory is just incresing by 8KB each time and lot of time malloc and free happens before reaching 150MB. hence very slow performance.\r\nsolution: directly check and allocate the required size.\r\n\r\n**3. Jvm crash when data size is more than 128 MB in unsafe sort step.**\r\nroot cause: unsafeCarbonRowPage is of 128MB, so if data is more than 128MB  for one row, we access block beyond allocated, leading to JVM crash.\r\nsolution: validate the size before access and prompt user to increase unsafe memory. (by carbon property)\r\n\r\nBe sure to do all of the following checklist to help us incorporate \r\nyour contribution quickly and easily:\r\n\r\n - [ ] Any interfaces changed? NA\r\n \r\n - [ ] Any backward compatibility impacted? NA\r\n \r\n - [ ] Document update required? NA\r\n\r\n - [ ] Testing done\r\ndone       \r\n - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/carbondata/issues/comments/420499786","html_url":"https://github.com/apache/carbondata/pull/2706#issuecomment-420499786","issue_url":"https://api.github.com/repos/apache/carbondata/issues/2706","id":420499786,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDQ5OTc4Ng==","user":{"login":"xuchuanyin","id":10445758,"node_id":"MDQ6VXNlcjEwNDQ1NzU4","avatar_url":"https://avatars1.githubusercontent.com/u/10445758?v=4","gravatar_id":"","url":"https://api.github.com/users/xuchuanyin","html_url":"https://github.com/xuchuanyin","followers_url":"https://api.github.com/users/xuchuanyin/followers","following_url":"https://api.github.com/users/xuchuanyin/following{/other_user}","gists_url":"https://api.github.com/users/xuchuanyin/gists{/gist_id}","starred_url":"https://api.github.com/users/xuchuanyin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuchuanyin/subscriptions","organizations_url":"https://api.github.com/users/xuchuanyin/orgs","repos_url":"https://api.github.com/users/xuchuanyin/repos","events_url":"https://api.github.com/users/xuchuanyin/events{/privacy}","received_events_url":"https://api.github.com/users/xuchuanyin/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T03:26:17Z","updated_at":"2018-09-12T03:26:17Z","author_association":"CONTRIBUTOR","body":"@ajantha-bhat \r\nHi, I think the main problem may be that you set the 'rowbuffer' as static which should not be shared among different data loadings.\r\n\r\nBesides, the judgement for increasing rowBuffer size per row per column may decrease data loading performance.\r\n\r\nAs a result, I'd like to implement this in an easier way.\r\n\r\nWe can add a table propery or load option for the size of row buffer. Just keep the previous row-buffer related code as it is. All you need is to change the initial size of the rowbuffer based on the table property or load option.\r\n\r\n@kumarvishal09 @ravipesala How do you think?"}}
{"actor":{"display_login":"xuchuanyin"}}