{"action":"created","issue":{"url":"https://api.github.com/repos/apache/carbondata/issues/2654","repository_url":"https://api.github.com/repos/apache/carbondata","labels_url":"https://api.github.com/repos/apache/carbondata/issues/2654/labels{/name}","comments_url":"https://api.github.com/repos/apache/carbondata/issues/2654/comments","events_url":"https://api.github.com/repos/apache/carbondata/issues/2654/events","html_url":"https://github.com/apache/carbondata/pull/2654","id":353493209,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEwNTMwODg5","number":2654,"title":"[CARBONDATA-2896] Adaptive Encoding for Primitive data types","user":{"login":"dhatchayani","id":21074817,"node_id":"MDQ6VXNlcjIxMDc0ODE3","avatar_url":"https://avatars2.githubusercontent.com/u/21074817?v=4","gravatar_id":"","url":"https://api.github.com/users/dhatchayani","html_url":"https://github.com/dhatchayani","followers_url":"https://api.github.com/users/dhatchayani/followers","following_url":"https://api.github.com/users/dhatchayani/following{/other_user}","gists_url":"https://api.github.com/users/dhatchayani/gists{/gist_id}","starred_url":"https://api.github.com/users/dhatchayani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhatchayani/subscriptions","organizations_url":"https://api.github.com/users/dhatchayani/orgs","repos_url":"https://api.github.com/users/dhatchayani/repos","events_url":"https://api.github.com/users/dhatchayani/events{/privacy}","received_events_url":"https://api.github.com/users/dhatchayani/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":91,"created_at":"2018-08-23T18:07:17Z","updated_at":"2018-09-04T05:53:27Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/carbondata/pulls/2654","html_url":"https://github.com/apache/carbondata/pull/2654","diff_url":"https://github.com/apache/carbondata/pull/2654.diff","patch_url":"https://github.com/apache/carbondata/pull/2654.patch"},"body":"Currently Encoding and Decoding is present only for Dictionary, Measure Columns, but for no dictionary Primitive types encoding is absent.\r\n\r\nWith this PR, we support encoding on the no dictionary primitive data types also.\r\n\r\n[CARBONDATA-2896] Adaptive Encoding for Primitive data types\r\n\r\nLoading configurations and settings\r\n(1) Parse data as like that of measure, so change in FieldEncoderFactory to take up measure flow\r\n(2) While creating loading configurations, no dictionary, sort columns should be taken care in all the needed flows\r\n\r\nSort rows preparation\r\n(1) Prepare the row to be sorted with original data for no dictionary columns\r\n(2) Use data type based comparators for the no dictionary sort columns in all the flows like Intermediate Sort, Final sort, Unsafe sort\r\n(3) Handle read write of row with no dictionary primitive data types to intermediate files and in the final file merger, as we will be reading and writing as original data\r\n(4) Get the no dictionary sort data types from the load configurations what we set in LOAD step\r\n\r\n\r\nAdding to Column page and apply adaptive encoding\r\n(1) Add the no dictionary primitive datatypes data as original data\r\n(2) Apply adaptive encoding to the page\r\n(3) Reuse the adaptive encoding techniques existing for measure column\r\n\r\nWriting inverted index to adaptive encoded page\r\n(1) Prepare in the inverted inverted list based on the datatype based comparison\r\n(2) Apply RLE on the inverted index\r\n(3) Write the inverted index to the encoded page\r\n\r\nCreate decoder while querying\r\n(1) Create proper decoder for the no dictionary column pages\r\n(2) Uncompress the column page and also the inverted index\r\n\r\n\r\nFilter flow changes\r\n(1) FilterValues will be in bytes, so convert the data to bytes for comparison\r\n(2) Change the isScanRequired to compare min/max values based on the data type\r\n\r\nFill output row in case of queries\r\n(1) Change the noDictionaryKeys to Object, now it can be datatypes based data for no dictionary primitive data types\r\n\r\nBloom filter changes\r\n(1) Change bloom filter load\r\n(2) While rebuilding the data map, the load expects the data to original data. Therefore a conversion is used\r\n(3) Fill the no dictionary primitive data as original data\r\n\r\nCompaction Changes\r\nCompaction will get the rows from the result collectors. But the result collectors will give bytes as no dictionary columns.\r\nSo a conversion is needed to convert the bytes to original data based on the data type.\r\n\r\n - [ ] Any interfaces changed?\r\n \r\n - [ ] Any backward compatibility impacted?\r\n \r\n - [ ] Document update required?\r\n\r\n - [ ] Testing done\r\n       \r\n - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/carbondata/issues/comments/418249116","html_url":"https://github.com/apache/carbondata/pull/2654#issuecomment-418249116","issue_url":"https://api.github.com/repos/apache/carbondata/issues/2654","id":418249116,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODI0OTExNg==","user":{"login":"CarbonDataQA","id":24402133,"node_id":"MDQ6VXNlcjI0NDAyMTMz","avatar_url":"https://avatars0.githubusercontent.com/u/24402133?v=4","gravatar_id":"","url":"https://api.github.com/users/CarbonDataQA","html_url":"https://github.com/CarbonDataQA","followers_url":"https://api.github.com/users/CarbonDataQA/followers","following_url":"https://api.github.com/users/CarbonDataQA/following{/other_user}","gists_url":"https://api.github.com/users/CarbonDataQA/gists{/gist_id}","starred_url":"https://api.github.com/users/CarbonDataQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CarbonDataQA/subscriptions","organizations_url":"https://api.github.com/users/CarbonDataQA/orgs","repos_url":"https://api.github.com/users/CarbonDataQA/repos","events_url":"https://api.github.com/users/CarbonDataQA/events{/privacy}","received_events_url":"https://api.github.com/users/CarbonDataQA/received_events","type":"User","site_admin":false},"created_at":"2018-09-04T05:53:27Z","updated_at":"2018-09-04T05:53:27Z","author_association":"NONE","body":"Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8280/\n"}}
{"actor":{"display_login":"CarbonDataQA"}}