{"action":"created","comment":{"url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/comments/64105176","id":64105176,"diff_hunk":"@@ -1,22 +1,26 @@\n # CarbonData\n-CarbonData is a fully indexed columnar and hadoop native datastore designed for fast analytics and multi-dimension query on big data.In customer benchmarks, CarbonData has been shown to manage Petabyte of data running on extraordinarily low-cost hardware and answers queries around 10 times faster than the current open source solutions (column-oriented SQL on Hadoop data-stores). \n+CarbonData is a new Apache Hadoop native file format for faster \n+interactive query using advanced columnar storage, index, compression \n+and encoding techniques to improve computing efficiency, in turn it will \n+help speedup queries an order of magnitude faster over PetaBytes of data. \n \n ### Why CarbonData\n-The CaronData file format provides a highly efficient way to store structured data,it was designed to overcome limitations of the other Hadoop file formats. \n-* CarbonData stores data along with index,the index is not stored separately but the carbondata itself is the index.Indexing helps to accelerate query performance and reduces the I/O scans to the minimum and reduces the CPU required for computation on the data.\n-* CarbonData is designed to support very efficient compression and global encoding schemes,CarbonData can support actionable compression across data files for query engines to perform all processing on compressed/encoded data without having to convert the data. This improves the processing speed, especially for analytical queries. The data can be converted back to the user readable format just before returning the results to the user.\n-* CarbonData is designed with various types of usecases in mind and provides flexible storage options. Data can be stored in completely columnar or row based formats or even in a mix of columnar and row based hybrid format.\n-* CarbonData is designed to integrate into big data ecosystem, leverages Apache Hadoop,Apache Spark etc. for distributed query processing.\n+Based on the below requirements, we investigated existing file formats in the Hadoop eco-system, but we could not find a suitable solution that satisfying requirements all at the same time, so we start designing CarbonData. \n+* Requirement1:Support big scan & only fetch a few columns \n+* Requirement2:Support primary key lookup response in sub-second. \n+* Requirement3:Support interactive OLAP-style query over big data which involve many filters in a query, this type of workload should response in seconds. \n+* Requirement4:Support fast individual record extraction which fetch all columns of the record. \n+* Requirement5:Support HDFS so that customer can leverage existing Hadoop cluster. \n \n ### Features\n-* Fast analytic queries in seconds using built-in index, optimized for interactive OLAP-style query, high through put scan query, low latency point query. \n-* Fast data loading speed and support incremental load in period of minutes. 5 mins near realtime loading should be supported\n-* Support concurrent query.\n-* Support time based data retention. \n-* Supports SQL based query interface.\n+CarbonData file format is a columnar store in HDFS, it has many features that a modern columnar format has, such as splittable, compression schema ,complex data type etc. And CarbonData has following unique features:\n+* Stores data along with index: it can significantly accelerate query performance and reduces the I/O scans and CPU resources, where there are filters in the query.  CarbonData index is consisted of multiple level, a processing framework can leverage this index to reduce the task it needs to schedule and process, and it can also do skip scan in more finer grain unit (called blocklet) in task side scanning instead of scanning the whole file. \n+* Operable encoded data :Through supporting efficient compression and global encoding schemes, can query on compressed/encoded data, the data can be converted just before returning the results to the users, which is \"late materialized\". \n+* Column group: Allow multiple columns form a column group to store as row format, thus cost of column reconstructing is reduced. ","path":"README.md","position":30,"original_position":30,"commit_id":"2740edbf65a608825b1f435a3ef22903c2859198","original_commit_id":"2740edbf65a608825b1f435a3ef22903c2859198","user":{"login":"Vimal-Das","id":11293485,"avatar_url":"https://avatars.githubusercontent.com/u/11293485?v=3","gravatar_id":"","url":"https://api.github.com/users/Vimal-Das","html_url":"https://github.com/Vimal-Das","followers_url":"https://api.github.com/users/Vimal-Das/followers","following_url":"https://api.github.com/users/Vimal-Das/following{/other_user}","gists_url":"https://api.github.com/users/Vimal-Das/gists{/gist_id}","starred_url":"https://api.github.com/users/Vimal-Das/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vimal-Das/subscriptions","organizations_url":"https://api.github.com/users/Vimal-Das/orgs","repos_url":"https://api.github.com/users/Vimal-Das/repos","events_url":"https://api.github.com/users/Vimal-Das/events{/privacy}","received_events_url":"https://api.github.com/users/Vimal-Das/received_events","type":"User","site_admin":false},"body":"Allow multiple columns to form a column group that would be stored as row format. This reduces the row reconstruction cost at query time.","created_at":"2016-05-20T20:48:02Z","updated_at":"2016-05-20T20:48:02Z","html_url":"https://github.com/HuaweiBigData/carbondata/pull/465#discussion_r64105176","pull_request_url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465","_links":{"self":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/comments/64105176"},"html":{"href":"https://github.com/HuaweiBigData/carbondata/pull/465#discussion_r64105176"},"pull_request":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465"}}},"pull_request":{"url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465","id":70914248,"html_url":"https://github.com/HuaweiBigData/carbondata/pull/465","diff_url":"https://github.com/HuaweiBigData/carbondata/pull/465.diff","patch_url":"https://github.com/HuaweiBigData/carbondata/pull/465.patch","issue_url":"https://api.github.com/repos/HuaweiBigData/carbondata/issues/465","number":465,"state":"open","locked":false,"title":"Update README for refining some description","user":{"login":"chenliang613","id":8075709,"avatar_url":"https://avatars.githubusercontent.com/u/8075709?v=3","gravatar_id":"","url":"https://api.github.com/users/chenliang613","html_url":"https://github.com/chenliang613","followers_url":"https://api.github.com/users/chenliang613/followers","following_url":"https://api.github.com/users/chenliang613/following{/other_user}","gists_url":"https://api.github.com/users/chenliang613/gists{/gist_id}","starred_url":"https://api.github.com/users/chenliang613/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenliang613/subscriptions","organizations_url":"https://api.github.com/users/chenliang613/orgs","repos_url":"https://api.github.com/users/chenliang613/repos","events_url":"https://api.github.com/users/chenliang613/events{/privacy}","received_events_url":"https://api.github.com/users/chenliang613/received_events","type":"User","site_admin":false},"body":"Update README for refining some description","created_at":"2016-05-20T20:40:21Z","updated_at":"2016-05-20T20:48:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b6dc6003dbddc6bee0ab79cf8e5ee37b95836a79","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465/commits","review_comments_url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465/comments","review_comment_url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/comments{/number}","comments_url":"https://api.github.com/repos/HuaweiBigData/carbondata/issues/465/comments","statuses_url":"https://api.github.com/repos/HuaweiBigData/carbondata/statuses/2740edbf65a608825b1f435a3ef22903c2859198","head":{"label":"chenliang613:readme-5.20","ref":"readme-5.20","sha":"2740edbf65a608825b1f435a3ef22903c2859198","user":{"login":"chenliang613","id":8075709,"avatar_url":"https://avatars.githubusercontent.com/u/8075709?v=3","gravatar_id":"","url":"https://api.github.com/users/chenliang613","html_url":"https://github.com/chenliang613","followers_url":"https://api.github.com/users/chenliang613/followers","following_url":"https://api.github.com/users/chenliang613/following{/other_user}","gists_url":"https://api.github.com/users/chenliang613/gists{/gist_id}","starred_url":"https://api.github.com/users/chenliang613/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenliang613/subscriptions","organizations_url":"https://api.github.com/users/chenliang613/orgs","repos_url":"https://api.github.com/users/chenliang613/repos","events_url":"https://api.github.com/users/chenliang613/events{/privacy}","received_events_url":"https://api.github.com/users/chenliang613/received_events","type":"User","site_admin":false},"repo":{"id":58723239,"name":"carbondata","full_name":"chenliang613/carbondata","owner":{"login":"chenliang613","id":8075709,"avatar_url":"https://avatars.githubusercontent.com/u/8075709?v=3","gravatar_id":"","url":"https://api.github.com/users/chenliang613","html_url":"https://github.com/chenliang613","followers_url":"https://api.github.com/users/chenliang613/followers","following_url":"https://api.github.com/users/chenliang613/following{/other_user}","gists_url":"https://api.github.com/users/chenliang613/gists{/gist_id}","starred_url":"https://api.github.com/users/chenliang613/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenliang613/subscriptions","organizations_url":"https://api.github.com/users/chenliang613/orgs","repos_url":"https://api.github.com/users/chenliang613/repos","events_url":"https://api.github.com/users/chenliang613/events{/privacy}","received_events_url":"https://api.github.com/users/chenliang613/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/chenliang613/carbondata","description":"CarbonData is a new hadoop native file format for faster interactive query","fork":true,"url":"https://api.github.com/repos/chenliang613/carbondata","forks_url":"https://api.github.com/repos/chenliang613/carbondata/forks","keys_url":"https://api.github.com/repos/chenliang613/carbondata/keys{/key_id}","collaborators_url":"https://api.github.com/repos/chenliang613/carbondata/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/chenliang613/carbondata/teams","hooks_url":"https://api.github.com/repos/chenliang613/carbondata/hooks","issue_events_url":"https://api.github.com/repos/chenliang613/carbondata/issues/events{/number}","events_url":"https://api.github.com/repos/chenliang613/carbondata/events","assignees_url":"https://api.github.com/repos/chenliang613/carbondata/assignees{/user}","branches_url":"https://api.github.com/repos/chenliang613/carbondata/branches{/branch}","tags_url":"https://api.github.com/repos/chenliang613/carbondata/tags","blobs_url":"https://api.github.com/repos/chenliang613/carbondata/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/chenliang613/carbondata/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/chenliang613/carbondata/git/refs{/sha}","trees_url":"https://api.github.com/repos/chenliang613/carbondata/git/trees{/sha}","statuses_url":"https://api.github.com/repos/chenliang613/carbondata/statuses/{sha}","languages_url":"https://api.github.com/repos/chenliang613/carbondata/languages","stargazers_url":"https://api.github.com/repos/chenliang613/carbondata/stargazers","contributors_url":"https://api.github.com/repos/chenliang613/carbondata/contributors","subscribers_url":"https://api.github.com/repos/chenliang613/carbondata/subscribers","subscription_url":"https://api.github.com/repos/chenliang613/carbondata/subscription","commits_url":"https://api.github.com/repos/chenliang613/carbondata/commits{/sha}","git_commits_url":"https://api.github.com/repos/chenliang613/carbondata/git/commits{/sha}","comments_url":"https://api.github.com/repos/chenliang613/carbondata/comments{/number}","issue_comment_url":"https://api.github.com/repos/chenliang613/carbondata/issues/comments{/number}","contents_url":"https://api.github.com/repos/chenliang613/carbondata/contents/{+path}","compare_url":"https://api.github.com/repos/chenliang613/carbondata/compare/{base}...{head}","merges_url":"https://api.github.com/repos/chenliang613/carbondata/merges","archive_url":"https://api.github.com/repos/chenliang613/carbondata/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/chenliang613/carbondata/downloads","issues_url":"https://api.github.com/repos/chenliang613/carbondata/issues{/number}","pulls_url":"https://api.github.com/repos/chenliang613/carbondata/pulls{/number}","milestones_url":"https://api.github.com/repos/chenliang613/carbondata/milestones{/number}","notifications_url":"https://api.github.com/repos/chenliang613/carbondata/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/chenliang613/carbondata/labels{/name}","releases_url":"https://api.github.com/repos/chenliang613/carbondata/releases{/id}","deployments_url":"https://api.github.com/repos/chenliang613/carbondata/deployments","created_at":"2016-05-13T09:25:08Z","updated_at":"2016-05-13T09:25:12Z","pushed_at":"2016-05-20T20:38:43Z","git_url":"git://github.com/chenliang613/carbondata.git","ssh_url":"git@github.com:chenliang613/carbondata.git","clone_url":"https://github.com/chenliang613/carbondata.git","svn_url":"https://github.com/chenliang613/carbondata","homepage":"","size":31316,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"HuaweiBigData:master","ref":"master","sha":"68981568bf167ccc15c320fda7badb27777d7646","user":{"login":"HuaweiBigData","id":13159946,"avatar_url":"https://avatars.githubusercontent.com/u/13159946?v=3","gravatar_id":"","url":"https://api.github.com/users/HuaweiBigData","html_url":"https://github.com/HuaweiBigData","followers_url":"https://api.github.com/users/HuaweiBigData/followers","following_url":"https://api.github.com/users/HuaweiBigData/following{/other_user}","gists_url":"https://api.github.com/users/HuaweiBigData/gists{/gist_id}","starred_url":"https://api.github.com/users/HuaweiBigData/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HuaweiBigData/subscriptions","organizations_url":"https://api.github.com/users/HuaweiBigData/orgs","repos_url":"https://api.github.com/users/HuaweiBigData/repos","events_url":"https://api.github.com/users/HuaweiBigData/events{/privacy}","received_events_url":"https://api.github.com/users/HuaweiBigData/received_events","type":"Organization","site_admin":false},"repo":{"id":52943921,"name":"carbondata","full_name":"HuaweiBigData/carbondata","owner":{"login":"HuaweiBigData","id":13159946,"avatar_url":"https://avatars.githubusercontent.com/u/13159946?v=3","gravatar_id":"","url":"https://api.github.com/users/HuaweiBigData","html_url":"https://github.com/HuaweiBigData","followers_url":"https://api.github.com/users/HuaweiBigData/followers","following_url":"https://api.github.com/users/HuaweiBigData/following{/other_user}","gists_url":"https://api.github.com/users/HuaweiBigData/gists{/gist_id}","starred_url":"https://api.github.com/users/HuaweiBigData/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HuaweiBigData/subscriptions","organizations_url":"https://api.github.com/users/HuaweiBigData/orgs","repos_url":"https://api.github.com/users/HuaweiBigData/repos","events_url":"https://api.github.com/users/HuaweiBigData/events{/privacy}","received_events_url":"https://api.github.com/users/HuaweiBigData/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/HuaweiBigData/carbondata","description":"CarbonData is a new hadoop native file format for faster interactive query","fork":false,"url":"https://api.github.com/repos/HuaweiBigData/carbondata","forks_url":"https://api.github.com/repos/HuaweiBigData/carbondata/forks","keys_url":"https://api.github.com/repos/HuaweiBigData/carbondata/keys{/key_id}","collaborators_url":"https://api.github.com/repos/HuaweiBigData/carbondata/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/HuaweiBigData/carbondata/teams","hooks_url":"https://api.github.com/repos/HuaweiBigData/carbondata/hooks","issue_events_url":"https://api.github.com/repos/HuaweiBigData/carbondata/issues/events{/number}","events_url":"https://api.github.com/repos/HuaweiBigData/carbondata/events","assignees_url":"https://api.github.com/repos/HuaweiBigData/carbondata/assignees{/user}","branches_url":"https://api.github.com/repos/HuaweiBigData/carbondata/branches{/branch}","tags_url":"https://api.github.com/repos/HuaweiBigData/carbondata/tags","blobs_url":"https://api.github.com/repos/HuaweiBigData/carbondata/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/HuaweiBigData/carbondata/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/HuaweiBigData/carbondata/git/refs{/sha}","trees_url":"https://api.github.com/repos/HuaweiBigData/carbondata/git/trees{/sha}","statuses_url":"https://api.github.com/repos/HuaweiBigData/carbondata/statuses/{sha}","languages_url":"https://api.github.com/repos/HuaweiBigData/carbondata/languages","stargazers_url":"https://api.github.com/repos/HuaweiBigData/carbondata/stargazers","contributors_url":"https://api.github.com/repos/HuaweiBigData/carbondata/contributors","subscribers_url":"https://api.github.com/repos/HuaweiBigData/carbondata/subscribers","subscription_url":"https://api.github.com/repos/HuaweiBigData/carbondata/subscription","commits_url":"https://api.github.com/repos/HuaweiBigData/carbondata/commits{/sha}","git_commits_url":"https://api.github.com/repos/HuaweiBigData/carbondata/git/commits{/sha}","comments_url":"https://api.github.com/repos/HuaweiBigData/carbondata/comments{/number}","issue_comment_url":"https://api.github.com/repos/HuaweiBigData/carbondata/issues/comments{/number}","contents_url":"https://api.github.com/repos/HuaweiBigData/carbondata/contents/{+path}","compare_url":"https://api.github.com/repos/HuaweiBigData/carbondata/compare/{base}...{head}","merges_url":"https://api.github.com/repos/HuaweiBigData/carbondata/merges","archive_url":"https://api.github.com/repos/HuaweiBigData/carbondata/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/HuaweiBigData/carbondata/downloads","issues_url":"https://api.github.com/repos/HuaweiBigData/carbondata/issues{/number}","pulls_url":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls{/number}","milestones_url":"https://api.github.com/repos/HuaweiBigData/carbondata/milestones{/number}","notifications_url":"https://api.github.com/repos/HuaweiBigData/carbondata/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/HuaweiBigData/carbondata/labels{/name}","releases_url":"https://api.github.com/repos/HuaweiBigData/carbondata/releases{/id}","deployments_url":"https://api.github.com/repos/HuaweiBigData/carbondata/deployments","created_at":"2016-03-02T07:44:41Z","updated_at":"2016-05-20T11:56:53Z","pushed_at":"2016-05-20T20:40:21Z","git_url":"git://github.com/HuaweiBigData/carbondata.git","ssh_url":"git@github.com:HuaweiBigData/carbondata.git","clone_url":"https://github.com/HuaweiBigData/carbondata.git","svn_url":"https://github.com/HuaweiBigData/carbondata","homepage":"","size":61240,"stargazers_count":25,"watchers_count":25,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":21,"mirror_url":null,"open_issues_count":108,"forks":21,"open_issues":108,"watchers":25,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465"},"html":{"href":"https://github.com/HuaweiBigData/carbondata/pull/465"},"issue":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/issues/465"},"comments":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/issues/465/comments"},"review_comments":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465/comments"},"review_comment":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/pulls/465/commits"},"statuses":{"href":"https://api.github.com/repos/HuaweiBigData/carbondata/statuses/2740edbf65a608825b1f435a3ef22903c2859198"}}}}
{"actor":{"display_login":"Vimal-Das"}}