{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/51801490","id":51801490,"diff_hunk":"@@ -0,0 +1,46 @@\n+---\n+layout: doc_page\n+---\n+\n+# Loading Data\n+\n+## Choosing an ingestion method\n+\n+Druid supports streaming (real-time) and file-based (batch) ingestion methods. The most\n+popular configurations are:\n+\n+- [Files](batch-ingestion.html) - Load data from HDFS, S3, local files, or any supported Hadoop\n+filesystem in batches. We recommend this method if your dataset is already in flat files.\n+\n+- [Stream push](stream-ingestion.html#stream-push) - Push a data stream into Druid in real-time\n+using [Tranquility](http://github.com/druid-io/tranquility), a client library for sending streams\n+to Druid. We recommend this method if your dataset originates in a streaming system like Kafka,\n+Storm, Spark Streaming, or your own system.\n+\n+- [Stream pull](stream-ingestion.html#stream-pull) - Pull a data stream directly from an external\n+data source into Druid using Realtime Nodes.\n+\n+## Getting started\n+\n+The easiest ways to get started with loading your own data are the three included tutorials.\n+\n+- [Files-based tutorial](tutorial-batch.html) showing you how to load files from your local disk.\n+- [Streams-based tutorial](tutorial-streams.html) showing you how to push data over HTTP.\n+- [Kafka-based tutorial](tutorial-kafka.html) showing you how to load data from Kafka.\n+\n+## Hybrid batch/streaming\n+\n+You can combine batch and streaming methods in a hybrid batch/streaming architecture. In a hybrid\n+architecture, you use a streaming method to do initial ingestion, and then periodically re-ingest\n+older data in batch mode (typically every few hours, or nightly).\n+\n+Hybrid architectures are simple with Druid, since batch loaded data for a particular time range\n+automatically replaces streaming loaded data for that same time range. All Druid queries seamlessly","path":"docs/content/tutorials/ingestion.md","position":38,"original_position":38,"commit_id":"11e0fee7eb52f0fb55afdb4f53ee063ac09b3aac","original_commit_id":"9963166cabc5f7aaf7338c2d929db1c04abb6dcc","user":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"body":"it can be hard to describe the  hybrid approach without good docs on how the segment timeline works... I should add that in another PR","created_at":"2016-02-03T22:50:26Z","updated_at":"2016-02-03T22:50:26Z","html_url":"https://github.com/druid-io/druid/pull/2216#discussion_r51801490","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/2216","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/51801490"},"html":{"href":"https://github.com/druid-io/druid/pull/2216#discussion_r51801490"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2216"}}},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/2216","id":55278854,"html_url":"https://github.com/druid-io/druid/pull/2216","diff_url":"https://github.com/druid-io/druid/pull/2216.diff","patch_url":"https://github.com/druid-io/druid/pull/2216.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/2216","number":2216,"state":"open","locked":false,"title":"Completely rework the Druid getting started process","user":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"body":"- rejigger the distribution packaging to make more sense\r\n- new quickstart tutorial\r\n- new load batch data tutorial\r\n- new load streaming data tutorial\r\n- new load kafka tutorial\r\n- new clustering tutorial\r\n- new streaming ingestion overview page\r\n- new stream push page\r\n- new stream pull page\r\n- lots of new information about using Druid and Tranquility\r\n- added a new query optimization page\r\n- added a new caching info page\r\n\r\nThis PR depends on some CSS changes to Druid docs which are coming in a separate PR. The updated pages will not render correctly without those changes.\r\n\r\nThis will rework the Druid getting started process to be very similar to Imply's recommended getting started process, which was mostly written by @gianm . The packaging of Druid will also be similar to what Imply is doing.\r\n","created_at":"2016-01-06T23:13:49Z","updated_at":"2016-02-03T22:50:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a5472598cb465a034817c242f3476c494ef15cf5","assignee":null,"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/14","html_url":"https://github.com/druid-io/druid/milestones/0.9.0","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/14/labels","id":1092727,"number":14,"title":"0.9.0","description":"","creator":{"login":"xvrl","id":815147,"avatar_url":"https://avatars.githubusercontent.com/u/815147?v=3","gravatar_id":"","url":"https://api.github.com/users/xvrl","html_url":"https://github.com/xvrl","followers_url":"https://api.github.com/users/xvrl/followers","following_url":"https://api.github.com/users/xvrl/following{/other_user}","gists_url":"https://api.github.com/users/xvrl/gists{/gist_id}","starred_url":"https://api.github.com/users/xvrl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xvrl/subscriptions","organizations_url":"https://api.github.com/users/xvrl/orgs","repos_url":"https://api.github.com/users/xvrl/repos","events_url":"https://api.github.com/users/xvrl/events{/privacy}","received_events_url":"https://api.github.com/users/xvrl/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":106,"state":"open","created_at":"2015-05-01T16:53:57Z","updated_at":"2016-02-03T22:29:23Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/2216/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/2216/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/2216/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/11e0fee7eb52f0fb55afdb4f53ee063ac09b3aac","head":{"label":"druid-io:new-tutorials","ref":"new-tutorials","sha":"11e0fee7eb52f0fb55afdb4f53ee063ac09b3aac","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","deployments_url":"https://api.github.com/repos/druid-io/druid/deployments","created_at":"2012-10-23T19:08:07Z","updated_at":"2016-02-03T17:09:08Z","pushed_at":"2016-02-03T22:45:03Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":125366,"stargazers_count":2411,"watchers_count":2411,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":571,"mirror_url":null,"open_issues_count":302,"forks":571,"open_issues":302,"watchers":2411,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"8af67e76d3c0b2d18644084779e7033246775e5e","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","deployments_url":"https://api.github.com/repos/druid-io/druid/deployments","created_at":"2012-10-23T19:08:07Z","updated_at":"2016-02-03T17:09:08Z","pushed_at":"2016-02-03T22:45:03Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":125366,"stargazers_count":2411,"watchers_count":2411,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":571,"mirror_url":null,"open_issues_count":302,"forks":571,"open_issues":302,"watchers":2411,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2216"},"html":{"href":"https://github.com/druid-io/druid/pull/2216"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/2216"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/2216/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2216/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/2216/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/11e0fee7eb52f0fb55afdb4f53ee063ac09b3aac"}}}}