{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/186184183","pull_request_review_id":117722943,"id":186184183,"diff_hunk":"@@ -147,3 +147,34 @@ An example is shown below:\n |type|This should be \"timed\"|yes|\n |shutoffTime|time at which the firehose should shut down, in ISO8601 format|yes|\n |delegate|firehose to use|yes|\n+\n+#### SqlFirehose\n+\n+SqlFirehoseFactory can be used to ingest events residing in RDBMS. The database connection information is provided as part of the ingestion spec.\n+An example is shown below:\n+\n+```json\n+{\n+    \"type\" : \"sql\",\n+    \"database\": {\n+        \"type\": \"mysql\",\n+        \"connectorConfig\" : {\n+        \"connectURI\" : \"jdbc:mysql://host:port/schema\",\n+        \"user\" : \"user\",\n+        \"password\" : \"password\"\n+        }\n+     },\n+    \"sqls\" : [\"SELECT * FROM table1\", \"SELECT * FROM table2\"]\n+}\n+```\n+\n+\n+|property|description|default|required?|\n+|--------|-----------|-------|---------|\n+|type|This should be \"sql\".||Yes|\n+|database|Specifies the database connection details.||Yes|","path":"docs/content/ingestion/firehose.md","position":29,"original_position":29,"commit_id":"c27b32f813f824427f1c6a345f4f134b970f3531","original_commit_id":"c27b32f813f824427f1c6a345f4f134b970f3531","user":{"login":"a2l007","id":4603202,"avatar_url":"https://avatars0.githubusercontent.com/u/4603202?v=4","gravatar_id":"","url":"https://api.github.com/users/a2l007","html_url":"https://github.com/a2l007","followers_url":"https://api.github.com/users/a2l007/followers","following_url":"https://api.github.com/users/a2l007/following{/other_user}","gists_url":"https://api.github.com/users/a2l007/gists{/gist_id}","starred_url":"https://api.github.com/users/a2l007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a2l007/subscriptions","organizations_url":"https://api.github.com/users/a2l007/orgs","repos_url":"https://api.github.com/users/a2l007/repos","events_url":"https://api.github.com/users/a2l007/events{/privacy}","received_events_url":"https://api.github.com/users/a2l007/received_events","type":"User","site_admin":false},"body":"Added info about database.","created_at":"2018-05-04T18:54:57Z","updated_at":"2018-05-04T18:54:57Z","html_url":"https://github.com/druid-io/druid/pull/5441#discussion_r186184183","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/5441","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/186184183"},"html":{"href":"https://github.com/druid-io/druid/pull/5441#discussion_r186184183"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/5441"}},"in_reply_to_id":185648453},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/5441","id":172057998,"html_url":"https://github.com/druid-io/druid/pull/5441","diff_url":"https://github.com/druid-io/druid/pull/5441.diff","patch_url":"https://github.com/druid-io/druid/pull/5441.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/5441","number":5441,"state":"open","locked":false,"title":"Local indexing from RDBMS","user":{"login":"a2l007","id":4603202,"avatar_url":"https://avatars0.githubusercontent.com/u/4603202?v=4","gravatar_id":"","url":"https://api.github.com/users/a2l007","html_url":"https://github.com/a2l007","followers_url":"https://api.github.com/users/a2l007/followers","following_url":"https://api.github.com/users/a2l007/following{/other_user}","gists_url":"https://api.github.com/users/a2l007/gists{/gist_id}","starred_url":"https://api.github.com/users/a2l007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a2l007/subscriptions","organizations_url":"https://api.github.com/users/a2l007/orgs","repos_url":"https://api.github.com/users/a2l007/repos","events_url":"https://api.github.com/users/a2l007/events{/privacy}","received_events_url":"https://api.github.com/users/a2l007/received_events","type":"User","site_admin":false},"body":"This is based of proposal #5238. \r\nIntroduces changes that support local batch indexing of data residing in RDBMS. A sample ingestion spec snippet would be as follows:\r\n\r\n```\r\n{\r\n      \"type\" : \"index\",\r\n        \"firehose\" : {\r\n        \"type\" : \"sql\",\r\n        \"database\": {\r\n        \"type\": \"mysql\",\r\n        \"connectorConfig\" : {\r\n        \"connectURI\" : \"jdbc:mysql://localhost:3306/sqltest\",\r\n        \"user\" : \"druid\",\r\n        \"password\" : \"diurd\"\r\n        }\r\n        },\r\n        \"sqls\" : [\"SELECT * FROM index_data_a\", \"SELECT * FROM index_data_b\"],\r\n        }\r\n     },\r\n     :\r\n     :\r\n     \"parser\" : {\r\n        \"type\" : \"map\",\r\n        \"parseSpec\" :\r\n        \t:\r\n        \t:\r\n\r\n```\r\n`SqlFirehoseFactory` is introduced which would execute the SQL query, read all the results to a file locally and index from the file. This avoids having to keep a persistent connection to the database until all the inputrows are read.\r\nWhen prefetching is enabled and there are multiple sql queries to index from, the sql results are prefetched in the background similiar to PrefetchableTextFileFirehoseFactory.\r\nUpdated docs and added tests.","created_at":"2018-02-28T20:04:38Z","updated_at":"2018-05-04T18:54:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"17d10bcf02ba1b1afd3612a25899809772451eb1","assignee":null,"assignees":[],"requested_reviewers":[{"login":"nishantmonu51","id":5023786,"avatar_url":"https://avatars3.githubusercontent.com/u/5023786?v=4","gravatar_id":"","url":"https://api.github.com/users/nishantmonu51","html_url":"https://github.com/nishantmonu51","followers_url":"https://api.github.com/users/nishantmonu51/followers","following_url":"https://api.github.com/users/nishantmonu51/following{/other_user}","gists_url":"https://api.github.com/users/nishantmonu51/gists{/gist_id}","starred_url":"https://api.github.com/users/nishantmonu51/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nishantmonu51/subscriptions","organizations_url":"https://api.github.com/users/nishantmonu51/orgs","repos_url":"https://api.github.com/users/nishantmonu51/repos","events_url":"https://api.github.com/users/nishantmonu51/events{/privacy}","received_events_url":"https://api.github.com/users/nishantmonu51/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/5441/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/5441/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/5441/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/c27b32f813f824427f1c6a345f4f134b970f3531","head":{"label":"a2l007:rdbmsindex","ref":"rdbmsindex","sha":"c27b32f813f824427f1c6a345f4f134b970f3531","user":{"login":"a2l007","id":4603202,"avatar_url":"https://avatars0.githubusercontent.com/u/4603202?v=4","gravatar_id":"","url":"https://api.github.com/users/a2l007","html_url":"https://github.com/a2l007","followers_url":"https://api.github.com/users/a2l007/followers","following_url":"https://api.github.com/users/a2l007/following{/other_user}","gists_url":"https://api.github.com/users/a2l007/gists{/gist_id}","starred_url":"https://api.github.com/users/a2l007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a2l007/subscriptions","organizations_url":"https://api.github.com/users/a2l007/orgs","repos_url":"https://api.github.com/users/a2l007/repos","events_url":"https://api.github.com/users/a2l007/events{/privacy}","received_events_url":"https://api.github.com/users/a2l007/received_events","type":"User","site_admin":false},"repo":{"id":97611468,"name":"druid","full_name":"a2l007/druid","owner":{"login":"a2l007","id":4603202,"avatar_url":"https://avatars0.githubusercontent.com/u/4603202?v=4","gravatar_id":"","url":"https://api.github.com/users/a2l007","html_url":"https://github.com/a2l007","followers_url":"https://api.github.com/users/a2l007/followers","following_url":"https://api.github.com/users/a2l007/following{/other_user}","gists_url":"https://api.github.com/users/a2l007/gists{/gist_id}","starred_url":"https://api.github.com/users/a2l007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a2l007/subscriptions","organizations_url":"https://api.github.com/users/a2l007/orgs","repos_url":"https://api.github.com/users/a2l007/repos","events_url":"https://api.github.com/users/a2l007/events{/privacy}","received_events_url":"https://api.github.com/users/a2l007/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/a2l007/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":true,"url":"https://api.github.com/repos/a2l007/druid","forks_url":"https://api.github.com/repos/a2l007/druid/forks","keys_url":"https://api.github.com/repos/a2l007/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/a2l007/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/a2l007/druid/teams","hooks_url":"https://api.github.com/repos/a2l007/druid/hooks","issue_events_url":"https://api.github.com/repos/a2l007/druid/issues/events{/number}","events_url":"https://api.github.com/repos/a2l007/druid/events","assignees_url":"https://api.github.com/repos/a2l007/druid/assignees{/user}","branches_url":"https://api.github.com/repos/a2l007/druid/branches{/branch}","tags_url":"https://api.github.com/repos/a2l007/druid/tags","blobs_url":"https://api.github.com/repos/a2l007/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/a2l007/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/a2l007/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/a2l007/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/a2l007/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/a2l007/druid/languages","stargazers_url":"https://api.github.com/repos/a2l007/druid/stargazers","contributors_url":"https://api.github.com/repos/a2l007/druid/contributors","subscribers_url":"https://api.github.com/repos/a2l007/druid/subscribers","subscription_url":"https://api.github.com/repos/a2l007/druid/subscription","commits_url":"https://api.github.com/repos/a2l007/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/a2l007/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/a2l007/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/a2l007/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/a2l007/druid/contents/{+path}","compare_url":"https://api.github.com/repos/a2l007/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/a2l007/druid/merges","archive_url":"https://api.github.com/repos/a2l007/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/a2l007/druid/downloads","issues_url":"https://api.github.com/repos/a2l007/druid/issues{/number}","pulls_url":"https://api.github.com/repos/a2l007/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/a2l007/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/a2l007/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/a2l007/druid/labels{/name}","releases_url":"https://api.github.com/repos/a2l007/druid/releases{/id}","deployments_url":"https://api.github.com/repos/a2l007/druid/deployments","created_at":"2017-07-18T14:57:15Z","updated_at":"2018-04-30T14:03:36Z","pushed_at":"2018-04-30T14:03:22Z","git_url":"git://github.com/a2l007/druid.git","ssh_url":"git@github.com:a2l007/druid.git","clone_url":"https://github.com/a2l007/druid.git","svn_url":"https://github.com/a2l007/druid","homepage":"http://druid.io","size":135376,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"ca3f833426231623cd7eb47a6eff9bd2886b6a87","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars0.githubusercontent.com/u/4203474?v=4","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars0.githubusercontent.com/u/4203474?v=4","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","deployments_url":"https://api.github.com/repos/druid-io/druid/deployments","created_at":"2012-10-23T19:08:07Z","updated_at":"2018-05-04T17:55:49Z","pushed_at":"2018-05-04T17:33:14Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":137429,"stargazers_count":6448,"watchers_count":6448,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1582,"mirror_url":null,"archived":false,"open_issues_count":1007,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":1582,"open_issues":1007,"watchers":6448,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/5441"},"html":{"href":"https://github.com/druid-io/druid/pull/5441"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/5441"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/5441/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/5441/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/5441/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/c27b32f813f824427f1c6a345f4f134b970f3531"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"a2l007"}}