{"action":"created","comment":{"url":"https://api.github.com/repos/apache/storm/pulls/comments/49000550","id":49000550,"diff_hunk":"@@ -0,0 +1,55 @@\n+---\n+title: The Internals of Storm SQL\n+layout: documentation\n+documentation: true\n+---\n+\n+This page describes the design and the implementation of the Storm SQL integration.\n+\n+## Overview\n+\n+SQL is a well-adopted yet complicated standard. Several projects including Drill, Hive, Phoenix and Spark have invested significantly in their SQL layers. One of the main design goal of StormSQL is to leverage the existing investments for these projects. StormSQL leverages [Apache Calcite](///calcite.apache.org) to implement the SQL standard. StormSQL focuses on compiling the SQL statements to Storm / Trident topologies so that they can be executed in Storm clusters.\n+\n+Figure 1 describes the workflow of executing a SQL query in StormSQL. First, users provide a sequence of SQL statements. StormSQL parses the SQL statements and translates them to a Calcite logical plan. A logical plan consists of a sequence of SQL logical operators that describe how the query should be executed irrespective to the underlying execution engines. Some examples of logical operators include `TableScan`, `Filter`, `Projection` and `GroupBy`.\n+\n+<div align=\"center\">\n+<img title=\"Workflow of StormSQL\" src=\"images/storm-sql-internal-workflow.png\" style=\"max-width: 80rem\"/>\n+\n+<p>Figure 1: Workflow of StormSQL.</p>\n+</div>\n+\n+The next step is to compile the logical execution plan down to a physical execution plan. A physical plan consists of physical operators that describes how to execute the SQL query in *StormSQL*. Physical operators such as `Filter`, `Projection`, and `GroupBy` are directly mapped to operations in Trident topologies. StormSQL also compiles expressions in the SQL statements into Java byte codes and plugs them into the Trident topologies.\n+\n+Finally, StormSQL packages both the Java byte codes and the topology into a JAR and submits it to the Storm cluster. Storm schedules and executes the JAR in the same way of it executes other Storm topologies.\n+\n+The follow code blocks show an example query that filters and projects results from a Kafka stream.\n+\n+```\n+CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' ...\n+\n+CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 'kafka://localhost:2181/brokers?topic=large_orders' ...\n+\n+INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM ORDERS WHERE UNIT_PRICE * QUANTITY > 50\n+```\n+\n+The first two SQL statements define the inputs and outputs of external data. Figure 2 describes the processes of how StormSQL takes the last `SELECT` query and compiles it down to Trident topology.\n+\n+<div align=\"center\">\n+<img title=\"Compiling the example query to Trident topology\" src=\"images/storm-sql-internal-example.png\" style=\"max-width: 80rem\"/>\n+\n+<p>Figure 2: Compiling the example query to Trident topology.</p>\n+</div>\n+\n+\n+## Constraints of querying streaming tables\n+\n+There are several constraints when querying tables that represent a real-time data stream:\n+\n+* The `ORDER BY` clause cannot be applied to a stream.\n+* There is at least one monotonic field in the `GROUP BY` clauses to allow StormSQL bounds the size of the buffer.\n+\n+For more information please refer to https://calcite.incubator.apache.org/docs/stream.html.\n+\n+## Dependency\n+\n+StormSQL does not ship the dependency of the external data sources in the packaged JAR. The users have to provide the dependency in the `extlib` directory of the worker node.","path":"documentation/storm-sql-internal.md","position":55,"original_position":55,"commit_id":"99f4494439ef4ab9c8a056ea5742221da08065c8","original_commit_id":"99f4494439ef4ab9c8a056ea5742221da08065c8","user":{"login":"haohui","id":1192828,"avatar_url":"https://avatars.githubusercontent.com/u/1192828?v=3","gravatar_id":"","url":"https://api.github.com/users/haohui","html_url":"https://github.com/haohui","followers_url":"https://api.github.com/users/haohui/followers","following_url":"https://api.github.com/users/haohui/following{/other_user}","gists_url":"https://api.github.com/users/haohui/gists{/gist_id}","starred_url":"https://api.github.com/users/haohui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haohui/subscriptions","organizations_url":"https://api.github.com/users/haohui/orgs","repos_url":"https://api.github.com/users/haohui/repos","events_url":"https://api.github.com/users/haohui/events{/privacy}","received_events_url":"https://api.github.com/users/haohui/received_events","type":"User","site_admin":false},"body":"This is yet to be implemented as it requires figuring out the transitive dependency of jars. Filed STORM-1435 to track the effort.","created_at":"2016-01-06T19:47:13Z","updated_at":"2016-01-06T19:47:13Z","html_url":"https://github.com/apache/storm/pull/931#discussion_r49000550","pull_request_url":"https://api.github.com/repos/apache/storm/pulls/931","_links":{"self":{"href":"https://api.github.com/repos/apache/storm/pulls/comments/49000550"},"html":{"href":"https://github.com/apache/storm/pull/931#discussion_r49000550"},"pull_request":{"href":"https://api.github.com/repos/apache/storm/pulls/931"}}},"pull_request":{"url":"https://api.github.com/repos/apache/storm/pulls/931","id":52952856,"html_url":"https://github.com/apache/storm/pull/931","diff_url":"https://github.com/apache/storm/pull/931.diff","patch_url":"https://github.com/apache/storm/pull/931.patch","issue_url":"https://api.github.com/repos/apache/storm/issues/931","number":931,"state":"open","locked":false,"title":"STORM-1366. Add documentation for StormSQL into the website.","user":{"login":"haohui","id":1192828,"avatar_url":"https://avatars.githubusercontent.com/u/1192828?v=3","gravatar_id":"","url":"https://api.github.com/users/haohui","html_url":"https://github.com/haohui","followers_url":"https://api.github.com/users/haohui/followers","following_url":"https://api.github.com/users/haohui/following{/other_user}","gists_url":"https://api.github.com/users/haohui/gists{/gist_id}","starred_url":"https://api.github.com/users/haohui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haohui/subscriptions","organizations_url":"https://api.github.com/users/haohui/orgs","repos_url":"https://api.github.com/users/haohui/repos","events_url":"https://api.github.com/users/haohui/events{/privacy}","received_events_url":"https://api.github.com/users/haohui/received_events","type":"User","site_admin":false},"body":"This PR adds the documentation for StormSQL in the ASF website.","created_at":"2015-12-08T08:49:51Z","updated_at":"2016-01-06T19:47:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":"297c5d0eb14d679db28c34ef90d31eb8b5b17586","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/storm/pulls/931/commits","review_comments_url":"https://api.github.com/repos/apache/storm/pulls/931/comments","review_comment_url":"https://api.github.com/repos/apache/storm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/storm/issues/931/comments","statuses_url":"https://api.github.com/repos/apache/storm/statuses/99f4494439ef4ab9c8a056ea5742221da08065c8","head":{"label":"haohui:asf-site","ref":"asf-site","sha":"99f4494439ef4ab9c8a056ea5742221da08065c8","user":{"login":"haohui","id":1192828,"avatar_url":"https://avatars.githubusercontent.com/u/1192828?v=3","gravatar_id":"","url":"https://api.github.com/users/haohui","html_url":"https://github.com/haohui","followers_url":"https://api.github.com/users/haohui/followers","following_url":"https://api.github.com/users/haohui/following{/other_user}","gists_url":"https://api.github.com/users/haohui/gists{/gist_id}","starred_url":"https://api.github.com/users/haohui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haohui/subscriptions","organizations_url":"https://api.github.com/users/haohui/orgs","repos_url":"https://api.github.com/users/haohui/repos","events_url":"https://api.github.com/users/haohui/events{/privacy}","received_events_url":"https://api.github.com/users/haohui/received_events","type":"User","site_admin":false},"repo":{"id":42336271,"name":"storm","full_name":"haohui/storm","owner":{"login":"haohui","id":1192828,"avatar_url":"https://avatars.githubusercontent.com/u/1192828?v=3","gravatar_id":"","url":"https://api.github.com/users/haohui","html_url":"https://github.com/haohui","followers_url":"https://api.github.com/users/haohui/followers","following_url":"https://api.github.com/users/haohui/following{/other_user}","gists_url":"https://api.github.com/users/haohui/gists{/gist_id}","starred_url":"https://api.github.com/users/haohui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haohui/subscriptions","organizations_url":"https://api.github.com/users/haohui/orgs","repos_url":"https://api.github.com/users/haohui/repos","events_url":"https://api.github.com/users/haohui/events{/privacy}","received_events_url":"https://api.github.com/users/haohui/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/haohui/storm","description":"Mirror of Apache Storm","fork":true,"url":"https://api.github.com/repos/haohui/storm","forks_url":"https://api.github.com/repos/haohui/storm/forks","keys_url":"https://api.github.com/repos/haohui/storm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/haohui/storm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/haohui/storm/teams","hooks_url":"https://api.github.com/repos/haohui/storm/hooks","issue_events_url":"https://api.github.com/repos/haohui/storm/issues/events{/number}","events_url":"https://api.github.com/repos/haohui/storm/events","assignees_url":"https://api.github.com/repos/haohui/storm/assignees{/user}","branches_url":"https://api.github.com/repos/haohui/storm/branches{/branch}","tags_url":"https://api.github.com/repos/haohui/storm/tags","blobs_url":"https://api.github.com/repos/haohui/storm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/haohui/storm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/haohui/storm/git/refs{/sha}","trees_url":"https://api.github.com/repos/haohui/storm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/haohui/storm/statuses/{sha}","languages_url":"https://api.github.com/repos/haohui/storm/languages","stargazers_url":"https://api.github.com/repos/haohui/storm/stargazers","contributors_url":"https://api.github.com/repos/haohui/storm/contributors","subscribers_url":"https://api.github.com/repos/haohui/storm/subscribers","subscription_url":"https://api.github.com/repos/haohui/storm/subscription","commits_url":"https://api.github.com/repos/haohui/storm/commits{/sha}","git_commits_url":"https://api.github.com/repos/haohui/storm/git/commits{/sha}","comments_url":"https://api.github.com/repos/haohui/storm/comments{/number}","issue_comment_url":"https://api.github.com/repos/haohui/storm/issues/comments{/number}","contents_url":"https://api.github.com/repos/haohui/storm/contents/{+path}","compare_url":"https://api.github.com/repos/haohui/storm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/haohui/storm/merges","archive_url":"https://api.github.com/repos/haohui/storm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/haohui/storm/downloads","issues_url":"https://api.github.com/repos/haohui/storm/issues{/number}","pulls_url":"https://api.github.com/repos/haohui/storm/pulls{/number}","milestones_url":"https://api.github.com/repos/haohui/storm/milestones{/number}","notifications_url":"https://api.github.com/repos/haohui/storm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/haohui/storm/labels{/name}","releases_url":"https://api.github.com/repos/haohui/storm/releases{/id}","created_at":"2015-09-11T23:08:43Z","updated_at":"2015-09-11T23:13:16Z","pushed_at":"2015-12-08T21:31:05Z","git_url":"git://github.com/haohui/storm.git","ssh_url":"git@github.com:haohui/storm.git","clone_url":"https://github.com/haohui/storm.git","svn_url":"https://github.com/haohui/storm","homepage":null,"size":97320,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:asf-site","ref":"asf-site","sha":"2ba4f1799c60f30a70027a125e4c0a294bc90041","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":14135470,"name":"storm","full_name":"apache/storm","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/storm","description":"Mirror of Apache Storm","fork":false,"url":"https://api.github.com/repos/apache/storm","forks_url":"https://api.github.com/repos/apache/storm/forks","keys_url":"https://api.github.com/repos/apache/storm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/storm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/storm/teams","hooks_url":"https://api.github.com/repos/apache/storm/hooks","issue_events_url":"https://api.github.com/repos/apache/storm/issues/events{/number}","events_url":"https://api.github.com/repos/apache/storm/events","assignees_url":"https://api.github.com/repos/apache/storm/assignees{/user}","branches_url":"https://api.github.com/repos/apache/storm/branches{/branch}","tags_url":"https://api.github.com/repos/apache/storm/tags","blobs_url":"https://api.github.com/repos/apache/storm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/storm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/storm/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/storm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/storm/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/storm/languages","stargazers_url":"https://api.github.com/repos/apache/storm/stargazers","contributors_url":"https://api.github.com/repos/apache/storm/contributors","subscribers_url":"https://api.github.com/repos/apache/storm/subscribers","subscription_url":"https://api.github.com/repos/apache/storm/subscription","commits_url":"https://api.github.com/repos/apache/storm/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/storm/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/storm/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/storm/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/storm/contents/{+path}","compare_url":"https://api.github.com/repos/apache/storm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/storm/merges","archive_url":"https://api.github.com/repos/apache/storm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/storm/downloads","issues_url":"https://api.github.com/repos/apache/storm/issues{/number}","pulls_url":"https://api.github.com/repos/apache/storm/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/storm/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/storm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/storm/labels{/name}","releases_url":"https://api.github.com/repos/apache/storm/releases{/id}","created_at":"2013-11-05T08:00:14Z","updated_at":"2016-01-06T05:49:16Z","pushed_at":"2016-01-06T19:36:02Z","git_url":"git://github.com/apache/storm.git","ssh_url":"git@github.com:apache/storm.git","clone_url":"https://github.com/apache/storm.git","svn_url":"https://github.com/apache/storm","homepage":null,"size":155974,"stargazers_count":2705,"watchers_count":2705,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":2022,"mirror_url":"git://git.apache.org/storm.git","open_issues_count":76,"forks":2022,"open_issues":76,"watchers":2705,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/storm/pulls/931"},"html":{"href":"https://github.com/apache/storm/pull/931"},"issue":{"href":"https://api.github.com/repos/apache/storm/issues/931"},"comments":{"href":"https://api.github.com/repos/apache/storm/issues/931/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/storm/pulls/931/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/storm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/storm/pulls/931/commits"},"statuses":{"href":"https://api.github.com/repos/apache/storm/statuses/99f4494439ef4ab9c8a056ea5742221da08065c8"}}}}