{"action":"created","comment":{"url":"https://api.github.com/repos/apache/beam-site/pulls/comments/209469261","pull_request_review_id":145488097,"id":209469261,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwOTQ2OTI2MQ==","diff_hunk":"@@ -0,0 +1,224 @@\n+---\n+layout: post\n+title:  \"A review of input streaming connectors\"\n+date:   2018-08-XX 00:00:01 -0800\n+excerpt_separator: <!--more-->\n+categories: blog\n+authors:\n+  - lkuligin\n+  - jphalip\n+---\n+\n+In this post, you'll learn about the current state of support for input streaming connectors in [Apache Beam](https://beam.apache.org/). For more context, you'll also learn about the corresponding state of support in [Apache Spark](https://spark.apache.org/).<!--more-->\n+\n+With batch processing, you might load data from any source, including a database system. Even if there are no specific SDKs available for those database systems, you can often resort to using a [JDBC](https://en.wikipedia.org/wiki/Java_Database_Connectivity) driver. With streaming, implementing a proper data pipeline is arguably more challenging as generally fewer source types are available. For that reason, this article particularly focuses on the streaming use case.\n+\n+## Connectors for Java\n+\n+Beam has an official [Java SDK](https://beam.apache.org/documentation/sdks/java/) and has several execution engines, called [runners](https://beam.apache.org/documentation/runners/capability-matrix/). In most cases it is fairly easy to transfer existing Beam pipelines written in Java or Scala to a Spark environment by using the [Spark Runner](https://beam.apache.org/documentation/runners/spark/).\n+\n+Spark is written in Scala and has a [Java API](https://spark.apache.org/docs/latest/api/java/). Spark's source code compiles to [Java bytecode](https://en.wikipedia.org/wiki/Java_(programming_language)#Java_JVM_and_Bytecode) and the binaries are run by a [Java Virtual Machine](https://en.wikipedia.org/wiki/Java_virtual_machine). Scala code is interoperable with Java and therefore has native compatibility with Java libraries (and vice versa).\n+\n+Spark offers two approaches to streaming: [Discretized Streaming](https://spark.apache.org/docs/latest/streaming-programming-guide.html) (or DStreams) and [Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html). DStreams are a basic abstraction that represents a continuous series of [Resilient Distributed Datasets](https://spark.apache.org/docs/latest/rdd-programming-guide.html) (or RDDs). Structured Streaming was introduced more recently (the alpha release came with Spark 2.1.0) and is based on a [model](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#programming-model) where live data is continuously appended to a table structure.\n+\n+Spark Structured Streaming supports [file sources](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/streaming/DataStreamReader.html) (local filesystems and HDFS-compatible systems like Cloud Storage or S3) and [Kafka](https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html) as streaming inputs. Spark maintains built-in connectors for DStreams aimed at third-party services, such as Kafka or Flume, while other connectors are available through linking external dependencies, as shown in the table below.\n+\n+Below are the main streaming input connectors for available for Beam and Spark DStreams in Java:\n+\n+<table>\n+  <tr>\n+   <td>\n+   </td>\n+   <td>\n+   </td>\n+   <td><strong>Apache Beam</strong>\n+   </td>\n+   <td><strong>Apache Spark DStreams</strong>\n+   </td>\n+  </tr>\n+  <tr>\n+   <td rowspan=\"2\" >File Systems\n+   </td>\n+   <td>Local<br>(Using the <code>file://</code> URI)\n+   </td>\n+   <td><a href=\"https://beam.apache.org/documentation/sdks/javadoc/2.5.0/org/apache/beam/sdk/io/TextIO.html\">TextIO</a>\n+   </td>\n+   <td><a href=\"https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-\">textFileStream</a><br>(Spark treats most Unix systems as HDFS-compatible, but the location should be accessible from all nodes)\n+   </td>\n+  </tr>\n+  <tr>\n+   <td>HDFS<br>(Using the <code>hdfs://</code> URI)\n+   </td>\n+   <td><a href=\"https://beam.apache.org/documentation/sdks/javadoc/2.5.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html\">HadoopFileSystemOptions</a>\n+   </td>\n+   <td><a href=\"https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/util/HdfsUtils.html\">HdfsUtils</a>\n+   </td>\n+  </tr>\n+  <tr>\n+   <td rowspan=\"2\" >Object Stores\n+   </td>\n+   <td>Cloud Storage<br>(Using the <code>gs://</code> URI)\n+   </td>\n+   <td rowspan=\"2\" ><a href=\"https://beam.apache.org/documentation/sdks/javadoc/2.5.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html\">HadoopFileSystemOptions</a>","path":"src/_posts/2018-08-XX-review-input-streaming-connectors.md","position":62,"original_position":62,"commit_id":"ac155ca51bea9371e0d166984eac05bc4067380c","original_commit_id":"ac155ca51bea9371e0d166984eac05bc4067380c","user":{"login":"iemejia","id":79476,"node_id":"MDQ6VXNlcjc5NDc2","avatar_url":"https://avatars3.githubusercontent.com/u/79476?v=4","gravatar_id":"","url":"https://api.github.com/users/iemejia","html_url":"https://github.com/iemejia","followers_url":"https://api.github.com/users/iemejia/followers","following_url":"https://api.github.com/users/iemejia/following{/other_user}","gists_url":"https://api.github.com/users/iemejia/gists{/gist_id}","starred_url":"https://api.github.com/users/iemejia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iemejia/subscriptions","organizations_url":"https://api.github.com/users/iemejia/orgs","repos_url":"https://api.github.com/users/iemejia/repos","events_url":"https://api.github.com/users/iemejia/events{/privacy}","received_events_url":"https://api.github.com/users/iemejia/received_events","type":"User","site_admin":false},"body":"SImilar to above, maybe (GcsOptions or GcsFileSystem) and (S3Options or S3FileSystem) ?","created_at":"2018-08-12T22:05:20Z","updated_at":"2018-08-12T22:26:17Z","html_url":"https://github.com/apache/beam-site/pull/521#discussion_r209469261","pull_request_url":"https://api.github.com/repos/apache/beam-site/pulls/521","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/apache/beam-site/pulls/comments/209469261"},"html":{"href":"https://github.com/apache/beam-site/pull/521#discussion_r209469261"},"pull_request":{"href":"https://api.github.com/repos/apache/beam-site/pulls/521"}}},"pull_request":{"url":"https://api.github.com/repos/apache/beam-site/pulls/521","id":206085613,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA2MDg1NjEz","html_url":"https://github.com/apache/beam-site/pull/521","diff_url":"https://github.com/apache/beam-site/pull/521.diff","patch_url":"https://github.com/apache/beam-site/pull/521.patch","issue_url":"https://api.github.com/repos/apache/beam-site/issues/521","number":521,"state":"open","locked":false,"title":"Add blog post \"A review of input streaming connectors\"","user":{"login":"jphalip","id":68544,"node_id":"MDQ6VXNlcjY4NTQ0","avatar_url":"https://avatars0.githubusercontent.com/u/68544?v=4","gravatar_id":"","url":"https://api.github.com/users/jphalip","html_url":"https://github.com/jphalip","followers_url":"https://api.github.com/users/jphalip/followers","following_url":"https://api.github.com/users/jphalip/following{/other_user}","gists_url":"https://api.github.com/users/jphalip/gists{/gist_id}","starred_url":"https://api.github.com/users/jphalip/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jphalip/subscriptions","organizations_url":"https://api.github.com/users/jphalip/orgs","repos_url":"https://api.github.com/users/jphalip/repos","events_url":"https://api.github.com/users/jphalip/events{/privacy}","received_events_url":"https://api.github.com/users/jphalip/received_events","type":"User","site_admin":false},"body":"This is a re-post of an article that I recently published on the GCP blog: https://cloud.google.com/blog/products/data-analytics/review-of-input-streaming-connectors-for-apache-beam-and-apache-spark\r\n\r\nThis is a slightly edited version from the other article to make it relevant to a broader audience beyond just GCP.","created_at":"2018-08-03T16:54:55Z","updated_at":"2018-08-12T22:26:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bc1f908903043d03d2af7e11229dce8bd4f5f2aa","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/beam-site/pulls/521/commits","review_comments_url":"https://api.github.com/repos/apache/beam-site/pulls/521/comments","review_comment_url":"https://api.github.com/repos/apache/beam-site/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/beam-site/issues/521/comments","statuses_url":"https://api.github.com/repos/apache/beam-site/statuses/ac155ca51bea9371e0d166984eac05bc4067380c","head":{"label":"jphalip:blog-post-streaming-connectors","ref":"blog-post-streaming-connectors","sha":"ac155ca51bea9371e0d166984eac05bc4067380c","user":{"login":"jphalip","id":68544,"node_id":"MDQ6VXNlcjY4NTQ0","avatar_url":"https://avatars0.githubusercontent.com/u/68544?v=4","gravatar_id":"","url":"https://api.github.com/users/jphalip","html_url":"https://github.com/jphalip","followers_url":"https://api.github.com/users/jphalip/followers","following_url":"https://api.github.com/users/jphalip/following{/other_user}","gists_url":"https://api.github.com/users/jphalip/gists{/gist_id}","starred_url":"https://api.github.com/users/jphalip/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jphalip/subscriptions","organizations_url":"https://api.github.com/users/jphalip/orgs","repos_url":"https://api.github.com/users/jphalip/repos","events_url":"https://api.github.com/users/jphalip/events{/privacy}","received_events_url":"https://api.github.com/users/jphalip/received_events","type":"User","site_admin":false},"repo":{"id":143212048,"node_id":"MDEwOlJlcG9zaXRvcnkxNDMyMTIwNDg=","name":"beam-site","full_name":"jphalip/beam-site","owner":{"login":"jphalip","id":68544,"node_id":"MDQ6VXNlcjY4NTQ0","avatar_url":"https://avatars0.githubusercontent.com/u/68544?v=4","gravatar_id":"","url":"https://api.github.com/users/jphalip","html_url":"https://github.com/jphalip","followers_url":"https://api.github.com/users/jphalip/followers","following_url":"https://api.github.com/users/jphalip/following{/other_user}","gists_url":"https://api.github.com/users/jphalip/gists{/gist_id}","starred_url":"https://api.github.com/users/jphalip/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jphalip/subscriptions","organizations_url":"https://api.github.com/users/jphalip/orgs","repos_url":"https://api.github.com/users/jphalip/repos","events_url":"https://api.github.com/users/jphalip/events{/privacy}","received_events_url":"https://api.github.com/users/jphalip/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jphalip/beam-site","description":"Apache Beam Site","fork":true,"url":"https://api.github.com/repos/jphalip/beam-site","forks_url":"https://api.github.com/repos/jphalip/beam-site/forks","keys_url":"https://api.github.com/repos/jphalip/beam-site/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jphalip/beam-site/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jphalip/beam-site/teams","hooks_url":"https://api.github.com/repos/jphalip/beam-site/hooks","issue_events_url":"https://api.github.com/repos/jphalip/beam-site/issues/events{/number}","events_url":"https://api.github.com/repos/jphalip/beam-site/events","assignees_url":"https://api.github.com/repos/jphalip/beam-site/assignees{/user}","branches_url":"https://api.github.com/repos/jphalip/beam-site/branches{/branch}","tags_url":"https://api.github.com/repos/jphalip/beam-site/tags","blobs_url":"https://api.github.com/repos/jphalip/beam-site/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jphalip/beam-site/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jphalip/beam-site/git/refs{/sha}","trees_url":"https://api.github.com/repos/jphalip/beam-site/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jphalip/beam-site/statuses/{sha}","languages_url":"https://api.github.com/repos/jphalip/beam-site/languages","stargazers_url":"https://api.github.com/repos/jphalip/beam-site/stargazers","contributors_url":"https://api.github.com/repos/jphalip/beam-site/contributors","subscribers_url":"https://api.github.com/repos/jphalip/beam-site/subscribers","subscription_url":"https://api.github.com/repos/jphalip/beam-site/subscription","commits_url":"https://api.github.com/repos/jphalip/beam-site/commits{/sha}","git_commits_url":"https://api.github.com/repos/jphalip/beam-site/git/commits{/sha}","comments_url":"https://api.github.com/repos/jphalip/beam-site/comments{/number}","issue_comment_url":"https://api.github.com/repos/jphalip/beam-site/issues/comments{/number}","contents_url":"https://api.github.com/repos/jphalip/beam-site/contents/{+path}","compare_url":"https://api.github.com/repos/jphalip/beam-site/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jphalip/beam-site/merges","archive_url":"https://api.github.com/repos/jphalip/beam-site/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jphalip/beam-site/downloads","issues_url":"https://api.github.com/repos/jphalip/beam-site/issues{/number}","pulls_url":"https://api.github.com/repos/jphalip/beam-site/pulls{/number}","milestones_url":"https://api.github.com/repos/jphalip/beam-site/milestones{/number}","notifications_url":"https://api.github.com/repos/jphalip/beam-site/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jphalip/beam-site/labels{/name}","releases_url":"https://api.github.com/repos/jphalip/beam-site/releases{/id}","deployments_url":"https://api.github.com/repos/jphalip/beam-site/deployments","created_at":"2018-08-01T21:49:45Z","updated_at":"2018-08-01T21:50:08Z","pushed_at":"2018-08-06T07:39:08Z","git_url":"git://github.com/jphalip/beam-site.git","ssh_url":"git@github.com:jphalip/beam-site.git","clone_url":"https://github.com/jphalip/beam-site.git","svn_url":"https://github.com/jphalip/beam-site","homepage":"","size":67495,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"asf-site"}},"base":{"label":"apache:asf-site","ref":"asf-site","sha":"e568ceabc4cf5f1dda5d6cc1b0655e74048e92a2","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":52588453,"node_id":"MDEwOlJlcG9zaXRvcnk1MjU4ODQ1Mw==","name":"beam-site","full_name":"apache/beam-site","owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/beam-site","description":"Apache Beam Site","fork":false,"url":"https://api.github.com/repos/apache/beam-site","forks_url":"https://api.github.com/repos/apache/beam-site/forks","keys_url":"https://api.github.com/repos/apache/beam-site/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/beam-site/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/beam-site/teams","hooks_url":"https://api.github.com/repos/apache/beam-site/hooks","issue_events_url":"https://api.github.com/repos/apache/beam-site/issues/events{/number}","events_url":"https://api.github.com/repos/apache/beam-site/events","assignees_url":"https://api.github.com/repos/apache/beam-site/assignees{/user}","branches_url":"https://api.github.com/repos/apache/beam-site/branches{/branch}","tags_url":"https://api.github.com/repos/apache/beam-site/tags","blobs_url":"https://api.github.com/repos/apache/beam-site/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/beam-site/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/beam-site/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/beam-site/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/beam-site/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/beam-site/languages","stargazers_url":"https://api.github.com/repos/apache/beam-site/stargazers","contributors_url":"https://api.github.com/repos/apache/beam-site/contributors","subscribers_url":"https://api.github.com/repos/apache/beam-site/subscribers","subscription_url":"https://api.github.com/repos/apache/beam-site/subscription","commits_url":"https://api.github.com/repos/apache/beam-site/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/beam-site/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/beam-site/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/beam-site/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/beam-site/contents/{+path}","compare_url":"https://api.github.com/repos/apache/beam-site/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/beam-site/merges","archive_url":"https://api.github.com/repos/apache/beam-site/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/beam-site/downloads","issues_url":"https://api.github.com/repos/apache/beam-site/issues{/number}","pulls_url":"https://api.github.com/repos/apache/beam-site/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/beam-site/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/beam-site/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/beam-site/labels{/name}","releases_url":"https://api.github.com/repos/apache/beam-site/releases{/id}","deployments_url":"https://api.github.com/repos/apache/beam-site/deployments","created_at":"2016-02-26T08:00:05Z","updated_at":"2018-08-12T20:29:14Z","pushed_at":"2018-08-12T20:29:11Z","git_url":"git://github.com/apache/beam-site.git","ssh_url":"git@github.com:apache/beam-site.git","clone_url":"https://github.com/apache/beam-site.git","svn_url":"https://github.com/apache/beam-site","homepage":"","size":72777,"stargazers_count":20,"watchers_count":20,"language":"HTML","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":122,"mirror_url":null,"archived":false,"open_issues_count":10,"license":null,"forks":122,"open_issues":10,"watchers":20,"default_branch":"asf-site"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/beam-site/pulls/521"},"html":{"href":"https://github.com/apache/beam-site/pull/521"},"issue":{"href":"https://api.github.com/repos/apache/beam-site/issues/521"},"comments":{"href":"https://api.github.com/repos/apache/beam-site/issues/521/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/beam-site/pulls/521/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/beam-site/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/beam-site/pulls/521/commits"},"statuses":{"href":"https://api.github.com/repos/apache/beam-site/statuses/ac155ca51bea9371e0d166984eac05bc4067380c"}},"author_association":"NONE"}}
{"actor":{"display_login":"iemejia"}}