{"action":"created","comment":{"url":"https://api.github.com/repos/basho/spark-riak-connector/pulls/comments/61471195","id":61471195,"diff_hunk":"@@ -533,6 +534,168 @@ conf.set(\"spark.riakts.write.bulk-size\", \"500\")\n conf.set(\"spark.riak.connections.min\", \"50\")\n ```\n \n+## Spark Streaming Example\n+\n+The Spark-Riak Connector can be used with Spark Streaming. To demonstrate this usage, we will work through two small Scala examples. These examples are located in the [examples folder](../examples/src/main/scala/com/basho/riak/spark/examples) of the Spark-Riak Connector repo.\n+\n+These examples require the use of Kafka. Please install Kafka and setup a Kafka broker prior to running this example. We will assume that there is a Kafka broker running at `127.0.0.1:9092` with a topic called `streaming`. Instructions on setting up Kafka topics can be found in [this guide](https://kafka.apache.org/documentation.html#quickstart). You can create a broker and topic with the following:\n+\n+```\n+path/to/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties\n+path/to/kafka/bin/kafka-server-start.sh config/server.properties\n+path/to/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic streaming\n+```\n+\n+We also assume Riak TS is installed and there is a Riak TS node running at `127.0.0.1:8087`. You can find instruction to do so [here](http://docs.basho.com/riak/ts/1.2.0/installing/mac-osx/). You will need to build the examples as well. Please follow the instructions on [building the examples](../examples#building-and-running-examplesdemos). After setting up, there are two examples to run: a KV bucket example and a TS table example.\n+\n+###Spark Streaming KV Buckets Example\n+\n+Now that we are set up, let's look at the KV bucket example [here](../examples/src/main/scala/com/basho/riak/spark/examples/streaming/StreamingKVExample.scala). \n+\n+In the first chunk of code in the main method, we are just setting up our local Spark Streaming context and setting the name for the KV bucket to `test-data`:\n+\n+```scala\n+val sparkConf = new SparkConf(true)\n+      .setAppName(\"Simple Spark Streaming to Riak KV Demo\")\n+setSparkOpt(sparkConf, \"spark.master\", \"local\")\n+setSparkOpt(sparkConf, \"spark.riak.connection.host\", \"127.0.0.1:8087\")\n+\n+val sc = new SparkContext(sparkConf)\n+val streamCtx = new StreamingContext(sc, Durations.seconds(15))\n+val sqlContext = new org.apache.spark.sql.SQLContext(sc)\n+import sqlContext.implicits._\n+val namespace = new Namespace(\"test-data\")\n+```\n+\n+Next we are setting up Kafka properties:\n+\n+```scala\n+val kafkaProps = Map[String, String](\n+      \"metadata.broker.list\" -> \"127.0.0.1:9092\",\n+      \"client.id\" -> UUID.randomUUID().toString\n+    )\n+```\n+\n+Then, we are using `KafkaUtils` to create a stream from the Kafka topic `streaming` into our KV bucket `test-data`:\n+\n+```scala\n+    KafkaUtils\n+      .createDirectStream[String, String, StringDecoder, StringDecoder](streamCtx, kafkaProps, Set[String](\"streaming\"))\n+      .foreachRDD { rdd =>\n+        val rows = sqlContext.read.json(rdd.values).map {\n+          line => val obj = RiakObjectConversionUtil.to(line)\n+            obj.setContentType(\"application/json\")\n+            obj\n+        }.saveToRiak(namespace)\n+      }\n+```\n+ And finally, we are starting the stream:\n+ \n+ ```scala\n+ streamCtx.start()\n+ streamCtx.awaitTermination()\n+ ```\n+\n+ Now that we understand the code, we can run the `StreamingKVExample.scala` example with:\n+ \n+ ```\n+ /path/to/spark-riak-connector-examples/bin/run-example streaming.StreamingKVExample\n+ ```\n+ \n+ This will start a stream from the Kafka topic `streaming` into the KV bucket `test-data` that we just created. This stream will run until terminated. Whenever a message is produced for Kafka topic `streaming`, the Spark Streaming context that the example creates will automatically stream the message from the topic into the KV bucket. To see this in action, we need to send a message to the Kafka topic `streaming` with the Kafka console producer script, which can be found in the Kafka directory.\n+ \n+ ```\n+ /path/to/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic streaming\n+ ```\n+ \n+ This will script read messages from the terminal and pass it to the topic. From the topic, the Spark Streaming context will write the message to Riak KV bucket `test-data`.  As an example put the following into the terminal:","path":"docs/using-connector.md","position":86,"original_position":86,"commit_id":"912904caad0549542f6a99038f3dd5545ee857b4","original_commit_id":"912904caad0549542f6a99038f3dd5545ee857b4","user":{"login":"mdigan","id":175687,"avatar_url":"https://avatars.githubusercontent.com/u/175687?v=3","gravatar_id":"","url":"https://api.github.com/users/mdigan","html_url":"https://github.com/mdigan","followers_url":"https://api.github.com/users/mdigan/followers","following_url":"https://api.github.com/users/mdigan/following{/other_user}","gists_url":"https://api.github.com/users/mdigan/gists{/gist_id}","starred_url":"https://api.github.com/users/mdigan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdigan/subscriptions","organizations_url":"https://api.github.com/users/mdigan/orgs","repos_url":"https://api.github.com/users/mdigan/repos","events_url":"https://api.github.com/users/mdigan/events{/privacy}","received_events_url":"https://api.github.com/users/mdigan/received_events","type":"User","site_admin":false},"body":"I think \"will script\" should be changed to \"script will\"","created_at":"2016-04-28T17:45:08Z","updated_at":"2016-04-28T17:45:08Z","html_url":"https://github.com/basho/spark-riak-connector/pull/147#discussion_r61471195","pull_request_url":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147","_links":{"self":{"href":"https://api.github.com/repos/basho/spark-riak-connector/pulls/comments/61471195"},"html":{"href":"https://github.com/basho/spark-riak-connector/pull/147#discussion_r61471195"},"pull_request":{"href":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147"}}},"pull_request":{"url":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147","id":67937997,"html_url":"https://github.com/basho/spark-riak-connector/pull/147","diff_url":"https://github.com/basho/spark-riak-connector/pull/147.diff","patch_url":"https://github.com/basho/spark-riak-connector/pull/147.patch","issue_url":"https://api.github.com/repos/basho/spark-riak-connector/issues/147","number":147,"state":"open","locked":false,"title":"Kclark docs update","user":{"login":"korry8911","id":10292569,"avatar_url":"https://avatars.githubusercontent.com/u/10292569?v=3","gravatar_id":"","url":"https://api.github.com/users/korry8911","html_url":"https://github.com/korry8911","followers_url":"https://api.github.com/users/korry8911/followers","following_url":"https://api.github.com/users/korry8911/following{/other_user}","gists_url":"https://api.github.com/users/korry8911/gists{/gist_id}","starred_url":"https://api.github.com/users/korry8911/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/korry8911/subscriptions","organizations_url":"https://api.github.com/users/korry8911/orgs","repos_url":"https://api.github.com/users/korry8911/repos","events_url":"https://api.github.com/users/korry8911/events{/privacy}","received_events_url":"https://api.github.com/users/korry8911/received_events","type":"User","site_admin":false},"body":"added spark streaming docs and sc.stop to quick start guides","created_at":"2016-04-26T18:48:06Z","updated_at":"2016-04-28T17:45:08Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4eb90fdc17f9abbb72457d59885c51e0c67e82da","assignee":{"login":"korry8911","id":10292569,"avatar_url":"https://avatars.githubusercontent.com/u/10292569?v=3","gravatar_id":"","url":"https://api.github.com/users/korry8911","html_url":"https://github.com/korry8911","followers_url":"https://api.github.com/users/korry8911/followers","following_url":"https://api.github.com/users/korry8911/following{/other_user}","gists_url":"https://api.github.com/users/korry8911/gists{/gist_id}","starred_url":"https://api.github.com/users/korry8911/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/korry8911/subscriptions","organizations_url":"https://api.github.com/users/korry8911/orgs","repos_url":"https://api.github.com/users/korry8911/repos","events_url":"https://api.github.com/users/korry8911/events{/privacy}","received_events_url":"https://api.github.com/users/korry8911/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147/commits","review_comments_url":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147/comments","review_comment_url":"https://api.github.com/repos/basho/spark-riak-connector/pulls/comments{/number}","comments_url":"https://api.github.com/repos/basho/spark-riak-connector/issues/147/comments","statuses_url":"https://api.github.com/repos/basho/spark-riak-connector/statuses/912904caad0549542f6a99038f3dd5545ee857b4","head":{"label":"basho:kclark-docs-update","ref":"kclark-docs-update","sha":"912904caad0549542f6a99038f3dd5545ee857b4","user":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"repo":{"id":35239038,"name":"spark-riak-connector","full_name":"basho/spark-riak-connector","owner":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/basho/spark-riak-connector","description":"The official Riak Spark Connector for Apache Spark with Riak TS and Riak KV","fork":false,"url":"https://api.github.com/repos/basho/spark-riak-connector","forks_url":"https://api.github.com/repos/basho/spark-riak-connector/forks","keys_url":"https://api.github.com/repos/basho/spark-riak-connector/keys{/key_id}","collaborators_url":"https://api.github.com/repos/basho/spark-riak-connector/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/basho/spark-riak-connector/teams","hooks_url":"https://api.github.com/repos/basho/spark-riak-connector/hooks","issue_events_url":"https://api.github.com/repos/basho/spark-riak-connector/issues/events{/number}","events_url":"https://api.github.com/repos/basho/spark-riak-connector/events","assignees_url":"https://api.github.com/repos/basho/spark-riak-connector/assignees{/user}","branches_url":"https://api.github.com/repos/basho/spark-riak-connector/branches{/branch}","tags_url":"https://api.github.com/repos/basho/spark-riak-connector/tags","blobs_url":"https://api.github.com/repos/basho/spark-riak-connector/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/basho/spark-riak-connector/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/basho/spark-riak-connector/git/refs{/sha}","trees_url":"https://api.github.com/repos/basho/spark-riak-connector/git/trees{/sha}","statuses_url":"https://api.github.com/repos/basho/spark-riak-connector/statuses/{sha}","languages_url":"https://api.github.com/repos/basho/spark-riak-connector/languages","stargazers_url":"https://api.github.com/repos/basho/spark-riak-connector/stargazers","contributors_url":"https://api.github.com/repos/basho/spark-riak-connector/contributors","subscribers_url":"https://api.github.com/repos/basho/spark-riak-connector/subscribers","subscription_url":"https://api.github.com/repos/basho/spark-riak-connector/subscription","commits_url":"https://api.github.com/repos/basho/spark-riak-connector/commits{/sha}","git_commits_url":"https://api.github.com/repos/basho/spark-riak-connector/git/commits{/sha}","comments_url":"https://api.github.com/repos/basho/spark-riak-connector/comments{/number}","issue_comment_url":"https://api.github.com/repos/basho/spark-riak-connector/issues/comments{/number}","contents_url":"https://api.github.com/repos/basho/spark-riak-connector/contents/{+path}","compare_url":"https://api.github.com/repos/basho/spark-riak-connector/compare/{base}...{head}","merges_url":"https://api.github.com/repos/basho/spark-riak-connector/merges","archive_url":"https://api.github.com/repos/basho/spark-riak-connector/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/basho/spark-riak-connector/downloads","issues_url":"https://api.github.com/repos/basho/spark-riak-connector/issues{/number}","pulls_url":"https://api.github.com/repos/basho/spark-riak-connector/pulls{/number}","milestones_url":"https://api.github.com/repos/basho/spark-riak-connector/milestones{/number}","notifications_url":"https://api.github.com/repos/basho/spark-riak-connector/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/basho/spark-riak-connector/labels{/name}","releases_url":"https://api.github.com/repos/basho/spark-riak-connector/releases{/id}","deployments_url":"https://api.github.com/repos/basho/spark-riak-connector/deployments","created_at":"2015-05-07T19:22:35Z","updated_at":"2016-04-27T06:03:19Z","pushed_at":"2016-04-28T17:37:56Z","git_url":"git://github.com/basho/spark-riak-connector.git","ssh_url":"git@github.com:basho/spark-riak-connector.git","clone_url":"https://github.com/basho/spark-riak-connector.git","svn_url":"https://github.com/basho/spark-riak-connector","homepage":"http://basho.com/products/spark/","size":17437,"stargazers_count":18,"watchers_count":18,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":8,"forks":6,"open_issues":8,"watchers":18,"default_branch":"master"}},"base":{"label":"basho:master","ref":"master","sha":"f5e4c423a886ee0a62d821a3e1d10eb7e15c051f","user":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"repo":{"id":35239038,"name":"spark-riak-connector","full_name":"basho/spark-riak-connector","owner":{"login":"basho","id":176293,"avatar_url":"https://avatars.githubusercontent.com/u/176293?v=3","gravatar_id":"","url":"https://api.github.com/users/basho","html_url":"https://github.com/basho","followers_url":"https://api.github.com/users/basho/followers","following_url":"https://api.github.com/users/basho/following{/other_user}","gists_url":"https://api.github.com/users/basho/gists{/gist_id}","starred_url":"https://api.github.com/users/basho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basho/subscriptions","organizations_url":"https://api.github.com/users/basho/orgs","repos_url":"https://api.github.com/users/basho/repos","events_url":"https://api.github.com/users/basho/events{/privacy}","received_events_url":"https://api.github.com/users/basho/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/basho/spark-riak-connector","description":"The official Riak Spark Connector for Apache Spark with Riak TS and Riak KV","fork":false,"url":"https://api.github.com/repos/basho/spark-riak-connector","forks_url":"https://api.github.com/repos/basho/spark-riak-connector/forks","keys_url":"https://api.github.com/repos/basho/spark-riak-connector/keys{/key_id}","collaborators_url":"https://api.github.com/repos/basho/spark-riak-connector/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/basho/spark-riak-connector/teams","hooks_url":"https://api.github.com/repos/basho/spark-riak-connector/hooks","issue_events_url":"https://api.github.com/repos/basho/spark-riak-connector/issues/events{/number}","events_url":"https://api.github.com/repos/basho/spark-riak-connector/events","assignees_url":"https://api.github.com/repos/basho/spark-riak-connector/assignees{/user}","branches_url":"https://api.github.com/repos/basho/spark-riak-connector/branches{/branch}","tags_url":"https://api.github.com/repos/basho/spark-riak-connector/tags","blobs_url":"https://api.github.com/repos/basho/spark-riak-connector/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/basho/spark-riak-connector/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/basho/spark-riak-connector/git/refs{/sha}","trees_url":"https://api.github.com/repos/basho/spark-riak-connector/git/trees{/sha}","statuses_url":"https://api.github.com/repos/basho/spark-riak-connector/statuses/{sha}","languages_url":"https://api.github.com/repos/basho/spark-riak-connector/languages","stargazers_url":"https://api.github.com/repos/basho/spark-riak-connector/stargazers","contributors_url":"https://api.github.com/repos/basho/spark-riak-connector/contributors","subscribers_url":"https://api.github.com/repos/basho/spark-riak-connector/subscribers","subscription_url":"https://api.github.com/repos/basho/spark-riak-connector/subscription","commits_url":"https://api.github.com/repos/basho/spark-riak-connector/commits{/sha}","git_commits_url":"https://api.github.com/repos/basho/spark-riak-connector/git/commits{/sha}","comments_url":"https://api.github.com/repos/basho/spark-riak-connector/comments{/number}","issue_comment_url":"https://api.github.com/repos/basho/spark-riak-connector/issues/comments{/number}","contents_url":"https://api.github.com/repos/basho/spark-riak-connector/contents/{+path}","compare_url":"https://api.github.com/repos/basho/spark-riak-connector/compare/{base}...{head}","merges_url":"https://api.github.com/repos/basho/spark-riak-connector/merges","archive_url":"https://api.github.com/repos/basho/spark-riak-connector/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/basho/spark-riak-connector/downloads","issues_url":"https://api.github.com/repos/basho/spark-riak-connector/issues{/number}","pulls_url":"https://api.github.com/repos/basho/spark-riak-connector/pulls{/number}","milestones_url":"https://api.github.com/repos/basho/spark-riak-connector/milestones{/number}","notifications_url":"https://api.github.com/repos/basho/spark-riak-connector/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/basho/spark-riak-connector/labels{/name}","releases_url":"https://api.github.com/repos/basho/spark-riak-connector/releases{/id}","deployments_url":"https://api.github.com/repos/basho/spark-riak-connector/deployments","created_at":"2015-05-07T19:22:35Z","updated_at":"2016-04-27T06:03:19Z","pushed_at":"2016-04-28T17:37:56Z","git_url":"git://github.com/basho/spark-riak-connector.git","ssh_url":"git@github.com:basho/spark-riak-connector.git","clone_url":"https://github.com/basho/spark-riak-connector.git","svn_url":"https://github.com/basho/spark-riak-connector","homepage":"http://basho.com/products/spark/","size":17437,"stargazers_count":18,"watchers_count":18,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":8,"forks":6,"open_issues":8,"watchers":18,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147"},"html":{"href":"https://github.com/basho/spark-riak-connector/pull/147"},"issue":{"href":"https://api.github.com/repos/basho/spark-riak-connector/issues/147"},"comments":{"href":"https://api.github.com/repos/basho/spark-riak-connector/issues/147/comments"},"review_comments":{"href":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147/comments"},"review_comment":{"href":"https://api.github.com/repos/basho/spark-riak-connector/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/basho/spark-riak-connector/pulls/147/commits"},"statuses":{"href":"https://api.github.com/repos/basho/spark-riak-connector/statuses/912904caad0549542f6a99038f3dd5545ee857b4"}}}}