{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/77779369","id":77779369,"diff_hunk":"@@ -0,0 +1,853 @@\n+---\n+title: \"SparkR - Practical Guide\"\n+output:\n+  html_document:\n+    theme: united\n+    toc: true\n+    toc_depth: 4\n+    toc_float: true\n+    highlight: textmate\n+---\n+\n+## Overview\n+\n+SparkR is an R package that provides a light-weight frontend to use Apache Spark from R. In Spark 2.0.0, SparkR provides a distributed data frame implementation that supports data processing operations like selection, filtering, aggregation etc. and distributed machine learning using [MLlib](http://spark.apache.org/mllib/).\n+\n+## Getting Started\n+\n+We begin with an example running on the local machine and provide an overview of the use of SparkR: data ingestion, data processing and machine learning.\n+\n+First, let's load and attach the package.\n+```{r, message=FALSE}\n+library(SparkR)\n+```\n+\n+`SparkSession` is the entry point into SparkR which connects your R program to a Spark cluster. You can create a `SparkSession` using `sparkR.session` and pass in options such as the application name, any Spark packages depended on, etc.\n+\n+We use default settings in which it runs in local mode. It auto downloads Spark package in the background if no previous installation is found. For more details about setup, see [Spark Session](#SetupSparkSession).\n+\n+```{r, message=FALSE, warning=FALSE}\n+sparkR.session()\n+```\n+\n+The operations in SparkR are centered around an R class called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood.\n+\n+`SparkDataFrame` can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing local R data frames. For example, we create a `SparkDataFrame` from a local R data frame,\n+\n+```{r}\n+cars <- cbind(model = rownames(mtcars), mtcars)\n+carsDF <- createDataFrame(cars)\n+```\n+\n+We can view the first few rows of the `SparkDataFrame` by `showDF` or `head` function.\n+```{r}\n+showDF(carsDF)\n+```\n+\n+Common data processing operations such as `filter`, `select` are supported on the `SparkDataFrame`.\n+```{r}\n+carsSubDF <- select(carsDF, \"model\", \"mpg\", \"hp\")\n+carsSubDF <- filter(carsSubDF, carsSubDF$hp >= 200)\n+showDF(carsSubDF)\n+```\n+\n+SparkR can use many common aggregation functions after grouping.\n+\n+```{r}\n+carsGPDF <- summarize(groupBy(carsDF, carsDF$gear), count = n(carsDF$gear))\n+showDF(carsGPDF)\n+```\n+\n+The results `carsDF` and `carsSubDF` are `SparkDataFrame` objects. To convert back to R `data.frame`, we can use `collect`.\n+```{r}\n+carsGP <- collect(carsGPDF)\n+class(carsGP)\n+```\n+\n+SparkR supports a number of commonly used machine learning algorithms. Under the hood, SparkR uses MLlib to train the model. Users can call `summary` to print a summary of the fitted model, `predict` to make predictions on new data, and `write.ml`/`read.ml` to save/load fitted models.\n+\n+SparkR supports a subset of R formula operators for model fitting, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. We use linear regression as an example.\n+```{r}\n+model <- spark.glm(carsDF, mpg ~ wt + cyl)\n+```\n+\n+```{r}\n+summary(model)\n+```\n+\n+The model can be saved by `write.ml` and loaded back using `read.ml`.\n+```{r, eval=FALSE}\n+write.ml(model, path = \"/HOME/tmp/mlModel/glmModel\")\n+```\n+\n+In the end, we can stop Spark Session by running\n+```{r, eval=FALSE}\n+sparkR.session.stop()\n+```\n+\n+## Setup\n+\n+### Installation\n+\n+Different from many other R packages, to use SparkR, you need an additional installation of Apache Spark. The Spark installation will be used to run a backend process that will compile and execute SparkR programs.\n+\n+If you don't have Spark installed on the computer, you may download it from [Apache Spark Website](http://spark.apache.org/downloads.html). Alternatively, we provide an easy-to-use function `install.spark` to complete this process.\n+\n+```{r, eval=FALSE}\n+install.spark()\n+```\n+\n+If you already have Spark installed, you don't have to install again and can pass the `sparkHome` argument to `sparkR.session` to let SparkR know where the Spark installation is.\n+\n+```{r, eval=FALSE}\n+sparkR.session(sparkHome = \"/HOME/spark\")\n+```\n+\n+### Spark Session {#SetupSparkSession}\n+\n+**For Windows users**: Due to different file prefixes across operating systems, to avoid the issue of potential wrong prefix, a current workaround is to specify `spark.sql.warehouse.dir` when starting the `SparkSession`.\n+\n+```{r, eval=FALSE}\n+spark_warehouse_path <- file.path(path.expand('~'), \"spark-warehouse\")\n+sparkR.session(spark.sql.warehouse.dir = spark_warehouse_path)\n+```\n+\n+In addition to `sparkHome`, many other options can be specified in `sparkR.session`. For a complete list, see the [SparkR API doc](http://spark.apache.org/docs/latest/api/R/sparkR.session.html).","path":"R/pkg/vignettes/sparkr-vignettes.Rmd","position":115,"original_position":115,"commit_id":"4386ab35b13c361bb471cbf039a980e8bddc1cc3","original_commit_id":"4386ab35b13c361bb471cbf039a980e8bddc1cc3","user":{"login":"junyangq","id":15318264,"avatar_url":"https://avatars.githubusercontent.com/u/15318264?v=3","gravatar_id":"","url":"https://api.github.com/users/junyangq","html_url":"https://github.com/junyangq","followers_url":"https://api.github.com/users/junyangq/followers","following_url":"https://api.github.com/users/junyangq/following{/other_user}","gists_url":"https://api.github.com/users/junyangq/gists{/gist_id}","starred_url":"https://api.github.com/users/junyangq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junyangq/subscriptions","organizations_url":"https://api.github.com/users/junyangq/orgs","repos_url":"https://api.github.com/users/junyangq/repos","events_url":"https://api.github.com/users/junyangq/events{/privacy}","received_events_url":"https://api.github.com/users/junyangq/received_events","type":"User","site_admin":false},"body":"That's true... Perhaps what we actually need is to add more stuff in `sparkR.session` help file?","created_at":"2016-09-07T08:28:12Z","updated_at":"2016-09-07T08:28:12Z","html_url":"https://github.com/apache/spark/pull/14980#discussion_r77779369","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/14980","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/77779369"},"html":{"href":"https://github.com/apache/spark/pull/14980#discussion_r77779369"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/14980"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14980","id":84178635,"html_url":"https://github.com/apache/spark/pull/14980","diff_url":"https://github.com/apache/spark/pull/14980.diff","patch_url":"https://github.com/apache/spark/pull/14980.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/14980","number":14980,"state":"open","locked":false,"title":"[SPARK-17317][SparkR] Add SparkR vignette","user":{"login":"junyangq","id":15318264,"avatar_url":"https://avatars.githubusercontent.com/u/15318264?v=3","gravatar_id":"","url":"https://api.github.com/users/junyangq","html_url":"https://github.com/junyangq","followers_url":"https://api.github.com/users/junyangq/followers","following_url":"https://api.github.com/users/junyangq/following{/other_user}","gists_url":"https://api.github.com/users/junyangq/gists{/gist_id}","starred_url":"https://api.github.com/users/junyangq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junyangq/subscriptions","organizations_url":"https://api.github.com/users/junyangq/orgs","repos_url":"https://api.github.com/users/junyangq/repos","events_url":"https://api.github.com/users/junyangq/events{/privacy}","received_events_url":"https://api.github.com/users/junyangq/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR tries to add a SparkR vignette, which works as a friendly guidance going through the functionality provided by SparkR. \r\n\r\n## How was this patch tested?\r\n\r\nManual test.\r\n\r\n","created_at":"2016-09-06T19:08:59Z","updated_at":"2016-09-07T08:28:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c48c8674022d16c4f5674df42964acb59ad38b39","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/14980/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/14980/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/14980/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/4386ab35b13c361bb471cbf039a980e8bddc1cc3","head":{"label":"junyangq:SPARKR-vignette","ref":"SPARKR-vignette","sha":"4386ab35b13c361bb471cbf039a980e8bddc1cc3","user":{"login":"junyangq","id":15318264,"avatar_url":"https://avatars.githubusercontent.com/u/15318264?v=3","gravatar_id":"","url":"https://api.github.com/users/junyangq","html_url":"https://github.com/junyangq","followers_url":"https://api.github.com/users/junyangq/followers","following_url":"https://api.github.com/users/junyangq/following{/other_user}","gists_url":"https://api.github.com/users/junyangq/gists{/gist_id}","starred_url":"https://api.github.com/users/junyangq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junyangq/subscriptions","organizations_url":"https://api.github.com/users/junyangq/orgs","repos_url":"https://api.github.com/users/junyangq/repos","events_url":"https://api.github.com/users/junyangq/events{/privacy}","received_events_url":"https://api.github.com/users/junyangq/received_events","type":"User","site_admin":false},"repo":{"id":45086006,"name":"spark","full_name":"junyangq/spark","owner":{"login":"junyangq","id":15318264,"avatar_url":"https://avatars.githubusercontent.com/u/15318264?v=3","gravatar_id":"","url":"https://api.github.com/users/junyangq","html_url":"https://github.com/junyangq","followers_url":"https://api.github.com/users/junyangq/followers","following_url":"https://api.github.com/users/junyangq/following{/other_user}","gists_url":"https://api.github.com/users/junyangq/gists{/gist_id}","starred_url":"https://api.github.com/users/junyangq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junyangq/subscriptions","organizations_url":"https://api.github.com/users/junyangq/orgs","repos_url":"https://api.github.com/users/junyangq/repos","events_url":"https://api.github.com/users/junyangq/events{/privacy}","received_events_url":"https://api.github.com/users/junyangq/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/junyangq/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/junyangq/spark","forks_url":"https://api.github.com/repos/junyangq/spark/forks","keys_url":"https://api.github.com/repos/junyangq/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/junyangq/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/junyangq/spark/teams","hooks_url":"https://api.github.com/repos/junyangq/spark/hooks","issue_events_url":"https://api.github.com/repos/junyangq/spark/issues/events{/number}","events_url":"https://api.github.com/repos/junyangq/spark/events","assignees_url":"https://api.github.com/repos/junyangq/spark/assignees{/user}","branches_url":"https://api.github.com/repos/junyangq/spark/branches{/branch}","tags_url":"https://api.github.com/repos/junyangq/spark/tags","blobs_url":"https://api.github.com/repos/junyangq/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/junyangq/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/junyangq/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/junyangq/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/junyangq/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/junyangq/spark/languages","stargazers_url":"https://api.github.com/repos/junyangq/spark/stargazers","contributors_url":"https://api.github.com/repos/junyangq/spark/contributors","subscribers_url":"https://api.github.com/repos/junyangq/spark/subscribers","subscription_url":"https://api.github.com/repos/junyangq/spark/subscription","commits_url":"https://api.github.com/repos/junyangq/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/junyangq/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/junyangq/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/junyangq/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/junyangq/spark/contents/{+path}","compare_url":"https://api.github.com/repos/junyangq/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/junyangq/spark/merges","archive_url":"https://api.github.com/repos/junyangq/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/junyangq/spark/downloads","issues_url":"https://api.github.com/repos/junyangq/spark/issues{/number}","pulls_url":"https://api.github.com/repos/junyangq/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/junyangq/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/junyangq/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/junyangq/spark/labels{/name}","releases_url":"https://api.github.com/repos/junyangq/spark/releases{/id}","deployments_url":"https://api.github.com/repos/junyangq/spark/deployments","created_at":"2015-10-28T03:18:33Z","updated_at":"2015-10-28T03:19:03Z","pushed_at":"2016-09-06T19:39:09Z","git_url":"git://github.com/junyangq/spark.git","ssh_url":"git@github.com:junyangq/spark.git","clone_url":"https://github.com/junyangq/spark.git","svn_url":"https://github.com/junyangq/spark","homepage":null,"size":182591,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"f7e26d788757f917b32749856bb29feb7b4c2987","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-09-07T07:10:03Z","pushed_at":"2016-09-07T06:56:08Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":523497,"stargazers_count":9916,"watchers_count":9916,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":9239,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":475,"forks":9239,"open_issues":475,"watchers":9916,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/14980"},"html":{"href":"https://github.com/apache/spark/pull/14980"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/14980"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/14980/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/14980/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/14980/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/4386ab35b13c361bb471cbf039a980e8bddc1cc3"}}}}
{"actor":{"display_login":"junyangq"}}