{"action":"created","comment":{"url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/comments/65785721","id":65785721,"diff_hunk":"@@ -12,14 +12,32 @@ The Sparkmagic project includes a set of magics for interactively running Spark\n ## Features\n \n * Run Spark code in multiple languages against any remote Spark cluster through Livy\n-\n+* Automatic SparkContext (`sc`) and HiveContext (`sqlContext`) creation\n+* Ability to execute SparkSQL queries\n * Automatic visualization of SQL queries with the `%%sql` magic in the PySpark and Spark kernels; use an easy visual interface to interactively construct visualizations, no code required\n-\n-* Capture the output of SQL queries as Pandas dataframes to work with them on your local machine\n+* Easy access to Spark application information and logs (`%%info` magic)\n+* Ability to capture the output of SQL queries as Pandas dataframes to interact with other Python libraries (e.g. matplotlib)\n \n ## Examples\n \n-Check out the [examples](examples) directory.\n+There are two ways to use sparkmagic. Head over to the [examples](examples) section for a demonstration on how to use both models of execution.\n+\n+### 1. Via the IPython kernel\n+\n+This model of execution works great for when you are working with data both small and large. The scenario is as follows:\n+\n+        You are a Data Scientist in the IPython kernel doing data analysis, and you have some Big Data need from time to time. For this, you would like to run some Spark code in your Spark cluster from the convenience of the IPython notebook you always use.\n+\n+sparkmagic allows you to do so by creating some IPython kernel magics that allow you to execute code in your remote Spark cluster and retrieve the results back to the IPython kernel.\n+\n+### 2. Via the PySpark and Spark kernels\n+\n+This model of execution allows you to execute all your code in the Spark cluster. These are, in essence, Python and Scala kernels completely dedicated to Spark code execution.\n+The scenario is as follows:\n+\n+        You are a Data Scientist that would like to explore Big-Data-size datasets via a Spark cluster. In order to do so, a dedicated Python or Scala kernel is better suited.\n+\n+If you want to see how to use all the features in these kernels, simply execute the `%%help` magic in them.","path":"README.md","position":33,"original_position":33,"commit_id":"18b343925e64cbd0eda481515421e4c7bf53d278","original_commit_id":"18b343925e64cbd0eda481515421e4c7bf53d278","user":{"login":"msftristew","id":13972471,"avatar_url":"https://avatars.githubusercontent.com/u/13972471?v=3","gravatar_id":"","url":"https://api.github.com/users/msftristew","html_url":"https://github.com/msftristew","followers_url":"https://api.github.com/users/msftristew/followers","following_url":"https://api.github.com/users/msftristew/following{/other_user}","gists_url":"https://api.github.com/users/msftristew/gists{/gist_id}","starred_url":"https://api.github.com/users/msftristew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msftristew/subscriptions","organizations_url":"https://api.github.com/users/msftristew/orgs","repos_url":"https://api.github.com/users/msftristew/repos","events_url":"https://api.github.com/users/msftristew/events{/privacy}","received_events_url":"https://api.github.com/users/msftristew/received_events","type":"User","site_admin":false},"body":"IMO these paragraphs are pretty wordy and fail to accurately communicate the use cases.  Try something like this:\r\n\r\n### 1. Via the IPython kernel\r\n\r\nThe sparkmagic library provides a `%%spark` magic that you can use to easily run code against a remote Spark cluster from a normal IPython notebook.  See (link to example notebook here)\r\n\r\n### 2. Via the PySpark and Spark kernels\r\n\r\nThe sparkmagic library also provides a set of kernels that allow you to automatically connect to a remote Spark cluster, run code and SQL queries, manage your Livy server and Spark job configuration, and generate automatic visualizations.  See (link to example notebook here)","created_at":"2016-06-03T22:37:34Z","updated_at":"2016-06-03T22:37:34Z","html_url":"https://github.com/jupyter-incubator/sparkmagic/pull/245#discussion_r65785721","pull_request_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245","_links":{"self":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/comments/65785721"},"html":{"href":"https://github.com/jupyter-incubator/sparkmagic/pull/245#discussion_r65785721"},"pull_request":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245"}}},"pull_request":{"url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245","id":72577588,"html_url":"https://github.com/jupyter-incubator/sparkmagic/pull/245","diff_url":"https://github.com/jupyter-incubator/sparkmagic/pull/245.diff","patch_url":"https://github.com/jupyter-incubator/sparkmagic/pull/245.patch","issue_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/245","number":245,"state":"open","locked":false,"title":"Add magics in ipython kernel sample notebook","user":{"login":"aggFTW","id":654204,"avatar_url":"https://avatars.githubusercontent.com/u/654204?v=3","gravatar_id":"","url":"https://api.github.com/users/aggFTW","html_url":"https://github.com/aggFTW","followers_url":"https://api.github.com/users/aggFTW/followers","following_url":"https://api.github.com/users/aggFTW/following{/other_user}","gists_url":"https://api.github.com/users/aggFTW/gists{/gist_id}","starred_url":"https://api.github.com/users/aggFTW/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aggFTW/subscriptions","organizations_url":"https://api.github.com/users/aggFTW/orgs","repos_url":"https://api.github.com/users/aggFTW/repos","events_url":"https://api.github.com/users/aggFTW/events{/privacy}","received_events_url":"https://api.github.com/users/aggFTW/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-06-03T22:16:04Z","updated_at":"2016-06-03T22:37:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2b593bf2bcd2eab5e5f11027ee977759d0b0f873","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245/commits","review_comments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245/comments","review_comment_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/comments{/number}","comments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/245/comments","statuses_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/statuses/18b343925e64cbd0eda481515421e4c7bf53d278","head":{"label":"jupyter-incubator:Samples","ref":"Samples","sha":"18b343925e64cbd0eda481515421e4c7bf53d278","user":{"login":"jupyter-incubator","id":14002361,"avatar_url":"https://avatars.githubusercontent.com/u/14002361?v=3","gravatar_id":"","url":"https://api.github.com/users/jupyter-incubator","html_url":"https://github.com/jupyter-incubator","followers_url":"https://api.github.com/users/jupyter-incubator/followers","following_url":"https://api.github.com/users/jupyter-incubator/following{/other_user}","gists_url":"https://api.github.com/users/jupyter-incubator/gists{/gist_id}","starred_url":"https://api.github.com/users/jupyter-incubator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jupyter-incubator/subscriptions","organizations_url":"https://api.github.com/users/jupyter-incubator/orgs","repos_url":"https://api.github.com/users/jupyter-incubator/repos","events_url":"https://api.github.com/users/jupyter-incubator/events{/privacy}","received_events_url":"https://api.github.com/users/jupyter-incubator/received_events","type":"Organization","site_admin":false},"repo":{"id":42875585,"name":"sparkmagic","full_name":"jupyter-incubator/sparkmagic","owner":{"login":"jupyter-incubator","id":14002361,"avatar_url":"https://avatars.githubusercontent.com/u/14002361?v=3","gravatar_id":"","url":"https://api.github.com/users/jupyter-incubator","html_url":"https://github.com/jupyter-incubator","followers_url":"https://api.github.com/users/jupyter-incubator/followers","following_url":"https://api.github.com/users/jupyter-incubator/following{/other_user}","gists_url":"https://api.github.com/users/jupyter-incubator/gists{/gist_id}","starred_url":"https://api.github.com/users/jupyter-incubator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jupyter-incubator/subscriptions","organizations_url":"https://api.github.com/users/jupyter-incubator/orgs","repos_url":"https://api.github.com/users/jupyter-incubator/repos","events_url":"https://api.github.com/users/jupyter-incubator/events{/privacy}","received_events_url":"https://api.github.com/users/jupyter-incubator/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/jupyter-incubator/sparkmagic","description":"Jupyter magics and kernels for working with remote Spark clusters","fork":false,"url":"https://api.github.com/repos/jupyter-incubator/sparkmagic","forks_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/forks","keys_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/teams","hooks_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/hooks","issue_events_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/events{/number}","events_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/events","assignees_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/assignees{/user}","branches_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/branches{/branch}","tags_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/tags","blobs_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/refs{/sha}","trees_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/statuses/{sha}","languages_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/languages","stargazers_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/stargazers","contributors_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/contributors","subscribers_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/subscribers","subscription_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/subscription","commits_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/commits{/sha}","git_commits_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/commits{/sha}","comments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/comments{/number}","issue_comment_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/comments{/number}","contents_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/contents/{+path}","compare_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/merges","archive_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/downloads","issues_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues{/number}","pulls_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls{/number}","milestones_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/milestones{/number}","notifications_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/labels{/name}","releases_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/releases{/id}","deployments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/deployments","created_at":"2015-09-21T15:35:25Z","updated_at":"2016-06-02T00:31:03Z","pushed_at":"2016-06-03T22:16:04Z","git_url":"git://github.com/jupyter-incubator/sparkmagic.git","ssh_url":"git@github.com:jupyter-incubator/sparkmagic.git","clone_url":"https://github.com/jupyter-incubator/sparkmagic.git","svn_url":"https://github.com/jupyter-incubator/sparkmagic","homepage":"","size":3107,"stargazers_count":67,"watchers_count":67,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"open_issues_count":11,"forks":10,"open_issues":11,"watchers":67,"default_branch":"master"}},"base":{"label":"jupyter-incubator:master","ref":"master","sha":"e1e4fc37b70bacba3bc81d954ef63f729dbab6e2","user":{"login":"jupyter-incubator","id":14002361,"avatar_url":"https://avatars.githubusercontent.com/u/14002361?v=3","gravatar_id":"","url":"https://api.github.com/users/jupyter-incubator","html_url":"https://github.com/jupyter-incubator","followers_url":"https://api.github.com/users/jupyter-incubator/followers","following_url":"https://api.github.com/users/jupyter-incubator/following{/other_user}","gists_url":"https://api.github.com/users/jupyter-incubator/gists{/gist_id}","starred_url":"https://api.github.com/users/jupyter-incubator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jupyter-incubator/subscriptions","organizations_url":"https://api.github.com/users/jupyter-incubator/orgs","repos_url":"https://api.github.com/users/jupyter-incubator/repos","events_url":"https://api.github.com/users/jupyter-incubator/events{/privacy}","received_events_url":"https://api.github.com/users/jupyter-incubator/received_events","type":"Organization","site_admin":false},"repo":{"id":42875585,"name":"sparkmagic","full_name":"jupyter-incubator/sparkmagic","owner":{"login":"jupyter-incubator","id":14002361,"avatar_url":"https://avatars.githubusercontent.com/u/14002361?v=3","gravatar_id":"","url":"https://api.github.com/users/jupyter-incubator","html_url":"https://github.com/jupyter-incubator","followers_url":"https://api.github.com/users/jupyter-incubator/followers","following_url":"https://api.github.com/users/jupyter-incubator/following{/other_user}","gists_url":"https://api.github.com/users/jupyter-incubator/gists{/gist_id}","starred_url":"https://api.github.com/users/jupyter-incubator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jupyter-incubator/subscriptions","organizations_url":"https://api.github.com/users/jupyter-incubator/orgs","repos_url":"https://api.github.com/users/jupyter-incubator/repos","events_url":"https://api.github.com/users/jupyter-incubator/events{/privacy}","received_events_url":"https://api.github.com/users/jupyter-incubator/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/jupyter-incubator/sparkmagic","description":"Jupyter magics and kernels for working with remote Spark clusters","fork":false,"url":"https://api.github.com/repos/jupyter-incubator/sparkmagic","forks_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/forks","keys_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/teams","hooks_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/hooks","issue_events_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/events{/number}","events_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/events","assignees_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/assignees{/user}","branches_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/branches{/branch}","tags_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/tags","blobs_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/refs{/sha}","trees_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/statuses/{sha}","languages_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/languages","stargazers_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/stargazers","contributors_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/contributors","subscribers_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/subscribers","subscription_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/subscription","commits_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/commits{/sha}","git_commits_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/git/commits{/sha}","comments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/comments{/number}","issue_comment_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/comments{/number}","contents_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/contents/{+path}","compare_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/merges","archive_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/downloads","issues_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues{/number}","pulls_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls{/number}","milestones_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/milestones{/number}","notifications_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/labels{/name}","releases_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/releases{/id}","deployments_url":"https://api.github.com/repos/jupyter-incubator/sparkmagic/deployments","created_at":"2015-09-21T15:35:25Z","updated_at":"2016-06-02T00:31:03Z","pushed_at":"2016-06-03T22:16:04Z","git_url":"git://github.com/jupyter-incubator/sparkmagic.git","ssh_url":"git@github.com:jupyter-incubator/sparkmagic.git","clone_url":"https://github.com/jupyter-incubator/sparkmagic.git","svn_url":"https://github.com/jupyter-incubator/sparkmagic","homepage":"","size":3107,"stargazers_count":67,"watchers_count":67,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"open_issues_count":11,"forks":10,"open_issues":11,"watchers":67,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245"},"html":{"href":"https://github.com/jupyter-incubator/sparkmagic/pull/245"},"issue":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/245"},"comments":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/issues/245/comments"},"review_comments":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245/comments"},"review_comment":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/pulls/245/commits"},"statuses":{"href":"https://api.github.com/repos/jupyter-incubator/sparkmagic/statuses/18b343925e64cbd0eda481515421e4c7bf53d278"}}}}
{"actor":{"display_login":"msftristew"}}