{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15659","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15659/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15659/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15659/events","html_url":"https://github.com/apache/spark/pull/15659","id":185607114,"number":15659,"title":"[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed","user":{"login":"holdenk","id":59893,"avatar_url":"https://avatars.githubusercontent.com/u/59893?v=3","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":130,"created_at":"2016-10-27T08:43:01Z","updated_at":"2016-11-12T20:02:32Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15659","html_url":"https://github.com/apache/spark/pull/15659","diff_url":"https://github.com/apache/spark/pull/15659.diff","patch_url":"https://github.com/apache/spark/pull/15659.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR aims to provide a pip installable PySpark package. This does a bunch of work to copy the jars over and package them with the Python code (to prevent challenges from trying to use different versions of the Python code with different versions of the JAR). It does not currently publish to PyPI but that is the natural follow up (SPARK-18129).\r\n\r\nDone:\r\n- pip installable on conda [manual tested]\r\n- setup.py installed on a non-pip managed system (RHEL) with YARN [manual tested]\r\n- Automated testing of this (virtualenv)\r\n- packaging and signing with release-build*\r\n\r\nPossible follow up work:\r\n- release-build update to publish to PyPI (SPARK-18128)\r\n- figure out who owns the pyspark package name on prod PyPI (is it someone with in the project or should we ask PyPI or should we choose a different name to publish with like ApachePySpark?)\r\n- Windows support and or testing ( SPARK-18136 )\r\n- investigate details of wheel caching and see if we can avoid cleaning the wheel cache during our test\r\n- consider how we want to number our dev/snapshot versions\r\n\r\nExplicitly out of scope:\r\n- Using pip installed PySpark to start a standalone cluster\r\n- Using pip installed PySpark for non-Python Spark programs\r\n\r\n*I've done some work to test release-build locally but as a non-committer I've just done local testing.\r\n## How was this patch tested?\r\n\r\nAutomated testing with virtualenv, manual testing with conda, a system wide install, and YARN integration.\r\n\r\nrelease-build changes tested locally as a non-committer (no testing of upload artifacts to Apache staging websites)\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/260144645","html_url":"https://github.com/apache/spark/pull/15659#issuecomment-260144645","issue_url":"https://api.github.com/repos/apache/spark/issues/15659","id":260144645,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-11-12T20:02:32Z","updated_at":"2016-11-12T20:02:32Z","body":"**[Test build #68570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68570/consoleFull)** for PR 15659 at commit [`3345eb9`](https://github.com/apache/spark/commit/3345eb9bd3f0a165ae1d20d76e5d36e88e5512cd)."}}
{"actor":{"display_login":"SparkQA"}}