{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12004","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12004/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12004/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12004/events","html_url":"https://github.com/apache/spark/pull/12004","id":144038933,"number":12004,"title":"[SPARK-7481] [build] Add spark-cloud module to pull in object store access, + documentation","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":66,"created_at":"2016-03-28T17:57:00Z","updated_at":"2016-12-01T14:40:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12004","html_url":"https://github.com/apache/spark/pull/12004","diff_url":"https://github.com/apache/spark/pull/12004.diff","patch_url":"https://github.com/apache/spark/pull/12004.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAdd a new `cloud` module and maven profile to pull in object store support from `hadoop-openstack`, `hadoop-aws` and `hadoop-azure` JARs, along with their dependencies. \r\n\r\nAs a result, it restores `s3n://` access to S3, adds its `s3a://` replacement , OpenStack `swift://` and azure `wasb://`.\r\n\r\nThe patch defines a new module, `spark-cloud` with transitive dependencies on the amazon and azure JARs. The spark assembly module JAR depends on this module to pull the relevant hadoop and dependent JARs into `SPARK_HOME/jars`. This module is only included in a build if the -`cloud` profile is set; there's an implicit dependency on the `hadoop-2.7` module too, because that is where the various hadoop-$CLOUD modules are imported.\r\n\r\nThere's a documentation page, `cloud_integration.md`, which covers the details of using Spark with object stores: API level, the different object stores, limitations, recommended tuning options. All the examples are in Scala. The docs try to be an introduction which avoids cut-and-paste text which will age fast against different versions, pointing at the Hadoop docs.\r\n\r\nHaving an explicit module enables downstream applications to declare their dependency upon `spark-cloud` and get object store access. This is used for the integration tests, which therefore implicitly test the correctness of the declared dependencies.\r\n\r\nThere is no import or coverage of the google cloud storage gs:// storage; that's not an ASF project & we've actually had some problems with gs; at the time of writing it hasn't implemented the standard Hadoop FS contract tests, so it's not clear what state is in. That could be future work.\r\n\r\nThere is a mention but no details on Amazon EMR's S3:// client. It's essentially says \"Amazon's code, their problem\", which is the case.\r\n\r\n## How was this patch tested?\r\n\r\nThere are no functional tests in this module; it is left to downstream projects to implement integration tests. I have such tests in [https://github.com/steveloughran/spark-cloud-examples/tree/master/cloud-examples](https://github.com/steveloughran/spark-cloud-examples/tree/master/cloud-examples)\r\n\r\nThe module has been tested against: Hadoop 2.7.3, branch-2.8 (2.8.0-SNAPSHOT), branch-2 (2.9.0-SNAPSHOT) and Hadoop trunk; \r\n\r\nTested endpoints: AWS S3 US-East, AWS S3 Ireland, Azure (US and EU), and Rackspace Swift US and EU,\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/264189673","html_url":"https://github.com/apache/spark/pull/12004#issuecomment-264189673","issue_url":"https://api.github.com/repos/apache/spark/issues/12004","id":264189673,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-12-01T14:40:50Z","updated_at":"2016-12-01T14:40:50Z","body":"**[Test build #69480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69480/consoleFull)** for PR 12004 at commit [`e9b8ed0`](https://github.com/apache/spark/commit/e9b8ed0b47b300cfe6bb64d2d2622b842fede142).\n * This patch **fails build dependency tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}