{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12004","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12004/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12004/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12004/events","html_url":"https://github.com/apache/spark/pull/12004","id":144038933,"number":12004,"title":"[SPARK-7481][build][WIP] Add Hadoop 2.6+ spark-cloud module to pull in object store access, plus tests","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-03-28T17:57:00Z","updated_at":"2016-04-26T13:37:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12004","html_url":"https://github.com/apache/spark/pull/12004","diff_url":"https://github.com/apache/spark/pull/12004.diff","patch_url":"https://github.com/apache/spark/pull/12004.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n[SPARK-7481] Add Hadoop 2.6+ profile to pull in object store FS accessors in hadoop-openstack, hadoop-aws and hadoop-azure. \r\n\r\nAs a result, it gets the Hadoop classes s3n:// support back into spark-assembly; adds s3a and openstack on Hadoop 2.6, and on Hadoop 2.7 adds azure support. It does not add the dependencies needed for s3a or azure\r\n\r\n- spark-assembly has an explicit dependency on jets3t; this is used by s3n\r\n- s3a needs a (large) amazon-aws JAR in Hadoop 2.6; Hadoop 2.7 has switched to a leaner amazon-aws-s3 JAR.\r\n- azure needs a microsoft azure storage JAR\r\n- openstack reuses JARs already in the assembly and adds one: commons-io. This is not excluded from spark-assembly, though it is easy to add that.\r\n\r\nThe patch defines a new module, \"cloud\" with transitive dependencies on the amazon (hadoop 2.6+) and azure (hadoop 2.7+) JARs. The spark assembly JAR pulls in spark-cloud and its hadoop dependencies (scoped at hadoop-provided) —but excludes those external dependencies. \r\n\r\nHaving an explicit module allows for followup work, specifically some tests. It also enables downstream applications to declare their dependency upon `spark-cloud` and get the the object store accessors (and anything else people choose to add in future)\r\n\r\n## How was this patch tested?\r\n\r\nThe dependency logic was verified via maven dependency checking; the inclusion of the hadoop code and exclusion of com.microsoft and com.amazon files checked by examining the contents of the assembly JAR; that check could be automated.\r\n\r\nFor testing that the spark integration with s3a, wasb, etc works, I'd propose a followup piece of work. That'd add to spark-cloud some tests and changes to the pom needed to pass down some environment options for running the tests, skipping them if the credentials are not provided."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/214747320","html_url":"https://github.com/apache/spark/pull/12004#issuecomment-214747320","issue_url":"https://api.github.com/repos/apache/spark/issues/12004","id":214747320,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-04-26T13:37:37Z","updated_at":"2016-04-26T13:37:37Z","body":"**[Test build #56998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56998/consoleFull)** for PR 12004 at commit [`8926acb`](https://github.com/apache/spark/commit/8926acb25e05f5d0748155c26262aae0d54fb3d0)."}}