{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12004","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12004/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12004/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12004/events","html_url":"https://github.com/apache/spark/pull/12004","id":144038933,"number":12004,"title":"[SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud module to pull in object store access, plus tests","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2016-03-28T17:57:00Z","updated_at":"2016-08-23T18:02:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12004","html_url":"https://github.com/apache/spark/pull/12004","diff_url":"https://github.com/apache/spark/pull/12004.diff","patch_url":"https://github.com/apache/spark/pull/12004.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n[SPARK-7481] Add Hadoop 2.6+ profile to pull in object store FS accessors in hadoop-openstack, hadoop-aws and hadoop-azure. \r\n\r\nAs a result, it gets the Hadoop classes s3n:// support back into spark-assembly; adds s3a and openstack on Hadoop 2.6, and on Hadoop 2.7 adds azure support. It does not add the dependencies needed for s3a or azure\r\n\r\n- spark-assembly has an explicit dependency on jets3t; this is used by s3n\r\n- s3a needs a (large) amazon-aws JAR in Hadoop 2.6; Hadoop 2.7 has switched to a leaner amazon-aws-s3 JAR.\r\n- azure needs a microsoft azure storage JAR\r\n- openstack reuses JARs already in the assembly and adds one: commons-io. This is not excluded from spark-assembly, though it is easy to add that.\r\n\r\nThe patch defines a new module, \"cloud\" with transitive dependencies on the amazon (hadoop 2.6+) and amazon + azure (hadoop 2.7+) JARs. The spark assembly JAR pulls in spark-cloud and its hadoop dependencies (scoped at hadoop-provided) —but excludes those external dependencies. Accordingly the (small) Hadoop FS client code is ready whenever someone adds in the larger cloud-specific JARs and requests a specific filesystem.\r\n\r\nHaving an explicit module allows for followup work, specifically some tests. It also enables downstream applications to declare their dependency upon `spark-cloud` and get the the object store accessors (and anything else people choose to add in future)\r\n\r\n## How was this patch tested?\r\n\r\nThe dependency logic was verified via maven dependency checking; the inclusion of the hadoop code and exclusion of com.microsoft and com.amazon files checked by examining the contents of the assembly JAR; that check could be automated.\r\n\r\nThere are tests for basic operations against S3 and Azure; these need logins with the relevant providers.\r\nThe tests will run if the maven property `cloud.test.configuration.file` points to an Hadoop XML file containing (directly or [indirectly](http://steveloughran.blogspot.co.uk/2016/04/testing-against-s3-and-object-stores.html) the keys needed to work with the object stores. \r\nThere are tests for S3 which use a 20+MB csv file of Amazon's as a source for basic Spark queries, so verifying the entire read chain.\r\n\r\nThe test suites make individual test cases conditional on the relevant credentials for that filestore being present. They also have a short keyname for every test (which must be unique in that class), such as `CSVgz`; this can then be invoked on the command line: \r\n```\r\n -Dtest.method.keys=CSVgz\r\n```\r\n\r\nAll other methods in the suite get ignored. This feature is needed as the maven scalatest doesn't directly offer the Maven/Junit \"run a method by name feature\", and with some of the test runs taking a noticeable number of seconds, isolation helps review the what is happening.\r\n\r\nA test case looks as follows\r\n\r\n```scala\r\n  ctest(\"CSVgz\", \"Read compressed CSV\",\r\n    \"Read compressed CSV files through the spark context\") {\r\n    val source = SceneList\r\n    sc = new SparkContext(\"local\", \"test\", newSparkConf(source))\r\n    val sceneInfo = getFS(source).getFileStatus(source)\r\n    logInfo(s\"Compressed size = ${sceneInfo.getLen}\")\r\n    validateCSV(sc, source)\r\n    logInfo(s\"Filesystem statistics ${getFS(source)}\")\r\n  }\r\n```\r\nThe module has been tested against: Hadoop 2.7.1, branch-2.8 (2.8.0-SNAPSHOT) and branch-2 (2.9.0-SNAPSHOT); the latter two have better performance and will print collected statistics in their `toString()` calls, which is why the tests log them.\r\n\r\nWhen run against Hadoop 2.6, while the tests complete, performance against s3a is atrocious. This is due to [HADOOP-11584](https://issues.apache.org/jira/browse/HADOOP-11584); the fact that the test shows the problem demonstrates why full stack tests are needed here."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/241820455","html_url":"https://github.com/apache/spark/pull/12004#issuecomment-241820455","issue_url":"https://api.github.com/repos/apache/spark/issues/12004","id":241820455,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-08-23T18:02:12Z","updated_at":"2016-08-23T18:02:12Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}