{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/89346124","pull_request_review_id":9892030,"id":89346124,"diff_hunk":"@@ -0,0 +1,29 @@\n+/*","path":"cloud/src/test/scala/org/apache/spark/cloud/AzureInstantiationSuite.scala","position":1,"original_position":1,"commit_id":"9726d6c857439b870e08d49889a5da3c35a708e3","original_commit_id":"9726d6c857439b870e08d49889a5da3c35a708e3","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"body":"In the absence of any real tests, these do check that the transient stuff is picked up, and, at a surface level, there aren't fundamental differences between jackson versions.\r\n\r\nI have a copy of the same tests in my downstream integration tests, where they are the first stop for detecting problems, the advantage being you don't need any credentials to run the tests —and they validate the spark-cloud exports, rather than the spark-cloud test CP. I'll cut them from here if you want","created_at":"2016-11-23T16:02:10Z","updated_at":"2016-11-23T16:02:10Z","html_url":"https://github.com/apache/spark/pull/12004#discussion_r89346124","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/12004","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/89346124"},"html":{"href":"https://github.com/apache/spark/pull/12004#discussion_r89346124"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/12004"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12004","id":64350614,"html_url":"https://github.com/apache/spark/pull/12004","diff_url":"https://github.com/apache/spark/pull/12004.diff","patch_url":"https://github.com/apache/spark/pull/12004.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/12004","number":12004,"state":"open","locked":false,"title":"[SPARK-7481] [build] Add spark-cloud module to pull in object store access, + documentation","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nAdd a new `cloud` module and maven profile to pull in object store support from `hadoop-openstack`, `hadoop-aws` and `hadoop-azure` JARs, along with their dependencies. \r\n\r\nAs a result, it restores `s3n://` access to S3, adds its `s3a://` replacement , OpenStack `swift://` and azure `wasb://`.\r\n\r\nThe patch defines a new module, cloud\" with transitive dependencies on the amazon and azure JARs. The spark assembly module JAR depends on this module to pull the relevant hadoop and dependent JARs into `SPARK_HOME/jars`. This module is only included in a build if the -`cloud` profile is set; there's an implicit dependency on the `hadoop-2.7` module too, because that is where the various hadoop-$CLOUD modules are imported.\r\n\r\nThere's a documentation page, `cloud_integration.md`, which covers the details of using Spark with object stores: API level, the different object stores, limitations, recommended tuning options. All the examples are in Scala. The docs try to be an introduction which avoids cut-and-paste text which will age fast against different versions, pointing at the Hadoop docs.\r\n\r\nHaving an explicit module enables downstream applications to declare their dependency upon `spark-cloud` and get object store access. This is used for the integration tests, which therefore implicitly test the correctness of the declared dependencies.\r\n\r\nThere is no import or coverage of the google cloud storage gs:// storage; that's not an ASF project & we've actually had some problems with gs; at the time of writing it hasn't implemented the standard Hadoop FS contract tests, so it's not clear what state is in. That could be future work.\r\n\r\nThere is a mention but no details on Amazon EMR's S3:// client. It's essentially says \"Amazon's code, their problem\", which is the case.\r\n\r\n## How was this patch tested?\r\n\r\nThere are no functional tests in this module; it is left to downstream projects to implement integration tests. I have such tests in [https://github.com/steveloughran/spark-cloud-examples/tree/master/cloud-examples](https://github.com/steveloughran/spark-cloud-examples/tree/master/cloud-examples)\r\n\r\nThe module has been tested against: Hadoop 2.7.3, branch-2.8 (2.8.0-SNAPSHOT), branch-2 (2.9.0-SNAPSHOT) and Hadoop trunk; \r\n\r\nTested endpoints: AWS S3 US-East, AWS S3 Ireland, Azure (US and EU), and Rackspace Swift US and EU,\r\n","created_at":"2016-03-28T17:57:00Z","updated_at":"2016-11-23T16:02:10Z","closed_at":null,"merged_at":null,"merge_commit_sha":"80bcc18381aae98a5aeb1b4e0ea10d194a029854","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/12004/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/12004/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/12004/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/9726d6c857439b870e08d49889a5da3c35a708e3","head":{"label":"steveloughran:features/SPARK-7481-cloud","ref":"features/SPARK-7481-cloud","sha":"9726d6c857439b870e08d49889a5da3c35a708e3","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"repo":{"id":32387210,"name":"spark","full_name":"steveloughran/spark","owner":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/steveloughran/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/steveloughran/spark","forks_url":"https://api.github.com/repos/steveloughran/spark/forks","keys_url":"https://api.github.com/repos/steveloughran/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/steveloughran/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/steveloughran/spark/teams","hooks_url":"https://api.github.com/repos/steveloughran/spark/hooks","issue_events_url":"https://api.github.com/repos/steveloughran/spark/issues/events{/number}","events_url":"https://api.github.com/repos/steveloughran/spark/events","assignees_url":"https://api.github.com/repos/steveloughran/spark/assignees{/user}","branches_url":"https://api.github.com/repos/steveloughran/spark/branches{/branch}","tags_url":"https://api.github.com/repos/steveloughran/spark/tags","blobs_url":"https://api.github.com/repos/steveloughran/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/steveloughran/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/steveloughran/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/steveloughran/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/steveloughran/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/steveloughran/spark/languages","stargazers_url":"https://api.github.com/repos/steveloughran/spark/stargazers","contributors_url":"https://api.github.com/repos/steveloughran/spark/contributors","subscribers_url":"https://api.github.com/repos/steveloughran/spark/subscribers","subscription_url":"https://api.github.com/repos/steveloughran/spark/subscription","commits_url":"https://api.github.com/repos/steveloughran/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/steveloughran/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/steveloughran/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/steveloughran/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/steveloughran/spark/contents/{+path}","compare_url":"https://api.github.com/repos/steveloughran/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/steveloughran/spark/merges","archive_url":"https://api.github.com/repos/steveloughran/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/steveloughran/spark/downloads","issues_url":"https://api.github.com/repos/steveloughran/spark/issues{/number}","pulls_url":"https://api.github.com/repos/steveloughran/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/steveloughran/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/steveloughran/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/steveloughran/spark/labels{/name}","releases_url":"https://api.github.com/repos/steveloughran/spark/releases{/id}","deployments_url":"https://api.github.com/repos/steveloughran/spark/deployments","created_at":"2015-03-17T10:36:33Z","updated_at":"2015-03-17T10:36:52Z","pushed_at":"2016-11-21T12:56:33Z","git_url":"git://github.com/steveloughran/spark.git","ssh_url":"git@github.com:steveloughran/spark.git","clone_url":"https://github.com/steveloughran/spark.git","svn_url":"https://github.com/steveloughran/spark","homepage":null,"size":238779,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"658547974915ebcaae83e13e4c3bdf68d5426fda","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-11-23T14:34:16Z","pushed_at":"2016-11-23T15:25:29Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":613661,"stargazers_count":10836,"watchers_count":10836,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":10108,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":452,"forks":10108,"open_issues":452,"watchers":10836,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/12004"},"html":{"href":"https://github.com/apache/spark/pull/12004"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/12004"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/12004/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/12004/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/12004/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/9726d6c857439b870e08d49889a5da3c35a708e3"}}}}
{"actor":{"display_login":"steveloughran"}}