{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18329","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18329/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18329/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18329/events","html_url":"https://github.com/apache/spark/pull/18329","id":236502574,"number":18329,"title":"[SPARK-19909][SS] Disabling the usage of a temporary directory for the checkpoint location if the temporary directory is on a filesystem different from the default one.","user":{"login":"mgaido91","id":8821783,"avatar_url":"https://avatars1.githubusercontent.com/u/8821783?v=4","gravatar_id":"","url":"https://api.github.com/users/mgaido91","html_url":"https://github.com/mgaido91","followers_url":"https://api.github.com/users/mgaido91/followers","following_url":"https://api.github.com/users/mgaido91/following{/other_user}","gists_url":"https://api.github.com/users/mgaido91/gists{/gist_id}","starred_url":"https://api.github.com/users/mgaido91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgaido91/subscriptions","organizations_url":"https://api.github.com/users/mgaido91/orgs","repos_url":"https://api.github.com/users/mgaido91/repos","events_url":"https://api.github.com/users/mgaido91/events{/privacy}","received_events_url":"https://api.github.com/users/mgaido91/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-06-16T14:39:47Z","updated_at":"2018-03-15T03:23:50Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18329","html_url":"https://github.com/apache/spark/pull/18329","diff_url":"https://github.com/apache/spark/pull/18329.diff","patch_url":"https://github.com/apache/spark/pull/18329.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAs stated in [SPARK-19909](https://issues.apache.org/jira/browse/SPARK-19909), if the `checkpointLocation` is not set, for some formats Spark falls back using a directory in the `java.io.tmp` dir to store the metadata. This fails with a weird exception (permission denied) if the default filesystem is different from the temp one. This is the case if the default filesystem is HDFS, for instance, and the temp directory is written on the local filesystem. \r\n\r\nThe change proposed in this PR disables the usage of the temp directory in this case. Thus, a meaningful `AnalysisException` is thrown, suggesting the proper fix to the user, i.e. to set a value for `checkpointLocation`, instead of a `PermissionDenied` exception which is hard to understand and fix.\r\n\r\nThis behavior is consistent with what happens in all the cases in which the `checkpointLocation` needs to be set.\r\n\r\n## How was this patch tested?\r\n\r\nA unit test was added to check the patch and all the unit tests have been run.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/373248638","html_url":"https://github.com/apache/spark/pull/18329#issuecomment-373248638","issue_url":"https://api.github.com/repos/apache/spark/issues/18329","id":373248638,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T03:23:50Z","updated_at":"2018-03-15T03:23:50Z","author_association":"NONE","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88252/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}