{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16098","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16098/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16098/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16098/events","html_url":"https://github.com/apache/spark/pull/16098","id":192798334,"number":16098,"title":"[SPARK-18672][CORE] Close recordwriter in SparkHadoopMapReduceWriter before committing","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-12-01T09:49:58Z","updated_at":"2016-12-02T16:53:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16098","html_url":"https://github.com/apache/spark/pull/16098","diff_url":"https://github.com/apache/spark/pull/16098.diff","patch_url":"https://github.com/apache/spark/pull/16098.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIt seems some APIs such as `PairRDDFunctions.saveAsHadoopDataset()` do not close the record writer before issuing the commit for the task.\r\n\r\nOn Windows, the output in the temp directory is being open and output committer tries to rename it from temp directory to the output directory after finishing writing. \r\n\r\nSo, it fails to move the file. It seems we should close the writer actually before committing the task like the other writers such as `FileFormatWriter`.\r\n\r\nIdentified failure was as below:\r\n\r\n```\r\nFAILURE! - in org.apache.spark.JavaAPISuite\r\nwriteWithNewAPIHadoopFile(org.apache.spark.JavaAPISuite)  Time elapsed: 0.25 sec  <<< ERROR!\r\norg.apache.spark.SparkException: Job aborted.\r\n\tat org.apache.spark.JavaAPISuite.writeWithNewAPIHadoopFile(JavaAPISuite.java:1231)\r\nCaused by: org.apache.spark.SparkException: \r\nJob aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows\r\n\tat org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:182)\r\n\tat org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:100)\r\n\tat org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:99)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.IOException: Could not rename file:/C:/projects/spark/core/target/tmp/1480553515529-0/output/_temporary/0/_temporary/attempt_20161201005155_0000_r_000000_0 to file:/C:/projects/spark/core/target/tmp/1480553515529-0/output/_temporary/0/task_20161201005155_0000_r_000000\r\n\tat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:436)\r\n\tat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:415)\r\n\tat org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)\r\n\tat org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:76)\r\n\tat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:153)\r\n\tat org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:167)\r\n\tat org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:156)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)\r\n\tat org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:168)\r\n\t... 8 more\r\nDriver stacktrace:\r\n\tat org.apache.spark.JavaAPISuite.writeWithNewAPIHadoopFile(JavaAPISuite.java:1231)\r\nCaused by: org.apache.spark.SparkException: Task failed while writing rows\r\nCaused by: java.io.IOException: Could not rename file:/C:/projects/spark/core/target/tmp/1480553515529-0/output/_temporary/0/_temporary/attempt_20161201005155_0000_r_000000_0 to file:/C:/projects/spark/core/target/tmp/1480553515529-0/output/_temporary/0/task_20161201005155_0000_r_000000\r\n```\r\n\r\nThis PR proposes to close this before committing the task.\r\n\r\n## How was this patch tested?\r\n\r\nManually tested via AppVeyor. \r\n\r\n**Before**\r\n\r\nhttps://ci.appveyor.com/project/spark-test/spark/build/94-scala-tests\r\n\r\n**After**\r\n\r\nhttps://ci.appveyor.com/project/spark-test/spark/build/93-scala-tests\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/264502788","html_url":"https://github.com/apache/spark/pull/16098#issuecomment-264502788","issue_url":"https://api.github.com/repos/apache/spark/issues/16098","id":264502788,"user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T16:53:00Z","updated_at":"2016-12-02T16:53:00Z","body":"Build started: [CORE] `org.apache.spark.JavaAPISuite` [![PR-16098](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=C6FA0845-A500-4CB8-A511-B7B5C418691F&svg=true)](https://ci.appveyor.com/project/spark-test/spark/branch/C6FA0845-A500-4CB8-A511-B7B5C418691F)\r\nDiff: https://github.com/apache/spark/compare/master...spark-test:C6FA0845-A500-4CB8-A511-B7B5C418691F"}}
{"actor":{"display_login":"HyukjinKwon"}}