{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16305","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16305/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16305/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16305/events","html_url":"https://github.com/apache/spark/pull/16305","id":195994357,"number":16305,"title":"[SPARK-18895][TESTS] Fix resource-closing-related and path-related test failures in identified ones on Windows","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2016-12-16T07:03:25Z","updated_at":"2016-12-17T01:02:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16305","html_url":"https://github.com/apache/spark/pull/16305","diff_url":"https://github.com/apache/spark/pull/16305.diff","patch_url":"https://github.com/apache/spark/pull/16305.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThere are several tests failing due to resource-closing-related and path-related  problems on Windows as below.\r\n\r\n- `RPackageUtilsSuite`:\r\n\r\n```\r\n- build an R package from a jar end to end *** FAILED *** (1 second, 625 milliseconds)\r\n  java.io.IOException: Unable to delete file: C:\\projects\\spark\\target\\tmp\\1481729427517-0\\a\\dep2\\d\\dep2-d.jar\r\n  at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)\r\n  at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)\r\n  at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)\r\n\r\n- faulty R package shows documentation *** FAILED *** (359 milliseconds)\r\n  java.io.IOException: Unable to delete file: C:\\projects\\spark\\target\\tmp\\1481729428970-0\\dep1-c.jar\r\n  at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)\r\n  at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)\r\n  at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)\r\n\r\n- SparkR zipping works properly *** FAILED *** (47 milliseconds)\r\n  java.util.regex.PatternSyntaxException: Unknown character property name {r} near index 4\r\n\r\nC:\\projects\\spark\\target\\tmp\\1481729429282-0\r\n\r\n    ^\r\n  at java.util.regex.Pattern.error(Pattern.java:1955)\r\n  at java.util.regex.Pattern.charPropertyNodeFor(Pattern.java:2781)\r\n```\r\n\r\n\r\n- `InputOutputMetricsSuite`:\r\n\r\n```\r\n- input metrics for old hadoop with coalesce *** FAILED *** (240 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input metrics with cache and coalesce *** FAILED *** (109 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input metrics for new Hadoop API with coalesce *** FAILED *** (0 milliseconds)\r\n  java.lang.IllegalArgumentException: Wrong FS: file://C:\\projects\\spark\\target\\tmp\\spark-9366ec94-dac7-4a5c-a74b-3e7594a692ab\\test\\InputOutputMetricsSuite.txt, expected: file:///\r\n  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)\r\n  at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:462)\r\n  at org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:114)\r\n\r\n- input metrics when reading text file *** FAILED *** (110 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input metrics on records read - simple *** FAILED *** (125 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input metrics on records read - more stages *** FAILED *** (110 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input metrics on records - New Hadoop API *** FAILED *** (16 milliseconds)\r\n  java.lang.IllegalArgumentException: Wrong FS: file://C:\\projects\\spark\\target\\tmp\\spark-3f10a1a4-7820-4772-b821-25fd7523bf6f\\test\\InputOutputMetricsSuite.txt, expected: file:///\r\n  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)\r\n  at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:462)\r\n  at org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:114)\r\n\r\n- input metrics on records read with cache *** FAILED *** (93 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input read/write and shuffle read/write metrics all line up *** FAILED *** (93 milliseconds)\r\n  java.io.IOException: Not a file: file:/C:/projects/spark/core/ignored\r\n  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:277)\r\n  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)\r\n  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)\r\n\r\n- input metrics with interleaved reads *** FAILED *** (0 milliseconds)\r\n  java.lang.IllegalArgumentException: Wrong FS: file://C:\\projects\\spark\\target\\tmp\\spark-2638d893-e89b-47ce-acd0-bbaeee78dd9b\\InputOutputMetricsSuite_cart.txt, expected: file:///\r\n  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)\r\n  at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:462)\r\n  at org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:114)\r\n\r\n- input metrics with old CombineFileInputFormat *** FAILED *** (157 milliseconds)\r\n  17947 was not greater than or equal to 300000 (InputOutputMetricsSuite.scala:324)\r\n  org.scalatest.exceptions.TestFailedException:\r\n  at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\r\n  at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\r\n  at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:466)\r\n\r\n- input metrics with new CombineFileInputFormat *** FAILED *** (16 milliseconds)\r\n  java.lang.IllegalArgumentException: Wrong FS: file://C:\\projects\\spark\\target\\tmp\\spark-11920c08-19d8-4c7c-9fba-28ed72b79f80\\test\\InputOutputMetricsSuite.txt, expected: file:///\r\n  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)\r\n  at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:462)\r\n  at org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:114)\r\n```\r\n\r\n- `ReplayListenerSuite`:\r\n\r\n```\r\n- End-to-end replay *** FAILED *** (121 milliseconds)\r\n  java.io.IOException: No FileSystem for scheme: C\r\n  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)\r\n  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)\r\n\r\n\r\n- End-to-end replay with compression *** FAILED *** (516 milliseconds)\r\n  java.io.IOException: No FileSystem for scheme: C\r\n  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)\r\n  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)\r\n  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)\r\n```\r\n\r\n- `EventLoggingListenerSuite`:\r\n\r\n```\r\n- End-to-end event logging *** FAILED *** (7 seconds, 435 milliseconds)\r\n  java.io.IOException: No FileSystem for scheme: C\r\n  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)\r\n  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)\r\n  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)\r\n\r\n- End-to-end event logging with compression *** FAILED *** (1 second)\r\n  java.io.IOException: No FileSystem for scheme: C\r\n  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)\r\n  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)\r\n  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)\r\n\r\n- Event log name *** FAILED *** (16 milliseconds)\r\n  \"file:/[]base-dir/app1\" did not equal \"file:/[C:/]base-dir/app1\" (EventLoggingListenerSuite.scala:123)\r\n  org.scalatest.exceptions.TestFailedException:\r\n  at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\r\n  at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\r\n  at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:466)\r\n```\r\n\r\nThis PR proposes to fix the test failures on Windows\r\n\r\n## How was this patch tested?\r\n\r\nManually tested via AppVeyor\r\n\r\n**Before**\r\n\r\n`RPackageUtilsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/273-RPackageUtilsSuite-before\r\n`InputOutputMetricsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/272-InputOutputMetricsSuite-before\r\n`ReplayListenerSuite`: https://ci.appveyor.com/project/spark-test/spark/build/274-ReplayListenerSuite-before\r\n`EventLoggingListenerSuite`: https://ci.appveyor.com/project/spark-test/spark/build/275-EventLoggingListenerSuite-before\r\n\r\n\r\n**After**\r\n\r\n`RPackageUtilsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/270-RPackageUtilsSuite\r\n`InputOutputMetricsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/271-InputOutputMetricsSuite\r\n`ReplayListenerSuite`: https://ci.appveyor.com/project/spark-test/spark/build/277-ReplayListenerSuite-after\r\n`EventLoggingListenerSuite`: https://ci.appveyor.com/project/spark-test/spark/build/278-EventLoggingListenerSuite-after\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/267731606","html_url":"https://github.com/apache/spark/pull/16305#issuecomment-267731606","issue_url":"https://api.github.com/repos/apache/spark/issues/16305","id":267731606,"user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2016-12-17T01:02:37Z","updated_at":"2016-12-17T01:02:37Z","body":"Ah, thank you @shivaram and @shaneknapp."}}
{"actor":{"display_login":"HyukjinKwon"}}