{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/20590","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/20590/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/20590/comments","events_url":"https://api.github.com/repos/apache/spark/issues/20590/events","html_url":"https://github.com/apache/spark/pull/20590","id":296538283,"number":20590,"title":"[SPARK-23399][SQL] Register a task completion listener first for OrcColumnarBatchReader","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars2.githubusercontent.com/u/9700541?v=4","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-02-12T21:55:28Z","updated_at":"2018-02-13T15:38:05Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20590","html_url":"https://github.com/apache/spark/pull/20590","diff_url":"https://github.com/apache/spark/pull/20590.diff","patch_url":"https://github.com/apache/spark/pull/20590.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR aims to resolve an open file leakage issue reported at [SPARK-23390](https://issues.apache.org/jira/browse/SPARK-23390) by moving the listener registration position. Currently, the sequence is like the following.\r\n\r\n1. Create `batchReader`\r\n2. `batchReader.initialize` opens a ORC file.\r\n3. `batchReader.initBatch` may take a long time to alloc memory in some environment and cause errors.\r\n4. `Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ => iter.close()))`\r\n\r\nThis PR moves 4 before 2 and 3. To sum up, the new sequence is 1 -> 4 -> 2 -> 3.\r\n\r\n## How was this patch tested?\r\n\r\nManual. The following test case makes OOM intentionally to cause leaked filesystem connection in the current code base. With this patch, leakage doesn't occurs. However, this should be tested manually because the test suite will also die.\r\n\r\n```scala\r\n  // This should be tested manually because it raises OOM intentionally\r\n  // in order to cause `Leaked filesystem connection`. The test suite dies, too.\r\n  test(\"SPARK-23399 Register a task completion listener first for OrcColumnarBatchReader\") {\r\n    withSQLConf(SQLConf.ORC_VECTORIZED_READER_BATCH_SIZE.key -> s\"${Int.MaxValue}\") {\r\n      withTempDir { dir =>\r\n        val basePath = dir.getCanonicalPath\r\n        Seq(0).toDF(\"a\").write.format(\"orc\").save(new Path(basePath, \"first\").toString)\r\n        Seq(1).toDF(\"a\").write.format(\"orc\").save(new Path(basePath, \"second\").toString)\r\n        val df = spark.read.orc(\r\n          new Path(basePath, \"first\").toString,\r\n          new Path(basePath, \"second\").toString)\r\n        val e = intercept[SparkException] {\r\n          df.collect()\r\n        }\r\n        assert(e.getCause.isInstanceOf[OutOfMemoryError])\r\n      }\r\n    }\r\n  }\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/365304068","html_url":"https://github.com/apache/spark/pull/20590#issuecomment-365304068","issue_url":"https://api.github.com/repos/apache/spark/issues/20590","id":365304068,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-02-13T15:38:05Z","updated_at":"2018-02-13T15:38:05Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/859/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}