{"push_id":2853609123,"size":41,"distinct_size":13,"ref":"refs/heads/feature/SPARK-23539","head":"b0f64e91cb4f6306a7c0c60d4a17f1a0aacb3a51","before":"253a894bedbd3a9642e529ad937dcb99dae346c7","commits":[{"sha":"339859c4e4b27726ba8ce9a64451a981ef74de0c","author":{"name":"huangtengfei02","email":"9c520914170dfce460ce9b5ea8bc3ce6a432a4ef@baidu.com"},"message":"[SPARK-25261][MINOR][DOC] update the description for spark.executor|driver.memory in configuration.md\n\n## What changes were proposed in this pull request?\n\nAs described in [SPARK-25261](https://issues.apache.org/jira/projects/SPARK/issues/SPARK-25261)，the unit of spark.executor.memory and spark.driver.memory is parsed as bytes in some cases if no unit specified, while in https://spark.apache.org/docs/latest/configuration.html#application-properties, they are descibed as MiB, which may lead to some misunderstandings.\n\n## How was this patch tested?\n\nN/A\n\nCloses #22252 from ivoson/branch-correct-configuration.\n\nLead-authored-by: huangtengfei02 <huangtengfei02@baidu.com>\nCo-authored-by: Huang Tengfei <tengfei.h@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/339859c4e4b27726ba8ce9a64451a981ef74de0c"},{"sha":"7fc8881b0fbc3d85a524e0454fa89925e92c4fa4","author":{"name":"Xiao Li","email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com"},"message":"[SPARK-25296][SQL][TEST] Create ExplainSuite\n\n## What changes were proposed in this pull request?\nMove the output verification of Explain test cases to a new suite ExplainSuite.\n\n## How was this patch tested?\nN/A\n\nCloses #22300 from gatorsmile/test3200.\n\nAuthored-by: Xiao Li <gatorsmile@gmail.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/7fc8881b0fbc3d85a524e0454fa89925e92c4fa4"},{"sha":"32da87dfa451fff677ed9316f740be2abdbff6a4","author":{"name":"Maxim Gekk","email":"05b3389ec46e7c78a10351def567499e4feebe3f@databricks.com"},"message":"[SPARK-25286][CORE] Removing the dangerous parmap\n\n## What changes were proposed in this pull request?\n\nI propose to remove one of `parmap` methods which accepts an execution context as a parameter. The method should be removed to eliminate any deadlocks that can occur if `parmap` is called recursively on thread pools restricted by size.\n\nCloses #22292 from MaxGekk/remove-overloaded-parmap.\n\nAuthored-by: Maxim Gekk <maxim.gekk@databricks.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/32da87dfa451fff677ed9316f740be2abdbff6a4"},{"sha":"e1d72f2c07ecd6f1880299e9373daa21cb032017","author":{"name":"Ilan Filonenko","email":"b57c113d8c727c2bfe8baa74ecf12b524e79d7b3@cornell.edu"},"message":"[SPARK-25264][K8S] Fix comma-delineated arguments passed into PythonRunner and RRunner\n\n## What changes were proposed in this pull request?\n\nFixes the issue brought up in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/273 where the arguments were being comma-delineated, which was incorrect wrt to the PythonRunner and RRunner.\n\n## How was this patch tested?\n\nModified unit test to test this change.\n\nAuthor: Ilan Filonenko <if56@cornell.edu>\n\nCloses #22257 from ifilonenko/SPARK-25264.","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/e1d72f2c07ecd6f1880299e9373daa21cb032017"},{"sha":"c5583fdcd2289559ad98371475eb7288ced9b148","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23466][SQL] Remove redundant null checks in generated Java code by GenerateUnsafeProjection\n\n## What changes were proposed in this pull request?\n\nThis PR works for one of TODOs in `GenerateUnsafeProjection` \"if the nullability of field is correct, we can use it to save null check\" to simplify generated code.\nWhen `nullable=false` in `DataType`, `GenerateUnsafeProjection` removed code for null checks in the generated Java code.\n\n## How was this patch tested?\n\nAdded new test cases into `GenerateUnsafeProjectionSuite`\n\nCloses #20637 from kiszk/SPARK-23466.\n\nAuthored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/c5583fdcd2289559ad98371475eb7288ced9b148"},{"sha":"7c36ee46d974021474d8098f87f70440a10319ee","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-25290][CORE][TEST] Reduce the size of acquired arrays to avoid OOM error\n\n## What changes were proposed in this pull request?\n\n`BytesToBytesMapOnHeapSuite`.`randomizedStressTest` caused `OutOfMemoryError` on several test runs. Seems better to reduce memory usage in this test.\n\n## How was this patch tested?\n\nUnit tests.\n\nCloses #22297 from viirya/SPARK-25290.\n\nAuthored-by: Liang-Chi Hsieh <viirya@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/7c36ee46d974021474d8098f87f70440a10319ee"},{"sha":"6ad8d4c375772c0c907c25837de762b5b9266a8e","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-25289][ML] Avoid exception in ChiSqSelector with FDR when no feature is selected\n\n## What changes were proposed in this pull request?\n\nCurrently, when FDR is used for `ChiSqSelector` and no feature is selected an exception is thrown because the max operation fails.\n\nThe PR fixes the problem by handling this case and returning an empty array in that case, as sklearn (which was the reference for the initial implementation of FDR) does.\n\n## How was this patch tested?\n\nadded UT\n\nCloses #22303 from mgaido91/SPARK-25289.\n\nAuthored-by: Marco Gaido <marcogaido91@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/6ad8d4c375772c0c907c25837de762b5b9266a8e"},{"sha":"a3dccd24c2e932b90e647e678f351f5b5568305b","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-10697][ML] Add lift to Association rules\n\n## What changes were proposed in this pull request?\n\nThe PR adds the lift measure to Association rules.\n\n## How was this patch tested?\n\nexisting and modified UTs\n\nCloses #22236 from mgaido91/SPARK-10697.\n\nAuthored-by: Marco Gaido <marcogaido91@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/a3dccd24c2e932b90e647e678f351f5b5568305b"},{"sha":"a481794ca9a5edb87982679cd0e95146f668fe78","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-25007][R] Add array_intersect/array_except/array_union/shuffle to SparkR\n\n## What changes were proposed in this pull request?\n\nAdd the R version of array_intersect/array_except/array_union/shuffle\n\n## How was this patch tested?\nAdd test in test_sparkSQL.R\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #22291 from huaxingao/spark-25007.","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/a481794ca9a5edb87982679cd0e95146f668fe78"},{"sha":"64bbd134ea1db5ead72461a65e130ef486c24dbf","author":{"name":"Darcy Shen","email":"a9a536cfcb71f58b94de5571e1607dbc58071d16@zoho.com"},"message":"[SPARK-25304][SPARK-8489][SQL][TEST] Fix HiveSparkSubmitSuite test for Scala 2.12\n\n## What changes were proposed in this pull request?\n\nremove test-2.10.jar and add test-2.12.jar.\n\n## How was this patch tested?\n\n```\n$ sbt -Dscala-2.12\n> ++ 2.12.6\n> project hive\n> testOnly *HiveSparkSubmitSuite -- -z \"8489\"\n```\n\nCloses #22308 from sadhen/SPARK-8489-FOLLOWUP.\n\nAuthored-by: Darcy Shen <sadhen@zoho.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/64bbd134ea1db5ead72461a65e130ef486c24dbf"},{"sha":"39d3d6cc965bd09b1719d245e672b013b8cee6f7","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-25167][SPARKR][TEST][MINOR] Minor fixes for R sql tests (timestamp comparison)\n\n## What changes were proposed in this pull request?\nThe \"date function on DataFrame\" test fails consistently on my laptop. In this PR\ni am fixing it by changing the way we compare the two timestamp values. With this change i am able to run the tests clean.\n\n## How was this patch tested?\nFixed the failing test.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #22274 from dilipbiswal/r-sql-test-fix2.","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/39d3d6cc965bd09b1719d245e672b013b8cee6f7"},{"sha":"546683c21a23cd5e3827e69609ca91cf92bd9e02","author":{"name":"Darcy Shen","email":"a9a536cfcb71f58b94de5571e1607dbc58071d16@zoho.com"},"message":"[SPARK-25298][BUILD] Improve build definition for Scala 2.12\n\n## What changes were proposed in this pull request?\n\nImprove build for Scala 2.12. Current build for sbt fails on the subproject `repl`:\n\n```\n[info] Compiling 6 Scala sources to /Users/rendong/wdi/spark/repl/target/scala-2.12/classes...\n[error] /Users/rendong/wdi/spark/repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala:80: overriding lazy value importableSymbolsWithRenames in class ImportHandler of type List[(this.intp.global.Symbol, this.intp.global.Name)];\n[error]  lazy value importableSymbolsWithRenames needs `override' modifier\n[error]       lazy val importableSymbolsWithRenames: List[(Symbol, Name)] = {\n[error]                ^\n[warn] /Users/rendong/wdi/spark/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala:53: variable addedClasspath in class ILoop is deprecated (since 2.11.0): use reset, replay or require to update class path\n[warn]       if (addedClasspath != \"\") {\n[warn]           ^\n[warn] /Users/rendong/wdi/spark/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala:54: variable addedClasspath in class ILoop is deprecated (since 2.11.0): use reset, replay or require to update class path\n[warn]         settings.classpath append addedClasspath\n[warn]                                   ^\n[warn] two warnings found\n[error] one error found\n[error] (repl/compile:compileIncremental) Compilation failed\n[error] Total time: 93 s, completed 2018-9-3 10:07:26\n```\n\n## How was this patch tested?\n\n```\n./dev/change-scala-version.sh 2.12\n\n##  For Maven\n./build/mvn -Pscala-2.12 [mvn commands]\n##  For SBT\nsbt -Dscala.version=2.12.6\n```\n\nCloses #22310 from sadhen/SPARK-25298.\n\nAuthored-by: Darcy Shen <sadhen@zoho.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/546683c21a23cd5e3827e69609ca91cf92bd9e02"},{"sha":"8e2169696f4cd52e5e3f51626a512d25215cffa4","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-25308][SQL] ArrayContains function may return a error in the code generation phase.\n\n## What changes were proposed in this pull request?\nInvoking ArrayContains function with non nullable array type throws the following error in the code generation phase. Below is the error snippet.\n```SQL\nCode generation of array_contains([1,2,3], 1) failed:\njava.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 11: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 11: Expression \"isNull_0\" is not an rvalue\njava.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 11: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 11: Expression \"isNull_0\" is not an rvalue\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)\n\tat com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)\n\tat com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)\n\tat com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)\n\tat com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)\n\tat com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)\n\tat com.google.common.cache.LocalCache.get(LocalCache.java:4000)\n\tat com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)\n\tat com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1305)\n\n```\n## How was this patch tested?\nAdded test in CollectionExpressionSuite.\n\nCloses #22315 from dilipbiswal/SPARK-25308.\n\nAuthored-by: Dilip Biswal <dbiswal@us.ibm.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/8e2169696f4cd52e5e3f51626a512d25215cffa4"},{"sha":"b60ee3a337e0484e1d3e978197685d7d5ab858e7","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-25307][SQL] ArraySort function may return an error in the code generation phase\n\n## What changes were proposed in this pull request?\nSorting array of booleans (not nullable) returns a compilation error in the code generation phase. Below is the compilation error :\n```SQL\njava.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 51, Column 23: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 51, Column 23: No applicable constructor/method found for actual parameters \"boolean[]\"; candidates are: \"public static void java.util.Arrays.sort(long[])\", \"public static void java.util.Arrays.sort(long[], int, int)\", \"public static void java.util.Arrays.sort(byte[], int, int)\", \"public static void java.util.Arrays.sort(float[])\", \"public static void java.util.Arrays.sort(float[], int, int)\", \"public static void java.util.Arrays.sort(char[])\", \"public static void java.util.Arrays.sort(char[], int, int)\", \"public static void java.util.Arrays.sort(short[], int, int)\", \"public static void java.util.Arrays.sort(short[])\", \"public static void java.util.Arrays.sort(byte[])\", \"public static void java.util.Arrays.sort(java.lang.Object[], int, int, java.util.Comparator)\", \"public static void java.util.Arrays.sort(java.lang.Object[], java.util.Comparator)\", \"public static void java.util.Arrays.sort(int[])\", \"public static void java.util.Arrays.sort(java.lang.Object[], int, int)\", \"public static void java.util.Arrays.sort(java.lang.Object[])\", \"public static void java.util.Arrays.sort(double[])\", \"public static void java.util.Arrays.sort(double[], int, int)\", \"public static void java.util.Arrays.sort(int[], int, int)\"\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)\n\tat com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)\n\tat com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)\n\tat com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)\n\tat com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)\n\tat com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)\n\tat com.google.common.cache.LocalCache.get(LocalCache.java:4000)\n\tat com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)\n\tat com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1305)\n\n```\n\n## How was this patch tested?\nAdded test in collectionExpressionSuite\n\nCloses #22314 from dilipbiswal/SPARK-25307.\n\nAuthored-by: Dilip Biswal <dbiswal@us.ibm.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/b60ee3a337e0484e1d3e978197685d7d5ab858e7"},{"sha":"4cb2ff9d8a58da5170744a634f672ba07b0a6a24","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-25310][SQL] ArraysOverlap may throw a CompilationException\n\n## What changes were proposed in this pull request?\n\nThis PR fixes a problem that `ArraysOverlap` function throws a `CompilationException` with non-nullable array type.\n\nThe following is the stack trace of the original problem:\n\n```\nCode generation of arrays_overlap([1,2,3], [4,5,3]) failed:\njava.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56, Column 11: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56, Column 11: Expression \"isNull_0\" is not an rvalue\njava.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56, Column 11: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56, Column 11: Expression \"isNull_0\" is not an rvalue\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)\n\tat com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)\n\tat com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)\n\tat com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)\n\tat com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)\n\tat com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)\n\tat com.google.common.cache.LocalCache.get(LocalCache.java:4000)\n\tat com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)\n\tat com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1305)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$.create(GenerateMutableProjection.scala:143)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$.create(GenerateMutableProjection.scala:48)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$.create(GenerateMutableProjection.scala:32)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:1260)\n```\n\n## How was this patch tested?\n\nAdded test in `CollectionExpressionSuite`.\n\nCloses #22317 from kiszk/SPARK-25310.\n\nAuthored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/4cb2ff9d8a58da5170744a634f672ba07b0a6a24"},{"sha":"e319ac92e597d95eba5b787bb7a5d5499bb3f87c","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-24962][SQL] Refactor CodeGenerator.createUnsafeArray, ArraySetLike, and ArrayDistinct\n\n## What changes were proposed in this pull request?\n\nThis PR integrates handling of `UnsafeArrayData` and `GenericArrayData` into one. The current `CodeGenerator.createUnsafeArray` handles only allocation of `UnsafeArrayData`.\nThis PR introduces a new method `createArrayData` that returns a code to allocate `UnsafeArrayData` or `GenericArrayData` and to assign a value into the allocated array.\n\nThis PR also reduce the size of generated code by calling a runtime helper.\n\nThis PR replaced `createArrayData` with `createUnsafeArray`. This PR also refactor `ArraySetLike` that can be used for `ArrayDistinct`, too.\nThis PR also refactors`ArrayDistinct` to use `ArraryBuilder`.\n\n## How was this patch tested?\n\nExisting tests\n\nCloses #21912 from kiszk/SPARK-24962.\n\nLead-authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nCo-authored-by: Takuya UESHIN <ueshin@happy-camper.st>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/e319ac92e597d95eba5b787bb7a5d5499bb3f87c"},{"sha":"0b9b6b7d10c8aa3d5bfb013d824255f67791c97d","author":{"name":"blueszheng","email":"0a20c728e29c1bd1cf6c6ce38753c97d7233019b@163.com"},"message":"[DOC] Update some outdated links\n\n## What changes were proposed in this pull request?\n\nThese links are outdated:\n - http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version\n - http://spark.apache.org/docs/latest/building-spark.html#building-with-buildmvn\n\nFix files which use these links.\n\nCloses #22321 from kisimple/docfix.\n\nAuthored-by: blueszheng <kisimple@163.com>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/0b9b6b7d10c8aa3d5bfb013d824255f67791c97d"},{"sha":"3aa60282cc84d471ea32ef240ec84e5b6e3e231b","author":{"name":"Xingbo Jiang","email":"19a442ba0283f6f4c465cc281093aec56203af87@databricks.com"},"message":"[SPARK-19355][SQL][FOLLOWUP][TEST] Properly recycle SparkSession on TakeOrderedAndProjectSuite finishes\n\n## What changes were proposed in this pull request?\n\nPreviously in `TakeOrderedAndProjectSuite` the SparkSession will not get recycled when the test suite finishes.\n\n## How was this patch tested?\n\nN/A\n\nCloses #22330 from jiangxb1987/SPARK-19355.\n\nAuthored-by: Xingbo Jiang <xingbo.jiang@databricks.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/3aa60282cc84d471ea32ef240ec84e5b6e3e231b"},{"sha":"061bb01d9b99911353e66a90abc3164c467fcae1","author":{"name":"Xiangrui Meng","email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com"},"message":"[SPARK-25248][CORE] Audit barrier Scala APIs for 2.4\n\n## What changes were proposed in this pull request?\n\nI made one pass over barrier APIs added to Spark 2.4 and updates some scopes and docs. I will update Python docs once Scala doc was reviewed.\n\nOne major issue is that `BarrierTaskContext` implements `TaskContextImpl` that exposes some public methods. And internally there were several direct references to `TaskContextImpl` methods instead of `TaskContext`. This PR moved some methods from `TaskContextImpl` to `TaskContext`, remaining package private, and used delegate methods to avoid inheriting `TaskContextImp` and exposing unnecessary APIs.\n\nTODOs:\n- [x] scala doc\n- [x] python doc (#22261 ).\n\nCloses #22240 from mengxr/SPARK-25248.\n\nAuthored-by: Xiangrui Meng <meng@databricks.com>\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/061bb01d9b99911353e66a90abc3164c467fcae1"},{"sha":"103f513231d31cfa475aa34ce4defc63acc3cab6","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-25306][SQL] Avoid skewed filter trees to speed up `createFilter` in ORC\n\n## What changes were proposed in this pull request?\n\nIn both ORC data sources, `createFilter` function has exponential time complexity due to its skewed filter tree generation. This PR aims to improve it by using new `buildTree` function.\n\n**REPRODUCE**\n```scala\n// Create and read 1 row table with 1000 columns\nsql(\"set spark.sql.orc.filterPushdown=true\")\nval selectExpr = (1 to 1000).map(i => s\"id c$i\")\nspark.range(1).selectExpr(selectExpr: _*).write.mode(\"overwrite\").orc(\"/tmp/orc\")\nprint(s\"With 0 filters, \")\nspark.time(spark.read.orc(\"/tmp/orc\").count)\n\n// Increase the number of filters\n(20 to 30).foreach { width =>\n  val whereExpr = (1 to width).map(i => s\"c$i is not null\").mkString(\" and \")\n  print(s\"With $width filters, \")\n  spark.time(spark.read.orc(\"/tmp/orc\").where(whereExpr).count)\n}\n```\n\n**RESULT**\n```scala\nWith 0 filters, Time taken: 653 ms\nWith 20 filters, Time taken: 962 ms\nWith 21 filters, Time taken: 1282 ms\nWith 22 filters, Time taken: 1982 ms\nWith 23 filters, Time taken: 3855 ms\nWith 24 filters, Time taken: 6719 ms\nWith 25 filters, Time taken: 12669 ms\nWith 26 filters, Time taken: 25032 ms\nWith 27 filters, Time taken: 49585 ms\nWith 28 filters, Time taken: 98980 ms    // over 1 min 38 seconds\nWith 29 filters, Time taken: 198368 ms   // over 3 mins\nWith 30 filters, Time taken: 393744 ms   // over 6 mins\n```\n\n**AFTER THIS PR**\n```scala\nWith 0 filters, Time taken: 774 ms\nWith 20 filters, Time taken: 601 ms\nWith 21 filters, Time taken: 399 ms\nWith 22 filters, Time taken: 679 ms\nWith 23 filters, Time taken: 363 ms\nWith 24 filters, Time taken: 342 ms\nWith 25 filters, Time taken: 336 ms\nWith 26 filters, Time taken: 352 ms\nWith 27 filters, Time taken: 322 ms\nWith 28 filters, Time taken: 302 ms\nWith 29 filters, Time taken: 307 ms\nWith 30 filters, Time taken: 301 ms\n```\n\n## How was this patch tested?\n\nPass the Jenkins with newly added test cases.\n\nCloses #22313 from dongjoon-hyun/SPARK-25306.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/dongjinleekr/spark/commits/103f513231d31cfa475aa34ce4defc63acc3cab6"}]}
{"actor":{"display_login":"dongjinleekr"}}