{"push_id":3156790206,"size":2,"distinct_size":2,"ref":"refs/heads/master","head":"305e9b5ad22b428501fd42d3730d73d2e09ad4c5","before":"aa0d4ca8bab08a467645080a5b8a28bf6dd8a042","commits":[{"sha":"98ecda3e8ef9db5e21b5b9605df09d1653094b9c","author":{"name":"liuxian","email":"b507fe1e602c016180e481ef5e61b475642f6f2d@zte.com.cn"},"message":"[MINOR][SQL] Locality does not need to be implemented\n\n## What changes were proposed in this pull request?\n`HadoopFileWholeTextReader` and  `HadoopFileLinesReader` will be eventually called in `FileSourceScanExec`.\nIn fact,  locality has been implemented in `FileScanRDD`,  even if we implement it in `HadoopFileWholeTextReader ` and  `HadoopFileLinesReader`,  it would be useless.\nSo I think these `TODO` can be removed.\n\n## How was this patch tested?\nN/A\n\nCloses #23339 from 10110346/noneededtodo.\n\nAuthored-by: liuxian <liu.xian3@zte.com.cn>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/98ecda3e8ef9db5e21b5b9605df09d1653094b9c"},{"sha":"305e9b5ad22b428501fd42d3730d73d2e09ad4c5","author":{"name":"Hyukjin Kwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[SPARK-26422][R] Support to disable Hive support in SparkR even for Hadoop versions unsupported by Hive fork\n\n## What changes were proposed in this pull request?\n\nCurrently,  even if I explicitly disable Hive support in SparkR session as below:\n\n```r\nsparkSession <- sparkR.session(\"local[4]\", \"SparkR\", Sys.getenv(\"SPARK_HOME\"),\n                               enableHiveSupport = FALSE)\n```\n\nproduces when the Hadoop version is not supported by our Hive fork:\n\n```\njava.lang.reflect.InvocationTargetException\n...\nCaused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1.3.1.0.0-78\n\tat org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)\n\tat org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)\n\tat org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)\n\tat org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)\n\t... 43 more\nError in handleErrors(returnStatus, conn) :\n  java.lang.ExceptionInInitializerError\n\tat org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)\n\tat java.lang.Class.forName0(Native Method)\n\tat java.lang.Class.forName(Class.java:348)\n\tat org.apache.spark.util.Utils$.classForName(Utils.scala:193)\n\tat org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1116)\n\tat org.apache.spark.sql.api.r.SQLUtils$.getOrCreateSparkSession(SQLUtils.scala:52)\n\tat org.apache.spark.sql.api.r.SQLUtils.getOrCreateSparkSession(SQLUtils.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n```\n\nThe root cause is that:\n\n```\nSparkSession.hiveClassesArePresent\n```\n\ncheck if the class is loadable or not to check if that's in classpath but `org.apache.hadoop.hive.conf.HiveConf` has a check for Hadoop version as static logic which is executed right away. This throws an `IllegalArgumentException` and that's not caught:\n\nhttps://github.com/apache/spark/blob/36edbac1c8337a4719f90e4abd58d38738b2e1fb/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L1113-L1121\n\nSo, currently, if users have a Hive built-in Spark with unsupported Hadoop version by our fork (namely 3+), there's no way to use SparkR even though it could work.\n\nThis PR just propose to change the order of bool comparison so that we can don't execute `SparkSession.hiveClassesArePresent` when:\n\n  1. `enableHiveSupport` is explicitly disabled\n  2. `spark.sql.catalogImplementation` is `in-memory`\n\nso that we **only** check `SparkSession.hiveClassesArePresent` when Hive support is explicitly enabled by short circuiting.\n\n## How was this patch tested?\n\nIt's difficult to write a test since we don't run tests against Hadoop 3 yet. See https://github.com/apache/spark/pull/21588. Manually tested.\n\nCloses #23356 from HyukjinKwon/SPARK-26422.\n\nAuthored-by: Hyukjin Kwon <gurwls223@apache.org>\nSigned-off-by: Hyukjin Kwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/305e9b5ad22b428501fd42d3730d73d2e09ad4c5"}]}
{"actor":{"display_login":"ueshin"}}