{"push_id":3002379793,"size":6,"distinct_size":6,"ref":"refs/heads/master","head":"5bd5e1b9c84b5f7d4d67ab94e02d49ebdd02f177","before":"ca2fca143277deaff58a69b7f1e0360cfc70561f","commits":[{"sha":"4e990d9dd2407dc257712c4b12b507f0990ca4e9","author":{"name":"Bruce Robbins","email":"9fa1b8dc62c4e42f4b64eedaa587e59466a4067a@gmail.com"},"message":"[DOC] Fix doc for spark.sql.parquet.recordLevelFilter.enabled\n\n## What changes were proposed in this pull request?\n\nUpdated the doc string value for spark.sql.parquet.recordLevelFilter.enabled to indicate that spark.sql.parquet.enableVectorizedReader must be disabled.\n\nThe code in ParquetFileFormat uses spark.sql.parquet.recordLevelFilter.enabled only after falling back to parquet-mr (see else for this if statement): https://github.com/apache/spark/blob/d5573c578a1eea9ee04886d9df37c7178e67bb30/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L412\nhttps://github.com/apache/spark/blob/d5573c578a1eea9ee04886d9df37c7178e67bb30/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L427-L430\n\nTests also bear this out.\n\n## How was this patch tested?\n\nThis is just a doc string fix: I built Spark and ran a single test.\n\nCloses #22865 from bersprockets/confdocfix.\n\nAuthored-by: Bruce Robbins <bersprockets@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Halo9Pan/dive_spark/commits/4e990d9dd2407dc257712c4b12b507f0990ca4e9"},{"sha":"fbaf150507a289ec0ac02fdbf4009c42cd9bc164","author":{"name":"hyukjinkwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[SPARK-25179][PYTHON][DOCS] Document BinaryType support in Arrow conversion\n\n## What changes were proposed in this pull request?\n\nThis PR targets to document binary type in \"Apache Arrow in Spark\".\n\n## How was this patch tested?\n\nManually built the documentation and checked.\n\nCloses #22871 from HyukjinKwon/SPARK-25179.\n\nAuthored-by: hyukjinkwon <gurwls223@apache.org>\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/Halo9Pan/dive_spark/commits/fbaf150507a289ec0ac02fdbf4009c42cd9bc164"},{"sha":"409d688fb6169ecbc41f6296c6341ae3ed7d1ec8","author":{"name":"yucai","email":"053c978f3c27c11ff61cd92f4d337d976848b8ce@ebay.com"},"message":"[SPARK-25864][SQL][TEST] Make main args accessible for BenchmarkBase's subclass\n\n## What changes were proposed in this pull request?\n\nSet main args correctly in BenchmarkBase, to make it accessible for its subclass.\nIt will benefit:\n- BuiltInDataSourceWriteBenchmark\n- AvroWriteBenchmark\n\n## How was this patch tested?\n\nmanual tests\n\nCloses #22872 from yucai/main_args.\n\nAuthored-by: yucai <yyu1@ebay.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Halo9Pan/dive_spark/commits/409d688fb6169ecbc41f6296c6341ae3ed7d1ec8"},{"sha":"7fe5cff0581ca9d8221533215098f40f69362018","author":{"name":"Peter Toth","email":"63c2680800cb5c8e15ff6801589894e00f220268@gmail.com"},"message":"[SPARK-25767][SQL] Fix lazily evaluated stream of expressions in code generation\n\n## What changes were proposed in this pull request?\n\nCode generation is incorrect if `outputVars` parameter of `consume` method in `CodegenSupport` contains a lazily evaluated stream of expressions.\nThis PR fixes the issue by forcing the evaluation of `inputVars` before generating the code for UnsafeRow.\n\n## How was this patch tested?\n\nTested with the sample program provided in https://issues.apache.org/jira/browse/SPARK-25767\n\nCloses #22789 from peter-toth/SPARK-25767.\n\nAuthored-by: Peter Toth <peter.toth@gmail.com>\nSigned-off-by: Herman van Hovell <hvanhovell@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Halo9Pan/dive_spark/commits/7fe5cff0581ca9d8221533215098f40f69362018"},{"sha":"5e5d886a2bc291a707cf4a6c70ecc6de6f8e990d","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-25856][SQL][MINOR] Remove AverageLike and CountLike classes\n\n## What changes were proposed in this pull request?\nThese two classes were added for regr_ expression support (SPARK-23907). These have been removed and hence we can remove these base classes and inline the logic in the concrete classes.\n## How was this patch tested?\nExisting tests.\n\nCloses #22856 from dilipbiswal/average_cleanup.\n\nAuthored-by: Dilip Biswal <dbiswal@us.ibm.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Halo9Pan/dive_spark/commits/5e5d886a2bc291a707cf4a6c70ecc6de6f8e990d"},{"sha":"5bd5e1b9c84b5f7d4d67ab94e02d49ebdd02f177","author":{"name":"hyukjinkwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[MINOR][SQL] Avoid hardcoded configuration keys in SQLConf's `doc`\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to avoid hardcorded configuration keys in SQLConf's `doc.\n\n## How was this patch tested?\n\nManually verified.\n\nCloses #22877 from HyukjinKwon/minor-conf-name.\n\nAuthored-by: hyukjinkwon <gurwls223@apache.org>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/Halo9Pan/dive_spark/commits/5bd5e1b9c84b5f7d4d67ab94e02d49ebdd02f177"}]}
{"actor":{"display_login":"Halo9Pan"}}