{"push_id":3985827025,"size":25,"distinct_size":25,"ref":"refs/heads/master","head":"eb037a8180be4ab7570eda1fa9cbf3c84b92c3f7","before":"1b404b9b9928144e9f527ac7b1caa15f932c2649","commits":[{"sha":"74527868b27a3f22c7c71549be5297814289a1d6","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-28789][DOCS][SQL] Document ALTER DATABASE command\n\n### What changes were proposed in this pull request?\nDocument ALTER DATABSE statement in SQL Reference Guide.\n\n### Why are the changes needed?\nCurrently Spark lacks documentation on the supported SQL constructs causing\nconfusion among users who sometimes have to look at the code to understand the\nusage. This is aimed at addressing this issue.\n\n### Does this PR introduce any user-facing change?\nYes.\n\n**Before:**\nThere was no documentation for this.\n\n**After.**\n<img width=\"1234\" alt=\"Screen Shot 2019-08-28 at 1 51 13 PM\" src=\"https://user-images.githubusercontent.com/14225158/63891854-fc817580-c99a-11e9-918e-6b305edf92e6.png\">\n<img width=\"1234\" alt=\"Screen Shot 2019-08-28 at 1 51 27 PM\" src=\"https://user-images.githubusercontent.com/14225158/63891869-0acf9180-c99b-11e9-91a4-04d870474a40.png\">\n\n### How was this patch tested?\nTested using jykyll build --serve\n\nCloses #25523 from dilipbiswal/ref-doc-alterdb.\n\nLead-authored-by: Dilip Biswal <dbiswal@us.ibm.com>\nCo-authored-by: Xiao Li <gatorsmile@gmail.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/74527868b27a3f22c7c71549be5297814289a1d6"},{"sha":"137b20b964364c651a2f4f4b76b15fcbd6fa2226","author":{"name":"Matt Hawes","email":"e8097cd5be07183c96b718a25e4c7ccd5e27a529@palantir.com"},"message":"[SPARK-28818][SQL] Respect source column nullability in the arrays created by `freqItems()`\n\n### What changes were proposed in this pull request?\nThis PR replaces the hard-coded non-nullability of the array elements returned by `freqItems()` with a nullability that reflects the original schema. Essentially [the functional change](https://github.com/apache/spark/pull/25575/files#diff-bf59bb9f3dc351f5bf6624e5edd2dcf4R122) to the schema generation is:\n```\nStructField(name + \"_freqItems\", ArrayType(dataType, false))\n```\nBecomes:\n```\nStructField(name + \"_freqItems\", ArrayType(dataType, originalField.nullable))\n```\n\nRespecting the original nullability prevents issues when Spark depends on `ArrayType`'s `containsNull` being accurate. The example that uncovered this is calling `collect()` on the dataframe (see [ticket](https://issues.apache.org/jira/browse/SPARK-28818) for full repro). Though it's likely that there a several places where this could cause a problem.\n\nI've also refactored a small amount of the surrounding code to remove some unnecessary steps and group together related operations.\n\n### Why are the changes needed?\nI think it's pretty clear why this change is needed. It fixes a bug that currently prevents users from calling `df.freqItems.collect()` along with potentially causing other, as yet unknown, issues.\n\n### Does this PR introduce any user-facing change?\nNullability of columns when calling freqItems on them is now respected after the change.\n\n### How was this patch tested?\nI added a test that specifically tests the carry-through of the nullability as well as explicitly calling `collect()` to catch the exact regression that was observed. I also ran the test against the old version of the code and it fails as expected.\n\nCloses #25575 from MGHawes/mhawes/SPARK-28818.\n\nAuthored-by: Matt Hawes <mhawes@palantir.com>\nSigned-off-by: HyukjinKwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/137b20b964364c651a2f4f4b76b15fcbd6fa2226"},{"sha":"24655583f1cb5dae2e80bb572604fb4a9761ec07","author":{"name":"Gengliang Wang","email":"bfe1a7b36425bd89859a70a36586361d28ec35eb@databricks.com"},"message":"[SPARK-28495][SQL][FOLLOW-UP] Disallow conversions between timestamp and long in ASNI mode\n\n### What changes were proposed in this pull request?\n\nDisallow conversions between `timestamp` type and `long` type in table insertion with ANSI store assignment policy.\n\n### Why are the changes needed?\n\nIn the PR https://github.com/apache/spark/pull/25581, timestamp type is allowed to be converted to long type, since timestamp type is represented by long type internally, and both legacy mode and strict mode allows the conversion.\n\nAfter reconsideration, I think we should disallow it. As per ANSI SQL section \"4.4.2 Characteristics of numbers\":\n> A number is assignable only to sites of numeric type.\n\nIn PostgreSQL, the conversion between timestamp and long is also disallowed.\n\n### Does this PR introduce any user-facing change?\n\nConversion between timestamp and long is disallowed in table insertion with ANSI store assignment policy.\n\n### How was this patch tested?\n\nUnit test\n\nCloses #25615 from gengliangwang/disallowTimeStampToLong.\n\nAuthored-by: Gengliang Wang <gengliang.wang@databricks.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/24655583f1cb5dae2e80bb572604fb4a9761ec07"},{"sha":"3e09a0fce9d9fe7249b3192c036af04d7f6707d0","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-28786][DOC][SQL] Document INSERT statement in SQL Reference\n\n### What changes were proposed in this pull request?\nDocument INSERT statement in SQL Reference\n\n### Why are the changes needed?\nTo complete SQL reference.\n\n### Does this PR introduce any user-facing change?\nYes.\n\n### How was this patch tested?\nManually checked newly added doc.\n\nHere are the screen shots:\n\n![image](https://user-images.githubusercontent.com/13592258/63490232-0a01a180-c469-11e9-82de-cfdc7c2343e7.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63903006-cce56400-c9c0-11e9-9f24-badd586227a2.png)\n\n<img width=\"1100\" alt=\"Screen Shot 2019-08-27 at 5 01 48 PM\" src=\"https://user-images.githubusercontent.com/13592258/63816303-845c7680-c8ec-11e9-8c36-1b8e4d3e6286.png\">\n\n<img width=\"1100\" alt=\"Screen Shot 2019-08-27 at 5 03 22 PM\" src=\"https://user-images.githubusercontent.com/13592258/63816347-ac4bda00-c8ec-11e9-9470-fa99522e6f14.png\">\n\n![image](https://user-images.githubusercontent.com/13592258/63817393-fc2ca000-c8f0-11e9-9d66-dd9b22a9d900.png)\n\n<img width=\"1102\" alt=\"Screen Shot 2019-08-27 at 5 05 13 PM\" src=\"https://user-images.githubusercontent.com/13592258/63816423-ea48fe00-c8ec-11e9-8f66-5b226a1ff693.png\">\n\n![image](https://user-images.githubusercontent.com/13592258/63903080-0e760f00-c9c1-11e9-966a-f45b0b1c1ea6.png)\n\n<img width=\"1100\" alt=\"Screen Shot 2019-08-27 at 5 07 19 PM\" src=\"https://user-images.githubusercontent.com/13592258/63816494-37c56b00-c8ed-11e9-88e1-27a9101eb09d.png\">\n\n![image](https://user-images.githubusercontent.com/13592258/63816712-131dc300-c8ee-11e9-8ee7-d83b8ad07bf2.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63817479-5a598300-c8f1-11e9-8789-adae7df5535a.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63817900-4adb3980-c8f3-11e9-94fe-d60f7d61c4b4.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63903155-4da46000-c9c1-11e9-88dd-609d4fe685a9.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63817157-d652cb80-c8ef-11e9-944c-99391cf2fb0a.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63903259-aa077f80-c9c1-11e9-982f-b8590ce0270d.png)\n\n![image](https://user-images.githubusercontent.com/13592258/63903270-b1c72400-c9c1-11e9-85c6-6d8e8cd7f006.png)\n\nCloses #25525 from huaxingao/spark-28786.\n\nAuthored-by: Huaxin Gao <huaxing@us.ibm.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/3e09a0fce9d9fe7249b3192c036af04d7f6707d0"},{"sha":"fb1053d14a3691695539304457b50e7342020c1f","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-28807][DOCS][SQL] Document SHOW DATABASES in SQL Reference\n\n### What changes were proposed in this pull request?\nDocument SHOW DATABASES statement in SQL Reference Guide.\n\n### Why are the changes needed?\nCurrently Spark lacks documentation on the supported SQL constructs causing\nconfusion among users who sometimes have to look at the code to understand the\nusage. This is aimed at addressing this issue.\n\n### Does this PR introduce any user-facing change?\nYes.\n\n**Before:**\nThere was no documentation for this.\n\n**After.**\n<img width=\"1234\" alt=\"Screen Shot 2019-08-28 at 11 43 36 PM\" src=\"https://user-images.githubusercontent.com/14225158/63916727-dd600380-c9ed-11e9-8372-789110c9d2dc.png\">\n<img width=\"1234\" alt=\"Screen Shot 2019-08-28 at 11 43 57 PM\" src=\"https://user-images.githubusercontent.com/14225158/63916734-e0f38a80-c9ed-11e9-8ad4-d854febeaab8.png\">\n<img width=\"1234\" alt=\"Screen Shot 2019-08-28 at 11 44 13 PM\" src=\"https://user-images.githubusercontent.com/14225158/63916740-e4871180-c9ed-11e9-9cfc-199cd8a64852.png\">\n\n### How was this patch tested?\nTested using jykyll build --serve\n\nCloses #25526 from dilipbiswal/ref-doc-show-db.\n\nAuthored-by: Dilip Biswal <dbiswal@us.ibm.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/fb1053d14a3691695539304457b50e7342020c1f"},{"sha":"7d72c073ddd66ebed8861d41e6cb14f8bfa27e7b","author":{"name":"Gabor Somogyi","email":"40e6b04f18fce378745bb3a5827ff2f7df5c5829@gmail.com"},"message":"[SPARK-28760][SS][TESTS] Add Kafka delegation token end-to-end test with mini KDC\n\n### What changes were proposed in this pull request?\nAt the moment no end-to-end Kafka delegation token test exists which was mainly because of missing embedded KDC. KDC is missing in general from the testing side so I've discovered what kind of possibilities are there. The most obvious choice is the MiniKDC inside the Hadoop library where Apache Kerby runs in the background. What this PR contains:\n* Added MiniKDC as test dependency from Hadoop\n* Added `maven-bundle-plugin` because couple of dependencies are coming in bundle format\n* Added security mode to `KafkaTestUtils`. Namely start KDC -> start Zookeeper in secure mode -> start Kafka in secure mode\n* Added a roundtrip test (saves and reads back data from Kafka)\n\n### Why are the changes needed?\nNo such test exists + security testing with KDC is completely missing.\n\n### Does this PR introduce any user-facing change?\nNo.\n\n### How was this patch tested?\nExisting + additional unit tests.\nI've put the additional test into a loop and was consuming ~10 sec average.\n\nCloses #25477 from gaborgsomogyi/SPARK-28760.\n\nAuthored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/7d72c073ddd66ebed8861d41e6cb14f8bfa27e7b"},{"sha":"f8f7c52f1272a11e6549fc2e4f5a308e295d1de9","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[SPARK-28899][SQL][TEST] merge the testing in-memory v2 catalogs from catalyst and core\n\n### What changes were proposed in this pull request?\n\nThere are 2 in-memory `TableCatalog` and `Table` implementations for testing, in sql/catalyst and sql/core. This PR merges them.\n\nAfter merging, there are 3 classes:\n1. `InMemoryTable`\n2. `InMemoryTableCatalog`\n3. `StagingInMemoryTableCatalog`\n\nFor better maintainability, these 3 classes are put in 3 different files.\n\n### Why are the changes needed?\n\nreduce duplicated code\n\n### Does this PR introduce any user-facing change?\n\nno\n### How was this patch tested?\n\nN/A\n\nCloses #25610 from cloud-fan/dsv2-test.\n\nAuthored-by: Wenchen Fan <wenchen@databricks.com>\nSigned-off-by: Ryan Blue <blue@apache.org>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/f8f7c52f1272a11e6549fc2e4f5a308e295d1de9"},{"sha":"31b59bd80517f17f160eb8d6665b9bf8b3372d3f","author":{"name":"Ryan Blue","email":"4c9a82ce72ca2519f38d0af0abbb4cecb9fceca9@apache.org"},"message":"[SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python if not set\n\n### What changes were proposed in this pull request?\n\nWhen starting python processes, set `OMP_NUM_THREADS` to the number of cores allocated to an executor or driver if `OMP_NUM_THREADS` is not already set. Each python process will use the same `OMP_NUM_THREADS` setting, even if workers are not shared.\n\nThis avoids creating an OpenMP thread pool for parallel processing with a number of threads equal to the number of cores on the executor and [significantly reduces memory consumption](https://github.com/numpy/numpy/issues/10455). Instead, this threadpool should use the number of cores allocated to the executor, if available. If a setting for number of cores is not available, this doesn't change any behavior. OpenMP is used by numpy and pandas.\n\n### Why are the changes needed?\n\nTo reduce memory consumption for PySpark jobs.\n\n### Does this PR introduce any user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nValidated this reduces python worker memory consumption by more than 1GB on our cluster.\n\nCloses #25545 from rdblue/SPARK-28843-set-omp-num-cores.\n\nAuthored-by: Ryan Blue <blue@apache.org>\nSigned-off-by: HyukjinKwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/31b59bd80517f17f160eb8d6665b9bf8b3372d3f"},{"sha":"d502c80404c398d852dfa5f86a0e87c104a6286e","author":{"name":"Gabor Somogyi","email":"40e6b04f18fce378745bb3a5827ff2f7df5c5829@gmail.com"},"message":"[SPARK-28922][SS] Safe Kafka parameter redaction\n\n### What changes were proposed in this pull request?\nAt the moment Kafka parameter reduction is expecting `SparkEnv`.  This must exist in normal queries but several unit tests are not providing it to make things simple. As an end-result such tests are throwing similar exception:\n```\njava.lang.NullPointerException\n\tat org.apache.spark.kafka010.KafkaRedactionUtil$.redactParams(KafkaRedactionUtil.scala:29)\n\tat org.apache.spark.kafka010.KafkaRedactionUtilSuite.$anonfun$new$1(KafkaRedactionUtilSuite.scala:33)\n\tat org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)\n\tat org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)\n\tat org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)\n\tat org.scalatest.Transformer.apply(Transformer.scala:22)\n\tat org.scalatest.Transformer.apply(Transformer.scala:20)\n\tat org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)\n\tat org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)\n\tat org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)\n\tat org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)\n\tat org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)\n\tat org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)\n\tat org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)\n\tat org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)\n\tat org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)\n\tat org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)\n\tat org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)\n\tat org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)\n\tat org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)\n\tat org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)\n\tat org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)\n\tat org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)\n\tat org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)\n\tat org.scalatest.FunSuite.runTests(FunSuite.scala:1560)\n\tat org.scalatest.Suite.run(Suite.scala:1147)\n\tat org.scalatest.Suite.run$(Suite.scala:1129)\n\tat org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)\n\tat org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)\n\tat org.scalatest.SuperEngine.runImpl(Engine.scala:521)\n\tat org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)\n\tat org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)\n\tat org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)\n\tat org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)\n\tat org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)\n\tat org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)\n\tat org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)\n\tat org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)\n\tat org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1346)\n\tat org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1340)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1340)\n\tat org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1031)\n\tat org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1010)\n\tat org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)\n\tat org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)\n\tat org.scalatest.tools.Runner$.run(Runner.scala:850)\n\tat org.scalatest.tools.Runner.run(Runner.scala)\n\tat org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)\n\tat org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)\n```\nThese are annoying and only red herrings so I would like to make them disappear.\n\nThere are basically 2 ways to handle this situation:\n* Add default value for `SparkEnv` in `KafkaReductionUtil`\n* Add `SparkEnv` to all such tests => I think it would be overkill and would just increase number of lines without real value\n\nConsidering this I've chosen the first approach.\n\n### Why are the changes needed?\nCouple of tests are throwing exceptions even if no real problem.\n\n### Does this PR introduce any user-facing change?\nNo.\n\n### How was this patch tested?\nNew + additional unit tests.\n\nCloses #25621 from gaborgsomogyi/safe-reduct.\n\nAuthored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/d502c80404c398d852dfa5f86a0e87c104a6286e"},{"sha":"780aa717499aa1c81f7afe8e18c300b08ad6b15f","author":{"name":"Dongjoon Hyun","email":"9b02cb7e37158797a8e22c559f7c5f5ecd28ea57@apple.com"},"message":"[SPARK-28919][INFRA] Add more profiles for JDK8/11 build test for Github workflow\n\n### What changes were proposed in this pull request?\n\nThis PR aims to add `-Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-3.2 -Phadoop-cloud` profiles to GitHub workflow conf.\n\n### Why are the changes needed?\n\nCurrently, we build with JDK8 and test with JDK8/11 in Jenkins.\nAnd, we use GitHub Workflow for JDK8/JDK11 building test.\nTo test JDK11 fully, we need to enable `hive` and `hadoop-3.2` profiles for `Hive 2.3.6` and `Hadoop 3.2`. Also, this PR adds all resource manager modules, too.\n\n### Does this PR introduce any user-facing change?\n\nNo. In addition, Jenkins workload will be the same because this is specific to GitHub workflow.\n\n### How was this patch tested?\n\nSee the GitHub workflow run result.\n\nCloses #25624 from dongjoon-hyun/SPARK-JDK11-HIVE.\n\nAuthored-by: Dongjoon Hyun <dhyun@apple.com>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/780aa717499aa1c81f7afe8e18c300b08ad6b15f"},{"sha":"2db45cbd5a18a95af6e6a922382df42d1db98cef","author":{"name":"Liang-Chi Hsieh","email":"56d6f1a570b23ccf100f962eeec2ef1658eed1f8@uber.com"},"message":"[SPARK-28920][INFRA] Set up java version for github workflow\n\nThis patch adds java version parameter to GitHub workflow conf for JDK8/11.\n\nAs we want to build JDK8/11 on GitHub workflow, we might need to add java version according current matrix.\n\nNo\n\nSee the GitHub workflow run result.\n\nCloses #25625 from viirya/github-workflow-java.\n\nAuthored-by: Liang-Chi Hsieh <liangchi@uber.com>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/2db45cbd5a18a95af6e6a922382df42d1db98cef"},{"sha":"827969399b88a084072e7864269e7b0fd64df439","author":{"name":"Burak Yavuz","email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com"},"message":"[SPARK-28668][SQL] Support V2SessionCatalog for ALTER TABLE\n\n### What changes were proposed in this pull request?\n\nAdds support for the V2SessionCatalog for ALTER TABLE statements.\nImplementation changes are ~50 loc. The rest is just test refactoring.\n\n### Why are the changes needed?\nTo allow V2 DataSources to plug in through a configurable plugin interface without requiring the explicit use of catalog identifiers, and leverage ALTER TABLE statements.\n\n### How was this patch tested?\n\nBy re-using existing tests in DataSourceV2SQLSuite.\n\nCloses #25502 from brkyvz/alterV3.\n\nAuthored-by: Burak Yavuz <brkyvz@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/827969399b88a084072e7864269e7b0fd64df439"},{"sha":"2bd02e2b41619196d2385d0541fa908e600a030f","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-28866][ML] Persist item factors RDD when checkpointing in ALS\n\n### What changes were proposed in this pull request?\n\nIn ALS ML implementation, for non-implicit case, we checkpoint the RDD of item factors, between intervals. Before checkpointing (.checkpoint()) and materializing (.count()) RDD, this RDD was not persisted. It causes recomputation. In an experiment, there is performance difference between persisting and no persisting before checkpointing the RDD.\n\nThe performance difference is not big, but this change is not big too. The actual performance difference varies depending the interval of checkpoint, training dataset, etc.\n\n### Why are the changes needed?\n\nPersisting the RDD before checkpointing the RDD of item factors can avoid recomputation.\n\n### Does this PR introduce any user-facing change?\n\nNo\n\n### How was this patch tested?\n\nManual check RDD recomputation or not.\n\nTaking 30% MovieLens 20M Dataset as training dataset. Setting checkpoint dir for SparkContext. Fitting an ALS model like:\n\n```scala\nval als = new ALS()\n      .setMaxIter(100)\n      .setCheckpointInterval(5)\n      .setRegParam(0.01)\n      .setUserCol(\"userId\")\n      .setItemCol(\"movieId\")\n      .setRatingCol(\"rating\")\n\nval t0 = System.currentTimeMillis()\nval model = als.fit(training)\nval t1 = System.currentTimeMillis()\n```\n\nBefore this patch:  65.386 s\nAfter this patch: 61.022 s\n\nCloses #25576 from viirya/persist-item-factors.\n\nAuthored-by: Liang-Chi Hsieh <viirya@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/2bd02e2b41619196d2385d0541fa908e600a030f"},{"sha":"92cabf63067b211a7cb06d27da8b67fbfe72c4d2","author":{"name":"HyukjinKwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.2.0 and fix build profile on AppVeyor\n\n### What changes were proposed in this pull request?\n\nThis PR proposes to upgrade scala-maven-plugin from 3.4.4 to 4.2.0.\n\nUpgrade to 4.1.1 was reverted due to unexpected build failure on AppVeyor.\n\nThe root cause seems to be an issue specific to AppVeyor - loading the system library 'kernel32.dll' seems being failed.\n\n```\nSuppressed: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.platform.win32.Kernel32\n        at sbt.internal.io.WinMilli$.getHandle(Milli.scala:264)\n        at sbt.internal.io.WinMilli$.getModifiedTimeNative(Milli.scala:289)\n        at sbt.internal.io.WinMilli$.getModifiedTimeNative(Milli.scala:260)\n        at sbt.internal.io.MilliNative.getModifiedTime(Milli.scala:61)\n        at sbt.internal.io.Milli$.getModifiedTime(Milli.scala:360)\n        at sbt.io.IO$.$anonfun$getModifiedTimeOrZero$1(IO.scala:1373)\n        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)\n        at sbt.internal.io.Retry$.liftedTree2$1(Retry.scala:38)\n        at sbt.internal.io.Retry$.impl$1(Retry.scala:38)\n        at sbt.internal.io.Retry$.apply(Retry.scala:52)\n        at sbt.internal.io.Retry$.apply(Retry.scala:24)\n        at sbt.io.IO$.getModifiedTimeOrZero(IO.scala:1373)\n        at sbt.internal.inc.caching.ClasspathCache$.fromCacheOrHash$1(ClasspathCache.scala:44)\n        at sbt.internal.inc.caching.ClasspathCache$.$anonfun$hashClasspath$1(ClasspathCache.scala:53)\n        at scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:659)\n        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)\n        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)\n        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)\n        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)\n        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)\n        at scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)\n        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:170)\n        ... 25 more\n```\n\nBy setting `-Djna.nosys=true`, it directly loads the library from the jar instead of system's.\n\nIn this way, the build seems working fine.\n\n### Why are the changes needed?\n\nIt upgrades the plugin to fix bugs and fixes the CI build.\n\n### Does this PR introduce any user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nIt was tested at https://github.com/apache/spark/pull/25497\n\nCloses #25633 from HyukjinKwon/SPARK-28759.\n\nAuthored-by: HyukjinKwon <gurwls223@apache.org>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/92cabf63067b211a7cb06d27da8b67fbfe72c4d2"},{"sha":"ea90ea6ce7e1e07064279b2f78a0301fd2048f11","author":{"name":"mcheah","email":"2e9bf58ec2f93170404b3a28538fb27400f699db@palantir.com"},"message":"[SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter\n\n## What changes were proposed in this pull request?\n\nUse the shuffle writer APIs introduced in SPARK-28209 in the sort shuffle writer.\n\n## How was this patch tested?\n\nExisting unit tests were changed to use the plugin instead, and they used the local disk version to ensure that there were no regressions.\n\nCloses #25342 from mccheah/shuffle-writer-refactor-sort-shuffle-writer.\n\nLead-authored-by: mcheah <mcheah@palantir.com>\nCo-authored-by: mccheah <mcheah@palantir.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/ea90ea6ce7e1e07064279b2f78a0301fd2048f11"},{"sha":"3b07a4eb288ef1ab060d1fbed059a31e9da92609","author":{"name":"younggyu chun","email":"8e69a484eaff0986168dd0159eb9349ee56fddd2@gmail.com"},"message":"[SPARK-27931][SQL] Accept \"true\", \"yes\", \"1\", \"false\", \"no\", \"0\", and unique prefixes as input and trim input for the boolean data type\n\n## What changes were proposed in this pull request?\nThis PR aims to add \"true\", \"yes\", \"1\", \"false\", \"no\", \"0\", and unique prefixes as input for the boolean data type and ignore input whitespace. Please see the following what string representations are using for the boolean type in other databases.\n\nhttps://www.postgresql.org/docs/devel/datatype-boolean.html\nhttps://docs.aws.amazon.com/redshift/latest/dg/r_Boolean_type.html\n\n## How was this patch tested?\nAdded new tests to CastSuite.\n\nCloses #25458 from younggyuchun/SPARK-27931.\n\nAuthored-by: younggyu chun <younggyuchun@gmail.com>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/3b07a4eb288ef1ab060d1fbed059a31e9da92609"},{"sha":"7cc0f0e9a795431a79408644b3473df499118d1e","author":{"name":"HyukjinKwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results\n\n### What changes were proposed in this pull request?\n\nSee https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109834/testReport/junit/org.apache.spark.sql/SQLQueryTestSuite/\n\n![Screen Shot 2019-08-28 at 4 08 58 PM](https://user-images.githubusercontent.com/6477701/63833484-2a23ea00-c9ae-11e9-91a1-0859cb183fea.png)\n\n```xml\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuite hostname=\"C02Y52ZLJGH5\" name=\"org.apache.spark.sql.SQLQueryTestSuite\" tests=\"3\" errors=\"0\" failures=\"0\" skipped=\"0\" time=\"14.475\">\n    ...\n    <testcase classname=\"org.apache.spark.sql.SQLQueryTestSuite\" name=\"sql - Scala UDF\" time=\"6.703\">\n    </testcase>\n    <testcase classname=\"org.apache.spark.sql.SQLQueryTestSuite\" name=\"sql - Regular Python UDF\" time=\"4.442\">\n    </testcase>\n    <testcase classname=\"org.apache.spark.sql.SQLQueryTestSuite\" name=\"sql - Scalar Pandas UDF\" time=\"3.33\">\n    </testcase>\n    <system-out/>\n    <system-err/>\n</testsuite>\n```\n\nRoot cause seems a bug in SBT - it truncates the test name based on the last dot.\n\nhttps://github.com/sbt/sbt/issues/2949\nhttps://github.com/sbt/sbt/blob/v0.13.18/testing/src/main/scala/sbt/JUnitXmlTestsListener.scala#L71-L79\n\nI tried to find a better way but couldn't find. Therefore, this PR proposes a workaround by appending the test file name into the assert log:\n\n```diff\n  [info] - inner-join.sql *** FAILED *** (4 seconds, 306 milliseconds)\n+ [info]   inner-join.sql\n  [info]   Expected \"1\ta\n  [info]   1\ta\n  [info]   1\tb\n  [info]   1[]\", but got \"1\ta\n  [info]   1\ta\n  [info]   1\tb\n  [info]   1[\tb]\" Result did not match for query #6\n  [info]   SELECT tb.* FROM ta INNER JOIN tb ON ta.a = tb.a AND ta.tag = tb.tag (SQLQueryTestSuite.scala:377)\n  [info]   org.scalatest.exceptions.TestFailedException:\n  [info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)\n```\n\nIt will at least prevent us to search full logs to identify which test file is failed by clicking filed test.\n\nNote that this PR does not fully fix the issue but only fix the logs on its failed tests.\n\n### Why are the changes needed?\nTo debug Jenkins logs easier. Otherwise, we should open full logs and search which test was failed.\n\n### Does this PR introduce any user-facing change?\nIt will print out the file name of failed tests in Jenkins' test reports.\n\n### How was this patch tested?\nManually tested but Jenkins tests are required in this PR.\n\nNow it at least shows which file it is:\n\n![Screen Shot 2019-08-30 at 10 16 32 PM](https://user-images.githubusercontent.com/6477701/64023705-de22a200-cb73-11e9-8806-2e98ad35adef.png)\n\nCloses #25630 from HyukjinKwon/SPARK-28894-1.\n\nAuthored-by: HyukjinKwon <gurwls223@apache.org>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/7cc0f0e9a795431a79408644b3473df499118d1e"},{"sha":"a08f33be68d22387be3261a35601bd9f2e9c7338","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-28804][DOCS][SQL] Document DESCRIBE QUERY in SQL Reference\n\n### What changes were proposed in this pull request?\nDocument DESCRIBE QUERY statement in SQL Reference Guide.\n\n### Why are the changes needed?\nCurrently Spark lacks documentation on the supported SQL constructs causing\nconfusion among users who sometimes have to look at the code to understand the\nusage. This is aimed at addressing this issue.\n\n### Does this PR introduce any user-facing change?\nYes.\n\n**Before:**\nThere was no documentation for this.\n\n**After.**\n<img width=\"1234\" alt=\"Screen Shot 2019-08-29 at 5 47 51 PM\" src=\"https://user-images.githubusercontent.com/14225158/63985609-43e43080-ca85-11e9-8a1a-c9c15d988e24.png\">\n<img width=\"1234\" alt=\"Screen Shot 2019-08-29 at 5 48 06 PM\" src=\"https://user-images.githubusercontent.com/14225158/63985610-46468a80-ca85-11e9-882a-7163784f72c6.png\">\n<img width=\"1234\" alt=\"Screen Shot 2019-08-29 at 5 48 18 PM\" src=\"https://user-images.githubusercontent.com/14225158/63985617-49da1180-ca85-11e9-9e77-a6d6c7042a85.png\">\n\n### How was this patch tested?\nTested using jykyll build --serve\n\nCloses #25529 from dilipbiswal/ref-doc-desc-query.\n\nLead-authored-by: Dilip Biswal <dbiswal@us.ibm.com>\nCo-authored-by: Xiao Li <gatorsmile@gmail.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/a08f33be68d22387be3261a35601bd9f2e9c7338"},{"sha":"d5b7eed12fca3fca1b311674737eb148a6a7309e","author":{"name":"Sean Owen","email":"3f9d48d7c3f24f7009479c5fa630bd1131ffc724@databricks.com"},"message":"[SPARK-28903][STREAMING][PYSPARK][TESTS] Fix AWS JDK version conflict that breaks Pyspark Kinesis tests\n\nThe Pyspark Kinesis tests are failing, at least in master:\n\n```\n======================================================================\nERROR: test_kinesis_stream (pyspark.streaming.tests.test_kinesis.KinesisStreamTests)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"/home/jenkins/workspace/SparkPullRequestBuilder2/python/pyspark/streaming/tests/test_kinesis.py\", line 44, in test_kinesis_stream\n    kinesisTestUtils = self.ssc._jvm.org.apache.spark.streaming.kinesis.KinesisTestUtils(2)\n  File \"/home/jenkins/workspace/SparkPullRequestBuilder2/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py\", line 1554, in __call__\n    answer, self._gateway_client, None, self._fqn)\n  File \"/home/jenkins/workspace/SparkPullRequestBuilder2/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py\", line 328, in get_return_value\n    format(target_id, \".\", name), value)\nPy4JJavaError: An error occurred while calling None.org.apache.spark.streaming.kinesis.KinesisTestUtils.\n: java.lang.NoSuchMethodError: com.amazonaws.regions.Region.getAvailableEndpoints()Ljava/util/Collection;\n\tat org.apache.spark.streaming.kinesis.KinesisTestUtils$.$anonfun$getRegionNameByEndpoint$1(KinesisTestUtils.scala:211)\n\tat org.apache.spark.streaming.kinesis.KinesisTestUtils$.$anonfun$getRegionNameByEndpoint$1$adapted(KinesisTestUtils.scala:211)\n\tat scala.collection.Iterator.find(Iterator.scala:993)\n\tat scala.collection.Iterator.find$(Iterator.scala:990)\n\tat scala.collection.AbstractIterator.find(Iterator.scala:1429)\n\tat scala.collection.IterableLike.find(IterableLike.scala:81)\n\tat scala.collection.IterableLike.find$(IterableLike.scala:80)\n\tat scala.collection.AbstractIterable.find(Iterable.scala:56)\n\tat org.apache.spark.streaming.kinesis.KinesisTestUtils$.getRegionNameByEndpoint(KinesisTestUtils.scala:211)\n\tat org.apache.spark.streaming.kinesis.KinesisTestUtils.<init>(KinesisTestUtils.scala:46)\n...\n```\n\nThe non-Python Kinesis tests are fine though. It turns out that this is because Pyspark tests use the output of the Spark assembly, and it pulls in `hadoop-cloud`, which in turn pulls in an old AWS Java SDK.\n\nPer Steve Loughran (below), it seems like we can just resolve this by excluding the aws-java-sdk dependency. See the attached PR for some more detail about the debugging and other options.\n\nSee https://github.com/apache/spark/pull/25558#issuecomment-524042709\n\nCloses #25559 from srowen/KinesisTest.\n\nAuthored-by: Sean Owen <sean.owen@databricks.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/d5b7eed12fca3fca1b311674737eb148a6a7309e"},{"sha":"1f96ce5443e744bddaf09d913005763e534adf73","author":{"name":"Dongjoon Hyun","email":"9b02cb7e37158797a8e22c559f7c5f5ecd28ea57@apple.com"},"message":"[SPARK-28932][BUILD] Add scala-library test dependency to network-common module for JDK11\n\n### What changes were proposed in this pull request?\n\nThis PR adds `scala-library` test dependency to `network-common` module for JDK11.\n\n### Why are the changes needed?\n\nIn JDK11, the following command fails due to scala library.\n```\nmvn clean install -pl common/network-common -DskipTests\n```\n\n**BEFORE**\n```\n...\nerror: fatal error: object scala in compiler mirror not found.\none error found\n...\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD FAILURE\n[INFO] ------------------------------------------------------------------------\n```\n\n**AFTER**\n```\n[INFO] ------------------------------------------------------------------------\n[INFO] BUILD SUCCESS\n[INFO] ------------------------------------------------------------------------\n```\n\n### Does this PR introduce any user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nManual. On JDK11, do the following.\n```\nmvn clean install -pl common/network-common -DskipTests\n```\n\nCloses #25638 from dongjoon-hyun/SPARK-28932.\n\nAuthored-by: Dongjoon Hyun <dhyun@apple.com>\nSigned-off-by: Dongjoon Hyun <dhyun@apple.com>","distinct":true,"url":"https://api.github.com/repos/Tonix517/spark/commits/1f96ce5443e744bddaf09d913005763e534adf73"}]}
{"actor":{"display_login":"Tonix517"}}