{"push_id":2896967495,"size":339,"distinct_size":339,"ref":"refs/heads/bug/SPARK-22148","head":"4c881680fdde32244030b54b44125ac217dacb0d","before":"ffbc9c32d14a0c82036defb90eb18167f93bad4d","commits":[{"sha":"298e80f5c7a735206e070e03b03aa52712038128","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-25041][BUILD] upgrade genJavaDoc-plugin from 0.10 to 0.11\n\n## What changes were proposed in this pull request?\n\nThis PR fixes a build error with sbt using Scala-2.12. Since [`genJavaDoc-plugin`] (https://mvnrepository.com/artifact/com.typesafe.genjavadoc/genjavadoc-plugin) 0.10 is not prepared for Scala-2.12.6, the recent version of `genJavaDoc-plugin` is necessary.\n\nThe version 0.11 of `genJavaDoc-plugin` is also prepared for Scala-2.11.12.\n\n[genJavaDoc-0.10](https://index.scala-lang.org/lightbend/genjavadoc/genjavadoc-plugin/0.10)\n[genJavaDoc-0.11](https://index.scala-lang.org/lightbend/genjavadoc/genjavadoc-plugin/0.11)\n\n## How was this patch tested?\n\nManually tested for Scala-2.12.\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #22020 from kiszk/SPARK-25041.","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/298e80f5c7a735206e070e03b03aa52712038128"},{"sha":"cb6cb313637836737f8ec7de34e592e425efd57f","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-23937][SQL] Add map_filter SQL function\n\n## What changes were proposed in this pull request?\n\nThe PR adds the high order function `map_filter`, which filters the entries of a map and returns a new map which contains only the entries which satisfied the filter function.\n\n## How was this patch tested?\n\nadded UTs\n\nCloses #21986 from mgaido91/SPARK-23937.\n\nAuthored-by: Marco Gaido <marcogaido91@gmail.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/cb6cb313637836737f8ec7de34e592e425efd57f"},{"sha":"8c13cb2ae4f82c0eb04939a667310b9268b1c2a7","author":{"name":"invkrh","email":"dce863d051743e2f24271654bb0dd2d5db183ef1@gmail.com"},"message":"[SPARK-25031][SQL] Fix MapType schema print\n\n## What changes were proposed in this pull request?\n\nThe PR fix the bug in `buildFormattedString` function in `MapType`, which makes the printed schema misleading.\n\n## How was this patch tested?\n\nAdded UT\n\nCloses #22006 from invkrh/fix-map-schema-print.\n\nAuthored-by: invkrh <invkrh@gmail.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/8c13cb2ae4f82c0eb04939a667310b9268b1c2a7"},{"sha":"f6356f9bc0ddcb84827c468a3da677995abe033f","author":{"name":"Neal Song","email":"2a6f2640336e4bfd8a9acddb0861bc5189a2436b@126.com"},"message":"[SPARK-25046][SQL] Fix Alter View can excute sql like \"ALTER VIEW ... AS INSERT INTO\"\n\n## What changes were proposed in this pull request?\n\nAlter View  can excute sql  like \"ALTER VIEW ... AS INSERT INTO\" . We should throw ParseException(s\"Operation not allowed: $message\", ctx)  as Create View does.\n ```\noverride def visitCreateView(ctx: CreateViewContext): LogicalPlan = withOrigin(ctx) {\n    if (ctx.identifierList != null) {\n      operationNotAllowed(\"CREATE VIEW ... PARTITIONED ON\", ctx)\n    } else {\n      // CREATE VIEW ... AS INSERT INTO is not allowed.\n      ctx.query.queryNoWith match {\n        case s: SingleInsertQueryContext if s.insertInto != null =>\n          operationNotAllowed(\"CREATE VIEW ... AS INSERT INTO\", ctx)\n        case _: MultiInsertQueryContext =>\n          operationNotAllowed(\"CREATE VIEW ... AS FROM ... [INSERT INTO ...]+\", ctx)\n        case _ => // OK\n      }\n```\n\n```\noverride def visitAlterViewQuery(ctx: AlterViewQueryContext): LogicalPlan = withOrigin(ctx) {\n    // ALTER VIEW ... AS INSERT INTO is not allowed.\n    ctx.query.queryNoWith match {\n      case s: SingleInsertQueryContext if s.insertInto != null =>\n        operationNotAllowed(\"ALTER VIEW ... AS INSERT INTO\", ctx)\n      case _: MultiInsertQueryContext =>\n        operationNotAllowed(\"ALTER VIEW ... AS FROM ... [INSERT INTO ...]+\", ctx)\n      case _ => // OK\n    }\n    AlterViewAsCommand(\n      name = visitTableIdentifier(ctx.tableIdentifier),\n      originalText = source(ctx.query),\n      query = plan(ctx.query))\n  }\n```\n\n## How was this patch tested?\n\nUT has been added in SparkSqlParserSuite\n\nCloses #22028 from sddyljsx/SPARK-25046.\n\nLead-authored-by: Neal Song <neal_song@126.com>\nCo-authored-by: neal <neal_song@126.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/f6356f9bc0ddcb84827c468a3da677995abe033f"},{"sha":"66699c5c3061f54463bd1d0f7a8f8e168c2882c9","author":{"name":"Sean Owen","email":"d519fa33a62684a61ab04b24d26f83452d51b34c@gmail.com"},"message":"[SPARK-25029][TESTS] Scala 2.12 issues: TaskNotSerializable and Janino \"Two non-abstract methods ...\" errors\n\n## What changes were proposed in this pull request?\n\nFixes for test issues that arose after Scala 2.12 support was added -- ones that only affect the 2.12 build.\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #22004 from srowen/SPARK-25029.\n\nAuthored-by: Sean Owen <srowen@gmail.com>\nSigned-off-by: Sean Owen <srowen@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/66699c5c3061f54463bd1d0f7a8f8e168c2882c9"},{"sha":"d90f1336d87199aac56fe227a0fe14ab0ae3a332","author":{"name":"Xingbo Jiang","email":"19a442ba0283f6f4c465cc281093aec56203af87@databricks.com"},"message":"[SPARK-25045][CORE] Make `RDDBarrier.mapParititions` similar to `RDD.mapPartitions`\n\n## What changes were proposed in this pull request?\n\nSignature of the function passed to `RDDBarrier.mapPartitions()` is different from that of `RDD.mapPartitions`. The later doesn’t take a `TaskContext`. We shall make the function signature the same to avoid confusion and misusage.\n\nThis PR proposes the following API changes:\n- In `RDDBarrier`, migrate `mapPartitions` from\n   ```\n        def mapPartitions[S: ClassTag](\n            f: (Iterator[T], BarrierTaskContext) => Iterator[S],\n            preservesPartitioning: Boolean = false): RDD[S]\n        }\n   ```\n    to\n   ```\n        def mapPartitions[S: ClassTag](\n            f: Iterator[T] => Iterator[S],\n            preservesPartitioning: Boolean = false): RDD[S]\n        }\n   ```\n- Add new static method to get a `BarrierTaskContext`:\n   ```\n        object BarrierTaskContext {\n           def get(): BarrierTaskContext\n        }\n   ```\n\n## How was this patch tested?\n\nExisting test cases.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #22026 from jiangxb1987/mapPartitions.","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/d90f1336d87199aac56fe227a0fe14ab0ae3a332"},{"sha":"5fef6e3513d6023a837c427d183006d153c7102b","author":{"name":"Ryan Blue","email":"4c9a82ce72ca2519f38d0af0abbb4cecb9fceca9@apache.org"},"message":"[SPARK-24251][SQL] Add AppendData logical plan.\n\n## What changes were proposed in this pull request?\n\nThis adds a new logical plan, AppendData, that was proposed in SPARK-23521: Standardize SQL logical plans.\n\n* DataFrameWriter uses the new AppendData plan for DataSourceV2 appends\n* AppendData is resolved if its output columns match the incoming data frame\n* A new analyzer rule, ResolveOutputColumns, validates data before it is appended. This rule will add safe casts, rename columns, and checks nullability\n\n## How was this patch tested?\n\nExisting tests for v2 appends. Will add AppendData tests to validate logical plan analysis.\n\nCloses #21305 from rdblue/SPARK-24251-add-append-data.\n\nLead-authored-by: Ryan Blue <blue@apache.org>\nCo-authored-by: Ryan Blue <rdblue@users.noreply.github.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/5fef6e3513d6023a837c427d183006d153c7102b"},{"sha":"c7a229d655a0ac0658d9f40d388ad123f4ed8ce2","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-25010][SQL][FOLLOWUP] Shuffle should also produce different values for each execution in streaming query.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up pr of #21980.\n\n`Shuffle` can also be `ExpressionWithRandomSeed` to produce different values for each execution in streaming query.\n\n## How was this patch tested?\n\nAdded a test.\n\nCloses #22027 from ueshin/issues/SPARK-25010/random_seed.\n\nAuthored-by: Takuya UESHIN <ueshin@databricks.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/c7a229d655a0ac0658d9f40d388ad123f4ed8ce2"},{"sha":"f08f6f4314b16fb09c479f6537f99bda77e4c256","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23935][SQL][FOLLOWUP] mapEntry throws org.codehaus.commons.compiler.CompileException\n\n## What changes were proposed in this pull request?\n\nThis PR fixes an exception during the compilation of generated code of `mapEntry`. This error occurs since the current code uses `key` type to store a `value` when `key` and `value` types are primitive type.\n\n```\n     val mid0 = Literal.create(Map(1 -> 1.1, 2 -> 2.2), MapType(IntegerType, DoubleType))\n     checkEvaluation(MapEntries(mid0), Seq(r(1, 1.1), r(2, 2.2)))\n```\n\n```\n[info]   Code generation of map_entries(keys: [1,2], values: [1.1,2.2]) failed:\n[info]   java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: No applicable constructor/method found for actual parameters \"int, double\"; candidates are: \"public void org.apache.spark.sql.catalyst.expressions.UnsafeRow.setInt(int, int)\", \"public void org.apache.spark.sql.catalyst.InternalRow.setInt(int, int)\"\n[info]   java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 80, Column 20: No applicable constructor/method found for actual parameters \"int, double\"; candidates are: \"public void org.apache.spark.sql.catalyst.expressions.UnsafeRow.setInt(int, int)\", \"public void org.apache.spark.sql.catalyst.InternalRow.setInt(int, int)\"\n[info]   \tat com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n[info]   \tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n[info]   \tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)\n[info]   \tat com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)\n[info]   \tat com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)\n[info]   \tat com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)\n[info]   \tat com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)\n[info]   \tat com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)\n[info]   \tat com.google.common.cache.LocalCache.get(LocalCache.java:4000)\n[info]   \tat com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)\n[info]   \tat com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)\n[info]   \tat org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:1290)\n...\n```\n\n## How was this patch tested?\n\nAdded a new test to `CollectionExpressionsSuite`\n\nCloses #22033 from kiszk/SPARK-23935-followup.\n\nAuthored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/f08f6f4314b16fb09c479f6537f99bda77e4c256"},{"sha":"960af63913613ed7104cd76e477e325bd3020163","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-25036][SQL] avoid match may not be exhaustive in Scala-2.12\n\n## What changes were proposed in this pull request?\n\nThe PR remove the following compilation error using scala-2.12 with sbt by adding a default case to `match`.\n\n```\n/home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:63: match may not be exhaustive.\n[error] It would fail on the following inputs: (NumericValueInterval(_, _), _), (_, NumericValueInterval(_, _)), (_, _)\n[error] [warn]   def isIntersected(r1: ValueInterval, r2: ValueInterval): Boolean = (r1, r2) match {\n[error] [warn]\n[error] [warn] /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/ValueInterval.scala:79: match may not be exhaustive.\n[error] It would fail on the following inputs: (NumericValueInterval(_, _), _), (_, NumericValueInterval(_, _)), (_, _)\n[error] [warn]     (r1, r2) match {\n[error] [warn]\n[error] [warn] /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxCountDistinctForIntervals.scala:67: match may not be exhaustive.\n[error] It would fail on the following inputs: (ArrayType(_, _), _), (_, ArrayData()), (_, _)\n[error] [warn]     (endpointsExpression.dataType, endpointsExpression.eval()) match {\n[error] [warn]\n[error] [warn] /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:470: match may not be exhaustive.\n[error] It would fail on the following inputs: NewFunctionSpec(_, None, Some(_)), NewFunctionSpec(_, Some(_), None)\n[error] [warn]     newFunction match {\n[error] [warn]\n[error] [warn] [error] [warn] /home/ishizaki/Spark/PR/scala212/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala:709: match may not be exhaustive.\n[error] It would fail on the following input: Schema((x: org.apache.spark.sql.types.DataType forSome x not in org.apache.spark.sql.types.StructType), _)\n[error] [warn]   def attributesFor[T: TypeTag]: Seq[Attribute] = schemaFor[T] match {\n[error] [warn]\n```\n\n## How was this patch tested?\n\nExisting UTs with Scala-2.11.\nManually build with Scala-2.12\n\nCloses #22014 from kiszk/SPARK-25036b.\n\nAuthored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/960af63913613ed7104cd76e477e325bd3020163"},{"sha":"6f6a4200783ec4d9041421c1b5fc59d4c9a58adb","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-23911][SQL][FOLLOW-UP] Fix examples of aggregate function.\n\n## What changes were proposed in this pull request?\n\nThis pr is a follow-up pr of #21982 and fixes the examples.\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #22035 from ueshin/issues/SPARK-23911/fup1.\n\nAuthored-by: Takuya UESHIN <ueshin@databricks.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/6f6a4200783ec4d9041421c1b5fc59d4c9a58adb"},{"sha":"f62fe435de2228ad6c3a857c214e782b8f308516","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-25036][SQL][FOLLOW-UP] Avoid match may not be exhaustive in Scala-2.12.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up pr of #22014.\n\nWe still have some more compilation errors in scala-2.12 with sbt:\n\n```\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala:493: match may not be exhaustive.\n[error] It would fail on the following input: (_, _)\n[error] [warn]       val typeMatches = (targetType, f.dataType) match {\n[error] [warn]\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala:393: match may not be exhaustive.\n[error] It would fail on the following input: (_, _)\n[error] [warn]             prevBatchOff.get.toStreamProgress(sources).foreach {\n[error] [warn]\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:173: match may not be exhaustive.\n[error] It would fail on the following input: AggregateExpression(_, _, false, _)\n[error] [warn]     val rewrittenDistinctFunctions = functionsWithDistinct.map {\n[error] [warn]\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala:271: match may not be exhaustive.\n[error] It would fail on the following input: (_, _)\n[error] [warn]       keyWithIndexToValueMetrics.customMetrics.map {\n[error] [warn]\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:959: match may not be exhaustive.\n[error] It would fail on the following input: CatalogTableType(_)\n[error] [warn]     val tableTypeString = metadata.tableType match {\n[error] [warn]\n[error] [warn] /.../sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:923: match may not be exhaustive.\n[error] It would fail on the following input: CatalogTableType(_)\n[error] [warn]     hiveTable.setTableType(table.tableType match {\n[error] [warn]\n```\n\n## How was this patch tested?\n\nManually build with Scala-2.12.\n\nCloses #22039 from ueshin/issues/SPARK-25036/fix_match.\n\nAuthored-by: Takuya UESHIN <ueshin@databricks.com>\nSigned-off-by: Sean Owen <srowen@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/f62fe435de2228ad6c3a857c214e782b8f308516"},{"sha":"a40806d2bd84e9a0308165f0d6c97e9cf00aa4a3","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-23596][SQL] Test interpreted path on encoders test suites\n\n## What changes were proposed in this pull request?\n\nWe have completed a significant subset of the object related Expressions to provide an interpreted fallback. This PR is going to modify the tests to also test the interpreted code paths.\n\nOne concern right now is that by testing the interpreted code paths too, we will double current test time or more. Otherwise, we can only choose to test the interpreted code paths for just few test suites such as encoder related.\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #21535 from viirya/SPARK-23596.\n\nAuthored-by: Liang-Chi Hsieh <viirya@gmail.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/a40806d2bd84e9a0308165f0d6c97e9cf00aa4a3"},{"sha":"519e03d82e52d2d948f3ef25f5b85cc54bb11a75","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-25058][SQL] Use Block.isEmpty/nonEmpty to check whether the code is empty or not.\n\n## What changes were proposed in this pull request?\n\nWe should use `Block.isEmpty/nonEmpty` instead of comparing with empty string to check whether the code is empty or not.\n\n```\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala:278: org.apache.spark.sql.catalyst.expressions.codegen.Block and String are unrelated: they will most likely always compare unequal\n[error] [warn]       if (ev.code != \"\" && required.contains(attributes(i))) {\n[error] [warn]\n[error] [warn] /.../sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala:323: org.apache.spark.sql.catalyst.expressions.codegen.Block and String are unrelated: they will most likely never compare equal\n[error] [warn]          |  ${buildVars.filter(_.code == \"\").map(v => s\"${v.isNull} = true;\").mkString(\"\\n\")}\n[error] [warn]\n```\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #22041 from ueshin/issues/SPARK-25058/fix_comparison.\n\nAuthored-by: Takuya UESHIN <ueshin@databricks.com>\nSigned-off-by: Takuya UESHIN <ueshin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/519e03d82e52d2d948f3ef25f5b85cc54bb11a75"},{"sha":"56e9e97073cf1896e301371b3941c9307e42ff77","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[MINOR][DOC] Fix typo\n\n## What changes were proposed in this pull request?\n\nThis PR fixes typo regarding `auxiliary verb + verb[s]`. This is a follow-on of #21956.\n\n## How was this patch tested?\n\nN/A\n\nCloses #22040 from kiszk/spellcheck1.\n\nAuthored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/56e9e97073cf1896e301371b3941c9307e42ff77"},{"sha":"386fbd3aff95ce919567b1b94d5b19c5bcef266a","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23415][SQL][TEST] Make behavior of BufferHolderSparkSubmitSuite correct and stable\n\n## What changes were proposed in this pull request?\n\nThis PR addresses two issues in `BufferHolderSparkSubmitSuite`.\n\n1. While `BufferHolderSparkSubmitSuite` tried to allocate a large object several times, it actually allocated an object once and reused the object.\n2. `BufferHolderSparkSubmitSuite` may fail due to timeout\n\nTo assign a small object before allocating a large object each time solved issue 1 by avoiding reuse.\nTo increasing heap size from 4g to 7g solved issue 2. It can also avoid OOM after fixing issue 1.\n\n## How was this patch tested?\n\nUpdated existing `BufferHolderSparkSubmitSuite`\n\nCloses #20636 from kiszk/SPARK-23415.\n\nAuthored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/386fbd3aff95ce919567b1b94d5b19c5bcef266a"},{"sha":"b2950cef3c898f59a2c92e8800ff134c44263b9a","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"Revert \"[SPARK-24648][SQL] SqlMetrics should be threadsafe\"\n\nThis reverts commit 5264164a67df498b73facae207eda12ee133be7d.","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/b2950cef3c898f59a2c92e8800ff134c44263b9a"},{"sha":"1a7e747ce4f8c5253c5923045d23c62e43a6566b","author":{"name":"Sean Owen","email":"d519fa33a62684a61ab04b24d26f83452d51b34c@gmail.com"},"message":"[SPARK-25047][ML] Can't assign SerializedLambda to scala.Function1 in deserialization of BucketedRandomProjectionLSHModel\n\n## What changes were proposed in this pull request?\n\nConvert two function fields in ML classes to simple functions to avoi…d odd SerializedLambda deserialization problem\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #22032 from srowen/SPARK-25047.\n\nAuthored-by: Sean Owen <srowen@gmail.com>\nSigned-off-by: Sean Owen <srowen@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/1a7e747ce4f8c5253c5923045d23c62e43a6566b"},{"sha":"2949a835fae3f4ac6e3dae6f18cd8b6543b74601","author":{"name":"maryannxue","email":"a27c3be666bde3a2f598c62e1c810c9c07319551@apache.org"},"message":"[SPARK-25063][SQL] Rename class KnowNotNull to KnownNotNull\n\n## What changes were proposed in this pull request?\n\nCorrect the class name typo checked in through SPARK-24891\n\n## How was this patch tested?\n\nPassed all existing tests.\n\nCloses #22049 from maryannxue/known-not-null.\n\nAuthored-by: maryannxue <maryannxue@apache.org>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/2949a835fae3f4ac6e3dae6f18cd8b6543b74601"},{"sha":"d36539741ff6a12a6acde9274e9992a66cdd36e7","author":{"name":"Achuth17","email":"32e1c3ad80238151cedc62114efd95e40cb2da2a@gmail.com"},"message":"[SPARK-24626][SQL] Improve location size calculation in Analyze Table command\n\n## What changes were proposed in this pull request?\n\nCurrently, Analyze table calculates table size sequentially for each partition. We can parallelize size calculations over partitions.\n\nResults : Tested on a table with 100 partitions and data stored in S3.\nWith changes :\n- 10.429s\n- 10.557s\n- 10.439s\n- 9.893s \n\nWithout changes :\n- 110.034s\n- 99.510s\n- 100.743s\n- 99.106s\n\n## How was this patch tested?\n\nSimple unit test.\n\nCloses #21608 from Achuth17/improveAnalyze.\n\nLead-authored-by: Achuth17 <Achuth.narayan@gmail.com>\nCo-authored-by: arajagopal17 <arajagopal@qubole.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dhruve/spark/commits/d36539741ff6a12a6acde9274e9992a66cdd36e7"}]}
{"actor":{"display_login":"dhruve"}}