{"push_id":2827431597,"size":212,"distinct_size":23,"ref":"refs/heads/updateNewMemory","head":"e223446978707807621f4e6b5acfddbea31a19fc","before":"c31597272c10c695d74a7a497ee76fd310026cfd","commits":[{"sha":"275415777b84b82aa5409e6577e1efaff1d989e7","author":{"name":"Xingbo Jiang","email":"19a442ba0283f6f4c465cc281093aec56203af87@databricks.com"},"message":"[SPARK-24795][CORE][FOLLOWUP] Kill all running tasks when a task in a barrier stage fail\n\n## What changes were proposed in this pull request?\n\nKill all running tasks when a task in a barrier stage fail in the middle. `TaskScheduler`.`cancelTasks()` will also fail the job, so we implemented a new method `killAllTaskAttempts()` to just kill all running tasks of a stage without cancel the stage/job.\n\n## How was this patch tested?\n\nAdd new test cases in `TaskSchedulerImplSuite`.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #21943 from jiangxb1987/killAllTasks.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/275415777b84b82aa5409e6577e1efaff1d989e7"},{"sha":"a65736996b2b506f61cc8d599ec9f4c52a1b5312","author":{"name":"Stavros Kontopoulos","email":"12186fc1627a9017e058cb6022429b76efb90814@lightbend.com"},"message":"[SPARK-14540][CORE] Fix remaining major issues for Scala 2.12 Support\n\n## What changes were proposed in this pull request?\nThis PR addresses issues 2,3 in this [document](https://docs.google.com/document/d/1fbkjEL878witxVQpOCbjlvOvadHtVjYXeB-2mgzDTvk).\n\n* We modified the closure cleaner to identify closures that are implemented via the LambdaMetaFactory mechanism (serializedLambdas) (issue2).\n\n* We also fix the issue due to scala/bug#11016. There are two options for solving the Unit issue, either add () at the end of the closure or use the trick described in the doc. Otherwise overloading resolution does not work (we are not going to eliminate either of the methods) here. Compiler tries to adapt to Unit and makes these two methods candidates for overloading, when there is polymorphic overloading there is no ambiguity (that is the workaround implemented). This does not look that good but it serves its purpose as we need to support two different uses for method: `addTaskCompletionListener`. One that passes a TaskCompletionListener and one that passes a closure that is wrapped with a TaskCompletionListener later on (issue3).\n\nNote: regarding issue 1 in the doc the plan is:\n\n> Do Nothing. Don’t try to fix this as this is only a problem for Java users who would want to use 2.11 binaries. In that case they can cast to MapFunction to be able to utilize lambdas. In Spark 3.0.0 the API should be simplified so that this issue is removed.\n\n## How was this patch tested?\nThis was manually tested:\n```./dev/change-scala-version.sh 2.12\n./build/mvn -DskipTests -Pscala-2.12 clean package\n./build/mvn -Pscala-2.12 clean package -DwildcardSuites=org.apache.spark.serializer.ProactiveClosureSerializationSuite -Dtest=None\n./build/mvn -Pscala-2.12 clean package -DwildcardSuites=org.apache.spark.util.ClosureCleanerSuite -Dtest=None\n./build/mvn -Pscala-2.12 clean package -DwildcardSuites=org.apache.spark.streaming.DStreamClosureSuite -Dtest=None```\n\nAuthor: Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>\n\nCloses #21930 from skonto/scala2.12-sup.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/a65736996b2b506f61cc8d599ec9f4c52a1b5312"},{"sha":"46110a589f4e91cd7605c5a2c34c3db6b2635830","author":{"name":"Xiao Li","email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com"},"message":"[SPARK-24865][FOLLOW-UP] Remove AnalysisBarrier LogicalPlan Node\n\n## What changes were proposed in this pull request?\nRemove the AnalysisBarrier LogicalPlan node, which is useless now.\n\n## How was this patch tested?\nN/A\n\nAuthor: Xiao Li <gatorsmile@gmail.com>\n\nCloses #21962 from gatorsmile/refactor2.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/46110a589f4e91cd7605c5a2c34c3db6b2635830"},{"sha":"7be6fc3c77b00f0eefd276676524ec4e36bab868","author":{"name":"Kaya Kupferschmidt","email":"6c5bd875e714881318aa2ad33797e3013d39b601@dimajix.de"},"message":"[SPARK-24742] Fix NullPointerexception in Field Metadata\n\n## What changes were proposed in this pull request?\n\nThis pull request provides a fix for SPARK-24742: SQL Field MetaData was throwing an Exception in the hashCode method when a \"null\" Metadata was added via \"putNull\"\n\n## How was this patch tested?\n\nA new unittest is provided in org/apache/spark/sql/types/MetadataSuite.scala\n\nAuthor: Kaya Kupferschmidt <k.kupferschmidt@dimajix.de>\n\nCloses #21722 from kupferk/SPARK-24742.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/7be6fc3c77b00f0eefd276676524ec4e36bab868"},{"sha":"d182b3d34d6afade401b8a455b774059bae9d90f","author":{"name":"Kaya Kupferschmidt","email":"6c5bd875e714881318aa2ad33797e3013d39b601@dimajix.de"},"message":"[SPARK-24742] Fix NullPointerexception in Field Metadata\n\n## What changes were proposed in this pull request?\n\nThis pull request provides a fix for SPARK-24742: SQL Field MetaData was throwing an Exception in the hashCode method when a \"null\" Metadata was added via \"putNull\"\n\n## How was this patch tested?\n\nA new unittest is provided in org/apache/spark/sql/types/MetadataSuite.scala\n\nAuthor: Kaya Kupferschmidt <k.kupferschmidt@dimajix.de>\n\nCloses #21722 from kupferk/SPARK-24742.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/d182b3d34d6afade401b8a455b774059bae9d90f"},{"sha":"f04cd670943d0eb6eb688a0f50d56293cda554ef","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[MINOR] remove dead code in ExpressionEvalHelper\n\n## What changes were proposed in this pull request?\n\nThis addresses https://github.com/apache/spark/pull/21236/files#r207078480\n\nboth https://github.com/apache/spark/pull/21236 and https://github.com/apache/spark/pull/21838 add a InternalRow result check to ExpressionEvalHelper and becomes duplicated.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #21958 from cloud-fan/minor.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/f04cd670943d0eb6eb688a0f50d56293cda554ef"},{"sha":"15fc2372269159ea2556b028d4eb8860c4108650","author":{"name":"LucaCanali","email":"43b950174ad04a5c21e0ea03146906daec6b8d7f@cern.ch"},"message":"Updates to Accumulators","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/15fc2372269159ea2556b028d4eb8860c4108650"},{"sha":"ad2e63662885b67b1e94030b13fdae4f7366dc4a","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-24598][DOCS] State in the documentation the behavior when arithmetic operations cause overflow\n\n## What changes were proposed in this pull request?\n\nAccording to the discussion in https://github.com/apache/spark/pull/21599, changing the behavior of arithmetic operations so that they can check for overflow is not nice in a minor release. What we can do for 2.4 is warn users about the current behavior in the documentation, so that they are aware of the issue and can take proper actions.\n\n## How was this patch tested?\n\nNA\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #21967 from mgaido91/SPARK-24598_doc.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/ad2e63662885b67b1e94030b13fdae4f7366dc4a"},{"sha":"38e4699c978e56a0f24b8efb94fd3206cdd8b3fe","author":{"name":"Xingbo Jiang","email":"19a442ba0283f6f4c465cc281093aec56203af87@databricks.com"},"message":"[SPARK-24820][SPARK-24821][CORE] Fail fast when submitted job contains a barrier stage with unsupported RDD chain pattern\n\n## What changes were proposed in this pull request?\n\nCheck on job submit to make sure we don't launch a barrier stage with unsupported RDD chain pattern. The following patterns are not supported:\n- Ancestor RDDs that have different number of partitions from the resulting RDD (eg. union()/coalesce()/first()/PartitionPruningRDD);\n- An RDD that depends on multiple barrier RDDs (eg. barrierRdd1.zip(barrierRdd2)).\n\n## How was this patch tested?\n\nAdd test cases in `BarrierStageOnSubmittedSuite`.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #21927 from jiangxb1987/SPARK-24820.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/38e4699c978e56a0f24b8efb94fd3206cdd8b3fe"},{"sha":"0df6bf882907d7d76572f513168a144067d0e0ec","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[BUILD] Fix lint-python.\n\n## What changes were proposed in this pull request?\n\nThis pr fixes lint-python.\n\n```\n./python/pyspark/accumulators.py:231:9: E306 expected 1 blank line before a nested definition, found 0\n./python/pyspark/accumulators.py:257:101: E501 line too long (107 > 100 characters)\n./python/pyspark/accumulators.py:264:1: E302 expected 2 blank lines, found 1\n./python/pyspark/accumulators.py:281:1: E302 expected 2 blank lines, found 1\n```\n\n## How was this patch tested?\n\nExecuted lint-python manually.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #21973 from ueshin/issues/build/1/fix_lint-python.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/0df6bf882907d7d76572f513168a144067d0e0ec"},{"sha":"02f967795b7e8ccf2738d567928e47c38c1134e1","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-23908][SQL] Add transform function.\n\n## What changes were proposed in this pull request?\n\nThis pr adds `transform` function which transforms elements in an array using the function.\nOptionally we can take the index of each element as the second argument.\n\n```sql\n> SELECT transform(array(1, 2, 3), x -> x + 1);\n array(2, 3, 4)\n> SELECT transform(array(1, 2, 3), (x, i) -> x + i);\n array(1, 3, 5)\n```\n\n## How was this patch tested?\n\nAdded tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #21954 from ueshin/issues/SPARK-23908/transform.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/02f967795b7e8ccf2738d567928e47c38c1134e1"},{"sha":"efef55388fedef3f7954a385776e666ad4597a58","author":{"name":"Takeshi Yamamuro","email":"9770f4f4b25a32d3dfa34670a10767defeb974f3@apache.org"},"message":"[SPARK-24705][SQL] ExchangeCoordinator broken when duplicate exchanges reused\n\n## What changes were proposed in this pull request?\nIn the current master, `EnsureRequirements` sets the number of exchanges in `ExchangeCoordinator` before `ReuseExchange`. Then, `ReuseExchange` removes some duplicate exchange and the actual number of registered exchanges changes. Finally, the assertion in `ExchangeCoordinator` fails because the logical number of exchanges and the actual number of registered exchanges become different;\nhttps://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ExchangeCoordinator.scala#L201\n\nThis pr fixed the issue and the code to reproduce this is as follows;\n```\nscala> sql(\"SET spark.sql.adaptive.enabled=true\")\nscala> sql(\"SET spark.sql.autoBroadcastJoinThreshold=-1\")\nscala> val df = spark.range(1).selectExpr(\"id AS key\", \"id AS value\")\nscala> val resultDf = df.join(df, \"key\").join(df, \"key\")\nscala> resultDf.show\n...\n  at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:119)\n  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)\n  ... 101 more\nCaused by: java.lang.AssertionError: assertion failed\n  at scala.Predef$.assert(Predef.scala:156)\n  at org.apache.spark.sql.execution.exchange.ExchangeCoordinator.doEstimationIfNecessary(ExchangeCoordinator.scala:201)\n  at org.apache.spark.sql.execution.exchange.ExchangeCoordinator.postShuffleRDD(ExchangeCoordinator.scala:259)\n  at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:124)\n  at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:119)\n  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)\n...\n```\n\n## How was this patch tested?\nAdded tests in `ExchangeCoordinatorSuite`.\n\nAuthor: Takeshi Yamamuro <yamamuro@apache.org>\n\nCloses #21754 from maropu/SPARK-24705-2.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/efef55388fedef3f7954a385776e666ad4597a58"},{"sha":"d0bc3ed6797e0c06f688b7b2ef6c26282a25b175","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-24896][SQL] Uuid should produce different values for each execution in streaming query\n\n## What changes were proposed in this pull request?\n\n`Uuid`'s results depend on random seed given during analysis. Thus under streaming query, we will have the same uuids in each execution. This seems to be incorrect for streaming query execution.\n\n## How was this patch tested?\n\nAdded test.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #21854 from viirya/uuid_in_streaming.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/d0bc3ed6797e0c06f688b7b2ef6c26282a25b175"},{"sha":"bbdcc3bf61da39704650d4570c6307b5a46f7100","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-22219][SQL] Refactor code to get a value for \"spark.sql.codegen.comments\"\n\n## What changes were proposed in this pull request?\n\nThis PR refactors code to get a value for \"spark.sql.codegen.comments\" by avoiding `SparkEnv.get.conf`. This PR uses `SQLConf.get.codegenComments` since `SQLConf.get` always returns an instance of `SQLConf`.\n\n## How was this patch tested?\n\nAdded test case to `DebuggingSuite`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #19449 from kiszk/SPARK-22219.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/bbdcc3bf61da39704650d4570c6307b5a46f7100"},{"sha":"29077a1d15e49dfafe7f2eab963830ba9cc6b29a","author":{"name":"Xingbo Jiang","email":"19a442ba0283f6f4c465cc281093aec56203af87@databricks.com"},"message":"[SPARK-24795][CORE][FOLLOWUP] Combine BarrierTaskContext with BarrierTaskContextImpl\n\n## What changes were proposed in this pull request?\n\nAccording to https://github.com/apache/spark/pull/21758#discussion_r206746905 , current declaration of `BarrierTaskContext` didn't extend methods from `TaskContext`. Since `TaskContext` is an abstract class and we don't want to change it to a trait, we have to define class `BarrierTaskContext` directly.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #21972 from jiangxb1987/BarrierTaskContext.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/29077a1d15e49dfafe7f2eab963830ba9cc6b29a"},{"sha":"7cf16a7fa4eb4145c0c5d1dd2555f78a2fdd8d8b","author":{"name":"Gengliang Wang","email":"bfe1a7b36425bd89859a70a36586361d28ec35eb@databricks.com"},"message":"[SPARK-24773] Avro: support logical timestamp type with different precisions\n\n## What changes were proposed in this pull request?\n\nSupport reading/writing Avro logical timestamp type with different precisions\nhttps://avro.apache.org/docs/1.8.2/spec.html#Timestamp+%28millisecond+precision%29\n\nTo specify the output timestamp type, use Dataframe option `outputTimestampType`  or SQL config `spark.sql.avro.outputTimestampType`.  The supported values are\n* `TIMESTAMP_MICROS`\n* `TIMESTAMP_MILLIS`\n\nThe default output type is `TIMESTAMP_MICROS`\n## How was this patch tested?\n\nUnit test\n\nAuthor: Gengliang Wang <gengliang.wang@databricks.com>\n\nCloses #21935 from gengliangwang/avro_timestamp.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/7cf16a7fa4eb4145c0c5d1dd2555f78a2fdd8d8b"},{"sha":"b3f2911eebeb418631ce296f68a7cc68083659cd","author":{"name":"Maxim Gekk","email":"05b3389ec46e7c78a10351def567499e4feebe3f@databricks.com"},"message":"[SPARK-24945][SQL] Switching to uniVocity 2.7.3\n\n## What changes were proposed in this pull request?\n\nIn the PR, I propose to upgrade uniVocity parser from **2.6.3** to **2.7.3**. The recent version includes a fix for the SPARK-24645 issue and has better performance.\n\nBefore changes:\n```\nParsing quoted values:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nOne quoted string                           33336 / 34122          0.0      666727.0       1.0X\n\nWide rows with 1000 columns:             Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nSelect 1000 columns                         90287 / 91713          0.0       90286.9       1.0X\nSelect 100 columns                          31826 / 36589          0.0       31826.4       2.8X\nSelect one column                           25738 / 25872          0.0       25737.9       3.5X\ncount()                                       6931 / 7269          0.1        6931.5      13.0X\n```\nafter:\n```\nParsing quoted values:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nOne quoted string                           33411 / 33510          0.0      668211.4       1.0X\n\nWide rows with 1000 columns:             Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nSelect 1000 columns                         88028 / 89311          0.0       88028.1       1.0X\nSelect 100 columns                          29010 / 32755          0.0       29010.1       3.0X\nSelect one column                           22936 / 22953          0.0       22936.5       3.8X\ncount()                                       6657 / 6740          0.2        6656.6      13.5X\n```\nCloses #21892\n\n## How was this patch tested?\n\nIt was tested by `CSVSuite` and `CSVBenchmarks`\n\nAuthor: Maxim Gekk <maxim.gekk@databricks.com>\n\nCloses #21969 from MaxGekk/univocity-2_7_3.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/b3f2911eebeb418631ce296f68a7cc68083659cd"},{"sha":"73dd6cf9b558f9d752e1f3c13584344257ad7863","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-24966][SQL] Implement precedence rules for set operations.\n\n## What changes were proposed in this pull request?\n\nCurrently the set operations INTERSECT, UNION and EXCEPT are assigned the same precedence. This PR fixes the problem by giving INTERSECT  higher precedence than UNION and EXCEPT. UNION and EXCEPT operators are evaluated in the order in which they appear in the query from left to right.\n\nThis results in change in behavior because of the change in order of evaluations of set operators in a query. The old behavior is still preserved under a newly added config parameter.\n\nQuery `:`\n```\nSELECT * FROM t1\nUNION\nSELECT * FROM t2\nEXCEPT\nSELECT * FROM t3\nINTERSECT\nSELECT * FROM t4\n```\nParsed plan before the change `:`\n```\n== Parsed Logical Plan ==\n'Intersect false\n:- 'Except false\n:  :- 'Distinct\n:  :  +- 'Union\n:  :     :- 'Project [*]\n:  :     :  +- 'UnresolvedRelation `t1`\n:  :     +- 'Project [*]\n:  :        +- 'UnresolvedRelation `t2`\n:  +- 'Project [*]\n:     +- 'UnresolvedRelation `t3`\n+- 'Project [*]\n   +- 'UnresolvedRelation `t4`\n```\nParsed plan after the change `:`\n```\n== Parsed Logical Plan ==\n'Except false\n:- 'Distinct\n:  +- 'Union\n:     :- 'Project [*]\n:     :  +- 'UnresolvedRelation `t1`\n:     +- 'Project [*]\n:        +- 'UnresolvedRelation `t2`\n+- 'Intersect false\n   :- 'Project [*]\n   :  +- 'UnresolvedRelation `t3`\n   +- 'Project [*]\n      +- 'UnresolvedRelation `t4`\n```\n## How was this patch tested?\nAdded tests in PlanParserSuite, SQLQueryTestSuite.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #21941 from dilipbiswal/SPARK-24966.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/73dd6cf9b558f9d752e1f3c13584344257ad7863"},{"sha":"f45d60a5a1f1e97ecde36eda8202034d78f93d53","author":{"name":"Gengliang Wang","email":"bfe1a7b36425bd89859a70a36586361d28ec35eb@databricks.com"},"message":"[SPARK-25002][SQL] Avro: revise the output record namespace\n\n## What changes were proposed in this pull request?\n\nCurrently the output namespace is starting with \".\", e.g. `.topLevelRecord`\n\nAlthough it is valid according to Avro spec, we should remove the starting dot in case of failures when the output Avro file is read by other lib:\n\nhttps://github.com/linkedin/goavro/pull/96\n\n## How was this patch tested?\n\nUnit test\n\nAuthor: Gengliang Wang <gengliang.wang@databricks.com>\n\nCloses #21974 from gengliangwang/avro_namespace.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/f45d60a5a1f1e97ecde36eda8202034d78f93d53"},{"sha":"b0d6967d45f3260ed4ee9b2a49f801d799e81283","author":{"name":"Chris Horn","email":"8ff709064bcba627309592de5076f454fa636465@gmail.com"},"message":"[SPARK-24788][SQL] RelationalGroupedDataset.toString with unresolved exprs should not fail\n\n## What changes were proposed in this pull request?\nIn the current master, `toString` throws an exception when `RelationalGroupedDataset` has unresolved expressions;\n```\nscala> spark.range(0, 10).groupBy(\"id\")\nres4: org.apache.spark.sql.RelationalGroupedDataset = RelationalGroupedDataset: [grouping expressions: [id: bigint], value: [id: bigint], type: GroupBy]\n\nscala> spark.range(0, 10).groupBy('id)\norg.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'id\n  at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:105)\n  at org.apache.spark.sql.RelationalGroupedDataset$$anonfun$12.apply(RelationalGroupedDataset.scala:474)\n  at org.apache.spark.sql.RelationalGroupedDataset$$anonfun$12.apply(RelationalGroupedDataset.scala:473)\n  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n  at scala.collection.AbstractTraversable.map(Traversable.scala:104)\n  at org.apache.spark.sql.RelationalGroupedDataset.toString(RelationalGroupedDataset.scala:473)\n  at scala.runtime.ScalaRunTime$.scala$runtime$ScalaRunTime$$inner$1(ScalaRunTime.scala:332)\n  at scala.runtime.ScalaRunTime$.stringOf(ScalaRunTime.scala:337)\n  at scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345)\n```\nThis pr fixed code to handle the unresolved case in `RelationalGroupedDataset.toString`.\n\nCloses #21752\n\n## How was this patch tested?\nAdded tests in `DataFrameAggregateSuite`.\n\nAuthor: Chris Horn <chorn4033@gmail.com>\nAuthor: Takeshi Yamamuro <yamamuro@apache.org>\n\nCloses #21964 from maropu/SPARK-24788.","distinct":false,"url":"https://api.github.com/repos/heary-cao/spark/commits/b0d6967d45f3260ed4ee9b2a49f801d799e81283"}]}
{"actor":{"display_login":"heary-cao"}}