{"push_id":1121954370,"size":2,"distinct_size":2,"ref":"refs/heads/master","head":"0e70fd61b4bc92bd744fc44dd3cbe91443207c72","before":"dfa61f7b136ae060bbe04e3c0da1148da41018c7","commits":[{"sha":"2ba3ff044900d16d5f6331523526f785864c1e62","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-10216][SQL] Revert \"[] Avoid creating empty files during overwrit…\n\nThis reverts commit 8d05a7a from #12855, which seems to have caused regressions when working with empty DataFrames.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #13181 from marmbrus/revert12855.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/2ba3ff044900d16d5f6331523526f785864c1e62"},{"sha":"0e70fd61b4bc92bd744fc44dd3cbe91443207c72","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-15438][SQL] improve explain of whole stage codegen\n\n## What changes were proposed in this pull request?\n\nCurrently, the explain of a query with whole-stage codegen looks like this\n```\n>>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [id#1L]\n:     +- BroadcastHashJoin [id#1L], [id#4L], Inner, BuildRight, None\n:        :- Range 0, 1, 4, 1000, [id#1L]\n:        +- INPUT\n+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint]))\n   +- WholeStageCodegen\n      :  +- Range 0, 1, 4, 1000, [id#4L]\n```\n\nThe problem is that the plan looks much different than logical plan, make us hard to understand the plan (especially when the logical plan is not showed together).\n\nThis PR will change it to:\n\n```\n>>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain()\n== Physical Plan ==\n*Project [id#0L]\n+- *BroadcastHashJoin [id#0L], [id#3L], Inner, BuildRight, None\n   :- *Range 0, 1, 4, 1000, [id#0L]\n   +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]))\n      +- *Range 0, 1, 4, 1000, [id#3L]\n```\n\nThe `*`before the plan means that it's part of whole-stage codegen, it's easy to understand.\n\n## How was this patch tested?\n\nManually ran some queries and check the explain.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13204 from davies/explain_codegen.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/0e70fd61b4bc92bd744fc44dd3cbe91443207c72"}]}
{"actor":{"display_login":"dongjoon-hyun"}}