{"push_id":1276768729,"size":25,"distinct_size":25,"ref":"refs/heads/master","head":"5d84c7fd83502aeb551d46a740502db4862508fe","before":"095862a3cff73fd88db9ed37a63e7629e664ff64","commits":[{"sha":"1a48c0047bbdb6328c3ac5ec617a5e35e244d66d","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[BUILD] Closes some stale PRs.\n\n## What changes were proposed in this pull request?\n\nCloses #10995\nCloses #13658\nCloses #14505\nCloses #14536\nCloses #12753\nCloses #14449\nCloses #12694\nCloses #12695\nCloses #14810\nCloses #10572\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14849 from srowen/CloseStalePRs.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/1a48c0047bbdb6328c3ac5ec617a5e35e244d66d"},{"sha":"08913ce0002a80a989489a31b7353f5ec4a5849f","author":{"name":"Seigneurin, Alexis (CONT)","email":"b618d49b43b8b6901714bdc90f6846079c125b45@capitalone.com"},"message":"fixed a typo\n\nidempotant -> idempotent\n\nAuthor: Seigneurin, Alexis (CONT) <Alexis.Seigneurin@capitalone.com>\n\nCloses #14833 from aseigneurin/fix-typo.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/08913ce0002a80a989489a31b7353f5ec4a5849f"},{"sha":"6a0fda2c0590b455e8713da79cd5f2413e5d0f28","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARKR][MINOR] Fix LDA doc\n\n## What changes were proposed in this pull request?\n\nThis PR tries to fix the name of the `SparkDataFrame` used in the example. Also, it gives a reference url of an example data file so that users can play with.\n\n## How was this patch tested?\n\nManual test.\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #14853 from junyangq/SPARKR-FixLDADoc.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/6a0fda2c0590b455e8713da79cd5f2413e5d0f28"},{"sha":"48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-17063] [SQL] Improve performance of MSCK REPAIR TABLE with Hive metastore\n\n## What changes were proposed in this pull request?\n\nThis PR split the the single `createPartitions()` call into smaller batches, which could prevent Hive metastore from OOM (caused by millions of partitions).\n\nIt will also try to gather all the fast stats (number of files and total size of all files) in parallel to avoid the bottle neck of listing the files in metastore sequential, which is controlled by spark.sql.gatherFastStats (enabled by default).\n\n## How was this patch tested?\n\nTested locally with 10000 partitions and 100 files with embedded metastore, without gathering fast stats in parallel, adding partitions took 153 seconds, after enable that, gathering the fast stats took about 34 seconds, adding these partitions took 25 seconds (most of the time spent in object store), 59 seconds in total, 2.5X faster (with larger cluster, gathering will much faster).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #14607 from davies/repair_batch.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd"},{"sha":"736a7911cb0335cdb2b2f6c87f9e3c32047b5bbb","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"[SPARK-16581][SPARKR] Make JVM backend calling functions public\n\n## What changes were proposed in this pull request?\n\nThis change exposes a public API in SparkR to create objects, call methods on the Spark driver JVM\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\nUnit tests, CRAN checks\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #14775 from shivaram/sparkr-java-api.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/736a7911cb0335cdb2b2f6c87f9e3c32047b5bbb"},{"sha":"48b459ddd58affd5519856cb6e204398b7739a2a","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17301][SQL] Remove unused classTag field from AtomicType base class\n\nThere's an unused `classTag` val in the AtomicType base class which is causing unnecessary slowness in deserialization because it needs to grab ScalaReflectionLock and create a new runtime reflection mirror. Removing this unused code gives a small but measurable performance boost in SQL task deserialization.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14869 from JoshRosen/remove-unused-classtag.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/48b459ddd58affd5519856cb6e204398b7739a2a"},{"sha":"8fb445d9bdead6f0ff2bd9879145fe688b3bdc80","author":{"name":"frreiss","email":"b2f409dba498dd996d8771bf1c667af870613edc@us.ibm.com"},"message":"[SPARK-17303] Added spark-warehouse to dev/.rat-excludes\n\n## What changes were proposed in this pull request?\n\nExcludes the `spark-warehouse` directory from the Apache RAT checks that src/run-tests performs. `spark-warehouse` is created by some of the Spark SQL tests, as well as by `bin/spark-sql`.\n\n## How was this patch tested?\n\nRan src/run-tests twice. The second time, the script failed because the first iteration\nMade the change in this PR.\nRan src/run-tests a third time; RAT checks succeeded.\n\nAuthor: frreiss <frreiss@us.ibm.com>\n\nCloses #14870 from frreiss/fred-17303.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/8fb445d9bdead6f0ff2bd9879145fe688b3bdc80"},{"sha":"94922d79e9f90fac3777db0974ccf7566b8ac3b3","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-17289][SQL] Fix a bug to satisfy sort requirements in partial aggregations\n\n## What changes were proposed in this pull request?\nPartial aggregations are generated in `EnsureRequirements`, but the planner fails to\ncheck if partial aggregation satisfies sort requirements.\nFor the following query:\n```\nval df2 = (0 to 1000).map(x => (x % 2, x.toString)).toDF(\"a\", \"b\").createOrReplaceTempView(\"t2\")\nspark.sql(\"select max(b) from t2 group by a\").explain(true)\n```\nNow, the SortAggregator won't insert Sort operator before partial aggregation, this will break sort-based partial aggregation.\n```\n== Physical Plan ==\nSortAggregate(key=[a#5], functions=[max(b#6)], output=[max(b)#17])\n+- *Sort [a#5 ASC], false, 0\n   +- Exchange hashpartitioning(a#5, 200)\n      +- SortAggregate(key=[a#5], functions=[partial_max(b#6)], output=[a#5, max#19])\n         +- LocalTableScan [a#5, b#6]\n```\nActually, a correct plan is:\n```\n== Physical Plan ==\nSortAggregate(key=[a#5], functions=[max(b#6)], output=[max(b)#17])\n+- *Sort [a#5 ASC], false, 0\n   +- Exchange hashpartitioning(a#5, 200)\n      +- SortAggregate(key=[a#5], functions=[partial_max(b#6)], output=[a#5, max#19])\n         +- *Sort [a#5 ASC], false, 0\n            +- LocalTableScan [a#5, b#6]\n```\n\n## How was this patch tested?\nAdded tests in `PlannerSuite`.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #14865 from maropu/SPARK-17289.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/94922d79e9f90fac3777db0974ccf7566b8ac3b3"},{"sha":"bca79c823024c41731ec89f96a3722d7b1c99639","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-17234][SQL] Table Existence Checking when Index Table with the Same Name Exists\n\n### What changes were proposed in this pull request?\nHive Index tables are not supported by Spark SQL. Thus, we issue an exception when users try to access Hive Index tables. When the internal function `tableExists` tries to access Hive Index tables, it always gets the same error message: ```Hive index table is not supported```. This message could be confusing to users, since their SQL operations could be completely unrelated to Hive Index tables. For example, when users try to alter a table to a new name and there exists an index table with the same name, the expected exception should be a `TableAlreadyExistsException`.\n\nThis PR made the following changes:\n- Introduced a new `AnalysisException` type: `SQLFeatureNotSupportedException`. When users try to access an `Index Table`, we will issue a `SQLFeatureNotSupportedException`.\n- `tableExists` returns `true` when hitting a `SQLFeatureNotSupportedException` and the feature is `Hive index table`.\n- Add a checking `requireTableNotExists` for `SessionCatalog`'s `createTable` API; otherwise, the current implementation relies on the Hive's internal checking.\n\n### How was this patch tested?\nAdded a test case\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14801 from gatorsmile/tableExists.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/bca79c823024c41731ec89f96a3722d7b1c99639"},{"sha":"2d76cb11f51b20a57af7bdefb7a74fdfe41060aa","author":{"name":"Xin Ren","email":"82e6f9780faa35cac922f524c8a765f69b3ff7ff@126.com"},"message":"[SPARK-17276][CORE][TEST] Stop env params output on Jenkins job page\n\nhttps://issues.apache.org/jira/browse/SPARK-17276\n\n## What changes were proposed in this pull request?\n\nWhen trying to find error msg in a failed Jenkins build job, I'm annoyed by the huge env output.\nThe env parameter output should be muted.\n\n![screen shot 2016-08-26 at 10 52 07 pm](https://cloud.githubusercontent.com/assets/3925641/18025581/b8d567ba-6be2-11e6-9eeb-6aec223f1730.png)\n\n## How was this patch tested?\n\nTested manually on local laptop.\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #14848 from keypointt/SPARK-17276.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/2d76cb11f51b20a57af7bdefb7a74fdfe41060aa"},{"sha":"befab9c1c6b59ad90f63a7d10e12b186be897f15","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-17264][SQL] DataStreamWriter should document that it only supports Parquet for now\n\n## What changes were proposed in this pull request?\n\nClarify that only parquet files are supported by DataStreamWriter now\n\n## How was this patch tested?\n\n(Doc build -- no functional changes to test)\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14860 from srowen/SPARK-17264.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/befab9c1c6b59ad90f63a7d10e12b186be897f15"},{"sha":"d4eee9932edf1a489d7fe9120a0f003150834df6","author":{"name":"Dmitriy Sokolov","email":"6f206a163f4f72fa84b3bc432f138f20f1610280@gmail.com"},"message":"[MINOR][DOCS] Fix minor typos in python example code\n\n## What changes were proposed in this pull request?\n\nFix minor typos python example code in streaming programming guide\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Dmitriy Sokolov <silentsokolov@gmail.com>\n\nCloses #14805 from silentsokolov/fix-typos.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/d4eee9932edf1a489d7fe9120a0f003150834df6"},{"sha":"27209252f09ff73c58e60c6df8aaba73b308088c","author":{"name":"Xin Ren","email":"82e6f9780faa35cac922f524c8a765f69b3ff7ff@126.com"},"message":"[MINOR][MLLIB][SQL] Clean up unused variables and unused import\n\n## What changes were proposed in this pull request?\n\nClean up unused variables and unused import statements, unnecessary `return` and `toArray`, and some more style improvement,  when I walk through the code examples.\n\n## How was this patch tested?\n\nTestet manually on local laptop.\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #14836 from keypointt/codeWalkThroughML.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/27209252f09ff73c58e60c6df8aaba73b308088c"},{"sha":"4b4e329e49f8af28fa6301bd06c48d7097eaf9e6","author":{"name":"Ferdinand Xu","email":"002ddcadfd85b96aefbf8a3ec68ff6c631925754@intel.com"},"message":"[SPARK-5682][CORE] Add encrypted shuffle in spark\n\nThis patch is using Apache Commons Crypto library to enable shuffle encryption support.\n\nAuthor: Ferdinand Xu <cheng.a.xu@intel.com>\nAuthor: kellyzly <kellyzly@126.com>\n\nCloses #8880 from winningsix/SPARK-10771.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/4b4e329e49f8af28fa6301bd06c48d7097eaf9e6"},{"sha":"fb20084313470593d8507a43fcb2cde2a4c854d9","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17304] Fix perf. issue caused by TaskSetManager.abortIfCompletelyBlacklisted\n\nThis patch addresses a minor scheduler performance issue that was introduced in #13603. If you run\n\n```\nsc.parallelize(1 to 100000, 100000).map(identity).count()\n```\n\nthen most of the time ends up being spent in `TaskSetManager.abortIfCompletelyBlacklisted()`:\n\n![image](https://cloud.githubusercontent.com/assets/50748/18071032/428732b0-6e07-11e6-88b2-c9423cd61f53.png)\n\nWhen processing resource offers, the scheduler uses a nested loop which considers every task set at multiple locality levels:\n\n```scala\n   for (taskSet <- sortedTaskSets; maxLocality <- taskSet.myLocalityLevels) {\n      do {\n        launchedTask = resourceOfferSingleTaskSet(\n            taskSet, maxLocality, shuffledOffers, availableCpus, tasks)\n      } while (launchedTask)\n    }\n```\n\nIn order to prevent jobs with globally blacklisted tasks from hanging, #13603 added a `taskSet.abortIfCompletelyBlacklisted` call inside of  `resourceOfferSingleTaskSet`; if a call to `resourceOfferSingleTaskSet` fails to schedule any tasks, then `abortIfCompletelyBlacklisted` checks whether the tasks are completely blacklisted in order to figure out whether they will ever be schedulable. The problem with this placement of the call is that the last call to `resourceOfferSingleTaskSet` in the `while` loop will return `false`, implying that  `resourceOfferSingleTaskSet` will call `abortIfCompletelyBlacklisted`, so almost every call to `resourceOffers` will trigger the `abortIfCompletelyBlacklisted` check for every task set.\n\nInstead, I think that this call should be moved out of the innermost loop and should be called _at most_ once per task set in case none of the task set's tasks can be scheduled at any locality level.\n\nBefore this patch's changes, the microbenchmark example that I posted above took 35 seconds to run, but it now only takes 15 seconds after this change.\n\n/cc squito and kayousterhout for review.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14871 from JoshRosen/bail-early-if-no-cpus.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/fb20084313470593d8507a43fcb2cde2a4c854d9"},{"sha":"02ac379e8645ce5d32e033f6683136da16fbe584","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17314][CORE] Use Netty's DefaultThreadFactory to enable its fast ThreadLocal impl\n\n## What changes were proposed in this pull request?\n\nWhen a thread is a Netty's FastThreadLocalThread, Netty will use its fast ThreadLocal implementation. It has a better performance than JDK's (See the benchmark results in https://github.com/netty/netty/pull/4417, note: it's not a fix to Netty's FastThreadLocal. It just fixed an issue in Netty's benchmark codes)\n\nThis PR just changed the ThreadFactory to Netty's DefaultThreadFactory which will use FastThreadLocalThread. There is also a minor change to the thread names. See https://github.com/netty/netty/blob/netty-4.0.22.Final/common/src/main/java/io/netty/util/concurrent/DefaultThreadFactory.java#L94\n\n## How was this patch tested?\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14879 from zsxwing/netty-thread.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/02ac379e8645ce5d32e033f6683136da16fbe584"},{"sha":"f7beae6da02e6b70a34c714e93136becbde7679b","author":{"name":"Alex Bozarth","email":"850363088eed533ef66882756a87f03f1fc98fde@us.ibm.com"},"message":"[SPARK-17243][WEB UI] Spark 2.0 History Server won't load with very large application history\n\n## What changes were proposed in this pull request?\n\nWith the new History Server the summary page loads the application list via the the REST API, this makes it very slow to impossible to load with large (10K+) application history. This pr fixes this by adding the `spark.history.ui.maxApplications` conf to limit the number of applications the History Server displays. This is accomplished using a new optional `limit` param for the `applications` api. (Note this only applies to what the summary page displays, all the Application UI's are still accessible if the user knows the App ID and goes to the Application UI directly.)\n\nI've also added a new test for the `limit` param in `HistoryServerSuite.scala`\n\n## How was this patch tested?\n\nManual testing and dev/run-tests\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #14835 from ajbozarth/spark17243.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/f7beae6da02e6b70a34c714e93136becbde7679b"},{"sha":"231f973295129dca976f2e4a8222a63318d4aafe","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17318][TESTS] Fix ReplSuite replicating blocks of object with class defined in repl\n\n## What changes were proposed in this pull request?\n\nThere are a lot of failures recently: http://spark-tests.appspot.com/tests/org.apache.spark.repl.ReplSuite/replicating%20blocks%20of%20object%20with%20class%20defined%20in%20repl\n\nThis PR just changed the persist level to `MEMORY_AND_DISK_2` to avoid blocks being evicted from memory.\n\n## How was this patch tested?\n\nJenkins unit tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14884 from zsxwing/SPARK-17318.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/231f973295129dca976f2e4a8222a63318d4aafe"},{"sha":"d92cd227cf245be9ab8f9bce714386f8283a97cb","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-15985][SQL] Eliminate redundant cast from an array without null or a map without null\n\n## What changes were proposed in this pull request?\n\nThis PR eliminates redundant cast from an `ArrayType` with `containsNull = false` or a `MapType` with `containsNull = false`.\n\nFor example, in `ArrayType` case, current implementation leaves a cast `cast(value#63 as array<double>).toDoubleArray`. However, we can eliminate `cast(value#63 as array<double>)` if we know `value#63` does not include `null`. This PR apply this elimination for `ArrayType` and `MapType` in `SimplifyCasts` at a plan optimization phase.\n\nIn summary, we got 1.2-1.3x performance improvements over the code before applying this PR.\nHere are performance results of benchmark programs:\n```\n  test(\"Read array in Dataset\") {\n    import sparkSession.implicits._\n\n    val iters = 5\n    val n = 1024 * 1024\n    val rows = 15\n\n    val benchmark = new Benchmark(\"Read primnitive array\", n)\n\n    val rand = new Random(511)\n    val intDS = sparkSession.sparkContext.parallelize(0 until rows, 1)\n      .map(i => Array.tabulate(n)(i => i)).toDS()\n    intDS.count() // force to create ds\n    val lastElement = n - 1\n    val randElement = rand.nextInt(lastElement)\n\n    benchmark.addCase(s\"Read int array in Dataset\", numIters = iters)(iter => {\n      val idx0 = randElement\n      val idx1 = lastElement\n      intDS.map(a => a(0) + a(idx0) + a(idx1)).collect\n    })\n\n    val doubleDS = sparkSession.sparkContext.parallelize(0 until rows, 1)\n      .map(i => Array.tabulate(n)(i => i.toDouble)).toDS()\n    doubleDS.count() // force to create ds\n\n    benchmark.addCase(s\"Read double array in Dataset\", numIters = iters)(iter => {\n      val idx0 = randElement\n      val idx1 = lastElement\n      doubleDS.map(a => a(0) + a(idx0) + a(idx1)).collect\n    })\n\n    benchmark.run()\n  }\n\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.10.4\nIntel(R) Core(TM) i5-5257U CPU  2.70GHz\n\nwithout this PR\nRead primnitive array:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nRead int array in Dataset                      525 /  690          2.0         500.9       1.0X\nRead double array in Dataset                   947 / 1209          1.1         902.7       0.6X\n\nwith this PR\nRead primnitive array:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nRead int array in Dataset                      400 /  492          2.6         381.5       1.0X\nRead double array in Dataset                   788 /  870          1.3         751.4       0.5X\n```\n\nAn example program that originally caused this performance issue.\n```\nval ds = Seq(Array(1.0, 2.0, 3.0), Array(4.0, 5.0, 6.0)).toDS()\nval ds2 = ds.map(p => {\n     var s = 0.0\n     for (i <- 0 to 2) { s += p(i) }\n     s\n   })\nds2.show\nds2.explain(true)\n```\n\nPlans before this PR\n```\n== Parsed Logical Plan ==\n'SerializeFromObject [input[0, double, true] AS value#68]\n+- 'MapElements <function1>, obj#67: double\n   +- 'DeserializeToObject unresolveddeserializer(upcast(getcolumnbyordinal(0, ArrayType(DoubleType,false)), ArrayType(DoubleType,false), - root class: \"scala.Array\").toDoubleArray), obj#66: [D\n      +- LocalRelation [value#63]\n\n== Analyzed Logical Plan ==\nvalue: double\nSerializeFromObject [input[0, double, true] AS value#68]\n+- MapElements <function1>, obj#67: double\n   +- DeserializeToObject cast(value#63 as array<double>).toDoubleArray, obj#66: [D\n      +- LocalRelation [value#63]\n\n== Optimized Logical Plan ==\nSerializeFromObject [input[0, double, true] AS value#68]\n+- MapElements <function1>, obj#67: double\n   +- DeserializeToObject cast(value#63 as array<double>).toDoubleArray, obj#66: [D\n      +- LocalRelation [value#63]\n\n== Physical Plan ==\n*SerializeFromObject [input[0, double, true] AS value#68]\n+- *MapElements <function1>, obj#67: double\n   +- *DeserializeToObject cast(value#63 as array<double>).toDoubleArray, obj#66: [D\n      +- LocalTableScan [value#63]\n```\n\nPlans after this PR\n```\n== Parsed Logical Plan ==\n'SerializeFromObject [input[0, double, true] AS value#6]\n+- 'MapElements <function1>, obj#5: double\n   +- 'DeserializeToObject unresolveddeserializer(upcast(getcolumnbyordinal(0, ArrayType(DoubleType,false)), ArrayType(DoubleType,false), - root class: \"scala.Array\").toDoubleArray), obj#4: [D\n      +- LocalRelation [value#1]\n\n== Analyzed Logical Plan ==\nvalue: double\nSerializeFromObject [input[0, double, true] AS value#6]\n+- MapElements <function1>, obj#5: double\n   +- DeserializeToObject cast(value#1 as array<double>).toDoubleArray, obj#4: [D\n      +- LocalRelation [value#1]\n\n== Optimized Logical Plan ==\nSerializeFromObject [input[0, double, true] AS value#6]\n+- MapElements <function1>, obj#5: double\n   +- DeserializeToObject value#1.toDoubleArray, obj#4: [D\n      +- LocalRelation [value#1]\n\n== Physical Plan ==\n*SerializeFromObject [input[0, double, true] AS value#6]\n+- *MapElements <function1>, obj#5: double\n   +- *DeserializeToObject value#1.toDoubleArray, obj#4: [D\n      +- LocalTableScan [value#1]\n```\n\n## How was this patch tested?\n\nTested by new test cases in `SimplifyCastsSuite`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #13704 from kiszk/SPARK-15985.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/d92cd227cf245be9ab8f9bce714386f8283a97cb"},{"sha":"fa6347938fc1c72ddc03a5f3cd2e929b5694f0a6","author":{"name":"Jeff Zhang","email":"8540ccd5710ce8e6da3f6c1c8189a4ef2fa98c60@apache.org"},"message":"[SPARK-17178][SPARKR][SPARKSUBMIT] Allow to set sparkr shell command through --conf\n\n## What changes were proposed in this pull request?\n\nAllow user to set sparkr shell command through --conf spark.r.shell.command\n\n## How was this patch tested?\n\nUnit test is added and also verify it manually through\n```\nbin/sparkr --master yarn-client --conf spark.r.shell.command=/usr/local/bin/R\n```\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #14744 from zjffdu/SPARK-17178.","distinct":true,"url":"https://api.github.com/repos/keypointt/spark/commits/fa6347938fc1c72ddc03a5f3cd2e929b5694f0a6"}]}
{"actor":{"display_login":"keypointt"}}