{"push_id":980034953,"size":170,"distinct_size":170,"ref":"refs/heads/master","head":"5f37aad48cb729a80c4cc25347460f12aafec9fb","before":"e6a02c66d53f59ba2d5c1548494ae80a385f9f5c","commits":[{"sha":"dab246f7e4664d36073ec49d9df8a11c5e998cdb","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13098] [SQL] remove GenericInternalRowWithSchema\n\nThis class is only used for serialization of Python DataFrame. However, we don't require internal row there, so `GenericRowWithSchema` can also do the job.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10992 from cloud-fan/python.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/dab246f7e4664d36073ec49d9df8a11c5e998cdb"},{"sha":"289373b28cd2546165187de2e6a9185a1257b1e7","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-6363][BUILD] Make Scala 2.11 the default Scala version\n\nThis patch changes Spark's build to make Scala 2.11 the default Scala version. To be clear, this does not mean that Spark will stop supporting Scala 2.10: users will still be able to compile Spark for Scala 2.10 by following the instructions on the \"Building Spark\" page; however, it does mean that Scala 2.11 will be the default Scala version used by our CI builds (including pull request builds).\n\nThe Scala 2.11 compiler is faster than 2.10, so I think we'll be able to look forward to a slight speedup in our CI builds (it looks like it's about 2X faster for the Maven compile-only builds, for instance).\n\nAfter this patch is merged, I'll update Jenkins to add new compile-only jobs to ensure that Scala 2.10 compilation doesn't break.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10608 from JoshRosen/SPARK-6363.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/289373b28cd2546165187de2e6a9185a1257b1e7"},{"sha":"de283719980ae78b740e507e4d70c7ebbf6c5f74","author":{"email":"e28ad19e4e56395ff72e0397107366a06912a175@haizhi.com","name":"wangyang"},"message":"[SPARK-13100][SQL] improving the performance of stringToDate method in DateTimeUtils.scala\n\n In jdk1.7 TimeZone.getTimeZone() is synchronized, so use an instance variable to hold an GMT TimeZone object instead of instantiate it every time.\n\nAuthor: wangyang <wangyang@haizhi.com>\n\nCloses #10994 from wangyang1992/datetimeUtil.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/de283719980ae78b740e507e4d70c7ebbf6c5f74"},{"sha":"a1303de0a0e9d0c80327977abf52a79e2aa95e1f","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-13070][SQL] Better error message when Parquet schema merging fails\n\nMake sure we throw better error messages when Parquet schema merging fails.\n\nAuthor: Cheng Lian <lian@databricks.com>\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10979 from viirya/schema-merging-failure-message.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/a1303de0a0e9d0c80327977abf52a79e2aa95e1f"},{"sha":"0e6d92d042b0a2920d8df5959d5913ba0166a678","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12689][SQL] Migrate DDL parsing to the newly absorbed parser\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12689\n\nDDLParser processes three commands: createTable, describeTable and refreshTable.\nThis patch migrates the three commands to newly absorbed parser.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10723 from viirya/migrate-ddl-describe.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/0e6d92d042b0a2920d8df5959d5913ba0166a678"},{"sha":"5a8b978fabb60aa178274f86432c63680c8b351a","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-13049] Add First/last with ignore nulls to functions.scala\n\nThis PR adds the ability to specify the ```ignoreNulls``` option to the functions dsl, e.g:\n```df.select($\"id\", last($\"value\", ignoreNulls = true).over(Window.partitionBy($\"id\").orderBy($\"other\"))```\n\nThis PR is some where between a bug fix (see the JIRA) and a new feature. I am not sure if we should backport to 1.6.\n\ncc yhuai\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #10957 from hvanhovell/SPARK-13049.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/5a8b978fabb60aa178274f86432c63680c8b351a"},{"sha":"c1da4d421ab78772ffa52ad46e5bdfb4e5268f47","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13093] [SQL] improve null check in nullSafeCodeGen for unary, binary and ternary expression\n\nThe current implementation is sub-optimal:\n\n* If an expression is always nullable, e.g. `Unhex`, we can still remove null check for children if they are not nullable.\n* If an expression has some non-nullable children, we can still remove null check for these children and keep null check for others.\n\nThis PR improves this by making the null check elimination more fine-grained.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10987 from cloud-fan/null-check.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/c1da4d421ab78772ffa52ad46e5bdfb4e5268f47"},{"sha":"6075573a93176ee8c071888e4525043d9e73b061","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-6847][CORE][STREAMING] Fix stack overflow issue when updateStateByKey is followed by a checkpointed dstream\n\nAdd a local property to indicate if checkpointing all RDDs that are marked with the checkpoint flag, and enable it in Streaming\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10934 from zsxwing/recursive-checkpoint.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/6075573a93176ee8c071888e4525043d9e73b061"},{"sha":"33c8a490f7f64320c53530a57bd8d34916e3607c","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12989][SQL] Delaying Alias Cleanup after ExtractWindowExpressions\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12989\n\nIn the rule `ExtractWindowExpressions`, we simply replace alias by the corresponding attribute. However, this will cause an issue exposed by the following case:\n\n```scala\nval data = Seq((\"a\", \"b\", \"c\", 3), (\"c\", \"b\", \"a\", 3)).toDF(\"A\", \"B\", \"C\", \"num\")\n  .withColumn(\"Data\", struct(\"A\", \"B\", \"C\"))\n  .drop(\"A\")\n  .drop(\"B\")\n  .drop(\"C\")\n\nval winSpec = Window.partitionBy(\"Data.A\", \"Data.B\").orderBy($\"num\".desc)\ndata.select($\"*\", max(\"num\").over(winSpec) as \"max\").explain(true)\n```\nIn this case, both `Data.A` and `Data.B` are `alias` in `WindowSpecDefinition`. If we replace these alias expression by their alias names, we are unable to know what they are since they will not be put in `missingExpr` too.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\nAuthor: xiaoli <lixiao1983@gmail.com>\nAuthor: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>\n\nCloses #10963 from gatorsmile/seletStarAfterColDrop.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/33c8a490f7f64320c53530a57bd8d34916e3607c"},{"sha":"8f26eb5ef6853a6666d7d9481b333de70bc501ed","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12705][SPARK-10777][SQL] Analyzer Rule ResolveSortReferences\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12705\n\n**Scope:**\nThis PR is a general fix for sorting reference resolution when the child's `outputSet` does not have the order-by attributes (called, *missing attributes*):\n  - UnaryNode support is limited to `Project`, `Window`, `Aggregate`, `Distinct`, `Filter`, `RepartitionByExpression`.\n  - We will not try to resolve the missing references inside a subquery, unless the outputSet of this subquery contains it.\n\n**General Reference Resolution Rules:**\n  - Jump over the nodes with the following types: `Distinct`, `Filter`, `RepartitionByExpression`. Do not need to add missing attributes. The reason is their `outputSet` is decided by their `inputSet`, which is the `outputSet` of their children.\n  - Group-by expressions in `Aggregate`: missing order-by attributes are not allowed to be added into group-by expressions since it will change the query result. Thus, in RDBMS, it is not allowed.\n  - Aggregate expressions in `Aggregate`: if the group-by expressions in `Aggregate` contains the missing attributes but aggregate expressions do not have it, just add them into the aggregate expressions. This can resolve the analysisExceptions thrown by the three TCPDS queries.\n  - `Project` and `Window` are special. We just need to add the missing attributes to their `projectList`.\n\n**Implementation:**\n  1. Traverse the whole tree in a pre-order manner to find all the resolvable missing order-by attributes.\n  2. Traverse the whole tree in a post-order manner to add the found missing order-by attributes to the node if their `inputSet` contains the attributes.\n  3. If the origins of the missing order-by attributes are different nodes, each pass only resolves the missing attributes that are from the same node.\n\n**Risk:**\nLow. This rule will be trigger iff ```!s.resolved && child.resolved``` is true. Thus, very few cases are affected.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #10678 from gatorsmile/sortWindows.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/8f26eb5ef6853a6666d7d9481b333de70bc501ed"},{"sha":"da9146c91a33577ff81378ca7e7c38a4b1917876","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[DOCS] Fix the jar location of datanucleus in sql-programming-guid.md\n\nISTM `lib` is better because `datanucleus` jars are located in `lib` for release builds.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #10901 from maropu/DocFix.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/da9146c91a33577ff81378ca7e7c38a4b1917876"},{"sha":"711ce048a285403241bbc9eaabffc1314162e89c","author":{"email":"bb02c6aa60d4a0a091a9679b63db9e2367e07b50@me.com","name":"Lewuathe"},"message":"[ML][MINOR] Invalid MulticlassClassification reference in ml-guide\n\nIn [ml-guide](https://spark.apache.org/docs/latest/ml-guide.html#example-model-selection-via-cross-validation), there is invalid reference to `MulticlassClassificationEvaluator` apidoc.\n\nhttps://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.evaluation.MultiClassClassificationEvaluator\n\nAuthor: Lewuathe <lewuathe@me.com>\n\nCloses #10996 from Lewuathe/fix-typo-in-ml-guide.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/711ce048a285403241bbc9eaabffc1314162e89c"},{"sha":"51b03b71ffc390e67b32936efba61e614a8b0d86","author":{"email":"22a0dda5f6015b8ee84a30ff6553d35a91fe8732@gmail.com","name":"Timothy Chen"},"message":"[SPARK-12463][SPARK-12464][SPARK-12465][SPARK-10647][MESOS] Fix zookeeper dir with mesos conf and add docs.\n\nFix zookeeper dir configuration used in cluster mode, and also add documentation around these settings.\n\nAuthor: Timothy Chen <tnachen@gmail.com>\n\nCloses #10057 from tnachen/fix_mesos_dir.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/51b03b71ffc390e67b32936efba61e614a8b0d86"},{"sha":"a41b68b954ba47284a1df312f0aaea29b0721b0a","author":{"email":"18833a86cec08fc9c664b2901d12d72b5d5cf0f2@gmail.com","name":"Nilanjan Raychaudhuri"},"message":"[SPARK-12265][MESOS] Spark calls System.exit inside driver instead of throwing exception\n\nThis takes over #10729 and makes sure that `spark-shell` fails with a proper error message. There is a slight behavioral change: before this change `spark-shell` would exit, while now the REPL is still there, but `sc` and `sqlContext` are not defined and the error is visible to the user.\n\nAuthor: Nilanjan Raychaudhuri <nraychaudhuri@gmail.com>\nAuthor: Iulian Dragos <jaguarul@gmail.com>\n\nCloses #10921 from dragos/pr/10729.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/a41b68b954ba47284a1df312f0aaea29b0721b0a"},{"sha":"c9b89a0a0921ce3d52864afd4feb7f37b90f7b46","author":{"email":"53fdba43521c8d2c2b4ff3a6f81e9f27ad86e7ac@gmail.com","name":"Iulian Dragos"},"message":"[SPARK-12979][MESOS] Don’t resolve paths on the local file system in Mesos scheduler\n\nThe driver filesystem is likely different from where the executors will run, so resolving paths (and symlinks, etc.) will lead to invalid paths on executors.\n\nAuthor: Iulian Dragos <jaguarul@gmail.com>\n\nCloses #10923 from dragos/issue/canonical-paths.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/c9b89a0a0921ce3d52864afd4feb7f37b90f7b46"},{"sha":"064b029c6a15481fc4dfb147100c19a68cd1cc95","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-13043][SQL] Implement remaining catalyst types in ColumnarBatch.\n\nThis includes: float, boolean, short, decimal and calendar interval.\n\nDecimal is mapped to long or byte array depending on the size and calendar\ninterval is mapped to a struct of int and long.\n\nThe only remaining type is map. The schema mapping is straightforward but\nwe might want to revisit how we deal with this in the rest of the execution\nengine.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10961 from nongli/spark-13043.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/064b029c6a15481fc4dfb147100c19a68cd1cc95"},{"sha":"a2973fed30fbe9a0b12e1c1225359fdf55d322b4","author":{"email":"9749d1492af3d43f9c09e04c5c43f27bb909af51@japila.pl","name":"Jacek Laskowski"},"message":"Fix for [SPARK-12854][SQL] Implement complex types support in Columna…\n\n…rBatch\n\nFixes build for Scala 2.11.\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #10946 from jaceklaskowski/SPARK-12854-fix.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/a2973fed30fbe9a0b12e1c1225359fdf55d322b4"},{"sha":"be7a2fc0716b7d25327b6f8f683390fc62532e3b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13078][SQL] API and test cases for internal catalog\n\nThis pull request creates an internal catalog API. The creation of this API is the first step towards consolidating SQLContext and HiveContext. I envision we will have two different implementations in Spark 2.0: (1) a simple in-memory implementation, and (2) an implementation based on the current HiveClient (ClientWrapper).\n\nI took a look at what Hive's internal metastore interface/implementation, and then created this API based on it. I believe this is the minimal set needed in order to achieve all the needed functionality.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10982 from rxin/SPARK-13078.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/be7a2fc0716b7d25327b6f8f683390fc62532e3b"},{"sha":"715a19d56fc934d4aec5025739ff650daf4580b7","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-12637][CORE] Print stage info of finished stages properly\n\nImprove printing of StageInfo in onStageCompleted\n\nSee also https://github.com/apache/spark/pull/10585\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #10922 from srowen/SPARK-12637.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/715a19d56fc934d4aec5025739ff650daf4580b7"},{"sha":"0df3cfb8ab4d584c95db6c340694e199d7b59e9e","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-12790][CORE] Remove HistoryServer old multiple files format\n\nRemoved isLegacyLogDirectory code path and updated tests\nandrewor14\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #10860 from felixcheung/historyserverformat.","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/0df3cfb8ab4d584c95db6c340694e199d7b59e9e"}]}