{"push_id":959561213,"size":54,"distinct_size":54,"ref":"refs/heads/followup","head":"0df3cfb8ab4d584c95db6c340694e199d7b59e9e","before":"4db255c7aa756daa224d61905db745b6bccc9173","commits":[{"sha":"90b0e562406a8bac529e190472e7f5da4030bf5c","author":{"email":"07c870ffe79a3d57c30fc67475f42d361ec88a03@gmail.com","name":"BenFradet"},"message":"[SPARK-12983][CORE][DOC] Correct metrics.properties.template\n\nThere are some typos or plain unintelligible sentences in the metrics template.\n\nAuthor: BenFradet <benjamin.fradet@gmail.com>\n\nCloses #10902 from BenFradet/SPARK-12983.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/90b0e562406a8bac529e190472e7f5da4030bf5c"},{"sha":"093291cf9b8729c0bd057cf67aed840b11f8c94a","author":{"email":"7e98c31fc4f71e4467497b580441a4c51679660e@gmail.com","name":"Andrew"},"message":"[SPARK-1680][DOCS] Explain environment variables for running on YARN in cluster mode\n\nJIRA 1680 added a property called spark.yarn.appMasterEnv.  This PR draws users' attention to this special case by adding an explanation in configuration.html#environment-variables\n\nAuthor: Andrew <weiner.andrew.j@gmail.com>\n\nCloses #10869 from weineran/branch-yarn-docs.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/093291cf9b8729c0bd057cf67aed840b11f8c94a"},{"sha":"41f0c85f9be264103c066935e743f59caf0fe268","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13023][PROJECT INFRA] Fix handling of root module in modules_to_test()\n\nThere's a minor bug in how we handle the `root` module in the `modules_to_test()` function in `dev/run-tests.py`: since `root` now depends on `build` (since every test needs to run on any build test), we now need to check for the presence of root in `modules_to_test` instead of `changed_modules`.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10933 from JoshRosen/build-module-fix.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/41f0c85f9be264103c066935e743f59caf0fe268"},{"sha":"edd473751b59b55fa3daede5ed7bc19ea8bd7170","author":{"email":"69d9c961933324ebb2a99682f1939032e0ff256e@us.ibm.com","name":"Jason Lee"},"message":"[SPARK-10847][SQL][PYSPARK] Pyspark - DataFrame - Optional Metadata with `None` triggers cryptic failure\n\nThe error message is now changed from \"Do not support type class scala.Tuple2.\" to \"Do not support type class org.json4s.JsonAST$JNull$\" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.\n\nAuthor: Jason Lee <cjlee@us.ibm.com>\n\nCloses #8969 from jasoncl/SPARK-10847.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/edd473751b59b55fa3daede5ed7bc19ea8bd7170"},{"sha":"87abcf7df921a5937fdb2bae8bfb30bfabc4970a","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-12895][SPARK-12896] Migrate TaskMetrics to accumulators\n\nThe high level idea is that instead of having the executors send both accumulator updates and TaskMetrics, we should have them send only accumulator updates. This eliminates the need to maintain both code paths since one can be implemented in terms of the other. This effort is split into two parts:\n\n**SPARK-12895: Implement TaskMetrics using accumulators.** TaskMetrics is basically just a bunch of accumulable fields. This patch makes TaskMetrics a syntactic wrapper around a collection of accumulators so we don't need to send TaskMetrics from the executors to the driver.\n\n**SPARK-12896: Send only accumulator updates to the driver.** Now that TaskMetrics are expressed in terms of accumulators, we can capture all TaskMetrics values if we just send accumulator updates from the executors to the driver. This completes the parent issue SPARK-10620.\n\nWhile an effort has been made to preserve as much of the public API as possible, there were a few known breaking DeveloperApi changes that would be very awkward to maintain. I will gather the full list shortly and post it here.\n\nNote: This was once part of #10717. This patch is split out into its own patch from there to make it easier for others to review. Other smaller pieces of already been merged into master.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10835 from andrewor14/task-metrics-use-accums.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/87abcf7df921a5937fdb2bae8bfb30bfabc4970a"},{"sha":"32f741115bda5d7d7dbfcd9fe827ecbea7303ffa","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13021][CORE] Fail fast when custom RDDs violate RDD.partition's API contract\n\nSpark's `Partition` and `RDD.partitions` APIs have a contract which requires custom implementations of `RDD.partitions` to ensure that for all `x`, `rdd.partitions(x).index == x`; in other words, the `index` reported by a repartition needs to match its position in the partitions array.\n\nIf a custom RDD implementation violates this contract, then Spark has the potential to become stuck in an infinite recomputation loop when recomputing a subset of an RDD's partitions, since the tasks that are actually run will not correspond to the missing output partitions that triggered the recomputation. Here's a link to a notebook which demonstrates this problem: https://rawgit.com/JoshRosen/e520fb9a64c1c97ec985/raw/5e8a5aa8d2a18910a1607f0aa4190104adda3424/Violating%2520RDD.partitions%2520contract.html\n\nIn order to guard against this infinite loop behavior, this patch modifies Spark so that it fails fast and refuses to compute RDDs' whose `partitions` violate the API contract.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10932 from JoshRosen/SPARK-13021.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/32f741115bda5d7d7dbfcd9fe827ecbea7303ffa"},{"sha":"680afabe78b77e4e63e793236453d69567d24290","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12938][SQL] DataFrame API for Bloom filter\n\nThis PR integrates Bloom filter from spark-sketch into DataFrame. This version resorts to RDD.aggregate for building the filter. A more performant UDAF version can be built in future follow-up PRs.\n\nThis PR also add 2 specify `put` version(`putBinary` and `putLong`) into `BloomFilter`, which makes it easier to build a Bloom filter over a `DataFrame`.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10937 from cloud-fan/bloom-filter.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/680afabe78b77e4e63e793236453d69567d24290"},{"sha":"ef96cd3c521c175878c38a1ed6eeeab0ed8346b5","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-12865][SPARK-12866][SQL] Migrate SparkSQLParser/ExtendedHiveQlParser commands to new Parser\n\nThis PR moves all the functionality provided by the SparkSQLParser/ExtendedHiveQlParser to the new Parser hierarchy (SparkQl/HiveQl). This also improves the current SET command parsing: the current implementation swallows ```set role ...``` and ```set autocommit ...``` commands, this PR respects these commands (and passes them on to Hive).\n\nThis PR and https://github.com/apache/spark/pull/10723 end the use of Parser-Combinator parsers for SQL parsing. As a result we can also remove the ```AbstractSQLParser``` in Catalyst.\n\nThe PR is marked WIP as long as it doesn't pass all tests.\n\ncc rxin viirya winningsix (this touches https://github.com/apache/spark/pull/10144)\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #10905 from hvanhovell/SPARK-12866.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/ef96cd3c521c175878c38a1ed6eeeab0ed8346b5"},{"sha":"d702f0c170d5c39df501e173813f8a7718e3b3c6","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[HOTFIX] Fix Scala 2.11 compilation\n\nby explicitly marking annotated parameters as vals (SI-8813).\n\nCaused by #10835.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10955 from andrewor14/fix-scala211.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/d702f0c170d5c39df501e173813f8a7718e3b3c6"},{"sha":"4a091232122b51f10521a68de8b1d9eb853b563d","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-13045] [SQL] Remove ColumnVector.Struct in favor of ColumnarBatch.Row\n\nThese two classes became identical as the implementation progressed.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10952 from nongli/spark-13045.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/4a091232122b51f10521a68de8b1d9eb853b563d"},{"sha":"c2204436a15838f2dce44e3cfb0fe58236ef6196","author":{"email":"d994670fb673e572968ac5ea8945d354483bbdc7@users.noreply.github.com","name":"James Lohse"},"message":"Provide same info as in spark-submit --help\n\nthis is stated for --packages and --repositories. Without stating it for --jars, people expect a standard java classpath to work, with expansion and using a different delimiter than a comma. Currently this is only state in the --help for spark-submit \"Comma-separated list of local jars to include on the driver and executor classpaths.\"\n\nAuthor: James Lohse <jimlohse@users.noreply.github.com>\n\nCloses #10890 from jimlohse/patch-1.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/c2204436a15838f2dce44e3cfb0fe58236ef6196"},{"sha":"415d0a859b7a76f3a866ec62ab472c4050f2a01b","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12818][SQL] Specialized integral and string types for Count-min Sketch\n\nThis PR is a follow-up of #10911. It adds specialized update methods for `CountMinSketch` so that we can avoid doing internal/external row format conversion in `DataFrame.countMinSketch()`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10968 from liancheng/cms-specialized.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/415d0a859b7a76f3a866ec62ab472c4050f2a01b"},{"sha":"676803963fcc08aa988aa6f14be3751314e006ca","author":{"email":"6ea7d3fc9c1075bfb0a62c61a167edd3b77f94cd@fb.com","name":"Tejas Patil"},"message":"[SPARK-12926][SQL] SQLContext to display warning message when non-sql configs are being set\n\nUsers unknowingly try to set core Spark configs in SQLContext but later realise that it didn't work. eg. sqlContext.sql(\"SET spark.shuffle.memoryFraction=0.4\"). This PR adds a warning message when such operations are done.\n\nAuthor: Tejas Patil <tejasp@fb.com>\n\nCloses #10849 from tejasapatil/SPARK-12926.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/676803963fcc08aa988aa6f14be3751314e006ca"},{"sha":"cc18a7199240bf3b03410c1ba6704fe7ce6ae38e","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13031] [SQL] cleanup codegen and improve test coverage\n\n1. enable whole stage codegen during tests even there is only one operator supports that.\n2. split doProduce() into two APIs: upstream() and doProduce()\n3. generate prefix for fresh names of each operator\n4. pass UnsafeRow to parent directly (avoid getters and create UnsafeRow again)\n5. fix bugs and tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10944 from davies/gen_refactor.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/cc18a7199240bf3b03410c1ba6704fe7ce6ae38e"},{"sha":"df78a934a07a4ce5af43243be9ba5fe60b91eee6","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9835][ML] Implement IterativelyReweightedLeastSquares solver\n\nImplement ```IterativelyReweightedLeastSquares``` solver for GLM. I consider it as a solver rather than estimator, it only used internal so I keep it ```private[ml]```.\nThere are two limitations in the current implementation compared with R:\n* It can not support ```Tuple``` as response for ```Binomial``` family, such as the following code:\n```\nglm( cbind(using, notUsing) ~  age + education + wantsMore , family = binomial)\n```\n* It does not support ```offset```.\n\nBecause I considered that ```RFormula``` did not support ```Tuple``` as label and ```offset``` keyword, so I simplified the implementation. But to add support for these two functions is not very hard, I can do it in follow-up PR if it is necessary. Meanwhile, we can also add R-like statistic summary for IRLS.\nThe implementation refers R, [statsmodels](https://github.com/statsmodels/statsmodels) and [sparkGLM](https://github.com/AlteryxLabs/sparkGLM).\nPlease focus on the main structure and overpass minor issues/docs that I will update later. Any comments and opinions will be appreciated.\n\ncc mengxr jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10639 from yanboliang/spark-9835.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/df78a934a07a4ce5af43243be9ba5fe60b91eee6"},{"sha":"abae889f08eb412cb897e4e63614ec2c93885ffd","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-12401][SQL] Add integration tests for postgres enum types\n\nWe can handle posgresql-specific enum types as strings in jdbc.\nSo, we should just add tests and close the corresponding JIRA ticket.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #10596 from maropu/AddTestsInIntegration.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/abae889f08eb412cb897e4e63614ec2c93885ffd"},{"sha":"3a40c0e575fd4215302ea60c9821d31a5a138b8a","author":{"email":"48125774e1ba05a013278c93767e0e53161dec0b@gmail.com","name":"Brandon Bradley"},"message":"[SPARK-12749][SQL] add json option to parse floating-point types as DecimalType\n\nI tried to add this via `USE_BIG_DECIMAL_FOR_FLOATS` option from Jackson with no success.\n\nAdded test for non-complex types. Should I add a test for complex types?\n\nAuthor: Brandon Bradley <bradleytastic@gmail.com>\n\nCloses #10936 from blbradley/spark-12749.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/3a40c0e575fd4215302ea60c9821d31a5a138b8a"},{"sha":"4637fc08a3733ec313218fb7e4d05064d9a6262d","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11955][SQL] Mark optional fields in merging schema for safely pushdowning filters in Parquet\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-11955\n\nCurrently we simply skip pushdowning filters in parquet if we enable schema merging.\n\nHowever, we can actually mark particular fields in merging schema for safely pushdowning filters in parquet.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #9940 from viirya/safe-pushdown-parquet-filters.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/4637fc08a3733ec313218fb7e4d05064d9a6262d"},{"sha":"b9dfdcc63bb12bc24de96060e756889c2ceda519","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"Revert \"[SPARK-13031] [SQL] cleanup codegen and improve test coverage\"\n\nThis reverts commit cc18a7199240bf3b03410c1ba6704fe7ce6ae38e.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/b9dfdcc63bb12bc24de96060e756889c2ceda519"},{"sha":"66449b8dcdbc3dca126c34b42c4d0419c7648696","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12968][SQL] Implement command to set current database\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12968\n\nImplement command to set current database.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10916 from viirya/ddl-use-database.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/66449b8dcdbc3dca126c34b42c4d0419c7648696"}]}