{"push_id":1080971131,"size":19,"distinct_size":19,"ref":"refs/heads/master","head":"8df8a81825709dbefe5aecd7642748c1b3a38e99","before":"a55fbe2a16aa0866ff8aca25bf9f772e6eb516a1","commits":[{"sha":"be0d5d3bbebf0912c27bf41ce27b5ba214e61e19","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-14873][CORE] Java sampleByKey methods take ju.Map but with Scala Double values; results in type Object\n\n## What changes were proposed in this pull request?\n\nJava `sampleByKey` methods should accept `Map` with `java.lang.Double` values\n\n## How was this patch tested?\n\nExisting (updated) Jenkins tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #12637 from srowen/SPARK-14873.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/be0d5d3bbebf0912c27bf41ce27b5ba214e61e19"},{"sha":"6acc72a02961982f217ceaea5bde4eca4e2d7db8","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes some open PRs that have been requested to close.\n\nCloses #7647\nCloses #8195\nCloses #8741\nCloses #8972\nCloses #9490\nCloses #10419\nCloses #10761\nCloses #11003\nCloses #11201\nCloses #11803\nCloses #12111\nCloses #12442","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/6acc72a02961982f217ceaea5bde4eca4e2d7db8"},{"sha":"39d3bc62a7ba16c646bed8d524cf9b929374a790","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-14594][SPARKR] check execution return status code\n\n## What changes were proposed in this pull request?\n\nWhen JVM backend fails without going proper error handling (eg. process crashed), the R error message could be ambiguous.\n\n```\nError in if (returnStatus != 0) { : argument is of length zero\n```\n\nThis change attempts to make it more clear (however, one would still need to investigate why JVM fails)\n\n## How was this patch tested?\n\nmanually\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #12622 from felixcheung/rreturnstatus.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/39d3bc62a7ba16c646bed8d524cf9b929374a790"},{"sha":"f0bba7447f7f9e5bcc4e029e0a74742880798546","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Turn script transformation back on.\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12565 from rxin/test-flaky.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/f0bba7447f7f9e5bcc4e029e0a74742880798546"},{"sha":"ee6b209a9dda3d5afedee8a31ed5a8d2ded3a1a0","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"[HOTFIX] disable generated aggregate map","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/ee6b209a9dda3d5afedee8a31ed5a8d2ded3a1a0"},{"sha":"fddd3aee0df2cf2244780ce81c370ff3863bb65c","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14871][SQL] Disable StatsReportListener to declutter output\n\n## What changes were proposed in this pull request?\nSpark SQL inherited from Shark to use the StatsReportListener. Unfortunately this clutters the spark-sql CLI output and makes it very difficult to read the actual query results.\n\n## How was this patch tested?\nBuilt and tested in spark-sql CLI.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12635 from rxin/SPARK-14871.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/fddd3aee0df2cf2244780ce81c370ff3863bb65c"},{"sha":"5c8a0ec99bded2271481f8d6cf5443fea5da4bbd","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14872][SQL] Restructure command package\n\n## What changes were proposed in this pull request?\nThis patch restructures sql.execution.command package to break the commands into multiple files, in some logical organization: databases, tables, views, functions.\n\nI also renamed basicOperators.scala to basicLogicalOperators.scala and basicPhysicalOperators.scala.\n\n## How was this patch tested?\nN/A - all I did was moving code around.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12636 from rxin/SPARK-14872.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/5c8a0ec99bded2271481f8d6cf5443fea5da4bbd"},{"sha":"890abd1279014d692548c9f3b557483644a0ee32","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14869][SQL] Don't mask exceptions in ResolveRelations\n\n## What changes were proposed in this pull request?\nIn order to support running SQL directly on files, we added some code in ResolveRelations to catch the exception thrown by catalog.lookupRelation and ignore it. This unfortunately masks all the exceptions. This patch changes the logic to simply test the table's existence.\n\n## How was this patch tested?\nI manually hacked some bugs into Spark and made sure the exceptions were being propagated up.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12634 from rxin/SPARK-14869.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/890abd1279014d692548c9f3b557483644a0ee32"},{"sha":"e3c1366bbcf712f8d7a91640eb11e67a4419e4be","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14865][SQL] Better error handling for view creation.\n\n## What changes were proposed in this pull request?\nThis patch improves error handling in view creation. CreateViewCommand itself will analyze the view SQL query first, and if it cannot successfully analyze it, throw an AnalysisException.\n\nIn addition, I also added the following two conservative guards for easier identification of Spark bugs:\n\n1. If there is a bug and the generated view SQL cannot be analyzed, throw an exception at runtime. Note that this is not an AnalysisException because it is not caused by the user and more likely indicate a bug in Spark.\n2. SQLBuilder when it gets an unresolved plan, it will also show the plan in the error message.\n\nI also took the chance to simplify the internal implementation of CreateViewCommand, and *removed* a fallback path that would've masked an exception from before.\n\n## How was this patch tested?\n1. Added a unit test for the user facing error handling.\n2. Manually introduced some bugs in Spark to test the internal defensive error handling.\n3. Also added a test case to test nested views (not super relevant).\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12633 from rxin/SPARK-14865.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/e3c1366bbcf712f8d7a91640eb11e67a4419e4be"},{"sha":"162e12b085692d67fd3b2bd1254152cb56db0722","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14877][SQL] Remove HiveMetastoreTypes class\n\n## What changes were proposed in this pull request?\nIt is unnecessary as DataType.catalogString largely replaces the need for this class.\n\n## How was this patch tested?\nMostly removing dead code and should be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12644 from rxin/SPARK-14877.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/162e12b085692d67fd3b2bd1254152cb56db0722"},{"sha":"bebb0240e6ece6ec7c79bdd62098529146fcb0c0","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[MINOR] [SQL] Fix error message string in nullSafeEvel of TernaryExpression\n\n## What changes were proposed in this pull request?\n\nTernaryExpressions should thows proper error message for itself.\n```scala\n   protected def nullSafeEval(input1: Any, input2: Any, input3: Any): Any =\n-    sys.error(s\"BinaryExpressions must override either eval or nullSafeEval\")\n+    sys.error(s\"TernaryExpressions must override either eval or nullSafeEval\")\n```\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12642 from dongjoon-hyun/minor_fix_error_msg_in_ternaryexpression.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/bebb0240e6ece6ec7c79bdd62098529146fcb0c0"},{"sha":"b45819ac43311dc830a97f15d1278936efca1cc1","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-14856] Correct message in assertion for 'returning batch for wide table'\n\n## What changes were proposed in this pull request?\n\nThere was a typo in the message for second assertion in \"returning batch for wide table\" test\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #12639 from tedyu/master.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/b45819ac43311dc830a97f15d1278936efca1cc1"},{"sha":"1b7eab74e64f554bbf892c8ef7b7ec00b359d2c0","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-12148][SPARKR] fix doc after renaming DataFrame to SparkDataFrame\n\n## What changes were proposed in this pull request?\n\nFixed inadvertent roxygen2 doc changes, added class name change to programming guide\nFollow up of #12621\n\n## How was this patch tested?\n\nmanually checked\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #12647 from felixcheung/rdataframe.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/1b7eab74e64f554bbf892c8ef7b7ec00b359d2c0"},{"sha":"ba5e0b87a043e46e9599695c82d90e7572185aa5","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-14838] [SQL] Set default size for ObjecType to avoid failure when estimating sizeInBytes in ObjectProducer\n\n## What changes were proposed in this pull request?\n\nWe have logical plans that produce domain objects which are `ObjectType`. As we can't estimate the size of `ObjectType`, we throw an `UnsupportedOperationException` if trying to do that. We should set a default size for `ObjectType` to avoid this failure.\n\n## How was this patch tested?\n\n`DatasetSuite`.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #12599 from viirya/skip-broadcast-objectproducer.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/ba5e0b87a043e46e9599695c82d90e7572185aa5"},{"sha":"28538596558b7f69f9d22eb0902d0e609d98be88","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-14833][SQL][STREAMING][TEST] Refactor StreamTests to test for source fault-tolerance correctly.\n\n## What changes were proposed in this pull request?\n\nCurrent StreamTest allows testing of a streaming Dataset generated explicitly wraps a source. This is different from the actual production code path where the source object is dynamically created through a DataSource object every time a query is started. So all the fault-tolerance testing in FileSourceSuite and FileSourceStressSuite is not really testing the actual code path as they are just reusing the FileStreamSource object.\n\nThis PR fixes StreamTest and the FileSource***Suite to test this correctly. Instead of maintaining a mapping of source --> expected offset in StreamTest (which requires reuse of source object), it now maintains a mapping of source index --> offset, so that it is independent of the source object.\n\nSummary of changes\n- StreamTest refactored to keep track of offset by source index instead of source\n- AddData, AddTextData and AddParquetData updated to find the FileStreamSource object from an active query, so that it can work with sources generated when query is started.\n- Refactored unit tests in FileSource***Suite to test using DataFrame/Dataset generated with public, rather than reusing the same FileStreamSource. This correctly tests fault tolerance.\n\nThe refactoring changed a lot of indents in FileSourceSuite, so its recommended to hide whitespace changes with this - https://github.com/apache/spark/pull/12592/files?w=1\n\n## How was this patch tested?\n\nRefactored unit tests.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #12592 from tdas/SPARK-14833.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/28538596558b7f69f9d22eb0902d0e609d98be88"},{"sha":"1672149c2644b5670b4b9a4086a4456fb8279a55","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-14879][SQL] Move CreateMetastoreDataSource and CreateMetastoreDataSourceAsSelect to sql/core\n\n## What changes were proposed in this pull request?\n\nCreateMetastoreDataSource and CreateMetastoreDataSourceAsSelect are not Hive-specific. So, this PR moves them from sql/hive to sql/core. Also, I am adding `Command` suffix to these two classes.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #12645 from yhuai/moveCreateDataSource.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/1672149c2644b5670b4b9a4086a4456fb8279a55"},{"sha":"902c15c5e6da55754501c2e56bd6379b9d5f1194","author":{"email":"e31abd0e9476b35820e2a59a347f6b55883097d1@nuance.com","name":"mathieu longtin"},"message":"Support single argument version of sqlContext.getConf\n\n## What changes were proposed in this pull request?\n\nIn Python, sqlContext.getConf didn't allow getting the system default (getConf with one parameter).\n\nNow the following are supported:\n```\nsqlContext.getConf(confName)  # System default if not locally set, this is new\nsqlContext.getConf(confName, myDefault)  # myDefault if not locally set, old behavior\n```\n\nI also added doctests to this function. The original behavior does not change.\n\n## How was this patch tested?\n\nManually, but doctests were added.\n\nAuthor: mathieu longtin <mathieu.longtin@nuance.com>\n\nCloses #12488 from mathieulongtin/pyfixgetconf3.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/902c15c5e6da55754501c2e56bd6379b9d5f1194"},{"sha":"db7113b1d37e86253d8584b88ed66672f3620254","author":{"email":"ec969b9b5c2f2a49a0d9175cec61f09554863a3d@hortonworks.com","name":"Steve Loughran"},"message":"[SPARK-13267][WEB UI] document the ?param arguments of the REST API; lift the…\n\nAdd to the REST API details on the ? args and examples from the test suite.\n\nI've used the existing table, adding all the fields to the second table.\n\nsee [in the pr](https://github.com/steveloughran/spark/blob/history/SPARK-13267-doc-params/docs/monitoring.md).\n\nThere's a slightly more sophisticated option: make the table 3 columns wide, and for all existing entries, have the initial `td` span 2 columns. The new entries would then have an empty 1st column, param in 2nd and text in 3rd, with any examples after a `br` entry.\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\n\nCloses #11152 from steveloughran/history/SPARK-13267-doc-params.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/db7113b1d37e86253d8584b88ed66672f3620254"},{"sha":"8df8a81825709dbefe5aecd7642748c1b3a38e99","author":{"email":"9749d1492af3d43f9c09e04c5c43f27bb909af51@japila.pl","name":"Jacek Laskowski"},"message":"[DOCS][MINOR] Screenshot + minor fixes to improve reading for accumulators\n\n## What changes were proposed in this pull request?\n\nAdded screenshot + minor fixes to improve reading\n\n## How was this patch tested?\n\nManual\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #12569 from jaceklaskowski/docs-accumulators.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/8df8a81825709dbefe5aecd7642748c1b3a38e99"}]}