{"push_id":1064983750,"size":1000,"distinct_size":11,"ref":"refs/heads/nullable","head":"f8c0bfecf37111b73c87bf42870994d5362c84ec","before":"8cf99b81cfef07fa2e12c6af931e82d1ef64cfe9","commits":[{"sha":"be7a2fc0716b7d25327b6f8f683390fc62532e3b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13078][SQL] API and test cases for internal catalog\n\nThis pull request creates an internal catalog API. The creation of this API is the first step towards consolidating SQLContext and HiveContext. I envision we will have two different implementations in Spark 2.0: (1) a simple in-memory implementation, and (2) an implementation based on the current HiveClient (ClientWrapper).\n\nI took a look at what Hive's internal metastore interface/implementation, and then created this API based on it. I believe this is the minimal set needed in order to achieve all the needed functionality.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10982 from rxin/SPARK-13078.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/be7a2fc0716b7d25327b6f8f683390fc62532e3b"},{"sha":"715a19d56fc934d4aec5025739ff650daf4580b7","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-12637][CORE] Print stage info of finished stages properly\n\nImprove printing of StageInfo in onStageCompleted\n\nSee also https://github.com/apache/spark/pull/10585\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #10922 from srowen/SPARK-12637.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/715a19d56fc934d4aec5025739ff650daf4580b7"},{"sha":"0df3cfb8ab4d584c95db6c340694e199d7b59e9e","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-12790][CORE] Remove HistoryServer old multiple files format\n\nRemoved isLegacyLogDirectory code path and updated tests\nandrewor14\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #10860 from felixcheung/historyserverformat.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/0df3cfb8ab4d584c95db6c340694e199d7b59e9e"},{"sha":"0fff5c6e6325357a241d311e72db942c4850af34","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13130][SQL] Make codegen variable names easier to read\n\n1. Use lower case\n2. Change long prefixes to something shorter (in this case I am changing only one: TungstenAggregate -> agg).\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11017 from rxin/SPARK-13130.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/0fff5c6e6325357a241d311e72db942c4850af34"},{"sha":"b8666fd0e2a797924eb2e94ac5558aba2a9b5140","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #10662. Closes #10661","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b8666fd0e2a797924eb2e94ac5558aba2a9b5140"},{"sha":"22ba21348b28d8b1909ccde6fe17fb9e68531e5a","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-13087][SQL] Fix group by function for sort based aggregation\n\nIt is not valid to call `toAttribute` on a `NamedExpression` unless we know for sure that the child produced that `NamedExpression`.  The current code worked fine when the grouping expressions were simple, but when they were a derived value this blew up at execution time.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #11013 from marmbrus/groupByFunction-master.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/22ba21348b28d8b1909ccde6fe17fb9e68531e5a"},{"sha":"12a20c144f14e80ef120ddcfb0b455a805a2da23","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-10820][SQL] Support for the continuous execution of structured queries\n\nThis is a follow up to 9aadcffabd226557174f3ff566927f873c71672e that extends Spark SQL to allow users to _repeatedly_ optimize and execute structured queries.  A `ContinuousQuery` can be expressed using SQL, DataFrames or Datasets.  The purpose of this PR is only to add some initial infrastructure which will be extended in subsequent PRs.\n\n## User-facing API\n\n- `sqlContext.streamFrom` and `df.streamTo` return builder objects that are analogous to the `read/write` interfaces already available to executing queries in a batch-oriented fashion.\n- `ContinuousQuery` provides an interface for interacting with a query that is currently executing in the background.\n\n## Internal Interfaces\n - `StreamExecution` - executes streaming queries in micro-batches\n\nThe following are currently internal, but public APIs will be provided in a future release.\n - `Source` - an interface for providers of continually arriving data.  A source must have a notion of an `Offset` that monotonically tracks what data has arrived.  For fault tolerance, a source must be able to replay data given a start offset.\n - `Sink` - an interface that accepts the results of a continuously executing query.  Also responsible for tracking the offset that should be resumed from in the case of a failure.\n\n## Testing\n - `MemoryStream` and `MemorySink` - simple implementations of source and sink that keep all data in memory and have methods for simulating durability failures\n - `StreamTest` - a framework for performing actions and checking invariants on a continuous query\n\nAuthor: Michael Armbrust <michael@databricks.com>\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\nAuthor: Josh Rosen <rosenville@gmail.com>\n\nCloses #11006 from marmbrus/structured-streaming.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/12a20c144f14e80ef120ddcfb0b455a805a2da23"},{"sha":"29d92181d0c49988c387d34e4a71b1afe02c29e2","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-13094][SQL] Add encoders for seq/array of primitives\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #11014 from marmbrus/seqEncoders.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/29d92181d0c49988c387d34e4a71b1afe02c29e2"},{"sha":"b93830126cc59a26e2cfb5d7b3c17f9cfbf85988","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-13114][SQL] Add a test for tokens more than the fields in schema\n\nhttps://issues.apache.org/jira/browse/SPARK-13114\n\nThis PR adds a test for tokens more than the fields in schema.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #11020 from HyukjinKwon/SPARK-13114.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b93830126cc59a26e2cfb5d7b3c17f9cfbf85988"},{"sha":"cba1d6b659288bfcd8db83a6d778155bab2bbecf","author":{"email":"060dd14c05a8ffa8f201883ef57d0676b7c9c04a@gmail.com","name":"Bryan Cutler"},"message":"[SPARK-12631][PYSPARK][DOC] PySpark clustering parameter desc to consistent format\n\nPart of task for [SPARK-11219](https://issues.apache.org/jira/browse/SPARK-11219) to make PySpark MLlib parameter description formatting consistent.  This is for the clustering module.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #10610 from BryanCutler/param-desc-consistent-cluster-SPARK-12631.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/cba1d6b659288bfcd8db83a6d778155bab2bbecf"},{"sha":"358300c795025735c3b2f96c5447b1b227d4abc1","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-13056][SQL] map column would throw NPE if value is null\n\nJira:\nhttps://issues.apache.org/jira/browse/SPARK-13056\n\nCreate a map like\n{ \"a\": \"somestring\", \"b\": null}\nQuery like\nSELECT col[\"b\"] FROM t1;\nNPE would be thrown.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #10964 from adrian-wang/npewriter.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/358300c795025735c3b2f96c5447b1b227d4abc1"},{"sha":"b1835d727234fdff42aa8cadd17ddcf43b0bed15","author":{"email":"a6aff63cf1928c737786fe9f14d6bc4bd2404f2b@codilime.com","name":"Grzegorz Chilkiewicz"},"message":"[SPARK-12711][ML] ML StopWordsRemover does not protect itself from column name duplication\n\nFixes problem and verifies fix by test suite.\nAlso - adds optional parameter: nullable (Boolean) to: SchemaUtils.appendColumn\nand deduplicates SchemaUtils.appendColumn functions.\n\nAuthor: Grzegorz Chilkiewicz <grzegorz.chilkiewicz@codilime.com>\n\nCloses #10741 from grzegorz-chilkiewicz/master.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b1835d727234fdff42aa8cadd17ddcf43b0bed15"},{"sha":"7f6e3ec79b77400f558ceffa10b2af011962115f","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13138][SQL] Add \"logical\" package prefix for ddl.scala\n\nddl.scala is defined in the execution package, and yet its reference of \"UnaryNode\" and \"Command\" are logical. This was fairly confusing when I was trying to understand the ddl code.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11021 from rxin/SPARK-13138.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7f6e3ec79b77400f558ceffa10b2af011962115f"},{"sha":"be5dd881f1eff248224a92d57cfd1309cb3acf38","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12913] [SQL] Improve performance of stat functions\n\nAs benchmarked and discussed here: https://github.com/apache/spark/pull/10786/files#r50038294, benefits from codegen, the declarative aggregate function could be much faster than imperative one.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10960 from davies/stddev.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/be5dd881f1eff248224a92d57cfd1309cb3acf38"},{"sha":"d0df2ca40953ba581dce199798a168af01283cdc","author":{"email":"1d6e1cf70ec6f9ab28d3ea4b27a49a77654d370e@nizzoli.net","name":"Gabriele Nizzoli"},"message":"[SPARK-13121][STREAMING] java mapWithState mishandles scala Option\n\nAlready merged into 1.6 branch, this PR is to commit to master the same change\n\nAuthor: Gabriele Nizzoli <mail@nizzoli.net>\n\nCloses #11028 from gabrielenizzoli/patch-1.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/d0df2ca40953ba581dce199798a168af01283cdc"},{"sha":"b377b03531d21b1d02a8f58b3791348962e1f31b","author":{"email":"629f1a1121929ad4e9b78df09dd6adf714a187a2@gmail.com","name":"Kevin (Sangwoo) Kim"},"message":"[DOCS] Update StructType.scala\n\nThe example will throw error like\n<console>:20: error: not found: value StructType\n\nNeed to add this line:\nimport org.apache.spark.sql.types._\n\nAuthor: Kevin (Sangwoo) Kim <sangwookim.me@gmail.com>\n\nCloses #10141 from swkimme/patch-1.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b377b03531d21b1d02a8f58b3791348962e1f31b"},{"sha":"6de6a97728408ee2619006decf2267cc43eeea0d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13150] [SQL] disable two flaky tests\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11037 from davies/disable_flaky.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/6de6a97728408ee2619006decf2267cc43eeea0d"},{"sha":"672032d0ab1e43bc5a25cecdb1b96dfd35c39778","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13020][SQL][TEST] fix random generator for map type\n\nwhen we generate map, we first randomly pick a length, then create a seq of key value pair with the expected length, and finally call `toMap`. However, `toMap` will remove all duplicated keys, which makes the actual map size much less than we expected.\n\nThis PR fixes this problem by put keys in a set first, to guarantee we have enough keys to build a map with expected length.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10930 from cloud-fan/random-generator.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/672032d0ab1e43bc5a25cecdb1b96dfd35c39778"},{"sha":"21112e8a14c042ccef4312079672108a1082a95e","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12992] [SQL] Update parquet reader to support more types when decoding to ColumnarBatch.\n\nThis patch implements support for more types when doing the vectorized decode. There are\na few more types remaining but they should be very straightforward after this. This code\nhas a few copy and paste pieces but they are difficult to eliminate due to performance\nconsiderations.\n\nSpecifically, this patch adds support for:\n  - String, Long, Byte types\n  - Dictionary encoding for those types.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10908 from nongli/spark-12992.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/21112e8a14c042ccef4312079672108a1082a95e"},{"sha":"ff71261b651a7b289ea2312abd6075da8b838ed9","author":{"email":"bf2bdb19a164be3120ec76ff9fecbc8dcbafc007@amazon.com","name":"Adam Budde"},"message":"[SPARK-13122] Fix race condition in MemoryStore.unrollSafely()\n\nhttps://issues.apache.org/jira/browse/SPARK-13122\n\nA race condition can occur in MemoryStore's unrollSafely() method if two threads that\nreturn the same value for currentTaskAttemptId() execute this method concurrently. This\nchange makes the operation of reading the initial amount of unroll memory used, performing\nthe unroll, and updating the associated memory maps atomic in order to avoid this race\ncondition.\n\nInitial proposed fix wraps all of unrollSafely() in a memoryManager.synchronized { } block. A cleaner approach might be introduce a mechanism that synchronizes based on task attempt ID. An alternative option might be to track unroll/pending unroll memory based on block ID rather than task attempt ID.\n\nAuthor: Adam Budde <budde@amazon.com>\n\nCloses #11012 from budde/master.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ff71261b651a7b289ea2312abd6075da8b838ed9"}]}