{"push_id":955771425,"size":12,"distinct_size":12,"ref":"refs/heads/master","head":"55561e7693dd2a5bf3c7f8026c725421801fd0ec","before":"c2204436a15838f2dce44e3cfb0fe58236ef6196","commits":[{"sha":"415d0a859b7a76f3a866ec62ab472c4050f2a01b","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12818][SQL] Specialized integral and string types for Count-min Sketch\n\nThis PR is a follow-up of #10911. It adds specialized update methods for `CountMinSketch` so that we can avoid doing internal/external row format conversion in `DataFrame.countMinSketch()`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10968 from liancheng/cms-specialized.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/415d0a859b7a76f3a866ec62ab472c4050f2a01b"},{"sha":"676803963fcc08aa988aa6f14be3751314e006ca","author":{"email":"6ea7d3fc9c1075bfb0a62c61a167edd3b77f94cd@fb.com","name":"Tejas Patil"},"message":"[SPARK-12926][SQL] SQLContext to display warning message when non-sql configs are being set\n\nUsers unknowingly try to set core Spark configs in SQLContext but later realise that it didn't work. eg. sqlContext.sql(\"SET spark.shuffle.memoryFraction=0.4\"). This PR adds a warning message when such operations are done.\n\nAuthor: Tejas Patil <tejasp@fb.com>\n\nCloses #10849 from tejasapatil/SPARK-12926.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/676803963fcc08aa988aa6f14be3751314e006ca"},{"sha":"cc18a7199240bf3b03410c1ba6704fe7ce6ae38e","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13031] [SQL] cleanup codegen and improve test coverage\n\n1. enable whole stage codegen during tests even there is only one operator supports that.\n2. split doProduce() into two APIs: upstream() and doProduce()\n3. generate prefix for fresh names of each operator\n4. pass UnsafeRow to parent directly (avoid getters and create UnsafeRow again)\n5. fix bugs and tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10944 from davies/gen_refactor.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/cc18a7199240bf3b03410c1ba6704fe7ce6ae38e"},{"sha":"df78a934a07a4ce5af43243be9ba5fe60b91eee6","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9835][ML] Implement IterativelyReweightedLeastSquares solver\n\nImplement ```IterativelyReweightedLeastSquares``` solver for GLM. I consider it as a solver rather than estimator, it only used internal so I keep it ```private[ml]```.\nThere are two limitations in the current implementation compared with R:\n* It can not support ```Tuple``` as response for ```Binomial``` family, such as the following code:\n```\nglm( cbind(using, notUsing) ~  age + education + wantsMore , family = binomial)\n```\n* It does not support ```offset```.\n\nBecause I considered that ```RFormula``` did not support ```Tuple``` as label and ```offset``` keyword, so I simplified the implementation. But to add support for these two functions is not very hard, I can do it in follow-up PR if it is necessary. Meanwhile, we can also add R-like statistic summary for IRLS.\nThe implementation refers R, [statsmodels](https://github.com/statsmodels/statsmodels) and [sparkGLM](https://github.com/AlteryxLabs/sparkGLM).\nPlease focus on the main structure and overpass minor issues/docs that I will update later. Any comments and opinions will be appreciated.\n\ncc mengxr jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10639 from yanboliang/spark-9835.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/df78a934a07a4ce5af43243be9ba5fe60b91eee6"},{"sha":"abae889f08eb412cb897e4e63614ec2c93885ffd","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-12401][SQL] Add integration tests for postgres enum types\n\nWe can handle posgresql-specific enum types as strings in jdbc.\nSo, we should just add tests and close the corresponding JIRA ticket.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #10596 from maropu/AddTestsInIntegration.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/abae889f08eb412cb897e4e63614ec2c93885ffd"},{"sha":"3a40c0e575fd4215302ea60c9821d31a5a138b8a","author":{"email":"48125774e1ba05a013278c93767e0e53161dec0b@gmail.com","name":"Brandon Bradley"},"message":"[SPARK-12749][SQL] add json option to parse floating-point types as DecimalType\n\nI tried to add this via `USE_BIG_DECIMAL_FOR_FLOATS` option from Jackson with no success.\n\nAdded test for non-complex types. Should I add a test for complex types?\n\nAuthor: Brandon Bradley <bradleytastic@gmail.com>\n\nCloses #10936 from blbradley/spark-12749.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/3a40c0e575fd4215302ea60c9821d31a5a138b8a"},{"sha":"4637fc08a3733ec313218fb7e4d05064d9a6262d","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11955][SQL] Mark optional fields in merging schema for safely pushdowning filters in Parquet\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-11955\n\nCurrently we simply skip pushdowning filters in parquet if we enable schema merging.\n\nHowever, we can actually mark particular fields in merging schema for safely pushdowning filters in parquet.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #9940 from viirya/safe-pushdown-parquet-filters.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/4637fc08a3733ec313218fb7e4d05064d9a6262d"},{"sha":"b9dfdcc63bb12bc24de96060e756889c2ceda519","author":{"email":"399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","name":"Davies Liu"},"message":"Revert \"[SPARK-13031] [SQL] cleanup codegen and improve test coverage\"\n\nThis reverts commit cc18a7199240bf3b03410c1ba6704fe7ce6ae38e.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/b9dfdcc63bb12bc24de96060e756889c2ceda519"},{"sha":"66449b8dcdbc3dca126c34b42c4d0419c7648696","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12968][SQL] Implement command to set current database\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12968\n\nImplement command to set current database.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #10916 from viirya/ddl-use-database.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/66449b8dcdbc3dca126c34b42c4d0419c7648696"},{"sha":"721ced28b522cc00b45ca7fa32a99e80ad3de2f7","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13067] [SQL] workaround for a weird scala reflection problem\n\nA simple workaround to avoid getting parameter types when convert a\nlogical plan to json.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10970 from cloud-fan/reflection.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/721ced28b522cc00b45ca7fa32a99e80ad3de2f7"},{"sha":"8d3cc3de7d116190911e7943ef3233fe3b7db1bf","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"[SPARK-13050][BUILD] Scalatest tags fail build with the addition of the sketch module\n\nA dependency on the spark test tags was left out of the sketch module pom file causing builds to fail when test tags were used. This dependency is found in the pom file for every other module in spark.\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #10954 from ajbozarth/spark13050.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/8d3cc3de7d116190911e7943ef3233fe3b7db1bf"},{"sha":"55561e7693dd2a5bf3c7f8026c725421801fd0ec","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13031][SQL] cleanup codegen and improve test coverage\n\n1. enable whole stage codegen during tests even there is only one operator supports that.\n2. split doProduce() into two APIs: upstream() and doProduce()\n3. generate prefix for fresh names of each operator\n4. pass UnsafeRow to parent directly (avoid getters and create UnsafeRow again)\n5. fix bugs and tests.\n\nThis PR re-open #10944 and fix the bug.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10977 from davies/gen_refactor.","distinct":true,"url":"https://api.github.com/repos/fanlu/spark/commits/55561e7693dd2a5bf3c7f8026c725421801fd0ec"}]}