{"push_id":1394906429,"size":105,"distinct_size":105,"ref":"refs/heads/master","head":"d479c5262276b47302659bd877a9e3467400bdb6","before":"04d417a7ca8ef694658b26fb697a035717414731","commits":[{"sha":"84f149e414475c2e60863898992001c21cfc13b2","author":{"name":"Pete Robbins","email":"65c4535b59e9806d6fbe65e3ae9fd980a3fbb159@gmail.com"},"message":"[SPARK-17827][SQL] maxColLength type should be Int for String and Binary\n\n## What changes were proposed in this pull request?\ncorrect the expected type from Length function to be Int\n\n## How was this patch tested?\nTest runs on little endian and big endian platforms\n\nAuthor: Pete Robbins <robbinspg@gmail.com>\n\nCloses #15464 from robbinspg/SPARK-17827.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/84f149e414475c2e60863898992001c21cfc13b2"},{"sha":"08eac356095c7faa2b19d52f2fb0cbc47eb7d1d1","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17834][SQL] Fetch the earliest offsets manually in KafkaSource instead of counting on KafkaConsumer\n\n## What changes were proposed in this pull request?\n\nBecause `KafkaConsumer.poll(0)` may update the partition offsets, this PR just calls `seekToBeginning` to manually set the earliest offsets for the KafkaSource initial offsets.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15397 from zsxwing/SPARK-17834.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/08eac356095c7faa2b19d52f2fb0cbc47eb7d1d1"},{"sha":"7106866c220c73960c6fe2a70e4911516617e21f","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-17731][SQL][STREAMING] Metrics for structured streaming\n\n## What changes were proposed in this pull request?\n\nMetrics are needed for monitoring structured streaming apps. Here is the design doc for implementing the necessary metrics.\nhttps://docs.google.com/document/d/1NIdcGuR1B3WIe8t7VxLrt58TJB4DtipWEbj5I_mzJys/edit?usp=sharing\n\nSpecifically, this PR adds the following public APIs changes.\n\n### New APIs\n- `StreamingQuery.status` returns a `StreamingQueryStatus` object (renamed from `StreamingQueryInfo`, see later)\n\n- `StreamingQueryStatus` has the following important fields\n  - inputRate - Current rate (rows/sec) at which data is being generated by all the sources\n  - processingRate - Current rate (rows/sec) at which the query is processing data from\n                                  all the sources\n  - ~~outputRate~~ - *Does not work with wholestage codegen*\n  - latency - Current average latency between the data being available in source and the sink writing the corresponding output\n  - sourceStatuses: Array[SourceStatus] - Current statuses of the sources\n  - sinkStatus: SinkStatus - Current status of the sink\n  - triggerStatus - Low-level detailed status of the last completed/currently active trigger\n    - latencies - getOffset, getBatch, full trigger, wal writes\n    - timestamps - trigger start, finish, after getOffset, after getBatch\n    - numRows - input, output, state total/updated rows for aggregations\n\n- `SourceStatus` has the following important fields\n  - inputRate - Current rate (rows/sec) at which data is being generated by the source\n  - processingRate - Current rate (rows/sec) at which the query is processing data from the source\n  - triggerStatus - Low-level detailed status of the last completed/currently active trigger\n\n- Python API for `StreamingQuery.status()`\n\n### Breaking changes to existing APIs\n**Existing direct public facing APIs**\n- Deprecated direct public-facing APIs `StreamingQuery.sourceStatuses` and `StreamingQuery.sinkStatus` in favour of `StreamingQuery.status.sourceStatuses/sinkStatus`.\n  - Branch 2.0 should have it deprecated, master should have it removed.\n\n**Existing advanced listener APIs**\n- `StreamingQueryInfo` renamed to `StreamingQueryStatus` for consistency with `SourceStatus`, `SinkStatus`\n   - Earlier StreamingQueryInfo was used only in the advanced listener API, but now it is used in direct public-facing API (StreamingQuery.status)\n\n- Field `queryInfo` in listener events `QueryStarted`, `QueryProgress`, `QueryTerminated` changed have name `queryStatus` and return type `StreamingQueryStatus`.\n\n- Field `offsetDesc` in `SourceStatus` was Option[String], converted it to `String`.\n\n- For `SourceStatus` and `SinkStatus` made constructor private instead of private[sql] to make them more java-safe. Instead added `private[sql] object SourceStatus/SinkStatus.apply()` which are harder to accidentally use in Java.\n\n## How was this patch tested?\n\nOld and new unit tests.\n- Rate calculation and other internal logic of StreamMetrics tested by StreamMetricsSuite.\n- New info in statuses returned through StreamingQueryListener is tested in StreamingQueryListenerSuite.\n- New and old info returned through StreamingQuery.status is tested in StreamingQuerySuite.\n- Source-specific tests for making sure input rows are counted are is source-specific test suites.\n- Additional tests to test minor additions in LocalTableScanExec, StateStore, etc.\n\nMetrics also manually tested using Ganglia sink\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #15307 from tdas/SPARK-17731.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/7106866c220c73960c6fe2a70e4911516617e21f"},{"sha":"adc112429d6fe671e6e8294824a0e41a2b1ec2e0","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17661][SQL] Consolidate various listLeafFiles implementations\n\n## What changes were proposed in this pull request?\nThere are 4 listLeafFiles-related functions in Spark:\n\n- ListingFileCatalog.listLeafFiles (which calls HadoopFsRelation.listLeafFilesInParallel if the number of paths passed in is greater than a threshold; if it is lower, then it has its own serial version implemented)\n- HadoopFsRelation.listLeafFiles (called only by HadoopFsRelation.listLeafFilesInParallel)\n- HadoopFsRelation.listLeafFilesInParallel (called only by ListingFileCatalog.listLeafFiles)\n\nIt is actually very confusing and error prone because there are effectively two distinct implementations for the serial version of listing leaf files. As an example, SPARK-17599 updated only one of the code path and ignored the other one.\n\nThis code can be improved by:\n\n- Move all file listing code into ListingFileCatalog, since it is the only class that needs this.\n- Keep only one function for listing files in serial.\n\n## How was this patch tested?\nThis change should be covered by existing unit and integration tests. I also moved a test case for HadoopFsRelation.shouldFilterOut from HadoopFsRelationSuite to ListingFileCatalogSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #15235 from petermaxlee/SPARK-17661.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/adc112429d6fe671e6e8294824a0e41a2b1ec2e0"},{"sha":"9dc0ca060d5925cd666b34021e62f7b38bb3aabb","author":{"name":"Jakob Odersky","email":"17d3c251b013cdb52ecefc0d9ef1a254297af19e@odersky.com"},"message":"[SPARK-17368][SQL] Add support for value class serialization and deserialization\n\n## What changes were proposed in this pull request?\nValue classes were unsupported because catalyst data types were\nobtained through reflection on erased types, which would resolve to a\nvalue class' wrapped type and hence lead to unavailable methods during\ncode generation.\n\nE.g. the following class\n```scala\ncase class Foo(x: Int) extends AnyVal\n```\nwould be seen as an `int` in catalyst and will cause instance cast failures when generated java code tries to treat it as a `Foo`.\n\nThis patch simply removes the erasure step when getting data types for\ncatalyst.\n\n## How was this patch tested?\nAdditional tests in `ExpressionEncoderSuite`.\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #15284 from jodersky/value-classes.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/9dc0ca060d5925cd666b34021e62f7b38bb3aabb"},{"sha":"44cbb61b34a98e3e0d8e2543a4eb6e950e0019a5","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-15957][FOLLOW-UP][ML][PYSPARK] Add Python API for RFormula forceIndexLabel.\n\n## What changes were proposed in this pull request?\nFollow-up work of #13675, add Python API for ```RFormula forceIndexLabel```.\n\n## How was this patch tested?\nUnit test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15430 from yanboliang/spark-15957-python.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/44cbb61b34a98e3e0d8e2543a4eb6e950e0019a5"},{"sha":"8543996c3f44098a521fc6b90ca0bb575f606e2a","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17927][SQL] Remove dead code in WriterContainer.\n\n## What changes were proposed in this pull request?\nspeculationEnabled and DATASOURCE_OUTPUTPATH seem like just dead code.\n\n## How was this patch tested?\nTests should fail if they are not dead code.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15477 from rxin/SPARK-17927.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/8543996c3f44098a521fc6b90ca0bb575f606e2a"},{"sha":"6c29b3de763115d8676ed91f896e75c490e8c5b2","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17925][SQL] Break fileSourceInterfaces.scala into multiple pieces\n\n## What changes were proposed in this pull request?\nThis patch does a few changes to the file structure of data sources:\n\n- Break fileSourceInterfaces.scala into multiple pieces (HadoopFsRelation, FileFormat, OutputWriter)\n- Move ParquetOutputWriter into its own file\n\nI created this as a separate patch so it'd be easier to review my future PRs that focus on refactoring this internal logic. This patch only moves code around, and has no logic changes.\n\n## How was this patch tested?\nN/A - should be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15473 from rxin/SPARK-17925.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/6c29b3de763115d8676ed91f896e75c490e8c5b2"},{"sha":"2fb12b0a33deeeadfac451095f64dea6c967caac","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17903][SQL] MetastoreRelation should talk to external catalog instead of hive client\n\n## What changes were proposed in this pull request?\n\n`HiveExternalCatalog` should be the only interface to talk to the hive metastore. In `MetastoreRelation` we can just use `ExternalCatalog` instead of `HiveClient` to interact with hive metastore,  and add missing API in `ExternalCatalog`.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15460 from cloud-fan/relation.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/2fb12b0a33deeeadfac451095f64dea6c967caac"},{"sha":"1db8feab8c564053c05e8bdc1a7f5026fd637d4f","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-15402][ML][PYSPARK] PySpark ml.evaluation should support save/load\n\n## What changes were proposed in this pull request?\nSince ```ml.evaluation``` has supported save/load at Scala side, supporting it at Python side is very straightforward and easy.\n\n## How was this patch tested?\nAdd python doctest.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13194 from yanboliang/spark-15402.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/1db8feab8c564053c05e8bdc1a7f5026fd637d4f"},{"sha":"a1b136d05c6c458ae8211b0844bfc98d7693fa42","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-14634][ML] Add BisectingKMeansSummary\n\n## What changes were proposed in this pull request?\nAdd BisectingKMeansSummary\n\n## How was this patch tested?\nunit test\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #12394 from zhengruifeng/biKMSummary.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/a1b136d05c6c458ae8211b0844bfc98d7693fa42"},{"sha":"c8b612decba28e51789891f7881b6d4ebc50e2bb","author":{"name":"Peng","email":"6c4c4b336ba0af8858d16e0da367ee314642f197@intel.com"},"message":"[SPARK-17870][MLLIB][ML] Change statistic to pValue for SelectKBest and SelectPercentile because of DoF difference\n\n## What changes were proposed in this pull request?\n\nFor feature selection method ChiSquareSelector, it is based on the ChiSquareTestResult.statistic (ChiSqure value) to select the features. It select the features with the largest ChiSqure value. But the Degree of Freedom (df) of ChiSqure value is different in Statistics.chiSqTest(RDD), and for different df, you cannot base on ChiSqure value to select features.\n\nSo we change statistic to pValue for SelectKBest and SelectPercentile\n\n## How was this patch tested?\nchange existing test\n\nAuthor: Peng <peng.meng@intel.com>\n\nCloses #15444 from mpjlu/chisqure-bug.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/c8b612decba28e51789891f7881b6d4ebc50e2bb"},{"sha":"28b645b1e643ae0f6c56cbe5a92356623306717f","author":{"name":"invkrh","email":"c17e4d6978edeb93e62587309ee6cf8b6ccede4c@gmail.com"},"message":"[SPARK-17855][CORE] Remove query string from jar url\n\n## What changes were proposed in this pull request?\n\nSpark-submit support jar url with http protocol. However, if the url contains any query strings, `worker.DriverRunner.downloadUserJar()` method will throw \"Did not see expected jar\" exception. This is because this method checks the existance of a downloaded jar whose name contains query strings. This is a problem when your jar is located on some web service which requires some additional information to retrieve the file.\n\nThis pr just removes query strings before checking jar existance on worker.\n\n## How was this patch tested?\n\nFor now, you can only test this patch by manual test.\n* Deploy a spark cluster locally\n* Make sure apache httpd service is on\n* Save an uber jar, e.g spark-job.jar under `/var/www/html/`\n* Use http://localhost/spark-job.jar?param=1 as jar url when running `spark-submit`\n* Job should be launched\n\nAuthor: invkrh <invkrh@gmail.com>\n\nCloses #15420 from invkrh/spark-17855.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/28b645b1e643ae0f6c56cbe5a92356623306717f"},{"sha":"7486442fe0b70f2aea21d569604e71d7ddf19a77","author":{"name":"wangzhenhua","email":"93943f521ced4ea3f21d64a737435e7a7e171ac6@huawei.com"},"message":"[SPARK-17073][SQL][FOLLOWUP] generate column-level statistics\n\n## What changes were proposed in this pull request?\nThis pr adds some test cases for statistics: case sensitive column names, non ascii column names, refresh table, and also improves some documentation.\n\n## How was this patch tested?\nadd test cases\n\nAuthor: wangzhenhua <wangzhenhua@huawei.com>\n\nCloses #15360 from wzhfy/colStats2.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/7486442fe0b70f2aea21d569604e71d7ddf19a77"},{"sha":"a0ebcb3a30ec64e01608ed6fa7b7ffb7acbd3af2","author":{"name":"Dhruve Ashar","email":"27f589a372d341e853c1e390c399cb5de87102c3@gmail.com"},"message":"[DOC] Fix typo in sql hive doc\n\nChange is too trivial to file a JIRA.\n\nAuthor: Dhruve Ashar <dhruveashar@gmail.com>\n\nCloses #15485 from dhruve/master.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/a0ebcb3a30ec64e01608ed6fa7b7ffb7acbd3af2"},{"sha":"fa37877af02a956203e8a00811b20f34af0278f7","author":{"name":"Andrew Ash","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@andrewash.com"},"message":"Typo: form -> from\n\n## What changes were proposed in this pull request?\n\nMinor typo fix\n\n## How was this patch tested?\n\nExisting unit tests on Jenkins\n\nAuthor: Andrew Ash <andrew@andrewash.com>\n\nCloses #15486 from ash211/patch-8.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/fa37877af02a956203e8a00811b20f34af0278f7"},{"sha":"05800b4b4e7873ebc445dfcd020b76d7539686e1","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[TEST] Ignore flaky test in StreamingQueryListenerSuite\n\n## What changes were proposed in this pull request?\n\nIgnoring the flaky test introduced in #15307\n\nhttps://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/1736/testReport/junit/org.apache.spark.sql.streaming/StreamingQueryListenerSuite/single_listener__check_trigger_statuses/\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #15491 from tdas/metrics-flaky-test.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/05800b4b4e7873ebc445dfcd020b76d7539686e1"},{"sha":"de1c1ca5c9d6064d3b7b3711e3bfb08fa018abe8","author":{"name":"sethah","email":"c41626c9d885a60b8db6c40a26d851740b23beef@gmail.com"},"message":"[SPARK-17941][ML][TEST] Logistic regression tests should use sample weights.\n\n## What changes were proposed in this pull request?\n\nThe sample weight testing for logistic regressions is not robust. Logistic regression suite already has many test cases comparing results to R glmnet. Since both libraries support sample weights, we should use sample weights in the test to increase coverage for sample weighting. This patch doesn't really add any code and makes the testing more complete.\n\nAlso fixed some errors with the R code that was referenced in the test suit. Changed `standardization=T` to `standardize=T` since the former is invalid.\n\n## How was this patch tested?\n\nExisting unit tests are modified. No non-test code is touched.\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #15488 from sethah/logreg_weight_tests.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/de1c1ca5c9d6064d3b7b3711e3bfb08fa018abe8"},{"sha":"7ab86244e30ca81eb4fa40ea77b4c2b8881cbab2","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables\n\n## What changes were proposed in this pull request?\nMake sure the hive.default.fileformat is used to when creating the storage format metadata.\n\nOutput\n``` SQL\nscala> spark.sql(\"SET hive.default.fileformat=orc\")\nres1: org.apache.spark.sql.DataFrame = [key: string, value: string]\n\nscala> spark.sql(\"CREATE TABLE tmp_default(id INT)\")\nres2: org.apache.spark.sql.DataFrame = []\n```\nBefore\n```SQL\nscala> spark.sql(\"DESC FORMATTED tmp_default\").collect.foreach(println)\n..\n[# Storage Information,,]\n[SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,]\n[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]\n[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]\n[Compressed:,No,]\n[Storage Desc Parameters:,,]\n[  serialization.format,1,]\n```\nAfter\n```SQL\nscala> spark.sql(\"DESC FORMATTED tmp_default\").collect.foreach(println)\n..\n[# Storage Information,,]\n[SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,]\n[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]\n[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]\n[Compressed:,No,]\n[Storage Desc Parameters:,,]\n[  serialization.format,1,]\n\n```\n\n## How was this patch tested?\nAdded new tests to HiveDDLCommandSuite\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #15190 from dilipbiswal/orc.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/7ab86244e30ca81eb4fa40ea77b4c2b8881cbab2"},{"sha":"522dd0d0e5af83e45a3c3526c191aa4b8bcaeeeb","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"Revert \"[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables\"\n\nThis reverts commit 7ab86244e30ca81eb4fa40ea77b4c2b8881cbab2.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/522dd0d0e5af83e45a3c3526c191aa4b8bcaeeeb"}]}
{"actor":{"display_login":"WangTaoTheTonic"}}