{"push_id":1299034465,"size":96,"distinct_size":6,"ref":"refs/heads/serde-2.0","head":"df83c2a2ed529e206f2c91bb3d50ed71891cc635","before":"8f979838ab40e24764ff278cb1e65b9a0f2de9db","commits":[{"sha":"b3a44306a36d6c1e5583e85961966fa5cf4f7e9a","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARKR][BUILD] ignore cran-check.out under R folder\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\nR add cran check which will generate the cran-check.out. This file should be ignored in git.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\nManual test it. Run clean test and git status to make sure the file is not included in git.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #14774 from wangmiao1981/ignore.\n\n(cherry picked from commit 9958ac0ce2b9e451d400604767bef2fe12a3399d)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/b3a44306a36d6c1e5583e85961966fa5cf4f7e9a"},{"sha":"ff2e270ebe3a74c19140cd96f96b7a62723002b1","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17205] Literal.sql should handle Infinity and NaN\n\nThis patch updates `Literal.sql` to properly generate SQL for `NaN` and `Infinity` float and double literals: these special values need to be handled differently from regular values, since simply appending a suffix to the value's `toString()` representation will not work for these values.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14777 from JoshRosen/SPARK-17205.\n\n(cherry picked from commit 3e4c7db4d11c474457e7886a5501108ebab0cf6d)\nSigned-off-by: Herman van Hovell <hvanhovell@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/ff2e270ebe3a74c19140cd96f96b7a62723002b1"},{"sha":"73014a2aa96b538d963f360fd41bac74f358ef46","author":{"name":"Michael Allman","email":"844aface8de49b32585c3c744b570659f1739c56@videoamp.com"},"message":"[SPARK-17231][CORE] Avoid building debug or trace log messages unless the respective log level is enabled\n\nThis is simply a backport of #14798 to `branch-2.0`. This backport omits the change to `ExternalShuffleBlockHandler.java`. In `branch-2.0`, that file does not contain the log message that was patched in `master`.\n\nAuthor: Michael Allman <michael@videoamp.com>\n\nCloses #14811 from mallman/spark-17231-logging_perf_improvements-2.0_backport.","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/73014a2aa96b538d963f360fd41bac74f358ef46"},{"sha":"27ed6d5dcd521b4ff1ebe777b03a03ba103d6e76","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17242][DOCUMENT] Update links of external dstream projects\n\n## What changes were proposed in this pull request?\n\nUpdated links of external dstream projects.\n\n## How was this patch tested?\n\nJust document changes.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14814 from zsxwing/dstream-link.\n\n(cherry picked from commit 341e0e778dff8c404b47d34ee7661b658bb91880)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/27ed6d5dcd521b4ff1ebe777b03a03ba103d6e76"},{"sha":"6f82d2da382cee2950a0797436e5d48805cbba5f","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARKR][MINOR] Fix example of spark.naiveBayes\n\n## What changes were proposed in this pull request?\n\nThe original example doesn't work because the features are not categorical. This PR fixes this by changing to another dataset.\n\n## How was this patch tested?\n\nManual test.\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #14820 from junyangq/SPARK-FixNaiveBayes.\n\n(cherry picked from commit 18832162357282ec81515b5b2ba93747be3ad18b)\nSigned-off-by: Felix Cheung <felixcheung@apache.org>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/6f82d2da382cee2950a0797436e5d48805cbba5f"},{"sha":"deb6a54cf0f69d4ac5b3e1d358bb81e49eea412d","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17165][SQL] FileStreamSource should not track the list of seen files indefinitely\n\n## What changes were proposed in this pull request?\nBefore this change, FileStreamSource uses an in-memory hash set to track the list of files processed by the engine. The list can grow indefinitely, leading to OOM or overflow of the hash set.\n\nThis patch introduces a new user-defined option called \"maxFileAge\", default to 24 hours. If a file is older than this age, FileStreamSource will purge it from the in-memory map that was used to track the list of files that have been processed.\n\n## How was this patch tested?\nAdded unit tests for the underlying utility, and also added an end-to-end test to validate the purge in FileStreamSourceSuite. Also verified the new test cases would fail when the timeout was set to a very large number.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14728 from petermaxlee/SPARK-17165.\n\n(cherry picked from commit 9812f7d5381f7cd8112fd30c7e45ae4f0eab6e88)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/deb6a54cf0f69d4ac5b3e1d358bb81e49eea412d"},{"sha":"52feb3fbf75a234d041703e3ac41884294ab0b64","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-17246][SQL] Add BigDecimal literal\n\n## What changes were proposed in this pull request?\nThis PR adds parser support for `BigDecimal` literals. If you append the suffix `BD` to a valid number then this will be interpreted as a `BigDecimal`, for example `12.0E10BD` will interpreted into a BigDecimal with scale -9 and precision 3. This is useful in situations where you need exact values.\n\n## How was this patch tested?\nAdded tests to `ExpressionParserSuite`, `ExpressionSQLBuilderSuite` and `SQLQueryTestSuite`.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #14819 from hvanhovell/SPARK-17246.\n\n(cherry picked from commit a11d10f1826b578ff721c4738224eef2b3c3b9f3)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/52feb3fbf75a234d041703e3ac41884294ab0b64"},{"sha":"dfdfc3092d1b6942eb9092e28e15fa4efb6ac084","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17235][SQL] Support purging of old logs in MetadataLog\n\n## What changes were proposed in this pull request?\nThis patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge function is currently unused, but I will use it to purge old execution and file source logs in follow-up patches. These changes are required in a production structured streaming job that runs for a long period of time.\n\n## How was this patch tested?\nAdded a unit test case in HDFSMetadataLogSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14802 from petermaxlee/SPARK-17235.\n\n(cherry picked from commit f64a1ddd09a34d5d867ccbaba46204d75fad038d)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/dfdfc3092d1b6942eb9092e28e15fa4efb6ac084"},{"sha":"9c0ac6b53abeb7b5f6fa2e8328d7559be8259300","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@cs.berkeley.edu"},"message":"[SPARK-17244] Catalyst should not pushdown non-deterministic join conditions\n\n## What changes were proposed in this pull request?\n\nGiven that non-deterministic expressions can be stateful, pushing them down the query plan during the optimization phase can cause incorrect behavior. This patch fixes that issue by explicitly disabling that.\n\n## How was this patch tested?\n\nA new test in `FilterPushdownSuite` that checks catalyst behavior for both deterministic and non-deterministic join conditions.\n\nAuthor: Sameer Agarwal <sameerag@cs.berkeley.edu>\n\nCloses #14815 from sameeragarwal/constraint-inputfile.\n\n(cherry picked from commit 540e91280147a61727f99592a66c0cbb12328fac)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/9c0ac6b53abeb7b5f6fa2e8328d7559be8259300"},{"sha":"94d52d76569f8b0782f424cfac959a4bb75c54c0","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17269][SQL] Move finish analysis optimization stage into its own file\n\nAs part of breaking Optimizer.scala apart, this patch moves various finish analysis optimization stage rules into a single file. I'm submitting separate pull requests so we can more easily merge this in branch-2.0 to simplify optimizer backports.\n\nThis should be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14838 from rxin/SPARK-17269.\n\n(cherry picked from commit dcefac438788c51d84641bfbc505efe095731a39)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/94d52d76569f8b0782f424cfac959a4bb75c54c0"},{"sha":"f91614f36472957355fad7d69d66327807fe80c8","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17270][SQL] Move object optimization rules into its own file (branch-2.0)\n\n## What changes were proposed in this pull request?\nAs part of breaking Optimizer.scala apart, this patch moves various Dataset object optimization rules into a single file. I'm submitting separate pull requests so we can more easily merge this in branch-2.0 to simplify optimizer backports.\n\nThis is https://github.com/apache/spark/pull/14839 but for branch-2.0.\n\n## How was this patch tested?\nThis should be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14843 from rxin/SPARK-17270-branch-2.0.","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/f91614f36472957355fad7d69d66327807fe80c8"},{"sha":"901ab06949addd05be6cb85df4eb6bd2104777e8","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17274][SQL] Move join optimizer rules into a separate file\n\n## What changes were proposed in this pull request?\nAs part of breaking Optimizer.scala apart, this patch moves various join rules into a single file.\n\n## How was this patch tested?\nThis should be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14846 from rxin/SPARK-17274.\n\n(cherry picked from commit 718b6bad2d698b76be6906d51da13626e9f3890e)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/901ab06949addd05be6cb85df4eb6bd2104777e8"},{"sha":"56a8426355b494bc085b649ae6c8245f0a039e3a","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-15382][SQL] Fix a bug in sampling with replacement\n\n## What changes were proposed in this pull request?\nThis pr to fix a bug below in sampling with replacement\n```\nval df = Seq((1, 0), (2, 0), (3, 0)).toDF(\"a\", \"b\")\ndf.sample(true, 2.0).withColumn(\"c\", monotonically_increasing_id).select($\"c\").show\n+---+\n|  c|\n+---+\n|  0|\n|  1|\n|  1|\n|  1|\n|  2|\n+---+\n```\n\n## How was this patch tested?\nAdded a test in `DataFrameSuite`.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #14800 from maropu/FixSampleBug.\n\n(cherry picked from commit cd0ed31ea9965563a9b1ea3e8bfbeaf8347cacd9)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/56a8426355b494bc085b649ae6c8245f0a039e3a"},{"sha":"7306c5f9098a25a7253b3b6fc27badb7d96b18d7","author":{"name":"Peng, Meng","email":"6c4c4b336ba0af8858d16e0da367ee314642f197@intel.com"},"message":"[ML][MLLIB] The require condition and message doesn't match in SparseMatrix.\n\n## What changes were proposed in this pull request?\nThe require condition and message doesn't match, and the condition also should be optimized.\nSmall change.  Please kindly let me know if JIRA required.\n\n## How was this patch tested?\nNo additional test required.\n\nAuthor: Peng, Meng <peng.meng@intel.com>\n\nCloses #14824 from mpjlu/smallChangeForMatrixRequire.\n\n(cherry picked from commit 40168dbe771ae662ed61851a1f3c677dd14fe344)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/7306c5f9098a25a7253b3b6fc27badb7d96b18d7"},{"sha":"5487fa0b8708deb31a872a943a6cccd6e922c575","author":{"name":"Robert Kruszewski","email":"88997ef7155e66173e74033f298d585f8feeac13@palantir.com"},"message":"[SPARK-17216][UI] fix event timeline bars length\n\n## What changes were proposed in this pull request?\n\nMake event timeline bar expand to full length of the bar (which is total time)\n\nThis issue occurs only on chrome, firefox looks fine. Haven't tested other browsers.\n\n## How was this patch tested?\nInspection in browsers\n\nBefore\n![screen shot 2016-08-24 at 3 38 24 pm](https://cloud.githubusercontent.com/assets/512084/17935104/0d6cda74-6a12-11e6-9c66-e00cfa855606.png)\n\nAfter\n![screen shot 2016-08-24 at 3 36 39 pm](https://cloud.githubusercontent.com/assets/512084/17935114/15740ea4-6a12-11e6-83a1-7c06eef6abb8.png)\n\nAuthor: Robert Kruszewski <robertk@palantir.com>\n\nCloses #14791 from robert3005/robertk/event-timeline.\n\n(cherry picked from commit 9fbced5b25c2f24d50c50516b4b7737f7e3eaf86)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/5487fa0b8708deb31a872a943a6cccd6e922c575"},{"sha":"eec03718db7e412f466ea72f3d9c2682915080bc","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16216][SQL][FOLLOWUP][BRANCH-2.0] Bacoport enabling timestamp type tests for JSON and verify all unsupported types in CSV\n\n## What changes were proposed in this pull request?\n\nThis backports https://github.com/apache/spark/pull/14829\n\n## How was this patch tested?\n\nTests in `JsonHadoopFsRelation` and `CSVSuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14840 from HyukjinKwon/SPARK-16216-followup-backport.","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/eec03718db7e412f466ea72f3d9c2682915080bc"},{"sha":"3d283f6c9d9daef53fa4e90b0ead2a94710a37a7","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-17063] [SQL] Improve performance of MSCK REPAIR TABLE with Hive metastore\n\nThis PR split the the single `createPartitions()` call into smaller batches, which could prevent Hive metastore from OOM (caused by millions of partitions).\n\nIt will also try to gather all the fast stats (number of files and total size of all files) in parallel to avoid the bottle neck of listing the files in metastore sequential, which is controlled by spark.sql.gatherFastStats (enabled by default).\n\nTested locally with 10000 partitions and 100 files with embedded metastore, without gathering fast stats in parallel, adding partitions took 153 seconds, after enable that, gathering the fast stats took about 34 seconds, adding these partitions took 25 seconds (most of the time spent in object store), 59 seconds in total, 2.5X faster (with larger cluster, gathering will much faster).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #14607 from davies/repair_batch.\n\n(cherry picked from commit 48caec2516ef35bfa1a3de2dc0a80d0dc819e6bd)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/3d283f6c9d9daef53fa4e90b0ead2a94710a37a7"},{"sha":"976a43dbf9d97b30d81576799470532b81b882f0","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"[SPARK-16581][SPARKR] Make JVM backend calling functions public\n\n## What changes were proposed in this pull request?\n\nThis change exposes a public API in SparkR to create objects, call methods on the Spark driver JVM\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\nUnit tests, CRAN checks\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #14775 from shivaram/sparkr-java-api.\n\n(cherry picked from commit 736a7911cb0335cdb2b2f6c87f9e3c32047b5bbb)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/976a43dbf9d97b30d81576799470532b81b882f0"},{"sha":"59032570fbd0985f758c27bdec5482221cc64af9","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17301][SQL] Remove unused classTag field from AtomicType base class\n\nThere's an unused `classTag` val in the AtomicType base class which is causing unnecessary slowness in deserialization because it needs to grab ScalaReflectionLock and create a new runtime reflection mirror. Removing this unused code gives a small but measurable performance boost in SQL task deserialization.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14869 from JoshRosen/remove-unused-classtag.\n\n(cherry picked from commit 48b459ddd58affd5519856cb6e204398b7739a2a)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/59032570fbd0985f758c27bdec5482221cc64af9"},{"sha":"f35b10ab1556e3ea76ce2322af7b6749b7f1357f","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-17264][SQL] DataStreamWriter should document that it only supports Parquet for now\n\n## What changes were proposed in this pull request?\n\nClarify that only parquet files are supported by DataStreamWriter now\n\n## How was this patch tested?\n\n(Doc build -- no functional changes to test)\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14860 from srowen/SPARK-17264.\n\n(cherry picked from commit befab9c1c6b59ad90f63a7d10e12b186be897f15)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/epahomov/spark/commits/f35b10ab1556e3ea76ce2322af7b6749b7f1357f"}]}
{"actor":{"display_login":"epahomov"}}