{"push_id":1289690873,"size":686,"distinct_size":686,"ref":"refs/heads/master","head":"78d5d4dd5ce5a537ed04cd1bf242c9e9ea2c391a","before":"5344bade8efb6f12aa43fbfbbbc2e3c0c7d16d98","commits":[{"sha":"07f46afc733b1718d528a6ea5c0d774f047024fa","author":{"name":"Sital Kedia","email":"e4343f82e25883a7c046475fe9553138fac5aa32@fb.com"},"message":"[SPARK-13850] Force the sorter to Spill when number of elements in th…\n\n## What changes were proposed in this pull request?\n\nForce the sorter to Spill when number of elements in the pointer array reach a certain size. This is to workaround the issue of timSort failing on large buffer size.\n\n## How was this patch tested?\n\nTested by running a job which was failing without this change due to TimSort bug.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #13107 from sitalkedia/fix_TimSort.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/07f46afc733b1718d528a6ea5c0d774f047024fa"},{"sha":"fdf9f94f8c8861a00cd8415073f842b857c397f7","author":{"name":"Imran Rashid","email":"63f21f1cb7b694d385ee53376e18c9c9be6c6f98@cloudera.com"},"message":"[SPARK-15865][CORE] Blacklist should not result in job hanging with less than 4 executors\n\n## What changes were proposed in this pull request?\n\nBefore this change, when you turn on blacklisting with `spark.scheduler.executorTaskBlacklistTime`, but you have fewer than `spark.task.maxFailures` executors, you can end with a job \"hung\" after some task failures.\n\nWhenever a taskset is unable to schedule anything on resourceOfferSingleTaskSet, we check whether the last pending task can be scheduled on *any* known executor.  If not, the taskset (and any corresponding jobs) are failed.\n* Worst case, this is O(maxTaskFailures + numTasks).  But unless many executors are bad, this should be small\n* This does not fail as fast as possible -- when a task becomes unschedulable, we keep scheduling other tasks.  This is to avoid an O(numPendingTasks * numExecutors) operation\n* Also, it is conceivable this fails too quickly.  You may be 1 millisecond away from unblacklisting a place for a task to run, or acquiring a new executor.\n\n## How was this patch tested?\n\nAdded unit test which failed before the change, ran new test 5k times manually, ran all scheduler tests manually, and the full suite via jenkins.\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #13603 from squito/progress_w_few_execs_and_blacklist.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/fdf9f94f8c8861a00cd8415073f842b857c397f7"},{"sha":"46395db80e3304e3f3a1ebdc8aadb8f2819b48b4","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16289][SQL] Implement posexplode table generating function\n\n## What changes were proposed in this pull request?\n\nThis PR implements `posexplode` table generating function. Currently, master branch raises the following exception for `map` argument. It's different from Hive.\n\n**Before**\n```scala\nscala> sql(\"select posexplode(map('a', 1, 'b', 2))\").show\norg.apache.spark.sql.AnalysisException: No handler for Hive UDF ... posexplode() takes an array as a parameter; line 1 pos 7\n```\n\n**After**\n```scala\nscala> sql(\"select posexplode(map('a', 1, 'b', 2))\").show\n+---+---+-----+\n|pos|key|value|\n+---+---+-----+\n|  0|  a|    1|\n|  1|  b|    2|\n+---+---+-----+\n```\n\nFor `array` argument, `after` is the same with `before`.\n```\nscala> sql(\"select posexplode(array(1, 2, 3))\").show\n+---+---+\n|pos|col|\n+---+---+\n|  0|  1|\n|  1|  2|\n|  2|  3|\n+---+---+\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests with newly added testcases.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13971 from dongjoon-hyun/SPARK-16289.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/46395db80e3304e3f3a1ebdc8aadb8f2819b48b4"},{"sha":"c62263340edb6976a10f274e716fde6cd2c5bf34","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-16212][STREAMING][KAFKA] code cleanup from review feedback\n\n## What changes were proposed in this pull request?\ncode cleanup in kafka-0-8 to match suggested changes for kafka-0-10 branch\n\n## How was this patch tested?\nunit tests\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #13908 from koeninger/kafka-0-8-cleanup.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/c62263340edb6976a10f274e716fde6cd2c5bf34"},{"sha":"5d00a7bc19ddeb1b5247733b55095a03ee7b1a30","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-16256][DOCS] Fix window operation diagram\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14001 from tdas/SPARK-16256-2.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/5d00a7bc19ddeb1b5247733b55095a03ee7b1a30"},{"sha":"fb41670c9263a89ec233861cc91a19cf1bb19073","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16336][SQL] Suggest doing table refresh upon FileNotFoundException\n\n## What changes were proposed in this pull request?\nThis patch appends a message to suggest users running refresh table or reloading data frames when Spark sees a FileNotFoundException due to stale, cached metadata.\n\n## How was this patch tested?\nAdded a unit test for this in MetadataCacheSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14003 from petermaxlee/SPARK-16336.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/fb41670c9263a89ec233861cc91a19cf1bb19073"},{"sha":"3d75a5b2a76eba0855d73476dc2fd579c612d521","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16313][SQL] Spark should not silently drop exceptions in file listing\n\n## What changes were proposed in this pull request?\nSpark silently drops exceptions during file listing. This is a very bad behavior because it can mask legitimate errors and the resulting plan will silently have 0 rows. This patch changes it to not silently drop the errors.\n\n## How was this patch tested?\nManually verified.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13987 from rxin/SPARK-16313.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/3d75a5b2a76eba0855d73476dc2fd579c612d521"},{"sha":"85f2303ecadd9bf6d9694a2743dda075654c5ccf","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16276][SQL] Implement elt SQL function\n\n## What changes were proposed in this pull request?\nThis patch implements the elt function, as it is implemented in Hive.\n\n## How was this patch tested?\nAdded expression unit test in StringExpressionsSuite and end-to-end test in StringFunctionsSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #13966 from petermaxlee/SPARK-16276.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/85f2303ecadd9bf6d9694a2743dda075654c5ccf"},{"sha":"dab10516138867b7c4fc6d42168497e82853b539","author":{"name":"Nick Pentreath","email":"01c7185e211fff3925ce587674cd3ea63724775a@za.ibm.com"},"message":"[SPARK-16328][ML][MLLIB][PYSPARK] Add 'asML' and 'fromML' conversion methods to PySpark linalg\n\nThe move to `ml.linalg` created `asML`/`fromML` utility methods in Scala/Java for converting between representations. These are missing in Python, this PR adds them.\n\n## How was this patch tested?\n\nNew doctests.\n\nAuthor: Nick Pentreath <nickp@za.ibm.com>\n\nCloses #13997 from MLnick/SPARK-16328-python-linalg-convert.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/dab10516138867b7c4fc6d42168497e82853b539"},{"sha":"4a981dc870a31d8b90aac5f6cb22884e02f6fbc6","author":{"name":"Nick Pentreath","email":"01c7185e211fff3925ce587674cd3ea63724775a@za.ibm.com"},"message":"[SPARK-15643][DOC][ML] Add breaking changes to ML migration guide\n\nThis PR adds the breaking changes from [SPARK-14810](https://issues.apache.org/jira/browse/SPARK-14810) to the migration guide.\n\n## How was this patch tested?\n\nBuilt docs locally.\n\nAuthor: Nick Pentreath <nickp@za.ibm.com>\n\nCloses #13924 from MLnick/SPARK-15643-migration-guide.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/4a981dc870a31d8b90aac5f6cb22884e02f6fbc6"},{"sha":"38f4d6f44eaa03bdc703662e4a7be9c09ba86e16","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15954][SQL] Disable loading test tables in Python tests\n\n## What changes were proposed in this pull request?\nThis patch introduces a flag to disable loading test tables in TestHiveSparkSession and disables that in Python. This fixes an issue in which python/run-tests would fail due to failure to load test tables.\n\nNote that these test tables are not used outside of HiveCompatibilitySuite. In the long run we should probably decouple the loading of test tables from the test Hive setup.\n\n## How was this patch tested?\nThis is a test only change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14005 from rxin/SPARK-15954.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/38f4d6f44eaa03bdc703662e4a7be9c09ba86e16"},{"sha":"aa6564f37f1d8de77c3b7bfa885000252efffea6","author":{"name":"Yuhao Yang","email":"718d4d5b3fde31fb7d3c3b83ad143ab39ee820cb@gmail.com"},"message":"[SPARK-14608][ML] transformSchema needs better documentation\n\n## What changes were proposed in this pull request?\njira: https://issues.apache.org/jira/browse/SPARK-14608\nPipelineStage.transformSchema currently has minimal documentation. It should have more to explain it can:\ncheck schema\ncheck parameter interactions\n\n## How was this patch tested?\nunit test\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\nAuthor: Yuhao Yang <yuhao.yang@intel.com>\n\nCloses #12384 from hhbyyh/transformSchemaDoc.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/aa6564f37f1d8de77c3b7bfa885000252efffea6"},{"sha":"14cf61e909598d9f6b9c3b920de7299e9bc828e0","author":{"name":"Hiroshi Inoue","email":"3e15e65404b975101e094443b1e5ff2d3461d1d4@jp.ibm.com"},"message":"[SPARK-16331][SQL] Reduce code generation time\n\n## What changes were proposed in this pull request?\nDuring the code generation, a `LocalRelation` often has a huge `Vector` object as `data`. In the simple example below, a `LocalRelation` has a Vector with 1000000 elements of `UnsafeRow`.\n\n```\nval numRows = 1000000\nval ds = (1 to numRows).toDS().persist()\nbenchmark.addCase(\"filter+reduce\") { iter =>\n  ds.filter(a => (a & 1) == 0).reduce(_ + _)\n}\n```\n\nAt `TreeNode.transformChildren`, all elements of the vector is unnecessarily iterated to check whether any children exist in the vector since `Vector` is Traversable. This part significantly increases code generation time.\n\nThis patch avoids this overhead by checking the number of children before iterating all elements; `LocalRelation` does not have children since it extends `LeafNode`.\n\nThe performance of the above example\n```\nwithout this patch\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_91-b14 on Mac OS X 10.11.5\nIntel(R) Core(TM) i5-5257U CPU  2.70GHz\ncompilationTime:                         Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nfilter+reduce                                 4426 / 4533          0.2        4426.0       1.0X\n\nwith this patch\ncompilationTime:                         Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nfilter+reduce                                 3117 / 3391          0.3        3116.6       1.0X\n```\n\n## How was this patch tested?\n\nusing existing unit tests\n\nAuthor: Hiroshi Inoue <inouehrs@jp.ibm.com>\n\nCloses #14000 from inouehrs/compilation-time-reduction.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/14cf61e909598d9f6b9c3b920de7299e9bc828e0"},{"sha":"fbfd0ab9d70f557c38c7bb8e704475bf19adaf02","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-12177][STREAMING][KAFKA] limit api surface area\n\n## What changes were proposed in this pull request?\nThis is an alternative to the refactoring proposed by https://github.com/apache/spark/pull/13996\n\n## How was this patch tested?\n\nunit tests\nalso tested under scala 2.10 via\nmvn -Dscala-2.10\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #13998 from koeninger/kafka-0-10-refactor.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/fbfd0ab9d70f557c38c7bb8e704475bf19adaf02"},{"sha":"2075bf8ef6035fd7606bcf20dc2cd7d7b9cda446","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16182][CORE] Utils.scala -- terminateProcess() should call Process.destroyForcibly() if and only if Process.destroy() fails\n\n## What changes were proposed in this pull request?\n\nUtils.terminateProcess should `destroy()` first and only fall back to `destroyForcibly()` if it fails. It's kind of bad that we're force-killing executors -- and only in Java 8. See JIRA for an example of the impact: no shutdown\n\nWhile here: `Utils.waitForProcess` should use the Java 8 method if available instead of a custom implementation.\n\n## How was this patch tested?\n\nExisting tests, which cover the force-kill case, and Amplab tests, which will cover both Java 7 and Java 8 eventually. However I tested locally on Java 8 and the PR builder will try Java 7 here.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #13973 from srowen/SPARK-16182.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/2075bf8ef6035fd7606bcf20dc2cd7d7b9cda446"},{"sha":"66283ee0b25de2a5daaa21d50a05a7fadec1de77","author":{"name":"MechCoder","email":"f3b3c28cb28c968d96a6fc2a6a7c7d189719947d@nyu.edu"},"message":"[SPARK-15761][MLLIB][PYSPARK] Load ipython when default python is Python3\n\n## What changes were proposed in this pull request?\n\nI would like to use IPython with Python 3.5. It is annoying when it fails with IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON when I have a version greater than 2.7\n\n## How was this patch tested\nIt now works with IPython and Python3\n\nAuthor: MechCoder <mks542@nyu.edu>\n\nCloses #13503 from MechCoder/spark-15761.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/66283ee0b25de2a5daaa21d50a05a7fadec1de77"},{"sha":"0ad6ce7e54b1d8f5946dde652fa5341d15059158","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16222][SQL] JDBC Sources - Handling illegal input values for `fetchsize` and `batchsize`\n\n#### What changes were proposed in this pull request?\nFor JDBC data sources, users can specify `batchsize` for multi-row inserts and `fetchsize` for multi-row fetch. A few issues exist:\n\n- The property keys are case sensitive. Thus, the existing test cases for `fetchsize` use incorrect names, `fetchSize`. Basically, the test cases are broken.\n- No test case exists for `batchsize`.\n- We do not detect the illegal input values for `fetchsize` and `batchsize`.\n\nFor example, when `batchsize` is zero, we got the following exception:\n```\nJob aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ArithmeticException: / by zero\n```\nwhen `fetchsize` is less than zero, we got the exception from the underlying JDBC driver:\n```\nJob aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): org.h2.jdbc.JdbcSQLException: Invalid value \"-1\" for parameter \"rows\" [90008-183]\n```\n\nThis PR fixes all the above issues, and issue the appropriate exceptions when detecting the illegal inputs for `fetchsize` and `batchsize`. Also update the function descriptions.\n\n#### How was this patch tested?\nTest cases are fixed and added.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13919 from gatorsmile/jdbcProperties.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/0ad6ce7e54b1d8f5946dde652fa5341d15059158"},{"sha":"c55397652ad1c6d047a8b8eb7fd92a8a1dc66306","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16208][SQL] Add `PropagateEmptyRelation` optimizer\n\n## What changes were proposed in this pull request?\n\nThis PR adds a new logical optimizer, `PropagateEmptyRelation`, to collapse a logical plans consisting of only empty LocalRelations.\n\n**Optimizer Targets**\n\n1. Binary(or Higher)-node Logical Plans\n   - Union with all empty children.\n   - Join with one or two empty children (including Intersect/Except).\n2. Unary-node Logical Plans\n   - Project/Filter/Sample/Join/Limit/Repartition with all empty children.\n   - Aggregate with all empty children and without AggregateFunction expressions, COUNT.\n   - Generate with Explode because other UserDefinedGenerators like Hive UDTF returns results.\n\n**Sample Query**\n```sql\nWITH t1 AS (SELECT a FROM VALUES 1 t(a)),\n     t2 AS (SELECT b FROM VALUES 1 t(b) WHERE 1=2)\nSELECT a,b\nFROM t1, t2\nWHERE a=b\nGROUP BY a,b\nHAVING a>1\nORDER BY a,b\n```\n\n**Before**\n```scala\nscala> sql(\"with t1 as (select a from values 1 t(a)), t2 as (select b from values 1 t(b) where 1=2) select a,b from t1, t2 where a=b group by a,b having a>1 order by a,b\").explain\n== Physical Plan ==\n*Sort [a#0 ASC, b#1 ASC], true, 0\n+- Exchange rangepartitioning(a#0 ASC, b#1 ASC, 200)\n   +- *HashAggregate(keys=[a#0, b#1], functions=[])\n      +- Exchange hashpartitioning(a#0, b#1, 200)\n         +- *HashAggregate(keys=[a#0, b#1], functions=[])\n            +- *BroadcastHashJoin [a#0], [b#1], Inner, BuildRight\n               :- *Filter (isnotnull(a#0) && (a#0 > 1))\n               :  +- LocalTableScan [a#0]\n               +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))\n                  +- *Filter (isnotnull(b#1) && (b#1 > 1))\n                     +- LocalTableScan <empty>, [b#1]\n```\n\n**After**\n```scala\nscala> sql(\"with t1 as (select a from values 1 t(a)), t2 as (select b from values 1 t(b) where 1=2) select a,b from t1, t2 where a=b group by a,b having a>1 order by a,b\").explain\n== Physical Plan ==\nLocalTableScan <empty>, [a#0, b#1]\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests (including a new testsuite).\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13906 from dongjoon-hyun/SPARK-16208.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/c55397652ad1c6d047a8b8eb7fd92a8a1dc66306"},{"sha":"26afb4ce4099e7942f8db1ead3817ed8fbf71ce3","author":{"name":"Narine Kokhlikyan","email":"320c8f16ec3d7c7efbde3043fd8d8758f3d71a34@slice.com"},"message":"[SPARK-16012][SPARKR] Implement gapplyCollect which will apply a R function on each group similar to gapply and collect the result back to R data.frame\n\n## What changes were proposed in this pull request?\ngapplyCollect() does gapply() on a SparkDataFrame and collect the result back to R. Compared to gapply() + collect(), gapplyCollect() offers performance optimization as well as programming convenience, as no schema is needed to be provided.\n\nThis is similar to dapplyCollect().\n\n## How was this patch tested?\nAdded test cases for gapplyCollect similar to dapplyCollect\n\nAuthor: Narine Kokhlikyan <narine@slice.com>\n\nCloses #13760 from NarineK/gapplyCollect.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/26afb4ce4099e7942f8db1ead3817ed8fbf71ce3"},{"sha":"e4fa58c43ce2bf8d76bffb0d9dc1132f8d0eae6a","author":{"name":"Sun Rui","email":"6f09da93fe840a44f8b7d461e5cd331e1e929387@gmail.com"},"message":"[SPARK-16299][SPARKR] Capture errors from R workers in daemon.R to avoid deletion of R session temporary directory.\n\n## What changes were proposed in this pull request?\nCapture errors from R workers in daemon.R to avoid deletion of R session temporary directory. See detailed description at https://issues.apache.org/jira/browse/SPARK-16299\n\n## How was this patch tested?\nSparkR unit tests.\n\nAuthor: Sun Rui <sunrui2016@gmail.com>\n\nCloses #13975 from sun-rui/SPARK-16299.","distinct":true,"url":"https://api.github.com/repos/zeocio/spark/commits/e4fa58c43ce2bf8d76bffb0d9dc1132f8d0eae6a"}]}
{"actor":{"display_login":"zeocio"}}