{"push_id":1024958328,"size":81,"distinct_size":81,"ref":"refs/heads/master","head":"ea9ca6f04ce1fb83612f907d6d0996d6fb362bd2","before":"250832c733602bcca034dd1fea6e325c55ccd1cd","commits":[{"sha":"9a1680c2c85da4096bad71743debb2ccacdfd79f","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13139][SQL] Follow-ups to #11573\n\nAddressing outstanding comments in #11573.\n\nJenkins, new test case in `DDLCommandSuite`\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11667 from andrewor14/ddl-parser-followups.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/9a1680c2c85da4096bad71743debb2ccacdfd79f"},{"sha":"9a87afd7d1dbcee993d58e648c74aa683d91f07e","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13833] Guard against race condition when re-caching disk blocks in memory\n\nWhen reading data from the DiskStore and attempting to cache it back into the memory store, we should guard against race conditions where multiple readers are attempting to re-cache the same block in memory.\n\nThis patch accomplishes this by synchronizing on the block's `BlockInfo` object while trying to re-cache a block.\n\n(Will file JIRA as soon as ASF JIRA stops being down / laggy).\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11660 from JoshRosen/concurrent-recaching-fixes.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/9a87afd7d1dbcee993d58e648c74aa683d91f07e"},{"sha":"45f8053be5c635b50c7b4ef5a0dc75d30f411291","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13578][CORE] Modify launch scripts to not use assemblies.\n\nInstead of looking for a specially-named assembly, the scripts now will\nblindly add all jars under the libs directory to the classpath. This\nlibs directory is still currently the old assembly dir, so things should\nkeep working the same way as before until we make more packaging changes.\n\nThe only lost feature is the detection of multiple assemblies; I consider\nthat a minor nicety that only really affects few developers, so it's probably\nok.\n\nTested locally by running spark-shell; also did some minor Win32 testing\n(just made sure spark-shell started).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11591 from vanzin/SPARK-13578.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/45f8053be5c635b50c7b4ef5a0dc75d30f411291"},{"sha":"63f642aea31fe0d202ce585681d51e7ac1715ba7","author":{"email":"4c9a82ce72ca2519f38d0af0abbb4cecb9fceca9@apache.org","name":"Ryan Blue"},"message":"[SPARK-13779][YARN] Avoid cancelling non-local container requests.\n\nTo maximize locality, the YarnAllocator would cancel any requests with a\nstale locality preference or no locality preference. This assumed that\nthe majority of tasks had locality preferences, but may not be the case\nwhen scanning S3. This caused container requests for S3 tasks to be\nconstantly cancelled and resubmitted.\n\nThis changes the allocator's logic to cancel only stale requests and\njust enough requests without locality preferences to submit requests\nwith locality preferences. This avoids cancelling requests without\nlocality preferences that would be resubmitted without locality\npreferences.\n\nWe've deployed this patch on our clusters and verified that jobs that couldn't get executors because they kept canceling and resubmitting requests are fixed. Large jobs are running fine.\n\nAuthor: Ryan Blue <blue@apache.org>\n\nCloses #11612 from rdblue/SPARK-13779-fix-yarn-allocator-requests.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/63f642aea31fe0d202ce585681d51e7ac1715ba7"},{"sha":"6a4bfcd62b7effcfbb865bdd301d41a0ba6e5c94","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13658][SQL] BooleanSimplification rule is slow with large boolean expressions\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13658\n\n## What changes were proposed in this pull request?\n\nQuoted from JIRA description: When run TPCDS Q3 [1] with lots predicates to filter out the partitions, the optimizer rule BooleanSimplification take about 2 seconds (it use lots of sematicsEqual, which require copy the whole tree).\n\nIt will great if we could speedup it.\n\n[1] https://github.com/cloudera/impala-tpcds-kit/blob/master/queries/q3.sql\n\nHow to speed up it:\n\nWhen we ask the canonicalized expression in `Expression`, it calls `Canonicalize.execute` on itself. `Canonicalize.execute` basically transforms up all expressions included in this expression. However, we don't keep the canonicalized versions for these children expressions. So in next time we ask the canonicalized expressions for the children expressions (e.g., `BooleanSimplification`), we will rerun `Canonicalize.execute` on each of them. It wastes much time.\n\nBy forcing the children expressions to get and keep their canonicalized versions first, we can avoid re-canonicalize these expressions.\n\nI simply benchmark it with an expression which is part of the where clause in TPCDS Q3:\n\n    val testRelation = LocalRelation('ss_sold_date_sk.int, 'd_moy.int, 'i_manufact_id.int, 'ss_item_sk.string, 'i_item_sk.string, 'd_date_sk.int)\n\n    val input = ('d_date_sk === 'ss_sold_date_sk) && ('ss_item_sk === 'i_item_sk) && ('i_manufact_id === 436) && ('d_moy === 12) && (('ss_sold_date_sk > 2415355 && 'ss_sold_date_sk < 2415385) || ('ss_sold_date_sk > 2415720 && 'ss_sold_date_sk < 2415750) || ('ss_sold_date_sk > 2416085 && 'ss_sold_date_sk < 2416115) || ('ss_sold_date_sk > 2416450 && 'ss_sold_date_sk < 2416480) || ('ss_sold_date_sk > 2416816 && 'ss_sold_date_sk < 2416846) || ('ss_sold_date_sk > 2417181 && 'ss_sold_date_sk < 2417211) || ('ss_sold_date_sk > 2417546 && 'ss_sold_date_sk < 2417576) || ('ss_sold_date_sk > 2417911 && 'ss_sold_date_sk < 2417941) || ('ss_sold_date_sk > 2418277 && 'ss_sold_date_sk < 2418307) || ('ss_sold_date_sk > 2418642 && 'ss_sold_date_sk < 2418672) || ('ss_sold_date_sk > 2419007 && 'ss_sold_date_sk < 2419037) || ('ss_sold_date_sk > 2419372 && 'ss_sold_date_sk < 2419402) || ('ss_sold_date_sk > 2419738 && 'ss_sold_date_sk < 2419768) || ('ss_sold_date_sk > 2420103 && 'ss_sold_date_sk < 2420133) || ('ss_sold_date_sk > 2420468 && 'ss_sold_date_sk < 2420498) || ('ss_sold_date_sk > 2420833 && 'ss_sold_date_sk < 2420863) || ('ss_sold_date_sk > 2421199 && 'ss_sold_date_sk < 2421229) || ('ss_sold_date_sk > 2421564 && 'ss_sold_date_sk < 2421594) || ('ss_sold_date_sk > 2421929 && 'ss_sold_date_sk < 2421959) || ('ss_sold_date_sk > 2422294 && 'ss_sold_date_sk < 2422324) || ('ss_sold_date_sk > 2422660 && 'ss_sold_date_sk < 2422690) || ('ss_sold_date_sk > 2423025 && 'ss_sold_date_sk < 2423055) || ('ss_sold_date_sk > 2423390 && 'ss_sold_date_sk < 2423420) || ('ss_sold_date_sk > 2423755 && 'ss_sold_date_sk < 2423785) || ('ss_sold_date_sk > 2424121 && 'ss_sold_date_sk < 2424151) || ('ss_sold_date_sk > 2424486 && 'ss_sold_date_sk < 2424516) || ('ss_sold_date_sk > 2424851 && 'ss_sold_date_sk < 2424881) || ('ss_sold_date_sk > 2425216 && 'ss_sold_date_sk < 2425246) || ('ss_sold_date_sk > 2425582 && 'ss_sold_date_sk < 2425612) || ('ss_sold_date_sk > 2425947 && 'ss_sold_date_sk < 2425977) || ('ss_sold_date_sk > 2426312 && 'ss_sold_date_sk < 2426342) || ('ss_sold_date_sk > 2426677 && 'ss_sold_date_sk < 2426707) || ('ss_sold_date_sk > 2427043 && 'ss_sold_date_sk < 2427073) || ('ss_sold_date_sk > 2427408 && 'ss_sold_date_sk < 2427438) || ('ss_sold_date_sk > 2427773 && 'ss_sold_date_sk < 2427803) || ('ss_sold_date_sk > 2428138 && 'ss_sold_date_sk < 2428168) || ('ss_sold_date_sk > 2428504 && 'ss_sold_date_sk < 2428534) || ('ss_sold_date_sk > 2428869 && 'ss_sold_date_sk < 2428899) || ('ss_sold_date_sk > 2429234 && 'ss_sold_date_sk < 2429264) || ('ss_sold_date_sk > 2429599 && 'ss_sold_date_sk < 2429629) || ('ss_sold_date_sk > 2429965 && 'ss_sold_date_sk < 2429995) || ('ss_sold_date_sk > 2430330 && 'ss_sold_date_sk < 2430360) || ('ss_sold_date_sk > 2430695 && 'ss_sold_date_sk < 2430725) || ('ss_sold_date_sk > 2431060 && 'ss_sold_date_sk < 2431090) || ('ss_sold_date_sk > 2431426 && 'ss_sold_date_sk < 2431456) || ('ss_sold_date_sk > 2431791 && 'ss_sold_date_sk < 2431821) || ('ss_sold_date_sk > 2432156 && 'ss_sold_date_sk < 2432186) || ('ss_sold_date_sk > 2432521 && 'ss_sold_date_sk < 2432551) || ('ss_sold_date_sk > 2432887 && 'ss_sold_date_sk < 2432917) || ('ss_sold_date_sk > 2433252 && 'ss_sold_date_sk < 2433282) || ('ss_sold_date_sk > 2433617 && 'ss_sold_date_sk < 2433647) || ('ss_sold_date_sk > 2433982 && 'ss_sold_date_sk < 2434012) || ('ss_sold_date_sk > 2434348 && 'ss_sold_date_sk < 2434378) || ('ss_sold_date_sk > 2434713 && 'ss_sold_date_sk < 2434743)))\n\n    val plan = testRelation.where(input).analyze\n    val actual = Optimize.execute(plan)\n\nWith this patch:\n\n    352 milliseconds\n    346 milliseconds\n    340 milliseconds\n\nWithout this patch:\n\n    585 milliseconds\n    880 milliseconds\n    677 milliseconds\n\n## How was this patch tested?\n\nExisting tests should pass.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11647 from viirya/improve-expr-canonicalize.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/6a4bfcd62b7effcfbb865bdd301d41a0ba6e5c94"},{"sha":"07cb323e7a128b87ef265ddc66f033365d9de463","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13848][SPARK-5185] Update to Py4J 0.9.2 in order to fix classloading issue\n\nThis patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185), a longstanding issue affecting the use of `--jars` and `--packages` in PySpark.\n\nIn order to demonstrate that the fix works, I removed the workarounds which were added as part of [SPARK-6027](https://issues.apache.org/jira/browse/SPARK-6027) / #4779 and other patches.\n\nPy4J diff: https://github.com/bartdag/py4j/compare/0.9.1...0.9.2\n\n/cc zsxwing tdas davies brkyvz\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11687 from JoshRosen/py4j-0.9.2.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/07cb323e7a128b87ef265ddc66f033365d9de463"},{"sha":"310981d49a332bd329303f610b150bbe02cf5f87","author":{"email":"a287260e530758aff99c3664b9757abed8299858@teralytics.net","name":"Bertrand Bossy"},"message":"[SPARK-12583][MESOS] Mesos shuffle service: Don't delete shuffle files before application has stopped\n\n## Problem description:\n\nMesos shuffle service is completely unusable since Spark 1.6.0 . The problem seems to occur since the move from akka to netty in the networking layer. Until now, a connection from the driver to each shuffle service was used as a signal for the shuffle service to determine, whether the driver is still running. Since 1.6.0, this connection is closed after spark.shuffle.io.connectionTimeout (or spark.network.timeout if the former is not set) due to it being idle. The shuffle service interprets this as a signal that the driver has stopped, despite the driver still being alive. Thus, shuffle files are deleted before the application has stopped.\n\n### Context and analysis:\n\nspark shuffle fails with mesos after 2mins: https://issues.apache.org/jira/browse/SPARK-12583\nExternal shuffle service broken w/ Mesos: https://issues.apache.org/jira/browse/SPARK-13159\n\nThis is a follow up on #11207 .\n\n## What changes were proposed in this pull request?\n\nThis PR adds a heartbeat signal from the Driver (in MesosExternalShuffleClient) to all registered external mesos shuffle service instances. In MesosExternalShuffleBlockHandler, a thread periodically checks whether a driver has timed out and cleans an application's shuffle files if this is the case.\n\n## How was the this patch tested?\n\nThis patch has been tested on a small mesos test cluster using the spark-shell. Log output from mesos shuffle service:\n```\n16/02/19 15:13:45 INFO mesos.MesosExternalShuffleBlockHandler: Received registration request from app 294def07-3249-4e0f-8d71-bf8c83c58a50-0018 (remote address /xxx.xxx.xxx.xxx:52391, heartbeat timeout 120000 ms).\n16/02/19 15:13:47 INFO shuffle.ExternalShuffleBlockResolver: Registered executor AppExecId{appId=294def07-3249-4e0f-8d71-bf8c83c58a50-0018, execId=3} with ExecutorShuffleInfo{localDirs=[/foo/blockmgr-c84c0697-a3f9-4f61-9c64-4d3ee227c047], subDirsPerLocalDir=64, shuffleManager=sort}\n16/02/19 15:13:47 INFO shuffle.ExternalShuffleBlockResolver: Registered executor AppExecId{appId=294def07-3249-4e0f-8d71-bf8c83c58a50-0018, execId=7} with ExecutorShuffleInfo{localDirs=[/foo/blockmgr-bf46497a-de80-47b9-88f9-563123b59e03], subDirsPerLocalDir=64, shuffleManager=sort}\n16/02/19 15:16:02 INFO mesos.MesosExternalShuffleBlockHandler: Application 294def07-3249-4e0f-8d71-bf8c83c58a50-0018 timed out. Removing shuffle files.\n16/02/19 15:16:02 INFO shuffle.ExternalShuffleBlockResolver: Application 294def07-3249-4e0f-8d71-bf8c83c58a50-0018 removed, cleanupLocalDirs = true\n16/02/19 15:16:02 INFO shuffle.ExternalShuffleBlockResolver: Cleaning up executor AppExecId{appId=294def07-3249-4e0f-8d71-bf8c83c58a50-0018, execId=3}'s 1 local dirs\n16/02/19 15:16:02 INFO shuffle.ExternalShuffleBlockResolver: Cleaning up executor AppExecId{appId=294def07-3249-4e0f-8d71-bf8c83c58a50-0018, execId=7}'s 1 local dirs\n```\nNote: there are 2 executors running on this slave.\n\nAuthor: Bertrand Bossy <bertrand.bossy@teralytics.net>\n\nCloses #11272 from bbossy/SPARK-12583-mesos-shuffle-service-heartbeat.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/310981d49a332bd329303f610b150bbe02cf5f87"},{"sha":"9f13f0fc1724e407d3be54152b82150a16395421","author":{"email":"e19996995266c8ed28ab7ebda5ebecb69b13fd3b@gmail.com","name":"Daniel Santana"},"message":"[MINOR][DOCS] Added Missing back slashes\n\n## What changes were proposed in this pull request?\n\nWhen studying spark many users just copy examples on the documentation and paste on their terminals\nand because of that the missing backlashes lead them run into some shell errors.\n\nThe added backslashes avoid that problem for spark users with that behavior.\n\n## How was this patch tested?\n\nI generated the documentation locally using jekyll and checked the generated pages\n\nAuthor: Daniel Santana <mestresan@gmail.com>\n\nCloses #11699 from danielsan/master.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/9f13f0fc1724e407d3be54152b82150a16395421"},{"sha":"e06493cb7b790623a9106241a8d496ecea703328","author":{"email":"bb0eadddd335e82bba4e0686d2c318a2ab1db8a0@gmail.com","name":"Bjorn Jonsson"},"message":"[MINOR][COMMON] Fix copy-paste oversight in variable naming\n\n## What changes were proposed in this pull request?\n\nJavaUtils.java has methods to convert time and byte strings for internal use, this change renames a variable used in byteStringAs(), from timeError to byteError.\n\nAuthor: Bjorn Jonsson <bjornjon@gmail.com>\n\nCloses #11695 from bjornjon/master.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/e06493cb7b790623a9106241a8d496ecea703328"},{"sha":"23385e853e7ca54332c6098cf83da7d0723546fe","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@prevailsail.corp.gq1.yahoo.com","name":"Thomas Graves"},"message":"[SPARK-13054] Always post TaskEnd event for tasks\n\nI am using dynamic container allocation and speculation and am seeing issues with the active task accounting. The Executor UI still shows active tasks on the an executor but the job/stage is all completed. I think its also affecting the dynamic allocation being able to release containers because it thinks there are still tasks.\nThere are multiple issues with this:\n-  If the task end for tasks (in this case probably because of speculation) comes in after the stage is finished, then the DAGScheduler.handleTaskCompletion will skip the task completion event\n\nAuthor: Thomas Graves <tgraves@prevailsail.corp.gq1.yahoo.com>\nAuthor: Thomas Graves <tgraves@staydecay.corp.gq1.yahoo.com>\nAuthor: Tom Graves <tgraves@yahoo-inc.com>\n\nCloses #10951 from tgravescs/SPARK-11701.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/23385e853e7ca54332c6098cf83da7d0723546fe"},{"sha":"a48296f4fe513b63041f1a26231cfe152b69657f","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-13686][MLLIB][STREAMING] Add a constructor parameter `reqParam` to (Streaming)LinearRegressionWithSGD\n\n## What changes were proposed in this pull request?\n\n`LinearRegressionWithSGD` and `StreamingLinearRegressionWithSGD` does not have `regParam` as their constructor arguments. They just depends on GradientDescent's default reqParam values.\nTo be consistent with other algorithms, we had better add them. The same default value is used.\n\n## How was this patch tested?\n\nPass the existing unit test.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11527 from dongjoon-hyun/SPARK-13686.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/a48296f4fe513b63041f1a26231cfe152b69657f"},{"sha":"38529d8f2350feb1f143aab0be336050c0f887f2","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10907][SPARK-6157] Remove pendingUnrollMemory from MemoryStore\n\nThis patch refactors the MemoryStore to remove the concept of `pendingUnrollMemory`. It also fixes fixes SPARK-6157: \"Unrolling with MEMORY_AND_DISK should always release memory\".\n\nKey changes:\n\n- Inline `MemoryStore.tryToPut` at its three call sites in the `MemoryStore`.\n- Inline `Memory.unrollSafely` at its only call site (in `MemoryStore.putIterator`).\n- Inline `MemoryManager.acquireStorageMemory` at its call sites.\n- Simplify the code as a result of this inlining (some parameters have fixed values after inlining, so lots of branches can be removed).\n- Remove the `pendingUnrollMemory` map by returning the amount of unrollMemory allocated when returning an iterator after a failed `putIterator` call.\n- Change `putIterator` to return an instance of `PartiallyUnrolledIterator`, a special iterator subclass which will automatically free the unroll memory of its partially-unrolled elements when the iterator is consumed. To handle cases where the iterator is not consumed (e.g. when a MEMORY_ONLY put fails), `PartiallyUnrolledIterator` exposes a `close()` method which may be called to discard the unrolled values and free their memory.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11613 from JoshRosen/cleanup-unroll-memory.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/38529d8f2350feb1f143aab0be336050c0f887f2"},{"sha":"8301fadd8d269da11e72870b7a889596e3337839","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13626][CORE] Avoid duplicate config deprecation warnings.\n\nThree different things were needed to get rid of spurious warnings:\n- silence deprecation warnings when cloning configuration\n- change the way SparkHadoopUtil instantiates SparkConf to silence\n  warnings\n- avoid creating new SparkConf instances where it's not needed.\n\nOn top of that, I changed the way that Logging.scala detects the repl;\nnow it uses a method that is overridden in the repl's Main class, and\nthe hack in Utils.scala is not needed anymore. This makes the 2.11 repl\nbehave like the 2.10 one and set the default log level to WARN, which\nis a lot better. Previously, this wasn't working because the 2.11 repl\ntriggers log initialization earlier than the 2.10 one.\n\nI also removed and simplified some other code in the 2.11 repl's Main\nto avoid replicating logic that already exists elsewhere in Spark.\n\nTested the 2.11 repl in local and yarn modes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11510 from vanzin/SPARK-13626.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/8301fadd8d269da11e72870b7a889596e3337839"},{"sha":"06dec37455c3f800897defee6fad0da623f26050","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13843][STREAMING] Remove streaming-flume, streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages\n\n## What changes were proposed in this pull request?\n\nCurrently there are a few sub-projects, each for integrating with different external sources for Streaming.  Now that we have better ability to include external libraries (spark packages) and with Spark 2.0 coming up, we can move the following projects out of Spark to https://github.com/spark-packages\n\n- streaming-flume\n- streaming-akka\n- streaming-mqtt\n- streaming-zeromq\n- streaming-twitter\n\nThey are just some ancillary packages and considering the overhead of maintenance, running tests and PR failures, it's better to maintain them out of Spark. In addition, these projects can have their different release cycles and we can release them faster.\n\nI have already copied these projects to https://github.com/spark-packages\n\n## How was this patch tested?\n\nJenkins tests\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11672 from zsxwing/remove-external-pkg.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/06dec37455c3f800897defee6fad0da623f26050"},{"sha":"992142b87ed5b507493e4f9fac3f72ba14fafbbc","author":{"email":"ed73627b221455d7ebc8c2eb75d4cb59d3b7de18@gmail.com","name":"Ehsan M.Kermani"},"message":"[SPARK-11826][MLLIB] Refactor add() and subtract() methods\n\nsrowen Could you please check this when you have time?\n\nAuthor: Ehsan M.Kermani <ehsanmo1367@gmail.com>\n\nCloses #9916 from ehsanmok/JIRA-11826.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/992142b87ed5b507493e4f9fac3f72ba14fafbbc"},{"sha":"17eec0a71ba8713c559d641e3f43a1be726b037c","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-13664][SQL] Add a strategy for planning partitioned and bucketed scans of files\n\nThis PR adds a new strategy, `FileSourceStrategy`, that can be used for planning scans of collections of files that might be partitioned or bucketed.\n\nCompared with the existing planning logic in `DataSourceStrategy` this version has the following desirable properties:\n - It removes the need to have `RDD`, `broadcastedHadoopConf` and other distributed concerns  in the public API of `org.apache.spark.sql.sources.FileFormat`\n - Partition column appending is delegated to the format to avoid an extra copy / devectorization when appending partition columns\n - It minimizes the amount of data that is shipped to each executor (i.e. it does not send the whole list of files to every worker in the form of a hadoop conf)\n - it natively supports bucketing files into partitions, and thus does not require coalescing / creating a `UnionRDD` with the correct partitioning.\n - Small files are automatically coalesced into fewer tasks using an approximate bin-packing algorithm.\n\nCurrently only a testing source is planned / tested using this strategy.  In follow-up PRs we will port the existing formats to this API.\n\nA stub for `FileScanRDD` is also added, but most methods remain unimplemented.\n\nOther minor cleanups:\n - partition pruning is pushed into `FileCatalog` so both the new and old code paths can use this logic.  This will also allow future implementations to use indexes or other tricks (i.e. a MySQL metastore)\n - The partitions from the `FileCatalog` now propagate information about file sizes all the way up to the planner so we can intelligently spread files out.\n - `Array` -> `Seq` in some internal APIs to avoid unnecessary `toArray` calls\n - Rename `Partition` to `PartitionDirectory` to differentiate partitions used earlier in pruning from those where we have already enumerated the files and their sizes.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #11646 from marmbrus/fileStrategy.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/17eec0a71ba8713c559d641e3f43a1be726b037c"},{"sha":"4bf460979562a7d8cec403f0bd603f88517fdb2b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13882][SQL] Remove org.apache.spark.sql.execution.local\n\n## What changes were proposed in this pull request?\nWe introduced some local operators in org.apache.spark.sql.execution.local package but never fully wired the engine to actually use these. We still plan to implement a full local mode, but it's probably going to be fairly different from what the current iterator-based local mode would look like. Based on what we know right now, we might want a push-based columnar version of these operators.\n\nLet's just remove them for now, and we can always re-introduced them in the future by looking at branch-1.6.\n\n## How was this patch tested?\nThis is simply dead code removal.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11705 from rxin/SPARK-13882.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/4bf460979562a7d8cec403f0bd603f88517fdb2b"},{"sha":"8e0b030606927741f91317660cd14a8a5ed6e5f9","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10380][SQL] Fix confusing documentation examples for astype/drop_duplicates.\n\n## What changes were proposed in this pull request?\nWe have seen users getting confused by the documentation for astype and drop_duplicates, because the examples in them do not use these functions (but do uses their aliases). This patch simply removes all examples for these functions, and say that they are aliases.\n\n## How was this patch tested?\nExisting PySpark unit tests.\n\nCloses #11543.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11698 from rxin/SPARK-10380.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/8e0b030606927741f91317660cd14a8a5ed6e5f9"},{"sha":"b5e3bd87f5cfa3dc59e5b68d032756feee6b4e25","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13791][SQL] Add MetadataLog and HDFSMetadataLog\n\n## What changes were proposed in this pull request?\n\n- Add a MetadataLog interface for  metadata reliably storage.\n- Add HDFSMetadataLog as a MetadataLog implementation based on HDFS.\n- Update FileStreamSource to use HDFSMetadataLog instead of managing metadata by itself.\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11625 from zsxwing/metadata-log.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/b5e3bd87f5cfa3dc59e5b68d032756feee6b4e25"},{"sha":"e76679a814f5a0903c5f93d9a482f5ddc56fe0d2","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13880][SPARK-13881][SQL] Rename DataFrame.scala Dataset.scala, and remove LegacyFunctions\n\n## What changes were proposed in this pull request?\n1. Rename DataFrame.scala Dataset.scala, since the class is now named Dataset.\n2. Remove LegacyFunctions. It was introduced in Spark 1.6 for backward compatibility, and can be removed in Spark 2.0.\n\n## How was this patch tested?\nShould be covered by existing unit/integration tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11704 from rxin/SPARK-13880.","distinct":true,"url":"https://api.github.com/repos/iostat/spark/commits/e76679a814f5a0903c5f93d9a482f5ddc56fe0d2"}]}