{"push_id":1104896697,"size":1000,"distinct_size":4529,"ref":"refs/heads/master","head":"a019e6efb71e4dce51ca91e41c3d293cf3a6ccb8","before":"40648c56cdaa52058a4771082f8f44a2d8e5a1ec","commits":[{"sha":"38529d8f2350feb1f143aab0be336050c0f887f2","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10907][SPARK-6157] Remove pendingUnrollMemory from MemoryStore\n\nThis patch refactors the MemoryStore to remove the concept of `pendingUnrollMemory`. It also fixes fixes SPARK-6157: \"Unrolling with MEMORY_AND_DISK should always release memory\".\n\nKey changes:\n\n- Inline `MemoryStore.tryToPut` at its three call sites in the `MemoryStore`.\n- Inline `Memory.unrollSafely` at its only call site (in `MemoryStore.putIterator`).\n- Inline `MemoryManager.acquireStorageMemory` at its call sites.\n- Simplify the code as a result of this inlining (some parameters have fixed values after inlining, so lots of branches can be removed).\n- Remove the `pendingUnrollMemory` map by returning the amount of unrollMemory allocated when returning an iterator after a failed `putIterator` call.\n- Change `putIterator` to return an instance of `PartiallyUnrolledIterator`, a special iterator subclass which will automatically free the unroll memory of its partially-unrolled elements when the iterator is consumed. To handle cases where the iterator is not consumed (e.g. when a MEMORY_ONLY put fails), `PartiallyUnrolledIterator` exposes a `close()` method which may be called to discard the unrolled values and free their memory.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11613 from JoshRosen/cleanup-unroll-memory.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/38529d8f2350feb1f143aab0be336050c0f887f2"},{"sha":"8301fadd8d269da11e72870b7a889596e3337839","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13626][CORE] Avoid duplicate config deprecation warnings.\n\nThree different things were needed to get rid of spurious warnings:\n- silence deprecation warnings when cloning configuration\n- change the way SparkHadoopUtil instantiates SparkConf to silence\n  warnings\n- avoid creating new SparkConf instances where it's not needed.\n\nOn top of that, I changed the way that Logging.scala detects the repl;\nnow it uses a method that is overridden in the repl's Main class, and\nthe hack in Utils.scala is not needed anymore. This makes the 2.11 repl\nbehave like the 2.10 one and set the default log level to WARN, which\nis a lot better. Previously, this wasn't working because the 2.11 repl\ntriggers log initialization earlier than the 2.10 one.\n\nI also removed and simplified some other code in the 2.11 repl's Main\nto avoid replicating logic that already exists elsewhere in Spark.\n\nTested the 2.11 repl in local and yarn modes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11510 from vanzin/SPARK-13626.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/8301fadd8d269da11e72870b7a889596e3337839"},{"sha":"06dec37455c3f800897defee6fad0da623f26050","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13843][STREAMING] Remove streaming-flume, streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages\n\n## What changes were proposed in this pull request?\n\nCurrently there are a few sub-projects, each for integrating with different external sources for Streaming.  Now that we have better ability to include external libraries (spark packages) and with Spark 2.0 coming up, we can move the following projects out of Spark to https://github.com/spark-packages\n\n- streaming-flume\n- streaming-akka\n- streaming-mqtt\n- streaming-zeromq\n- streaming-twitter\n\nThey are just some ancillary packages and considering the overhead of maintenance, running tests and PR failures, it's better to maintain them out of Spark. In addition, these projects can have their different release cycles and we can release them faster.\n\nI have already copied these projects to https://github.com/spark-packages\n\n## How was this patch tested?\n\nJenkins tests\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11672 from zsxwing/remove-external-pkg.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/06dec37455c3f800897defee6fad0da623f26050"},{"sha":"992142b87ed5b507493e4f9fac3f72ba14fafbbc","author":{"email":"ed73627b221455d7ebc8c2eb75d4cb59d3b7de18@gmail.com","name":"Ehsan M.Kermani"},"message":"[SPARK-11826][MLLIB] Refactor add() and subtract() methods\n\nsrowen Could you please check this when you have time?\n\nAuthor: Ehsan M.Kermani <ehsanmo1367@gmail.com>\n\nCloses #9916 from ehsanmok/JIRA-11826.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/992142b87ed5b507493e4f9fac3f72ba14fafbbc"},{"sha":"17eec0a71ba8713c559d641e3f43a1be726b037c","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-13664][SQL] Add a strategy for planning partitioned and bucketed scans of files\n\nThis PR adds a new strategy, `FileSourceStrategy`, that can be used for planning scans of collections of files that might be partitioned or bucketed.\n\nCompared with the existing planning logic in `DataSourceStrategy` this version has the following desirable properties:\n - It removes the need to have `RDD`, `broadcastedHadoopConf` and other distributed concerns  in the public API of `org.apache.spark.sql.sources.FileFormat`\n - Partition column appending is delegated to the format to avoid an extra copy / devectorization when appending partition columns\n - It minimizes the amount of data that is shipped to each executor (i.e. it does not send the whole list of files to every worker in the form of a hadoop conf)\n - it natively supports bucketing files into partitions, and thus does not require coalescing / creating a `UnionRDD` with the correct partitioning.\n - Small files are automatically coalesced into fewer tasks using an approximate bin-packing algorithm.\n\nCurrently only a testing source is planned / tested using this strategy.  In follow-up PRs we will port the existing formats to this API.\n\nA stub for `FileScanRDD` is also added, but most methods remain unimplemented.\n\nOther minor cleanups:\n - partition pruning is pushed into `FileCatalog` so both the new and old code paths can use this logic.  This will also allow future implementations to use indexes or other tricks (i.e. a MySQL metastore)\n - The partitions from the `FileCatalog` now propagate information about file sizes all the way up to the planner so we can intelligently spread files out.\n - `Array` -> `Seq` in some internal APIs to avoid unnecessary `toArray` calls\n - Rename `Partition` to `PartitionDirectory` to differentiate partitions used earlier in pruning from those where we have already enumerated the files and their sizes.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #11646 from marmbrus/fileStrategy.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/17eec0a71ba8713c559d641e3f43a1be726b037c"},{"sha":"4bf460979562a7d8cec403f0bd603f88517fdb2b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13882][SQL] Remove org.apache.spark.sql.execution.local\n\n## What changes were proposed in this pull request?\nWe introduced some local operators in org.apache.spark.sql.execution.local package but never fully wired the engine to actually use these. We still plan to implement a full local mode, but it's probably going to be fairly different from what the current iterator-based local mode would look like. Based on what we know right now, we might want a push-based columnar version of these operators.\n\nLet's just remove them for now, and we can always re-introduced them in the future by looking at branch-1.6.\n\n## How was this patch tested?\nThis is simply dead code removal.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11705 from rxin/SPARK-13882.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/4bf460979562a7d8cec403f0bd603f88517fdb2b"},{"sha":"8e0b030606927741f91317660cd14a8a5ed6e5f9","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-10380][SQL] Fix confusing documentation examples for astype/drop_duplicates.\n\n## What changes were proposed in this pull request?\nWe have seen users getting confused by the documentation for astype and drop_duplicates, because the examples in them do not use these functions (but do uses their aliases). This patch simply removes all examples for these functions, and say that they are aliases.\n\n## How was this patch tested?\nExisting PySpark unit tests.\n\nCloses #11543.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11698 from rxin/SPARK-10380.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/8e0b030606927741f91317660cd14a8a5ed6e5f9"},{"sha":"b5e3bd87f5cfa3dc59e5b68d032756feee6b4e25","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13791][SQL] Add MetadataLog and HDFSMetadataLog\n\n## What changes were proposed in this pull request?\n\n- Add a MetadataLog interface for  metadata reliably storage.\n- Add HDFSMetadataLog as a MetadataLog implementation based on HDFS.\n- Update FileStreamSource to use HDFSMetadataLog instead of managing metadata by itself.\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11625 from zsxwing/metadata-log.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/b5e3bd87f5cfa3dc59e5b68d032756feee6b4e25"},{"sha":"e76679a814f5a0903c5f93d9a482f5ddc56fe0d2","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13880][SPARK-13881][SQL] Rename DataFrame.scala Dataset.scala, and remove LegacyFunctions\n\n## What changes were proposed in this pull request?\n1. Rename DataFrame.scala Dataset.scala, since the class is now named Dataset.\n2. Remove LegacyFunctions. It was introduced in Spark 1.6 for backward compatibility, and can be removed in Spark 2.0.\n\n## How was this patch tested?\nShould be covered by existing unit/integration tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11704 from rxin/SPARK-13880.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/e76679a814f5a0903c5f93d9a482f5ddc56fe0d2"},{"sha":"9256840cb631cad50852b2b218a1ac71b567084a","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13661][SQL] avoid the copy in HashedRelation\n\n## What changes were proposed in this pull request?\n\nAvoid the copy in HashedRelation, since most of the HashedRelation are built with Array[Row], added the copy() for LeftSemiJoinHash. This could help to reduce the memory consumption for Broadcast join.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11666 from davies/remove_copy.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/9256840cb631cad50852b2b218a1ac71b567084a"},{"sha":"f72743d971a38d3d08984ef4b66e0955945d2f58","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13353][SQL] fast serialization for collecting DataFrame/Dataset\n\n## What changes were proposed in this pull request?\n\nWhen we call DataFrame/Dataset.collect(), Java serializer (or Kryo Serializer) will be used to serialize the UnsafeRows in executor, then deserialize them into UnsafeRows in driver. Java serializer (and Kyro serializer) are slow on millions rows, because they try to find out the same rows, but usually there is no same rows.\n\nThis PR will serialize the UnsafeRows as byte array by packing them together, then Java serializer (or Kyro serializer) serialize the bytes very fast (there are fewer blocks and byte array are not compared by content).\n\nThe UnsafeRow format is highly compressible, the serialized bytes are also compressed (configurable by spark.io.compression.codec).\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAdd a benchmark for collect, before this patch:\n```\nIntel(R) Core(TM) i7-4558U CPU  2.80GHz\ncollect:                        Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n-------------------------------------------------------------------------------------------\ncollect 1 million                      3991 / 4311          0.3        3805.7       1.0X\ncollect 2 millions                  10083 / 10637          0.1        9616.0       0.4X\ncollect 4 millions                  29551 / 30072          0.0       28182.3       0.1X\n```\n\n```\nIntel(R) Core(TM) i7-4558U CPU  2.80GHz\ncollect:                        Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n-------------------------------------------------------------------------------------------\ncollect 1 million                        775 / 1170          1.4         738.9       1.0X\ncollect 2 millions                     1153 / 1758          0.9        1099.3       0.7X\ncollect 4 millions                     4451 / 5124          0.2        4244.9       0.2X\n```\n\nWe can see about 5-7X speedup.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11664 from davies/serialize_row.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/f72743d971a38d3d08984ef4b66e0955945d2f58"},{"sha":"e64958001cb95d53c441131f8c7a92556f49fd7d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13884][SQL] Remove DescribeCommand's dependency on LogicalPlan\n\n## What changes were proposed in this pull request?\nThis patch removes DescribeCommand's dependency on LogicalPlan. After this patch, DescribeCommand simply accepts a TableIdentifier. It minimizes the dependency, and blocks my next patch (removes SQLContext dependency from SparkPlanner).\n\n## How was this patch tested?\nShould be covered by existing unit tests and Hive compatibility tests that run describe table.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11710 from rxin/SPARK-13884.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/e64958001cb95d53c441131f8c7a92556f49fd7d"},{"sha":"43304b1758dec141b7fe9ed33cac976d75efdf91","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13888][DOC] Remove Akka Receiver doc and refer to the DStream Akka project\n\n## What changes were proposed in this pull request?\n\nI have copied the docs of Streaming Akka to https://github.com/spark-packages/dstream-akka/blob/master/README.md\n\nSo we can remove them from Spark now.\n\n## How was this patch tested?\n\nOnly document changes.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11711 from zsxwing/remove-akka-doc.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/43304b1758dec141b7fe9ed33cac976d75efdf91"},{"sha":"a51f877b5dc56b7bb9ef95044a50024c6b64718e","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-13870][SQL] Add scalastyle escaping correctly in CVSSuite.scala\n\n## What changes were proposed in this pull request?\n\nWhen initial creating `CVSSuite.scala` in SPARK-12833, there was a typo on `scalastyle:on`: `scalstyle:on`. So, it turns off ScalaStyle checking for the rest of the file mistakenly. So, it can not find a violation on the code of `SPARK-12668` added recently. This issue fixes the existing escaping correctly and adds a new escaping for `SPARK-12668` code like the following.\n\n```scala\n   test(\"test aliases sep and encoding for delimiter and charset\") {\n+    // scalastyle:off\n     val cars = sqlContext\n...\n       .load(testFile(carsFile8859))\n+    // scalastyle:on\n```\nThis will prevent future potential problems, too.\n\n## How was this patch tested?\n\nPass the Jenkins test.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11700 from dongjoon-hyun/SPARK-13870.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/a51f877b5dc56b7bb9ef95044a50024c6b64718e"},{"sha":"276c2d51a3bbe2531763a11580adfec7e39fdd58","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13890][SQL] Remove some internal classes' dependency on SQLContext\n\n## What changes were proposed in this pull request?\nIn general it is better for internal classes to not depend on the external class (in this case SQLContext) to reduce coupling between user-facing APIs and the internal implementations. This patch removes SQLContext dependency from some internal classes such as SparkPlanner, SparkOptimizer.\n\nAs part of this patch, I also removed the following internal methods from SQLContext:\n```\nprotected[sql] def functionRegistry: FunctionRegistry\nprotected[sql] def optimizer: Optimizer\nprotected[sql] def sqlParser: ParserInterface\nprotected[sql] def planner: SparkPlanner\nprotected[sql] def continuousQueryManager\nprotected[sql] def prepareForExecution: RuleExecutor[SparkPlan]\n```\n\n## How was this patch tested?\nExisting unit/integration tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11712 from rxin/sqlContext-planner.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/276c2d51a3bbe2531763a11580adfec7e39fdd58"},{"sha":"99bd2f0e94657687834c5c59c4270c1484c9f595","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-13840][SQL] Split Optimizer Rule ColumnPruning to ColumnPruning and EliminateOperator\n\n#### What changes were proposed in this pull request?\n\nBefore this PR, two Optimizer rules `ColumnPruning` and `PushPredicateThroughProject` reverse each other's effects. Optimizer always reaches the max iteration when optimizing some queries. Extra `Project` are found in the plan. For example, below is the optimized plan after reaching 100 iterations:\n\n```\nJoin Inner, Some((cast(id1#16 as bigint) = id1#18L))\n:- Project [id1#16]\n:  +- Filter isnotnull(cast(id1#16 as bigint))\n:     +- Project [id1#16]\n:        +- Relation[id1#16,newCol#17] JSON part: struct<>, data: struct<id1:int,newCol:int>\n+- Filter isnotnull(id1#18L)\n   +- Relation[id1#18L] JSON part: struct<>, data: struct<id1:bigint>\n```\n\nThis PR splits the optimizer rule `ColumnPruning` to `ColumnPruning` and `EliminateOperators`\n\nThe issue becomes worse when having another rule `NullFiltering`, which could add extra Filters for `IsNotNull`. We have to be careful when introducing extra `Filter` if the benefit is not large enough. Another PR will be submitted by sameeragarwal to handle this issue.\n\ncc sameeragarwal marmbrus\n\nIn addition, `ColumnPruning` should not push `Project` through non-deterministic `Filter`. This could cause wrong results. This will be put in a separate PR.\n\ncc davies cloud-fan yhuai\n\n#### How was this patch tested?\n\nModified the existing test cases.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #11682 from gatorsmile/viewDuplicateNames.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/99bd2f0e94657687834c5c59c4270c1484c9f595"},{"sha":"10251a7457e44fa33f13274ccb2266bc5e931363","author":{"email":"623dda1cf42194a966de8b161dee064823eb61d3@126.com","name":"Xin Ren"},"message":"[SPARK-13660][SQL][TESTS] ContinuousQuerySuite floods the logs with garbage\n\n## What changes were proposed in this pull request?\n\nUse method 'testQuietly' to avoid ContinuousQuerySuite flooding the console logs with garbage\n\nMake ContinuousQuerySuite not output logs to the console. The logs will still output to unit-tests.log.\n\n## How was this patch tested?\n\nJust check Jenkins output.\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #11703 from keypointt/SPARK-13660.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/10251a7457e44fa33f13274ccb2266bc5e931363"},{"sha":"dafd70fbfe70702502ef198f2a8f529ef7557592","author":{"email":"8b686eb7d2bd6ecbcaccba98aff1768585dde44b@gmail.com","name":"sethah"},"message":"[SPARK-12379][ML][MLLIB] Copy GBT implementation to spark.ml\n\nCurrently, GBTs in spark.ml wrap the implementation in spark.mllib. This is preventing several improvements to GBTs in spark.ml, so we need to move the implementation to ml and use spark.ml decision trees in the implementation. At first, we should make minimal changes to the implementation.\nPerformance testing should be done to ensure there were no regressions.\n\nPerformance testing results are [here](https://docs.google.com/document/d/1dYd2mnfGdUKkQ3vZe2BpzsTnI5IrpSLQ-NNKDZhUkgw/edit?usp=sharing)\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #10607 from sethah/SPARK-12379.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/dafd70fbfe70702502ef198f2a8f529ef7557592"},{"sha":"bd5365bbe9ff6518cde9402ee8843ec1002fff5b","author":{"email":"fd726277649d3b2e3ee5d332dca6c89739b68aa5@gmail.com","name":"CodingCat"},"message":"[SPARK-13803] restore the changes in SPARK-3411\n\n## What changes were proposed in this pull request?\n\nThis patch contains the functionality to balance the load of the cluster-mode drivers among workers\n\nThis patch restores the changes in https://github.com/apache/spark/pull/1106 which was erased due to the merging of https://github.com/apache/spark/pull/731\n\n## How was this patch tested?\n\ntest with existing test cases\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #11702 from CodingCat/SPARK-13803.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/bd5365bbe9ff6518cde9402ee8843ec1002fff5b"},{"sha":"48978abfa4d8f2cf79a4b053cc8bc7254cc2d61b","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13576][BUILD] Don't create assembly for examples.\n\nAs part of the goal to stop creating assemblies in Spark, this change\nmodifies the mvn and sbt builds to not create an assembly for examples.\n\nInstead, dependencies are copied to the build directory (under\ntarget/scala-xx/jars), and in the final archive, into the \"examples/jars\"\ndirectory.\n\nTo avoid having to deal too much with Windows batch files, I made examples\nrun through the launcher library; the spark-submit launcher now has a\nspecial mode to run examples, which adds all the necessary jars to the\nspark-submit command line, and replaces the bash and batch scripts that\nwere used to run examples. The scripts are now just a thin wrapper around\nspark-submit; another advantage is that now all spark-submit options are\nsupported.\n\nThere are a few glitches; in the mvn build, a lot of duplicated dependencies\nget copied, because they are promoted to \"compile\" scope due to extra\ndependencies in the examples module (such as HBase). In the sbt build,\nall dependencies are copied, because there doesn't seem to be an easy\nway to filter things.\n\nI plan to clean some of this up when the rest of the tasks are finished.\nWhen the main assembly is replaced with jars, we can remove duplicate jars\nfrom the examples directory during packaging.\n\nTested by running SparkPi in: maven build, sbt build, dist created by\nmake-distribution.sh.\n\nFinally: note that running the \"assembly\" target in sbt doesn't build\nthe examples anymore. You need to run \"package\" for that.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11452 from vanzin/SPARK-13576.","distinct":true,"url":"https://api.github.com/repos/rnirmal/spark/commits/48978abfa4d8f2cf79a4b053cc8bc7254cc2d61b"}]}