{"push_id":1105831704,"size":20,"distinct_size":20,"ref":"refs/heads/branch-2.0","head":"d8c2da9a46f240438c5350f73a7b3daa07c101a8","before":"27bb51ca47acf6ffb6d5f5316393075cccdce710","commits":[{"sha":"58f77421b8dc3aec6252e504e7fd158fd2414acf","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-15215][SQL] Fix Explain Parsing and Output\n\n#### What changes were proposed in this pull request?\nThis PR is to address a few existing issues in `EXPLAIN`:\n- The `EXPLAIN` options `LOGICAL | FORMATTED | EXTENDED | CODEGEN` should not be 0 or more match. It should 0 or one match. Parser does not allow users to use more than one option in a single command.\n- The option `LOGICAL` is not supported. Issue an exception when users specify this option in the command.\n- The output of `EXPLAIN ` contains a weird empty line when the output of analyzed plan is empty. We should remove it. For example:\n  ```\n  == Parsed Logical Plan ==\n  CreateTable CatalogTable(`t`,CatalogTableType(MANAGED),CatalogStorageFormat(None,Some(org.apache.hadoop.mapred.TextInputFormat),Some(org.apache.hadoop.hive.ql.io.  HiveIgnoreKeyTextOutputFormat),None,false,Map()),List(CatalogColumn(col,int,true,None)),List(),List(),List(),-1,,1462725171656,-1,Map(),None,None,None), false\n\n  == Analyzed Logical Plan ==\n\n  CreateTable CatalogTable(`t`,CatalogTableType(MANAGED),CatalogStorageFormat(None,Some(org.apache.hadoop.mapred.TextInputFormat),Some(org.apache.hadoop.hive.ql.io.  HiveIgnoreKeyTextOutputFormat),None,false,Map()),List(CatalogColumn(col,int,true,None)),List(),List(),List(),-1,,1462725171656,-1,Map(),None,None,None), false\n\n  == Optimized Logical Plan ==\n  CreateTable CatalogTable(`t`,CatalogTableType(MANAGED),CatalogStorageFormat(None,Some(org.apache.hadoop.mapred.TextInputFormat),Some(org.apache.hadoop.hive.ql.io.  HiveIgnoreKeyTextOutputFormat),None,false,Map()),List(CatalogColumn(col,int,true,None)),List(),List(),List(),-1,,1462725171656,-1,Map(),None,None,None), false\n  ...\n  ```\n\n#### How was this patch tested?\nAdded and modified a few test cases\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #12991 from gatorsmile/explainCreateTable.\n\n(cherry picked from commit 570647267055cbe33291232b375e08fa1f5d8e7a)\nSigned-off-by: Herman van Hovell <hvanhovell@questtec.nl>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/58f77421b8dc3aec6252e504e7fd158fd2414acf"},{"sha":"ff2b715e0e27dade5feaedf916ea5d4a46854b06","author":{"email":"651d9e2ab320d6d21a117141758c2477389f223d@fb.com","name":"Sital Kedia"},"message":"[SPARK-14542][CORE] PipeRDD should allow configurable buffer size for…\n\n## What changes were proposed in this pull request?\n\nCurrently PipedRDD internally uses PrintWriter to write data to the stdin of the piped process, which by default uses a BufferedWriter of buffer size 8k. In our experiment, we have seen that 8k buffer size is too small and the job spends significant amount of CPU time in system calls to copy the data. We should have a way to configure the buffer size for the writer.\n\n## How was this patch tested?\nRan PipedRDDSuite tests.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #12309 from sitalkedia/bufferedPipedRDD.\n\n(cherry picked from commit a019e6efb71e4dce51ca91e41c3d293cf3a6ccb8)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/ff2b715e0e27dade5feaedf916ea5d4a46854b06"},{"sha":"841666d5d6af14b16e07c1cf3b1fdc88801a0eb2","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-14127][SQL] \"DESC <table>\": Extracts schema information from table properties for data source tables\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up of #12934 and #12844. This PR adds a set of utility methods in `DDLUtils` to help extract schema information (user-defined schema, partition columns, and bucketing information) from data source table properties. These utility methods are then used in `DescribeTableCommand` to refine output for data source tables. Before this PR, the aforementioned schema information are only shown as table properties, which are hard to read.\n\nSample output:\n\n```\n+----------------------------+---------------------------------------------------------+-------+\n|col_name                    |data_type                                                |comment|\n+----------------------------+---------------------------------------------------------+-------+\n|a                           |bigint                                                   |       |\n|b                           |bigint                                                   |       |\n|c                           |bigint                                                   |       |\n|d                           |bigint                                                   |       |\n|# Partition Information     |                                                         |       |\n|# col_name                  |                                                         |       |\n|d                           |                                                         |       |\n|                            |                                                         |       |\n|# Detailed Table Information|                                                         |       |\n|Database:                   |default                                                  |       |\n|Owner:                      |lian                                                     |       |\n|Create Time:                |Tue May 10 03:20:34 PDT 2016                             |       |\n|Last Access Time:           |Wed Dec 31 16:00:00 PST 1969                             |       |\n|Location:                   |file:/Users/lian/local/src/spark/workspace-a/target/...  |       |\n|Table Type:                 |MANAGED                                                  |       |\n|Table Parameters:           |                                                         |       |\n|  rawDataSize               |-1                                                       |       |\n|  numFiles                  |1                                                        |       |\n|  transient_lastDdlTime     |1462875634                                               |       |\n|  totalSize                 |684                                                      |       |\n|  spark.sql.sources.provider|parquet                                                  |       |\n|  EXTERNAL                  |FALSE                                                    |       |\n|  COLUMN_STATS_ACCURATE     |false                                                    |       |\n|  numRows                   |-1                                                       |       |\n|                            |                                                         |       |\n|# Storage Information       |                                                         |       |\n|SerDe Library:              |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe       |       |\n|InputFormat:                |org.apache.hadoop.mapred.SequenceFileInputFormat         |       |\n|OutputFormat:               |org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat|       |\n|Compressed:                 |No                                                       |       |\n|Num Buckets:                |2                                                        |       |\n|Bucket Columns:             |[b]                                                      |       |\n|Sort Columns:               |[c]                                                      |       |\n|Storage Desc Parameters:    |                                                         |       |\n|  path                      |file:/Users/lian/local/src/spark/workspace-a/target/...  |       |\n|  serialization.format      |1                                                        |       |\n+----------------------------+---------------------------------------------------------+-------+\n```\n\n## How was this patch tested?\n\nTest cases are added in `HiveDDLSuite` to check command output.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #13025 from liancheng/spark-14127-extract-schema-info.\n\n(cherry picked from commit 8a12580d25b1ce5abc45c600483fad69f90ca333)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/841666d5d6af14b16e07c1cf3b1fdc88801a0eb2"},{"sha":"4aa9052974087a77905981f8bbef416312a52e80","author":{"email":"73b9523a173a4d37b1985403702ac17bd8e586ea@gmail.com","name":"Pete Robbins"},"message":"[SPARK-15154] [SQL] Change key types to Long in tests\n\n## What changes were proposed in this pull request?\n\nAs reported in the Jira the 2 tests changed here are using a key of type Integer where the Spark sql code assumes the type is Long. This PR changes the tests to use the correct key types.\n\n## How was this patch tested?\n\nTest builds run on both Big Endian and Little Endian platforms\n\nAuthor: Pete Robbins <robbinspg@gmail.com>\n\nCloses #13009 from robbinspg/HashedRelationSuiteFix.\n\n(cherry picked from commit 2dfb9cd1f7e7f0438ce571aae7e3a7b77d4082b7)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/4aa9052974087a77905981f8bbef416312a52e80"},{"sha":"1a6272e2650087be4f553caf6f2932f6b989a0e6","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-14773] [SPARK-15179] [SQL] Fix SQL building and enable Hive tests\n\n## What changes were proposed in this pull request?\nThis PR fixes SQL building for predicate subqueries and correlated scalar subqueries. It also enables most Hive subquery tests.\n\n## How was this patch tested?\nEnabled new tests in HiveComparisionSuite.\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #12988 from hvanhovell/SPARK-14773.\n\n(cherry picked from commit 2646265368aab0f0b800d3052e557dea7c40c2d6)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1a6272e2650087be4f553caf6f2932f6b989a0e6"},{"sha":"a66ebbca0f4645de598c0aebf826bd86a8e363c6","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-13382][DOCS][PYSPARK] Update pyspark testing notes in build docs\n\n## What changes were proposed in this pull request?\n\nThe current build documents don't specify that for PySpark tests we need to include Hive in the assembly otherwise the ORC tests fail.\n\n## How was the this patch tested?\n\nManually built the docs locally. Ran the provided build command follow by the PySpark SQL tests.\n\n![pyspark2](https://cloud.githubusercontent.com/assets/59893/13190008/8829cde4-d70f-11e5-8ff5-a88b7894d2ad.png)\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #11278 from holdenk/SPARK-13382-update-pyspark-testing-notes-r2.\n\n(cherry picked from commit 488863d87340e0fb68e81ba0be6df7d7c9723992)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/a66ebbca0f4645de598c0aebf826bd86a8e363c6"},{"sha":"918bf6e1baa8682c21484f0a3088c3bd0bafeb5f","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13670][LAUNCHER] Propagate error from launcher to shell.\n\nbash doesn't really propagate errors from subshells when using redirection\nthe way spark-class does; so, instead, this change captures the exit code\nof the launcher process in the command array, and checks it before executing\nthe actual command.\n\nTested by injecting an error in Main.java (the launcher entry point) and\nverifying the shell gets the right exit code from spark-class.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #12910 from vanzin/SPARK-13670.\n\n(cherry picked from commit 36c5892b4631d1d5d70e569fce12efedd82bdef0)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/918bf6e1baa8682c21484f0a3088c3bd0bafeb5f"},{"sha":"af12b0a50a55595b9c6631bcee233ee4f3484917","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-11249][LAUNCHER] Throw error if app resource is not provided.\n\nWithout this, the code would build an invalid spark-submit command line,\nand a more cryptic error would be presented to the user. Also, expose\na constant that allows users to set a dummy resource in cases where\nthey don't need an actual resource file; for backwards compatibility,\nthat uses the same \"spark-internal\" resource that Spark itself uses.\n\nTested via unit tests, run-example, spark-shell, and running the\nthrift server with mixed spark and hive command line arguments.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #12909 from vanzin/SPARK-11249.\n\n(cherry picked from commit 0b9cae42426e14060bc6182c037fd715f35a2d23)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/af12b0a50a55595b9c6631bcee233ee4f3484917"},{"sha":"19a9c23c2d2ee6d16d8078db0730602ae5a591ed","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12837][CORE] reduce network IO for accumulators\n\nSending un-updated accumulators back to driver makes no sense, as merging a zero value accumulator is a no-op. We should only send back updated accumulators, to save network IO.\n\nnew test in `TaskContextSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #12899 from cloud-fan/acc.\n\n(cherry picked from commit bcfee153b1cacfe617e602f3b72c0877e0bdf1f7)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/19a9c23c2d2ee6d16d8078db0730602ae5a591ed"},{"sha":"5bf74b44d9efcb8b0f0c3e7d129bc5ba31419551","author":{"email":"ad9280c159074d9ec90899b584f520606e83d10e@techaddict.me","name":"Sandeep Singh"},"message":"[SPARK-15037][SQL][MLLIB] Use SparkSession instead of SQLContext in Scala/Java TestSuites\n\n## What changes were proposed in this pull request?\nUse SparkSession instead of SQLContext in Scala/Java TestSuites\nas this PR already very big working Python TestSuites in a diff PR.\n\n## How was this patch tested?\nExisting tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #12907 from techaddict/SPARK-15037.\n\n(cherry picked from commit ed0b4070fb50054b1ecf66ff6c32458a4967dfd3)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/5bf74b44d9efcb8b0f0c3e7d129bc5ba31419551"},{"sha":"42db140c5e134fe442d5160836f576f202aa17e5","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-14603][SQL] Verification of Metadata Operations by Session Catalog\n\nSince we cannot really trust if the underlying external catalog can throw exceptions when there is an invalid metadata operation, let's do it in SessionCatalog.\n\n- [X] The first step is to unify the error messages issued in Hive-specific Session Catalog and general Session Catalog.\n- [X] The second step is to verify the inputs of metadata operations for partitioning-related operations. This is moved to a separate PR: https://github.com/apache/spark/pull/12801\n- [X] The third step is to add database existence verification in `SessionCatalog`\n- [X] The fourth step is to add table existence verification in `SessionCatalog`\n- [X] The fifth step is to add function existence verification in `SessionCatalog`\n\nAdd test cases and verify the error messages we issued\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\nAuthor: xiaoli <lixiao1983@gmail.com>\nAuthor: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>\n\nCloses #12385 from gatorsmile/verifySessionAPIs.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/42db140c5e134fe442d5160836f576f202aa17e5"},{"sha":"bd7fd14c93746556bd99faa640fb6b95defef148","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-15037][HOTFIX] Replace `sqlContext` and `sparkSession` with `spark`.\n\nThis replaces `sparkSession` with `spark` in CatalogSuite.scala.\n\nPass the Jenkins tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13030 from dongjoon-hyun/hotfix_sparkSession.\n\n(cherry picked from commit db3b4a20150ff7fb1caaf62ab3d2a2f1e632af36)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/bd7fd14c93746556bd99faa640fb6b95defef148"},{"sha":"a432e80b81cf898dd1886860ec7d50aa1dc43e7d","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-15037][HOTFIX] Don't create 2 SparkSessions in constructor\n\n## What changes were proposed in this pull request?\n\nAfter #12907 `TestSparkSession` creates a spark session in one of the constructors just to get the `SparkContext` from it. This ends up creating 2 `SparkSession`s from one call, which is definitely not what we want.\n\n## How was this patch tested?\n\nJenkins.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13031 from andrewor14/sql-test.\n\n(cherry picked from commit 69641066ae1d35c33b082451cef636a7f2e646d9)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/a432e80b81cf898dd1886860ec7d50aa1dc43e7d"},{"sha":"82f69594fa625bc3fb643895543b0448d2c03f3e","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-15195][PYSPARK][DOCS] Update ml.tuning PyDocs\n\n## What changes were proposed in this pull request?\n\nTag classes in ml.tuning as experimental, add docs for kfolds avg metric, and copy TrainValidationSplit scaladoc for more detailed explanation.\n\n## How was this patch tested?\n\nbuilt docs locally\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #12967 from holdenk/SPARK-15195-pydoc-ml-tuning.\n\n(cherry picked from commit 93353b0113158c87e09f0bad91a663a92e9cf1bc)\nSigned-off-by: Nick Pentreath <nickp@za.ibm.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/82f69594fa625bc3fb643895543b0448d2c03f3e"},{"sha":"5a4a188fea2bdf3bc1096557b7caf7383e628037","author":{"email":"91c5214c0a28a5aafc3cf26ac6277e87cb71e65e@gmail.com","name":"Subhobrata Dey"},"message":"[SPARK-14642][SQL] import org.apache.spark.sql.expressions._ breaks udf under functions\n\n## What changes were proposed in this pull request?\n\nPR fixes the import issue which breaks udf functions.\n\nThe following code snippet throws an error\n\n```\nscala> import org.apache.spark.sql.functions._\nimport org.apache.spark.sql.functions._\n\nscala> import org.apache.spark.sql.expressions._\nimport org.apache.spark.sql.expressions._\n\nscala> udf((v: String) => v.stripSuffix(\"-abc\"))\n<console>:30: error: No TypeTag available for String\n       udf((v: String) => v.stripSuffix(\"-abc\"))\n```\n\nThis PR resolves the issue.\n\n## How was this patch tested?\n\npatch tested with unit tests.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: Subhobrata Dey <sbcd90@gmail.com>\n\nCloses #12458 from sbcd90/udfFuncBreak.\n\n(cherry picked from commit 89f73f674126bbc1cc101f0f5731b5750f1c90c8)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/5a4a188fea2bdf3bc1096557b7caf7383e628037"},{"sha":"0ab1958868e265dc13c066f5ec26e573a94a2490","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-14986][SQL] Return correct result for empty LATERAL VIEW OUTER\n\n## What changes were proposed in this pull request?\nA Generate with the `outer` flag enabled should always return one or more rows for every input row. The optimizer currently violates this by rewriting `outer` Generates that do not contain columns of the child plan into an unjoined generate, for example:\n```sql\nselect e from a lateral view outer explode(a.b) as e\n```\nThe result of this is that `outer` Generate does not produce output at all when the Generators' input expression is empty. This PR fixes this.\n\n## How was this patch tested?\nAdded test case to `SQLQuerySuite`.\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #12906 from hvanhovell/SPARK-14986.\n\n(cherry picked from commit d28c67544b26c38d51a31d1f8dac3fc23860e1ef)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/0ab1958868e265dc13c066f5ec26e573a94a2490"},{"sha":"95f254994d07a83178eecd668cccf90a12a4b956","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-6005][TESTS] Fix flaky test: o.a.s.streaming.kafka.DirectKafkaStreamSuite.offset recovery\n\n## What changes were proposed in this pull request?\n\nBecause this test extracts data from `DStream.generatedRDDs` before stopping, it may get data before checkpointing. Then after recovering from the checkpoint, `recoveredOffsetRanges` may contain something not in `offsetRangesBeforeStop`, which will fail the test. Adding `Thread.sleep(1000)` before `ssc.stop()` will reproduce this failure.\n\nThis PR just moves the logic of `offsetRangesBeforeStop` (also renamed to `offsetRangesAfterStop`) after `ssc.stop()` to fix the flaky test.\n\n## How was this patch tested?\n\nJenkins unit tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #12903 from zsxwing/SPARK-6005.\n\n(cherry picked from commit 9533f5390a3ad7ab96a7bea01cdb6aed89503a51)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/95f254994d07a83178eecd668cccf90a12a4b956"},{"sha":"1db027d113ada12d0f11cbd7e6da8a90b4ff027b","author":{"email":"ad9280c159074d9ec90899b584f520606e83d10e@techaddict.me","name":"Sandeep Singh"},"message":"[SPARK-15249][SQL] Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource\n\nUse FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource\nsee: TODO's here\nhttps://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L36\nhttps://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala#L42\n\nExisting tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #13024 from techaddict/SPARK-15249.\n\n(cherry picked from commit da02d006bbb5c4fe62abd5542b9fff7d1c58603c)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1db027d113ada12d0f11cbd7e6da8a90b4ff027b"},{"sha":"f021f346015a9c9fd045fcbea51748061273accd","author":{"email":"623dda1cf42194a966de8b161dee064823eb61d3@126.com","name":"Xin Ren"},"message":"[SPARK-14936][BUILD][TESTS] FlumePollingStreamSuite is slow\n\nhttps://issues.apache.org/jira/browse/SPARK-14936\n\n## What changes were proposed in this pull request?\n\nFlumePollingStreamSuite contains two tests which run for a minute each. This seems excessively slow and we should speed it up if possible.\n\nIn this PR, instead of creating `StreamingContext` directly from `conf`, here an underlying `SparkContext` is created before all and it is used to create  each`StreamingContext`.\n\nRunning time is reduced by avoiding multiple `SparkContext` creations and destroys.\n\n## How was this patch tested?\n\nTested on my local machine running `testOnly *.FlumePollingStreamSuite`\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #12845 from keypointt/SPARK-14936.\n\n(cherry picked from commit 86475520f88f90c9d3b71516f65ccc0e9d244863)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f021f346015a9c9fd045fcbea51748061273accd"},{"sha":"d8c2da9a46f240438c5350f73a7b3daa07c101a8","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-14837][SQL][STREAMING] Added support in file stream source for reading new files added to subdirs\n\n## What changes were proposed in this pull request?\nCurrently, file stream source can only find new files if they appear in the directory given to the source, but not if they appear in subdirs. This PR add support for providing glob patterns when creating file stream source so that it can find new files in nested directories based on the glob pattern.\n\n## How was this patch tested?\n\nUnit test that tests when new files are discovered with globs and partitioned directories.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #12616 from tdas/SPARK-14837.\n\n(cherry picked from commit d9ca9fd3e582f9d29f8887c095637c93a8b93651)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/d8c2da9a46f240438c5350f73a7b3daa07c101a8"}]}