{"push_id":1107774125,"size":21,"distinct_size":21,"ref":"refs/heads/branch-2.0","head":"114be703d5655b6456955e795e670cd62915b37e","before":"36f711dc6d577f585fed424da5c49f04c5226591","commits":[{"sha":"1e7d8ba5d6212c6e1e57a48f56d68c03c7386e66","author":{"email":"70018956f0f6b0e0ad9070192ca2405a35a8a6ae@intel.com","name":"mwws"},"message":"[SPARK-14976][STREAMING] make StreamingContext.textFileStream support wildcard\n\n## What changes were proposed in this pull request?\nmake StreamingContext.textFileStream support wildcard\nlike /home/user/*/file\n\n## How was this patch tested?\nI did manual test and added a new unit test case\n\nAuthor: mwws <wei.mao@intel.com>\nAuthor: unknown <maowei@maowei-MOBL.ccr.corp.intel.com>\n\nCloses #12752 from mwws/SPARK_FileStream.\n\n(cherry picked from commit 33597810ec256cd9bd363bad9239cc6d5b707a6f)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1e7d8ba5d6212c6e1e57a48f56d68c03c7386e66"},{"sha":"1753f6502ce7c81f43e120b0dc324674b4d5331c","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-15238] Clarify supported Python versions\n\nThis PR:\n* Clarifies that Spark *does* support Python 3, starting with Python 3.4.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #13017 from nchammas/supported-python-versions.\n\n(cherry picked from commit fafc95af79fa34f82964a86407c2ee046eda3814)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1753f6502ce7c81f43e120b0dc324674b4d5331c"},{"sha":"3bd7a89bd08bfcaf94039d3070ba60bdf2c9a2e4","author":{"email":"21ef94aec54fae1a36abbd011347596ae66d9959@foxmail.com","name":"Zheng RuiFeng"},"message":"[SPARK-15150][EXAMPLE][DOC] Update LDA examples\n\n## What changes were proposed in this pull request?\n1,create a libsvm-type dataset for lda: `data/mllib/sample_lda_libsvm_data.txt`\n2,add python example\n3,directly read the datafile in examples\n4,BTW, change to `SparkSession` in `aft_survival_regression.py`\n\n## How was this patch tested?\nmanual tests\n`./bin/spark-submit examples/src/main/python/ml/lda_example.py`\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #12927 from zhengruifeng/lda_pe.\n\n(cherry picked from commit d88afabdfa83be47f36d833105aadd6b818ceeee)\nSigned-off-by: Nick Pentreath <nickp@za.ibm.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/3bd7a89bd08bfcaf94039d3070ba60bdf2c9a2e4"},{"sha":"749c29bc099c20aa6156b843cf7c9216315cd5a6","author":{"email":"3aa38c00921c7acfa71de69042a28a39f706d59b@us.ibm.com","name":"xin Wu"},"message":"[SPARK-14933][SQL] Failed to create view out of a parquet or orc table\n\n## What changes were proposed in this pull request?\n#### Symptom\n If a table is created as parquet or ORC table with hive syntaxt DDL, such as\n```SQL\ncreate table t1 (c1 int, c2 string) stored as parquet\n```\nThe following command will fail\n```SQL\ncreate view v1 as select * from t1\n```\n#### Root Cause\nCurrently, `HiveMetaStoreCatalog` converts Paruqet/Orc tables to `LogicalRelation` without giving any `tableIdentifier`. `SQLBuilder` expects the `LogicalRelation` to have an associated `tableIdentifier`. However, the `LogicalRelation` created earlier does not have such a `tableIdentifier`. Thus, `SQLBuilder.toSQL` can not recognize this logical plan and issue an exception.\n\nThis PR is to assign a `TableIdentifier` to the `LogicalRelation` when resolving parquet or orc tables in `HiveMetaStoreCatalog`.\n\n## How was this patch tested?\ntestcases created and dev/run-tests is run.\n\nAuthor: xin Wu <xinwu@us.ibm.com>\n\nCloses #12716 from xwu0226/SPARK_14933.\n\n(cherry picked from commit 427c20dd6d84cb9de1aac322183bc6e7b72ca25d)\nSigned-off-by: Cheng Lian <lian@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/749c29bc099c20aa6156b843cf7c9216315cd5a6"},{"sha":"0858a82c141fe9b2d2c94a62c16657dcd6c3ec8b","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-15268][SQL] Make JavaTypeInference work with UDTRegistration\n\n## What changes were proposed in this pull request?\n\nWe have a private `UDTRegistration` API to register user defined type. Currently `JavaTypeInference` can't work with it. So `SparkSession.createDataFrame` from a bean class will not correctly infer the schema of the bean class.\n\n## How was this patch tested?\n`VectorUDTSuite`.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #13046 from viirya/fix-udt-registry-javatypeinference.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/0858a82c141fe9b2d2c94a62c16657dcd6c3ec8b"},{"sha":"403ba65133c3bb45ad6e236464218d8970b893c9","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14933][HOTFIX] Replace `sqlContext` with `spark`.\n\n## What changes were proposed in this pull request?\n\nThis fixes compile errors.\n\n## How was this patch tested?\n\nPass the Jenkins tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13053 from dongjoon-hyun/hotfix_sqlquerysuite.\n\n(cherry picked from commit e1576478bde2b9146bdbd4f2bae20a4011b20229)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/403ba65133c3bb45ad6e236464218d8970b893c9"},{"sha":"381a825890b74afa0bd7325265aa4e15bbc8f10f","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-15241] [SPARK-15242] [SQL] fix 2 decimal-related issues in RowEncoder\n\n## What changes were proposed in this pull request?\n\nSPARK-15241: We now support java decimal and catalyst decimal in external row, it makes sense to also support scala decimal.\n\nSPARK-15242: This is a long-standing bug, and is exposed after https://github.com/apache/spark/pull/12364, which eliminate the `If` expression if the field is not nullable:\n```\nval fieldValue = serializerFor(\n  GetExternalRowField(inputObject, i, externalDataTypeForInput(f.dataType)),\n  f.dataType)\nif (f.nullable) {\n  If(\n    Invoke(inputObject, \"isNullAt\", BooleanType, Literal(i) :: Nil),\n    Literal.create(null, f.dataType),\n    fieldValue)\n} else {\n  fieldValue\n}\n```\n\nPreviously, we always use `DecimalType.SYSTEM_DEFAULT` as the output type of converted decimal field, which is wrong as it doesn't match the real decimal type. However, it works well because we always put converted field into `If` expression to do the null check, and `If` use its `trueValue`'s data type as its output type.\nNow if we have a not nullable decimal field, then the converted field's output type will be `DecimalType.SYSTEM_DEFAULT`, and we will write wrong data into unsafe row.\n\nThe fix is simple, just use the given decimal type as the output type of converted decimal field.\n\nThese 2 issues was found at https://github.com/apache/spark/pull/13008\n\n## How was this patch tested?\n\nnew tests in RowEncoderSuite\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #13019 from cloud-fan/encoder-decimal.\n\n(cherry picked from commit d8935db5ecb7c959585411da9bf1e9a9c4d5cb37)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/381a825890b74afa0bd7325265aa4e15bbc8f10f"},{"sha":"1b90adc03bd88fbd664d4692092142672ac5a36b","author":{"email":"ad9280c159074d9ec90899b584f520606e83d10e@techaddict.me","name":"Sandeep Singh"},"message":"[SPARK-15037] [SQL] [MLLIB] Part2: Use SparkSession instead of SQLContext in Python TestSuites\n\n## What changes were proposed in this pull request?\nUse SparkSession instead of SQLContext in Python TestSuites\n\n## How was this patch tested?\nExisting tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #13044 from techaddict/SPARK-15037-python.\n\n(cherry picked from commit 29314379729de4082bd2297c9e5289e3e4a0115e)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/1b90adc03bd88fbd664d4692092142672ac5a36b"},{"sha":"e3703c41177c01c41516d1669c1ffa239074e59f","author":{"email":"cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@databricks.com","name":"Eric Liang"},"message":"[SPARK-15259] Sort time metric should not include spill and record insertion time\n\n## What changes were proposed in this pull request?\n\nAfter SPARK-14669 it seems the sort time metric includes both spill and record insertion time. This makes it not very useful since the metric becomes close to the total execution time of the node.\n\nWe should track just the time spent for in-memory sort, as before.\n\n## How was this patch tested?\n\nVerified metric in the UI, also unit test on UnsafeExternalRowSorter.\n\ncc davies\n\nAuthor: Eric Liang <ekl@databricks.com>\nAuthor: Eric Liang <ekhliang@gmail.com>\n\nCloses #13035 from ericl/fix-metrics.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/e3703c41177c01c41516d1669c1ffa239074e59f"},{"sha":"56e1e2f1706c857f72f519e51c51e39e30638eb6","author":{"email":"c4fe2b1d90ef8f2548c8aeabfa633434316f0305@koeninger.org","name":"cody koeninger"},"message":"[SPARK-15085][STREAMING][KAFKA] Rename streaming-kafka artifact\n\n## What changes were proposed in this pull request?\nRenaming the streaming-kafka artifact to include kafka version, in anticipation of needing a different artifact for later kafka versions\n\n## How was this patch tested?\nUnit tests\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #12946 from koeninger/SPARK-15085.\n\n(cherry picked from commit 89e67d6667d5f8be9c6fb6c120fbcd350ae2950d)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/56e1e2f1706c857f72f519e51c51e39e30638eb6"},{"sha":"6b36185d0eece2c438ac9892cc34547da90c3c1c","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-15248][SQL] Make MetastoreFileCatalog consider directories from partition specs of a partitioned metastore table\n\nTable partitions can be added with locations different from default warehouse location of a hive table.\n`CREATE TABLE parquetTable (a int) PARTITIONED BY (b int) STORED AS parquet `\n`ALTER TABLE parquetTable ADD PARTITION (b=1) LOCATION '/partition'`\nQuerying such a table throws error as the MetastoreFileCatalog does not list the added partition directory, it only lists the default base location.\n\n```\n[info] - SPARK-15248: explicitly added partitions should be readable *** FAILED *** (1 second, 8 milliseconds)\n[info]   java.util.NoSuchElementException: key not found: file:/Users/tdas/Projects/Spark/spark2/target/tmp/spark-b39ad224-c5d1-4966-8981-fb45a2066d61/partition\n[info]   at scala.collection.MapLike$class.default(MapLike.scala:228)\n[info]   at scala.collection.AbstractMap.default(Map.scala:59)\n[info]   at scala.collection.MapLike$class.apply(MapLike.scala:141)\n[info]   at scala.collection.AbstractMap.apply(Map.scala:59)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningAwareFileCatalog$$anonfun$listFiles$1.apply(PartitioningAwareFileCatalog.scala:59)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningAwareFileCatalog$$anonfun$listFiles$1.apply(PartitioningAwareFileCatalog.scala:55)\n[info]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n[info]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n[info]   at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n[info]   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n[info]   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n[info]   at scala.collection.AbstractTraversable.map(Traversable.scala:104)\n[info]   at org.apache.spark.sql.execution.datasources.PartitioningAwareFileCatalog.listFiles(PartitioningAwareFileCatalog.scala:55)\n[info]   at org.apache.spark.sql.execution.datasources.FileSourceStrategy$.apply(FileSourceStrategy.scala:93)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:59)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:59)\n[info]   at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)\n[info]   at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:60)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner.planLater(QueryPlanner.scala:55)\n[info]   at org.apache.spark.sql.execution.SparkStrategies$SpecialLimits$.apply(SparkStrategies.scala:55)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:59)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:59)\n[info]   at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)\n[info]   at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:60)\n[info]   at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:77)\n[info]   at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:75)\n[info]   at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:82)\n[info]   at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:82)\n[info]   at org.apache.spark.sql.QueryTest.assertEmptyMissingInput(QueryTest.scala:330)\n[info]   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:146)\n[info]   at org.apache.spark.sql.QueryTest.checkAnswer(QueryTest.scala:159)\n[info]   at org.apache.spark.sql.hive.ParquetMetastoreSuite$$anonfun$12$$anonfun$apply$mcV$sp$7$$anonfun$apply$mcV$sp$25.apply(parquetSuites.scala:554)\n[info]   at org.apache.spark.sql.hive.ParquetMetastoreSuite$$anonfun$12$$anonfun$apply$mcV$sp$7$$anonfun$apply$mcV$sp$25.apply(parquetSuites.scala:535)\n[info]   at org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:125)\n[info]   at org.apache.spark.sql.hive.ParquetPartitioningTest.withTempDir(parquetSuites.scala:726)\n[info]   at org.apache.spark.sql.hive.ParquetMetastoreSuite$$anonfun$12$$anonfun$apply$mcV$sp$7.apply$mcV$sp(parquetSuites.scala:535)\n[info]   at org.apache.spark.sql.test.SQLTestUtils$class.withTable(SQLTestUtils.scala:166)\n[info]   at org.apache.spark.sql.hive.ParquetPartitioningTest.withTable(parquetSuites.scala:726)\n[info]   at org.apache.spark.sql.hive.ParquetMetastoreSuite$$anonfun$12.apply$mcV$sp(parquetSuites.scala:534)\n[info]   at org.apache.spark.sql.hive.ParquetMetastoreSuite$$anonfun$12.apply(parquetSuites.scala:534)\n[info]   at org.apache.spark.sql.hive.ParquetMetastoreSuite$$anonfun$12.apply(parquetSuites.scala:534)\n```\n\nThe solution in this PR to get the paths to list from the partition spec and not rely on the default table path alone.\n\nunit tests.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #13022 from tdas/SPARK-15248.\n\n(cherry picked from commit 81c68eceba3a857ba7349c6892dc336c3ebd11dc)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/6b36185d0eece2c438ac9892cc34547da90c3c1c"},{"sha":"83050ddb84ce65b646a88d7d365a960f423eee19","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-15260] Atomically resize memory pools\n\n## What changes were proposed in this pull request?\n\nWhen we acquire execution memory, we do a lot of things between shrinking the storage memory pool and enlarging the execution memory pool. In particular, we call `memoryStore.evictBlocksToFreeSpace`, which may do a lot of I/O and can throw exceptions. If an exception is thrown, the pool sizes on that executor will be in a bad state.\n\nThis patch minimizes the things we do between the two calls to make the resizing more atomic.\n\n## How was this patch tested?\n\nJenkins.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13039 from andrewor14/safer-pool.\n\n(cherry picked from commit bb88ad4e0e870c88d474c71939a19541522a3023)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/83050ddb84ce65b646a88d7d365a960f423eee19"},{"sha":"6e08eb46951648345e6d21b85a46759422cb3e4b","author":{"email":"0c1e002f289accce3282ac32d1c7dd7c3f34c5b7@adpilot.pl","name":"Maciej Brynski"},"message":"[SPARK-12200][SQL] Add __contains__ implementation to Row\n\nhttps://issues.apache.org/jira/browse/SPARK-12200\n\nAuthor: Maciej Brynski <maciej.brynski@adpilot.pl>\nAuthor: Maciej Bryński <maciek-github@brynski.pl>\n\nCloses #10194 from maver1ck/master.\n\n(cherry picked from commit 7ecd496884f6f126ab186b9ceaa861a571d6155c)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/6e08eb46951648345e6d21b85a46759422cb3e4b"},{"sha":"2454f6abf29c938420dda8319a4e4afd758fc4e3","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-15262] Synchronize block manager / scheduler executor state\n\n## What changes were proposed in this pull request?\n\nIf an executor is still alive even after the scheduler has removed its metadata, we may receive a heartbeat from that executor and tell its block manager to reregister itself. If that happens, the block manager master will know about the executor, but the scheduler will not.\n\nThat is a dangerous situation, because when the executor does get disconnected later, the scheduler will not ask the block manager to also remove metadata for that executor. Later, when we try to clean up an RDD or a broadcast variable, we may try to send a message to that executor, triggering an exception.\n\n## How was this patch tested?\n\nJenkins.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13055 from andrewor14/block-manager-remove.\n\n(cherry picked from commit 40a949aae9c3040019a52482d091912a85b0f4d4)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/2454f6abf29c938420dda8319a4e4afd758fc4e3"},{"sha":"0699acc1bafcc2ce3f2c4eb795b0e4c6b3448185","author":{"email":"ad9280c159074d9ec90899b584f520606e83d10e@techaddict.me","name":"Sandeep Singh"},"message":"[SPARK-15270] [SQL] Use SparkSession Builder to build a session with HiveSupport\n\n## What changes were proposed in this pull request?\nBefore:\nCreating a hiveContext was failing\n```python\nfrom pyspark.sql import HiveContext\nhc = HiveContext(sc)\n```\nwith\n```\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"spark-2.0/python/pyspark/sql/context.py\", line 458, in __init__\n    sparkSession = SparkSession.withHiveSupport(sparkContext)\n  File \"spark-2.0/python/pyspark/sql/session.py\", line 192, in withHiveSupport\n    jsparkSession = sparkContext._jvm.SparkSession.withHiveSupport(sparkContext._jsc.sc())\n  File \"spark-2.0/python/lib/py4j-0.9.2-src.zip/py4j/java_gateway.py\", line 1048, in __getattr__\npy4j.protocol.Py4JError: org.apache.spark.sql.SparkSession.withHiveSupport does not exist in the JVM\n```\n\nNow:\n```python\n>>> from pyspark.sql import HiveContext\n>>> hc = HiveContext(sc)\n>>> hc.range(0, 100)\nDataFrame[id: bigint]\n>>> hc.range(0, 100).count()\n100\n```\n## How was this patch tested?\nExisting Tests, tested manually in python shell\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #13056 from techaddict/SPARK-15270.\n\n(cherry picked from commit de9c85ccaacd12de9837eb88eae0a7e7ededd679)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/0699acc1bafcc2ce3f2c4eb795b0e4c6b3448185"},{"sha":"b1e14d9bfb4df836b827b572e8dd07548a41bc3f","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-15278] [SQL] Remove experimental tag from Python DataFrame\n\n## What changes were proposed in this pull request?\nEarlier we removed experimental tag for Scala/Java DataFrames, but haven't done so for Python. This patch removes the experimental flag for Python and declares them stable.\n\n## How was this patch tested?\nN/A.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13062 from rxin/SPARK-15278.\n\n(cherry picked from commit 40ba87f769ab03721d01c7960b89a8c414fcfbca)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/b1e14d9bfb4df836b827b572e8dd07548a41bc3f"},{"sha":"4e56857ca0c3d61cc257ffd3bd1c59ed44c862d7","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-15257][SQL] Require CREATE EXTERNAL TABLE to specify LOCATION\n\n## What changes were proposed in this pull request?\n\nBefore:\n```sql\n-- uses warehouse dir anyway\nCREATE EXTERNAL TABLE my_tab\n-- doesn't actually delete the data\nDROP TABLE my_tab\n```\nAfter:\n```sql\n-- no location is provided, throws exception\nCREATE EXTERNAL TABLE my_tab\n-- creates an external table using that location\nCREATE EXTERNAL TABLE my_tab LOCATION '/path/to/something'\n-- doesn't delete the data, which is expected\nDROP TABLE my_tab\n```\n\n## How was this patch tested?\n\nNew test in `DDLCommandSuite`\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13032 from andrewor14/create-external-table-location.\n\n(cherry picked from commit 8881765ac7ac6ba6fe9ef0d0a669c08cca58ed93)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/4e56857ca0c3d61cc257ffd3bd1c59ed44c862d7"},{"sha":"f9ea54575659f489750a37e9a8c1105a825c94f8","author":{"email":"fbfd45890515f7eeeeabf87d7a80d3449c3bce83@gmail.com","name":"Nicholas Chammas"},"message":"[SPARK-15256] [SQL] [PySpark] Clarify DataFrameReader.jdbc() docstring\n\nThis PR:\n* Corrects the documentation for the `properties` parameter, which is supposed to be a dictionary and not a list.\n* Generally clarifies the Python docstring for DataFrameReader.jdbc() by pulling from the [Scala docstrings](https://github.com/apache/spark/blob/b28137764716f56fa1a923c4278624a56364a505/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L201-L251) and rephrasing things.\n* Corrects minor Sphinx typos.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #13034 from nchammas/SPARK-15256.\n\n(cherry picked from commit b9cf617a6fa8812b45ff33acd109757a59f91dfa)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f9ea54575659f489750a37e9a8c1105a825c94f8"},{"sha":"f763c14851f6e55e61de8ef79ae449a7257a547d","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-15276][SQL] CREATE TABLE with LOCATION should imply EXTERNAL\n\n## What changes were proposed in this pull request?\n\nBefore:\n```sql\n-- uses that location but issues a warning\nCREATE TABLE my_tab LOCATION /some/path\n-- deletes any existing data in the specified location\nDROP TABLE my_tab\n```\n\nAfter:\n```sql\n-- uses that location but creates an EXTERNAL table instead\nCREATE TABLE my_tab LOCATION /some/path\n-- does not delete the data at /some/path\nDROP TABLE my_tab\n```\n\nThis patch essentially makes the `EXTERNAL` field optional. This is related to #13032.\n\n## How was this patch tested?\n\nNew test in `DDLCommandSuite`.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13060 from andrewor14/location-implies-external.\n\n(cherry picked from commit f14c4ba001fbdbcc9faa46896f1f9d08a7d06609)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f763c14851f6e55e61de8ef79ae449a7257a547d"},{"sha":"f8804bb1082682e34d1ea225f931253247ca094e","author":{"email":"c692d6a10598e0a801576fdd4ecf3c37e45bfbc4@databricks.com","name":"Bill Chambers"},"message":"[SPARK-15264][SPARK-15274][SQL] CSV Reader Error on Blank Column Names\n\n## What changes were proposed in this pull request?\n\nWhen a CSV begins with:\n- `,,`\nOR\n- `\"\",\"\",`\n\nmeaning that the first column names are either empty or blank strings and `header` is specified to be `true`, then the column name is replaced with `C` + the index number of that given column. For example, if you were to read in the CSV:\n```\n\"\",\"second column\"\n\"hello\", \"there\"\n```\nThen column names would become `\"C0\", \"second column\"`.\n\nThis behavior aligns with what currently happens when `header` is specified to be `false` in recent versions of Spark.\n\n### Current Behavior in Spark <=1.6\nIn Spark <=1.6, a CSV with a blank column name becomes a blank string, `\"\"`, meaning that this column cannot be accessed. However the CSV reads in without issue.\n\n### Current Behavior in Spark 2.0\nSpark throws a NullPointerError and will not read in the file.\n\n#### Reproduction in 2.0\nhttps://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/346304/2828750690305044/484361/latest.html\n\n## How was this patch tested?\nA new test was added to `CSVSuite` to account for this issue. We then have asserts that test for being able to select both the empty column names as well as the regular column names.\n\nAuthor: Bill Chambers <bill@databricks.com>\nAuthor: Bill Chambers <wchambers@ischool.berkeley.edu>\n\nCloses #13041 from anabranch/master.\n\n(cherry picked from commit 603f4453a16825cc5773cfe24d6ae4cee5ec949a)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f8804bb1082682e34d1ea225f931253247ca094e"}]}