{"push_id":1292568224,"size":44,"distinct_size":3,"ref":"refs/heads/showCreateTableNew","head":"d3dcb564509fd2a32a3fadefb811495affaaa466","before":"c9cdf44b561c5e88a108cd09ad17842332d14162","commits":[{"sha":"8d08f43d09157b98e559c0be6ce6fd571a35e0d1","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17279][SQL] better error message for exceptions during ScalaUDF execution\n\n## What changes were proposed in this pull request?\n\nIf `ScalaUDF` throws exceptions during executing user code, sometimes it's hard for users to figure out what's wrong, especially when they use Spark shell. An example\n```\norg.apache.spark.SparkException: Job aborted due to stage failure: Task 12 in stage 325.0 failed 4 times, most recent failure: Lost task 12.3 in stage 325.0 (TID 35622, 10.0.207.202): java.lang.NullPointerException\n\tat line8414e872fb8b42aba390efc153d1611a12.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(<console>:40)\n\tat line8414e872fb8b42aba390efc153d1611a12.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(<console>:40)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n...\n```\nWe should catch these exceptions and rethrow them with better error message, to say that the exception is happened in scala udf.\n\nThis PR also does some clean up for `ScalaUDF` and add a unit test suite for it.\n\n## How was this patch tested?\n\nthe new test suite\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14850 from cloud-fan/npe.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/8d08f43d09157b98e559c0be6ce6fd571a35e0d1"},{"sha":"afb3d5d301d004fd748ad305b3d72066af4ebb6c","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-17369][SQL] MetastoreRelation toJSON throws AssertException due to missing otherCopyArgs\n\n## What changes were proposed in this pull request?\n\n`TreeNode.toJSON` requires a subclass to explicitly override otherCopyArgs to include currying construction arguments, otherwise it reports AssertException telling that the construction argument values' count doesn't match the construction argument names' count.\n\nFor class `MetastoreRelation`, it has a currying construction parameter `client: HiveClient`, but Spark forgets to add it to the list of otherCopyArgs.\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #14928 from clockfly/metastore_relation_toJSON.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/afb3d5d301d004fd748ad305b3d72066af4ebb6c"},{"sha":"64e826f91eabb1a22d3d163d71fbb7b6d2185f25","author":{"name":"Yadong Qi","email":"a4dd261dfdc77f6e3f21308e2b46fcb2783647d6@gmail.com"},"message":"[SPARK-17358][SQL] Cached table(parquet/orc) should be shard between beelines\n\n## What changes were proposed in this pull request?\nCached table(parquet/orc) couldn't be shard between beelines, because the `sameResult` method used by `CacheManager` always return false(`sparkSession` are different) when compare two `HadoopFsRelation` in different beelines. So we make `sparkSession` a curry parameter.\n\n## How was this patch tested?\nBeeline1\n```\n1: jdbc:hive2://localhost:10000> CACHE TABLE src_pqt;\n+---------+--+\n| Result  |\n+---------+--+\n+---------+--+\nNo rows selected (5.143 seconds)\n1: jdbc:hive2://localhost:10000> EXPLAIN SELECT * FROM src_pqt;\n+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+\n|                                                                                                                                                                                                            plan                                                                                                                                                                                                            |\n+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+\n| == Physical Plan ==\nInMemoryTableScan [key#49, value#50]\n   +- InMemoryRelation [key#49, value#50], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas), `src_pqt`\n         +- *FileScan parquet default.src_pqt[key#0,value#1] Batched: true, Format: ParquetFormat, InputPaths: hdfs://199.0.0.1:9000/qiyadong/src_pqt, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<key:int,value:string>  |\n+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+\n```\n\nBeeline2\n```\n0: jdbc:hive2://localhost:10000> EXPLAIN SELECT * FROM src_pqt;\n+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+\n|                                                                                                                                                                                                            plan                                                                                                                                                                                                            |\n+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+\n| == Physical Plan ==\nInMemoryTableScan [key#68, value#69]\n   +- InMemoryRelation [key#68, value#69], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas), `src_pqt`\n         +- *FileScan parquet default.src_pqt[key#0,value#1] Batched: true, Format: ParquetFormat, InputPaths: hdfs://199.0.0.1:9000/qiyadong/src_pqt, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<key:int,value:string>  |\n+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+\n```\n\nAuthor: Yadong Qi <qiyadong2010@gmail.com>\n\nCloses #14913 from watermen/SPARK-17358.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/64e826f91eabb1a22d3d163d71fbb7b6d2185f25"},{"sha":"c0ae6bc6ea38909730fad36e653d3c7ab0a84b44","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17361][SQL] file-based external table without path should not be created\n\n## What changes were proposed in this pull request?\n\nUsing the public `Catalog` API, users can create a file-based data source table, without giving the path options. For this case, currently we can create the table successfully, but fail when we read it. Ideally we should fail during creation.\n\nThis is because when we create data source table, we resolve the data source relation without validating path: `resolveRelation(checkPathExist = false)`.\n\nLooking back to why we add this trick(`checkPathExist`), it's because when we call `resolveRelation` for managed table, we add the path to data source options but the path is not created yet. So why we add this not-yet-created path to data source options? This PR fix the problem by adding path to options after we call `resolveRelation`. Then we can remove the `checkPathExist` parameter in `DataSource.resolveRelation` and do some related cleanups.\n\n## How was this patch tested?\n\nexisting tests and new test in `CatalogSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14921 from cloud-fan/check-path.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/c0ae6bc6ea38909730fad36e653d3c7ab0a84b44"},{"sha":"6f13aa7dfee12b1b301bd10a1050549008ecc67e","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-17356][SQL] Fix out of memory issue when generating JSON for TreeNode\n\n## What changes were proposed in this pull request?\n\nclass `org.apache.spark.sql.types.Metadata` is widely used in mllib to store some ml attributes. `Metadata` is commonly stored in `Alias` expression.\n\n```\ncase class Alias(child: Expression, name: String)(\n    val exprId: ExprId = NamedExpression.newExprId,\n    val qualifier: Option[String] = None,\n    val explicitMetadata: Option[Metadata] = None,\n    override val isGenerated: java.lang.Boolean = false)\n```\n\nThe `Metadata` can take a big memory footprint since the number of attributes is big ( in scale of million). When `toJSON` is called on `Alias` expression, the `Metadata` will also be converted to a big JSON string.\nIf a plan contains many such kind of `Alias` expressions, it may trigger out of memory error when `toJSON` is called, since converting all `Metadata` references to JSON will take huge memory.\n\nWith this PR, we will skip scanning Metadata when doing JSON conversion. For a reproducer of the OOM, and analysis, please look at jira https://issues.apache.org/jira/browse/SPARK-17356.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #14915 from clockfly/json_oom.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/6f13aa7dfee12b1b301bd10a1050549008ecc67e"},{"sha":"39d538dddf7d44bf4603c966d0f7b2c92f1e951a","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[MINOR][ML] Correct weights doc of MultilayerPerceptronClassificationModel.\n\n## What changes were proposed in this pull request?\n```weights``` of ```MultilayerPerceptronClassificationModel``` should be the output weights of layers rather than initial weights, this PR correct it.\n\n## How was this patch tested?\nDoc change.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14967 from yanboliang/mlp-weights.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/39d538dddf7d44bf4603c966d0f7b2c92f1e951a"},{"sha":"bc2767df2666ff615e7f44e980555afab06dd8a3","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-17374][SQL] Better error messages when parsing JSON using DataFrameReader\n\n## What changes were proposed in this pull request?\n\nThis PR adds better error messages for malformed record when reading a JSON file using DataFrameReader.\n\nFor example, for query:\n```\nimport org.apache.spark.sql.types._\nval corruptRecords = spark.sparkContext.parallelize(\"\"\"{\"a\":{, b:3}\"\"\" :: Nil)\nval schema = StructType(StructField(\"a\", StringType, true) :: Nil)\nval jsonDF = spark.read.schema(schema).json(corruptRecords)\n```\n\n**Before change:**\nWe silently replace corrupted line with null\n```\nscala> jsonDF.show\n+----+\n|   a|\n+----+\n|null|\n+----+\n```\n\n**After change:**\nAdd an explicit warning message:\n```\nscala> jsonDF.show\n16/09/02 14:43:16 WARN JacksonParser: Found at least one malformed records (sample: {\"a\":{, b:3}). The JSON reader will replace\nall malformed records with placeholder null in current PERMISSIVE parser mode.\nTo find out which corrupted records have been replaced with null, please use the\ndefault inferred schema instead of providing a custom schema.\n\nCode example to print all malformed records (scala):\n===================================================\n// The corrupted record exists in column _corrupt_record.\nval parsedJson = spark.read.json(\"/path/to/json/file/test.json\")\n\n+----+\n|   a|\n+----+\n|null|\n+----+\n```\n\n###\n\n## How was this patch tested?\n\nUnit test.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #14929 from clockfly/logwarning_if_schema_not_contain_corrupted_record.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/bc2767df2666ff615e7f44e980555afab06dd8a3"},{"sha":"f7e26d788757f917b32749856bb29feb7b4c2987","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16922] [SPARK-17211] [SQL] make the address of values portable in LongToUnsafeRowMap\n\n## What changes were proposed in this pull request?\n\nIn LongToUnsafeRowMap, we use offset of a value as pointer, stored in a array also in the page for chained values. The offset is not portable, because Platform.LONG_ARRAY_OFFSET will be different with different JVM Heap size, then the deserialized LongToUnsafeRowMap will be corrupt.\n\nThis PR will change to use portable address (without Platform.LONG_ARRAY_OFFSET).\n\n## How was this patch tested?\n\nAdded a test case with random generated keys, to improve the coverage. But this test is not a regression test, that could require a Spark cluster that have at least 32G heap in driver or executor.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #14927 from davies/longmap.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/f7e26d788757f917b32749856bb29feb7b4c2987"},{"sha":"6c08dbf683875ff1ba724447e0531f673bcff8ba","author":{"name":"Adam Roberts","email":"ac6c9ba1da70de03b556cc2f8b9c0fac7b1e271e@uk.ibm.com"},"message":"[SPARK-17378][BUILD] Upgrade snappy-java to 1.1.2.6\n\n## What changes were proposed in this pull request?\n\nUpgrades the Snappy version to 1.1.2.6 from 1.1.2.4, release notes: https://github.com/xerial/snappy-java/blob/master/Milestone.md mention \"Fix a bug in SnappyInputStream when reading compressed data that happened to have the same first byte with the stream magic header (#142)\"\n\n## How was this patch tested?\nExisting unit tests using the latest IBM Java 8 on Intel, Power and Z architectures (little and big-endian)\n\nAuthor: Adam Roberts <aroberts@uk.ibm.com>\n\nCloses #14958 from a-roberts/master.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/6c08dbf683875ff1ba724447e0531f673bcff8ba"},{"sha":"7775d9f224e22400c6c8c093652a383f4af66ee0","author":{"name":"Sandeep Singh","email":"69043e619d79bcf4ae068e5b131d66c33e73727d@techaddict.me"},"message":"[SPARK-17299] TRIM/LTRIM/RTRIM should not strips characters other than spaces\n\n## What changes were proposed in this pull request?\nTRIM/LTRIM/RTRIM should not strips characters other than spaces, we were trimming all chars small than ASCII 0x20(space)\n\n## How was this patch tested?\nfixed existing tests.\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #14924 from techaddict/SPARK-17299.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/7775d9f224e22400c6c8c093652a383f4af66ee0"},{"sha":"8bbb08a3001313b53a5d854ef442d82d4c7e1d5d","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[MINOR] Remove unnecessary check in MLSerDe\n\n## What changes were proposed in this pull request?\n1, remove unnecessary `require()`, because it will make following check useless.\n2, update the error msg.\n\n## How was this patch tested?\nno test\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #14972 from zhengruifeng/del_unnecessary_check.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/8bbb08a3001313b53a5d854ef442d82d4c7e1d5d"},{"sha":"29cfab3f1524c5690be675d24dda0a9a1806d6ff","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17110] Fix StreamCorruptionException in BlockManager.getRemoteValues()\n\n## What changes were proposed in this pull request?\n\nThis patch fixes a `java.io.StreamCorruptedException` error affecting remote reads of cached values when certain data types are used. The problem stems from #11801 / SPARK-13990, a patch to have Spark automatically pick the \"best\" serializer when caching RDDs. If PySpark cached a PythonRDD, then this would be cached as an `RDD[Array[Byte]]` and the automatic serializer selection would pick KryoSerializer for replication and block transfer. However, the `getRemoteValues()` / `getRemoteBytes()` code path did not pass proper class tags in order to enable the same serializer to be used during deserialization, causing Java to be inappropriately used instead of Kryo, leading to the StreamCorruptedException.\n\nWe already fixed a similar bug in #14311, which dealt with similar issues in block replication. Prior to that patch, it seems that we had no tests to ensure that block replication actually succeeded. Similarly, prior to this bug fix patch it looks like we had no tests to perform remote reads of cached data, which is why this bug was able to remain latent for so long.\n\nThis patch addresses the bug by modifying `BlockManager`'s `get()` and  `getRemoteValues()` methods to accept ClassTags, allowing the proper class tag to be threaded in the `getOrElseUpdate` code path (which is used by `rdd.iterator`)\n\n## How was this patch tested?\n\nExtended the caching tests in `DistributedSuite` to exercise the `getRemoteValues` path, plus manual testing to verify that the PySpark bug reproduction in SPARK-17110 is fixed.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14952 from JoshRosen/SPARK-17110.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/29cfab3f1524c5690be675d24dda0a9a1806d6ff"},{"sha":"4f769b903bc9822c262f0a15f5933cc05c67923f","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-17296][SQL] Simplify parser join processing.\n\n## What changes were proposed in this pull request?\nJoin processing in the parser relies on the fact that the grammar produces a right nested trees, for instance the parse tree for `select * from a join b join c` is expected to produce a tree similar to `JOIN(a, JOIN(b, c))`. However there are cases in which this (invariant) is violated, like:\n```sql\nSELECT COUNT(1)\nFROM test T1\n     CROSS JOIN test T2\n     JOIN test T3\n      ON T3.col = T1.col\n     JOIN test T4\n      ON T4.col = T1.col\n```\nIn this case the parser returns a tree in which Joins are located on both the left and the right sides of the parent join node.\n\nThis PR introduces a different grammar rule which does not make this assumption. The new rule takes a relation and searches for zero or more joined relations. As a bonus processing is much easier.\n\n## How was this patch tested?\nExisting tests and I have added a regression test to the plan parser suite.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #14867 from hvanhovell/SPARK-17296.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/4f769b903bc9822c262f0a15f5933cc05c67923f"},{"sha":"0bd00ff2454c5046e4cb084ee64d432c4d3dcbc3","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-15891][YARN] Clean up some logging in the YARN AM.\n\nTo make the log file more readable, rework some of the logging done\nby the AM:\n\n- log executor command / env just once, since they're all almost the same;\n  the information that changes, such as executor ID, is already available\n  in other log messages.\n- avoid printing logs when nothing happens, especially when updating the\n  container requests in the allocator.\n- print fewer log messages when requesting many unlocalized executors,\n  instead of repeating the same message multiple times.\n- removed some logs that seemed unnecessary.\n\nIn the process, I slightly fixed up the wording in a few log messages, and\ndid some minor clean up of method arguments that were redundant.\n\nTested by running existing unit tests, and analyzing the logs of an\napplication that exercises dynamic allocation by forcing executors\nto be allocated and be killed in waves.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14943 from vanzin/SPARK-15891.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/0bd00ff2454c5046e4cb084ee64d432c4d3dcbc3"},{"sha":"175b4344112b376cbbbd05265125ed0e1b87d507","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17316][CORE] Fix the 'ask' type parameter in 'removeExecutor'\n\n## What changes were proposed in this pull request?\n\nFix the 'ask' type parameter in 'removeExecutor' to eliminate a lot of error logs `Cannot cast java.lang.Boolean to scala.runtime.Nothing$`\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14983 from zsxwing/SPARK-17316-3.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/175b4344112b376cbbbd05265125ed0e1b87d507"},{"sha":"c07cbb3534a57834b9b78e1572d40fb2af930f5f","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-17371] Resubmitted shuffle outputs can get deleted by zombie map tasks\n\n## What changes were proposed in this pull request?\n\nIt seems that old shuffle map tasks hanging around after a stage resubmit will delete intended shuffle output files on stop(), causing downstream stages to fail even after successful resubmit completion. This can happen easily if the prior map task is waiting for a network timeout when its stage is resubmitted.\n\nThis can cause unnecessary stage resubmits, sometimes multiple times as fetch fails cause a cascade of shuffle file invalidations, and confusing FetchFailure messages that report shuffle index files missing from the local disk.\n\nGiven that IndexShuffleBlockResolver commits data atomically, it seems unnecessary to ever delete committed task output: even in the rare case that a task is failed after it finishes committing shuffle output, it should be safe to retain that output.\n\n## How was this patch tested?\n\nPrior to the fix proposed in https://github.com/apache/spark/pull/14931, I was able to reproduce this behavior by killing slaves in the middle of a large shuffle. After this patch, stages were no longer resubmitted multiple times due to shuffle index loss.\n\ncc JoshRosen vanzin\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #14932 from ericl/dont-remove-committed-files.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/c07cbb3534a57834b9b78e1572d40fb2af930f5f"},{"sha":"a40657bfd375bd27d65204bb42ed0cbd7bd1ebf2","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-17408][TEST] Flaky test: org.apache.spark.sql.hive.StatisticsSuite\n\n### What changes were proposed in this pull request?\nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64956/testReport/junit/org.apache.spark.sql.hive/StatisticsSuite/test_statistics_of_LogicalRelation_converted_from_MetastoreRelation/\n```\norg.apache.spark.sql.hive.StatisticsSuite.test statistics of LogicalRelation converted from MetastoreRelation\n\nFailing for the past 1 build (Since Failed#64956 )\nTook 1.4 sec.\nError Message\n\norg.scalatest.exceptions.TestFailedException: 6871 did not equal 4236\nStacktrace\n\nsbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: 6871 did not equal 4236\n\tat org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\n```\n\nThis fix does not check the exact value of `sizeInBytes`. Instead, we compare whether it is larger than zero and compare the values between different values.\n\nIn addition, we also combine `checkMetastoreRelationStats` and `checkLogicalRelationStats` into the same checking function.\n\n### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14978 from gatorsmile/spark17408.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/a40657bfd375bd27d65204bb42ed0cbd7bd1ebf2"},{"sha":"d6eede9a36766e2d2294951b054d7557008a5662","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17238][SQL] simplify the logic for converting data source table into hive compatible format\n\n## What changes were proposed in this pull request?\n\nPreviously we have 2 conditions to decide whether a data source table is hive-compatible:\n\n1. the data source is file-based and has a corresponding Hive serde\n2. have a `path` entry in data source options/storage properties\n\nHowever, if condition 1 is true, condition 2 must be true too, as we will put the default table path into data source options/storage properties for managed data source tables.\n\nThere is also a potential issue: we will set the `locationUri` even for managed table.\n\nThis PR removes the condition 2 and only set the `locationUri` for external data source tables.\n\nNote: this is also a first step to unify the `path` of data source tables and `locationUri` of hive serde tables. For hive serde tables, `locationUri` is only set for external table. For data source tables, `path` is always set. We can make them consistent after this PR.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14809 from cloud-fan/minor2.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/d6eede9a36766e2d2294951b054d7557008a5662"},{"sha":"eb1ab88a86ce35f3d6ba03b3a798099fbcf6b3fc","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-17372][SQL][STREAMING] Avoid serialization issues by using Arrays to save file names in FileStreamSource\n\n## What changes were proposed in this pull request?\n\nWhen we create a filestream on a directory that has partitioned subdirs (i.e. dir/x=y/), then ListingFileCatalog.allFiles returns the files in the dir as Seq[String] which internally is a Stream[String]. This is because of this [line](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileCatalog.scala#L93), where a LinkedHashSet.values.toSeq returns Stream. Then when the [FileStreamSource](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala#L79) filters this Stream[String] to remove the seen files, it creates a new Stream[String], which has a filter function that has a $outer reference to the FileStreamSource (in Scala 2.10). Trying to serialize this Stream[String] causes NotSerializableException. This will happened even if there is just one file in the dir.\n\nIts important to note that this behavior is different in Scala 2.11. There is no $outer reference to FileStreamSource, so it does not throw NotSerializableException. However, with a large sequence of files (tested with 10000 files), it throws StackOverflowError. This is because how Stream class is implemented. Its basically like a linked list, and attempting to serialize a long Stream requires *recursively* going through linked list, thus resulting in StackOverflowError.\n\nIn short, across both Scala 2.10 and 2.11, serialization fails when both the following conditions are true.\n- file stream defined on a partitioned directory\n- directory has 10k+ files\n\nThe right solution is to convert the seq to an array before writing to the log. This PR implements this fix in two ways.\n- Changing all uses for HDFSMetadataLog to ensure Array is used instead of Seq\n- Added a `require` in HDFSMetadataLog such that it is never used with type Seq\n\n## How was this patch tested?\n\nAdded unit test that test that ensures the file stream source can handle with 10000 files. This tests fails in both Scala 2.10 and 2.11 with different failures as indicated above.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14987 from tdas/SPARK-17372.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/eb1ab88a86ce35f3d6ba03b3a798099fbcf6b3fc"},{"sha":"9fccde4ff80fb0fd65a9e90eb3337965e4349de4","author":{"name":"Clark Fitzgerald","email":"481934d8b8c575ba8a8c1b8379a47effd1a070ae@gmail.com"},"message":"[SPARK-16785] R dapply doesn't return array or raw columns\n\n## What changes were proposed in this pull request?\n\nFixed bug in `dapplyCollect` by changing the `compute` function of `worker.R` to explicitly handle raw (binary) vectors.\n\ncc shivaram\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: Clark Fitzgerald <clarkfitzg@gmail.com>\n\nCloses #14783 from clarkfitzg/SPARK-16785.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/9fccde4ff80fb0fd65a9e90eb3337965e4349de4"}]}
{"actor":{"display_login":"gatorsmile"}}