{"push_id":1188363498,"size":4,"distinct_size":4,"ref":"refs/heads/master","head":"8f6cf00c697689174bcf522091e396b4fa5ef66d","before":"26283339786f38c50722a7488d0bca8573b9c352","commits":[{"sha":"a539b724c1d407083cb87abfa06d8bf213501057","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-16260][ML][EXAMPLE] PySpark ML Example Improvements and Cleanup\n\n## What changes were proposed in this pull request?\n1). Remove unused import in Scala example;\n\n2). Move spark session import outside example off;\n\n3). Change parameter setting the same as Scala;\n\n4). Change comment to be consistent;\n\n5). Make sure that Scala and python using the same data set;\n\nI did one pass and fixed the above issues. There are missing examples in python, which might be added later.\n\nTODO: For some examples, there are comments on how to run examples; But there are many missing. We can add them later.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\nManually test them\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #14021 from wangmiao1981/ann.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/a539b724c1d407083cb87abfa06d8bf213501057"},{"sha":"18fb57f58a04685823408f3a174a8722f155fd4d","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure\n\n## What changes were proposed in this pull request?\n\nCoincidentally, I discovered that a couple images were unused in `docs/`, and then searched and found more, and then realized some PNGs were pretty big and could be crushed, and before I knew it, had done the same for the ASF site (not committed yet).\n\nNo functional change at all, just less superfluous image data.\n\n## How was this patch tested?\n\n`jekyll serve`\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14029 from srowen/RemoveCompressImages.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/18fb57f58a04685823408f3a174a8722f155fd4d"},{"sha":"7dbffcdd6dc76b8e8d6a9cd6eeb24323a6b740c3","author":{"name":"Michael Allman","email":"844aface8de49b32585c3c744b570659f1739c56@videoamp.com"},"message":"[SPARK-16353][BUILD][DOC] Missing javadoc options for java unidoc\n\nLink to Jira issue: https://issues.apache.org/jira/browse/SPARK-16353\n\n## What changes were proposed in this pull request?\n\nThe javadoc options for the java unidoc generation are ignored when generating the java unidoc. For example, the generated `index.html` has the wrong HTML page title. This can be seen at http://spark.apache.org/docs/latest/api/java/index.html.\n\nI changed the relevant setting scope from `doc` to `(JavaUnidoc, unidoc)`.\n\n## How was this patch tested?\n\nI ran `docs/jekyll build` and verified that the java unidoc `index.html` has the correct HTML page title.\n\nAuthor: Michael Allman <michael@videoamp.com>\n\nCloses #14031 from mallman/spark-16353.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/7dbffcdd6dc76b8e8d6a9cd6eeb24323a6b740c3"},{"sha":"8f6cf00c697689174bcf522091e396b4fa5ef66d","author":{"name":"Michael Allman","email":"844aface8de49b32585c3c744b570659f1739c56@videoamp.com"},"message":"[SPARK-15968][SQL] Nonempty partitioned metastore tables are not cached\n\n(Please note this is a revision of PR #13686, which has been closed in favor of this PR.)\n\nThis PR addresses [SPARK-15968](https://issues.apache.org/jira/browse/SPARK-15968).\n\n## What changes were proposed in this pull request?\n\nThe `getCached` method of [HiveMetastoreCatalog](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala) computes `pathsInMetastore` from the metastore relation's catalog table. This only returns the table base path, which is incomplete/inaccurate for a nonempty partitioned table. As a result, cached lookups on nonempty partitioned tables always miss.\n\nRather than get `pathsInMetastore` from\n\n    metastoreRelation.catalogTable.storage.locationUri.toSeq\n\nI modified the `getCached` method to take a `pathsInMetastore` argument. Calls to this method pass in the paths computed from calls to the Hive metastore. This is how `getCached` was implemented in Spark 1.5:\n\nhttps://github.com/apache/spark/blob/e0c3212a9b42e3e704b070da4ac25b68c584427f/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala#L444.\n\nI also added a call in `InsertIntoHiveTable.scala` to invalidate the table from the SQL session catalog.\n\n## How was this patch tested?\n\nI've added a new unit test to `parquetSuites.scala`:\n\n    SPARK-15968: nonempty partitioned metastore Parquet table lookup should use cached relation\n\nNote that the only difference between this new test and the one above it in the file is that the new test populates its partitioned table with a single value, while the existing test leaves the table empty. This reveals a subtle, unexpected hole in test coverage present before this patch.\n\nNote I also modified a different but related unit test in `parquetSuites.scala`:\n\n    SPARK-15248: explicitly added partitions should be readable\n\nThis unit test asserts that Spark SQL should return data from a table partition which has been placed there outside a metastore query immediately after it is added. I changed the test so that, instead of adding the data as a parquet file saved in the partition's location, the data is added through a SQL `INSERT` query. I made this change because I could find no way to efficiently support partitioned table caching without failing that test.\n\nIn addition to my primary motivation, I can offer a few reasons I believe this is an acceptable weakening of that test. First, it still validates a fix for [SPARK-15248](https://issues.apache.org/jira/browse/SPARK-15248), the issue for which it was written. Second, the assertion made is stronger than that required for non-partitioned tables. If you write data to the storage location of a non-partitioned metastore table without using a proper SQL DML query, a subsequent call to show that data will not return it. I believe this is an intentional limitation put in place to make table caching feasible, but I'm only speculating.\n\nBuilding a large `HadoopFsRelation` requires `stat`-ing all of its data files. In our environment, where we have tables with 10's of thousands of partitions, the difference between using a cached relation versus a new one is a matter of seconds versus minutes. Caching partitioned table metadata vastly improves the usability of Spark SQL for these cases.\n\nThanks.\n\nAuthor: Michael Allman <michael@videoamp.com>\n\nCloses #13818 from mallman/spark-15968.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/8f6cf00c697689174bcf522091e396b4fa5ef66d"}]}
{"actor":{"display_login":"zzcclp"}}