{"push_id":1227311880,"size":16,"distinct_size":16,"ref":"refs/heads/master","head":"0dc4310b470c7e4355c0da67ca3373c3013cc9dd","before":"bc4851adeb386edc5bef47027a12ca44eda82b09","commits":[{"sha":"b14d7b5cf4f173a1e45a4b1ae2a5e4e7ac5e9bb1","author":{"name":"KevinGrealish","email":"1488b75c50c0f029b05cfb525b168d0202b3f733@microsoft.com"},"message":"[SPARK-16110][YARN][PYSPARK] Fix allowing python version to be specified per submit for cluster mode.\n\n## What changes were proposed in this pull request?\n\nThis fix allows submit of pyspark jobs to specify python 2 or 3.\n\nChange ordering in setup for application master environment so env vars PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON can be overridden by spark.yarn.appMasterEnv.* conf settings. This applies to YARN in cluster mode. This allows them to be set per submission without needing the unset the env vars (which is not always possible - e.g. batch submit with LIVY only exposes the arguments to spark-submit)\n\n## How was this patch tested?\nManual and existing unit tests.\n\nAuthor: KevinGrealish <KevinGre@microsoft.com>\n\nCloses #13824 from KevinGrealish/SPARK-16110.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/b14d7b5cf4f173a1e45a4b1ae2a5e4e7ac5e9bb1"},{"sha":"11d427c924d303e20af90c0179a105f6ff4d89e2","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16730][SQL] Implement function aliases for type casts\n\n## What changes were proposed in this pull request?\nSpark 1.x supports using the Hive type name as function names for doing casts, e.g.\n```sql\nSELECT int(1.0);\nSELECT string(2.0);\n```\n\nThe above query would work in Spark 1.x because Spark 1.x fail back to Hive for unimplemented functions, and break in Spark 2.0 because the fall back was removed.\n\nThis patch implements function aliases using an analyzer rule for the following cast functions:\n- boolean\n- tinyint\n- smallint\n- int\n- bigint\n- float\n- double\n- decimal\n- date\n- timestamp\n- binary\n- string\n\n## How was this patch tested?\nAdded end-to-end tests in SQLCompatibilityFunctionSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14364 from petermaxlee/SPARK-16730-2.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/11d427c924d303e20af90c0179a105f6ff4d89e2"},{"sha":"5c2ae79bfcf448d8dc9217efafa1409997c739de","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-15232][SQL] Add subquery SQL building tests to LogicalPlanToSQLSuite\n\n## What changes were proposed in this pull request?\n\nWe currently test subquery SQL building using the `HiveCompatibilitySuite`. The is not desired since SQL building is actually a part of `sql/core` and because we are slowly reducing our dependency on Hive. This PR adds the same tests from the whitelist of `HiveCompatibilitySuite` into `LogicalPlanToSQLSuite`.\n\n## How was this patch tested?\n\nThis adds more testcases. Pass the Jenkins tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14383 from dongjoon-hyun/SPARK-15232.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/5c2ae79bfcf448d8dc9217efafa1409997c739de"},{"sha":"762366fd8722f2b3fa98b8da9338b757a1821708","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16552][SQL] Store the Inferred Schemas into External Catalog Tables when Creating Tables\n\n#### What changes were proposed in this pull request?\n\nCurrently, in Spark SQL, the initial creation of schema can be classified into two groups. It is applicable to both Hive tables and Data Source tables:\n\n**Group A. Users specify the schema.**\n\n_Case 1 CREATE TABLE AS SELECT_: the schema is determined by the result schema of the SELECT clause. For example,\n```SQL\nCREATE TABLE tab STORED AS TEXTFILE\nAS SELECT * from input\n```\n\n_Case 2 CREATE TABLE_: users explicitly specify the schema. For example,\n```SQL\nCREATE TABLE jsonTable (_1 string, _2 string)\nUSING org.apache.spark.sql.json\n```\n\n**Group B. Spark SQL infers the schema at runtime.**\n\n_Case 3 CREATE TABLE_. Users do not specify the schema but the path to the file location. For example,\n```SQL\nCREATE TABLE jsonTable\nUSING org.apache.spark.sql.json\nOPTIONS (path '${tempDir.getCanonicalPath}')\n```\n\nBefore this PR, Spark SQL does not store the inferred schema in the external catalog for the cases in Group B. When users refreshing the metadata cache, accessing the table at the first time after (re-)starting Spark, Spark SQL will infer the schema and store the info in the metadata cache for improving the performance of subsequent metadata requests. However, the runtime schema inference could cause undesirable schema changes after each reboot of Spark.\n\nThis PR is to store the inferred schema in the external catalog when creating the table. When users intend to refresh the schema after possible changes on external files (table location), they issue `REFRESH TABLE`. Spark SQL will infer the schema again based on the previously specified table location and update/refresh the schema in the external catalog and metadata cache.\n\nIn this PR, we do not use the inferred schema to replace the user specified schema for avoiding external behavior changes . Based on the design, user-specified schemas (as described in Group A) can be changed by ALTER TABLE commands, although we do not support them now.\n\n#### How was this patch tested?\nTODO: add more cases to cover the changes.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14207 from gatorsmile/userSpecifiedSchema.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/762366fd8722f2b3fa98b8da9338b757a1821708"},{"sha":"9ade77c3fa2e1bf436b79368a97d5980c12fe215","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-16639][SQL] The query with having condition that contains grouping by column should work\n\n## What changes were proposed in this pull request?\n\nThe query with having condition that contains grouping by column will be failed during analysis. E.g.,\n\n    create table tbl(a int, b string);\n    select count(b) from tbl group by a + 1 having a + 1 = 2;\n\nHaving condition should be able to use grouping by column.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #14296 from viirya/having-contains-grouping-column.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/9ade77c3fa2e1bf436b79368a97d5980c12fe215"},{"sha":"1178d61ede816bf1c8d5bb3dbb3b965c9b944407","author":{"name":"Sylvain Zimmer","email":"028d819198e8f89377b4eb9de11e35c0685e4939@sylvainzimmer.com"},"message":"[SPARK-16740][SQL] Fix Long overflow in LongToUnsafeRowMap\n\n## What changes were proposed in this pull request?\n\nAvoid overflow of Long type causing a NegativeArraySizeException a few lines later.\n\n## How was this patch tested?\n\nUnit tests for HashedRelationSuite still pass.\n\nI can confirm the python script I included in https://issues.apache.org/jira/browse/SPARK-16740 works fine with this patch. Unfortunately I don't have the knowledge/time to write a Scala test case for HashedRelationSuite right now. As the patch is pretty obvious I hope it can be included without this.\n\nThanks!\n\nAuthor: Sylvain Zimmer <sylvain@sylvainzimmer.com>\n\nCloses #14373 from sylvinus/master.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/1178d61ede816bf1c8d5bb3dbb3b965c9b944407"},{"sha":"3fd39b87bda77f3c3a4622d854f23d4234683571","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@cs.berkeley.edu"},"message":"[SPARK-16764][SQL] Recommend disabling vectorized parquet reader on OutOfMemoryError\n\n## What changes were proposed in this pull request?\n\nWe currently don't bound or manage the data array size used by column vectors in the vectorized reader (they're just bound by INT.MAX) which may lead to OOMs while reading data. As a short term fix, this patch intercepts the OutOfMemoryError exception and suggest the user to disable the vectorized parquet reader.\n\n## How was this patch tested?\n\nExisting Tests\n\nAuthor: Sameer Agarwal <sameerag@cs.berkeley.edu>\n\nCloses #14387 from sameeragarwal/oom.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/3fd39b87bda77f3c3a4622d854f23d4234683571"},{"sha":"274f3b9ec86e4109c7678eef60f990d41dc3899f","author":{"name":"Nicholas Chammas","email":"3a05fc0c001acadf8f7c4ad23a983345f7ce437d@gmail.com"},"message":"[SPARK-16772] Correct API doc references to PySpark classes + formatting fixes\n\n## What's Been Changed\n\nThe PR corrects several broken or missing class references in the Python API docs. It also correct formatting problems.\n\nFor example, you can see [here](http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#pyspark.sql.SQLContext.registerFunction) how Sphinx is not picking up the reference to `DataType`. That's because the reference is relative to the current module, whereas `DataType` is in a different module.\n\nYou can also see [here](http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#pyspark.sql.SQLContext.createDataFrame) how the formatting for byte, tinyint, and so on is italic instead of monospace. That's because in ReST single backticks just make things italic, unlike in Markdown.\n\n## Testing\n\nI tested this PR by [building the Python docs](https://github.com/apache/spark/tree/master/docs#generating-the-documentation-html) and reviewing the results locally in my browser. I confirmed that the broken or missing class references were resolved, and that the formatting was corrected.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #14393 from nchammas/python-docstring-fixes.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/274f3b9ec86e4109c7678eef60f990d41dc3899f"},{"sha":"d1d5069aa3744d46abd3889abab5f15e9067382a","author":{"name":"Wesley Tang","email":"c52c3b098164201233f329aa2b18b552b34b71d6@mininglamp.com"},"message":"[SPARK-16664][SQL] Fix persist call on Data frames with more than 200…\n\n## What changes were proposed in this pull request?\n\nf12f11e578169b47e3f8b18b299948c0670ba585 introduced this bug, missed foreach as map\n\n## How was this patch tested?\n\nTest added\n\nAuthor: Wesley Tang <tangmingjun@mininglamp.com>\n\nCloses #14324 from breakdawn/master.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/d1d5069aa3744d46abd3889abab5f15e9067382a"},{"sha":"0557a45452f6e73877e5ec972110825ce8f3fbc5","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16750][ML] Fix GaussianMixture training failed due to feature column type mistake\n\n## What changes were proposed in this pull request?\nML ```GaussianMixture``` training failed due to feature column type mistake. The feature column type should be ```ml.linalg.VectorUDT``` but got ```mllib.linalg.VectorUDT``` by mistake.\nSee [SPARK-16750](https://issues.apache.org/jira/browse/SPARK-16750) for how to reproduce this bug.\nWhy the unit tests did not complain this errors? Because some estimators/transformers missed calling ```transformSchema(dataset.schema)``` firstly during ```fit``` or ```transform```. I will also add this function to all estimators/transformers who missed in this PR.\n\n## How was this patch tested?\nNo new tests, should pass existing ones.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14378 from yanboliang/spark-16750.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/0557a45452f6e73877e5ec972110825ce8f3fbc5"},{"sha":"04a2c072d94874f3f7ae9dd94c026e8826a75ccd","author":{"name":"Adam Roberts","email":"ac6c9ba1da70de03b556cc2f8b9c0fac7b1e271e@uk.ibm.com"},"message":"[SPARK-16751] Upgrade derby to 10.12.1.1\n\n## What changes were proposed in this pull request?\n\nVersion of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1\n\nThe CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs\n\n## How was this patch tested?\nExisting tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder.\n\nI then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present.\n\nI don't know if this would also remove it from the assembly jar in our 1.x branches.\n\nAuthor: Adam Roberts <aroberts@uk.ibm.com>\n\nCloses #14379 from a-roberts/patch-4.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/04a2c072d94874f3f7ae9dd94c026e8826a75ccd"},{"sha":"266b92faffb66af24d8ed2725beb80770a2d91f8","author":{"name":"Michael Gummelt","email":"997b1eeb9667009a1fe95939cfe9ef99c4f83395@mesosphere.io"},"message":"[SPARK-16637] Unified containerizer\n\n## What changes were proposed in this pull request?\n\nNew config var: spark.mesos.docker.containerizer={\"mesos\",\"docker\" (default)}\n\nThis adds support for running docker containers via the Mesos unified containerizer: http://mesos.apache.org/documentation/latest/container-image/\n\nThe benefit is losing the dependency on `dockerd`, and all the costs which it incurs.\n\nI've also updated the supported Mesos version to 0.28.2 for support of the required protobufs.\n\nThis is blocked on: https://github.com/apache/spark/pull/14167\n\n## How was this patch tested?\n\n- manually testing jobs submitted with both \"mesos\" and \"docker\" settings for the new config var.\n- spark/mesos integration test suite\n\nAuthor: Michael Gummelt <mgummelt@mesosphere.io>\n\nCloses #14275 from mgummelt/unified-containerizer.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/266b92faffb66af24d8ed2725beb80770a2d91f8"},{"sha":"2c15323ad026da64caa68787c5d103a8595f63a0","author":{"name":"Sun Dapeng","email":"475e951cfcbbfda22e3cd6b3aae5dd506f1c7328@apache.org"},"message":"[SPARK-16761][DOC][ML] Fix doc link in docs/ml-guide.md\n\n## What changes were proposed in this pull request?\n\nFix the link at http://spark.apache.org/docs/latest/ml-guide.html.\n\n## How was this patch tested?\n\nNone\n\nAuthor: Sun Dapeng <sdp@apache.org>\n\nCloses #14386 from sundapeng/doclink.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/2c15323ad026da64caa68787c5d103a8595f63a0"},{"sha":"2182e4322da6ba732f99ae75dce00f76f1cdc4d9","author":{"name":"Nicholas Chammas","email":"3a05fc0c001acadf8f7c4ad23a983345f7ce437d@gmail.com"},"message":"[SPARK-16772][PYTHON][DOCS] Restore \"datatype string\" to Python API docstrings\n\n## What changes were proposed in this pull request?\n\nThis PR corrects [an error made in an earlier PR](https://github.com/apache/spark/pull/14393/files#r72843069).\n\n## How was this patch tested?\n\n```sh\n$ ./dev/lint-python\nPEP8 checks passed.\nrm -rf _build/*\npydoc checks passed.\n```\n\nI also built the docs and confirmed that they looked good in my browser.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #14408 from nchammas/SPARK-16772.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/2182e4322da6ba732f99ae75dce00f76f1cdc4d9"},{"sha":"bbc247548ac6faeca15afc05c266cee37ef13416","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-16748][SQL] SparkExceptions during planning should not wrapped in TreeNodeException\n\n## What changes were proposed in this pull request?\nWe do not want SparkExceptions from job failures in the planning phase to create TreeNodeException. Hence do not wrap SparkException in TreeNodeException.\n\n## How was this patch tested?\nNew unit test\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14395 from tdas/SPARK-16748.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/bbc247548ac6faeca15afc05c266cee37ef13416"},{"sha":"0dc4310b470c7e4355c0da67ca3373c3013cc9dd","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16694][CORE] Use for/foreach rather than map for Unit expressions whose side effects are required\n\n## What changes were proposed in this pull request?\n\nUse foreach/for instead of map where operation requires execution of body, not actually defining a transformation\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14332 from srowen/SPARK-16694.","distinct":true,"url":"https://api.github.com/repos/krishnakalyan3/spark/commits/0dc4310b470c7e4355c0da67ca3373c3013cc9dd"}]}
{"actor":{"display_login":"krishnakalyan3"}}