{"push_id":1396905750,"size":1000,"distinct_size":3440,"ref":"refs/heads/SPARK-13027","head":"fe68b6c03300a37799ccaad2ee554bde005c8f6f","before":"17cdabdf75a759675bf7defb635017a384d1c880","commits":[{"sha":"274f3b9ec86e4109c7678eef60f990d41dc3899f","author":{"name":"Nicholas Chammas","email":"3a05fc0c001acadf8f7c4ad23a983345f7ce437d@gmail.com"},"message":"[SPARK-16772] Correct API doc references to PySpark classes + formatting fixes\n\n## What's Been Changed\n\nThe PR corrects several broken or missing class references in the Python API docs. It also correct formatting problems.\n\nFor example, you can see [here](http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#pyspark.sql.SQLContext.registerFunction) how Sphinx is not picking up the reference to `DataType`. That's because the reference is relative to the current module, whereas `DataType` is in a different module.\n\nYou can also see [here](http://spark.apache.org/docs/2.0.0/api/python/pyspark.sql.html#pyspark.sql.SQLContext.createDataFrame) how the formatting for byte, tinyint, and so on is italic instead of monospace. That's because in ReST single backticks just make things italic, unlike in Markdown.\n\n## Testing\n\nI tested this PR by [building the Python docs](https://github.com/apache/spark/tree/master/docs#generating-the-documentation-html) and reviewing the results locally in my browser. I confirmed that the broken or missing class references were resolved, and that the formatting was corrected.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #14393 from nchammas/python-docstring-fixes.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/274f3b9ec86e4109c7678eef60f990d41dc3899f"},{"sha":"d1d5069aa3744d46abd3889abab5f15e9067382a","author":{"name":"Wesley Tang","email":"c52c3b098164201233f329aa2b18b552b34b71d6@mininglamp.com"},"message":"[SPARK-16664][SQL] Fix persist call on Data frames with more than 200…\n\n## What changes were proposed in this pull request?\n\nf12f11e578169b47e3f8b18b299948c0670ba585 introduced this bug, missed foreach as map\n\n## How was this patch tested?\n\nTest added\n\nAuthor: Wesley Tang <tangmingjun@mininglamp.com>\n\nCloses #14324 from breakdawn/master.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/d1d5069aa3744d46abd3889abab5f15e9067382a"},{"sha":"0557a45452f6e73877e5ec972110825ce8f3fbc5","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16750][ML] Fix GaussianMixture training failed due to feature column type mistake\n\n## What changes were proposed in this pull request?\nML ```GaussianMixture``` training failed due to feature column type mistake. The feature column type should be ```ml.linalg.VectorUDT``` but got ```mllib.linalg.VectorUDT``` by mistake.\nSee [SPARK-16750](https://issues.apache.org/jira/browse/SPARK-16750) for how to reproduce this bug.\nWhy the unit tests did not complain this errors? Because some estimators/transformers missed calling ```transformSchema(dataset.schema)``` firstly during ```fit``` or ```transform```. I will also add this function to all estimators/transformers who missed in this PR.\n\n## How was this patch tested?\nNo new tests, should pass existing ones.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14378 from yanboliang/spark-16750.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/0557a45452f6e73877e5ec972110825ce8f3fbc5"},{"sha":"04a2c072d94874f3f7ae9dd94c026e8826a75ccd","author":{"name":"Adam Roberts","email":"ac6c9ba1da70de03b556cc2f8b9c0fac7b1e271e@uk.ibm.com"},"message":"[SPARK-16751] Upgrade derby to 10.12.1.1\n\n## What changes were proposed in this pull request?\n\nVersion of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1\n\nThe CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs\n\n## How was this patch tested?\nExisting tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder.\n\nI then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present.\n\nI don't know if this would also remove it from the assembly jar in our 1.x branches.\n\nAuthor: Adam Roberts <aroberts@uk.ibm.com>\n\nCloses #14379 from a-roberts/patch-4.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/04a2c072d94874f3f7ae9dd94c026e8826a75ccd"},{"sha":"266b92faffb66af24d8ed2725beb80770a2d91f8","author":{"name":"Michael Gummelt","email":"997b1eeb9667009a1fe95939cfe9ef99c4f83395@mesosphere.io"},"message":"[SPARK-16637] Unified containerizer\n\n## What changes were proposed in this pull request?\n\nNew config var: spark.mesos.docker.containerizer={\"mesos\",\"docker\" (default)}\n\nThis adds support for running docker containers via the Mesos unified containerizer: http://mesos.apache.org/documentation/latest/container-image/\n\nThe benefit is losing the dependency on `dockerd`, and all the costs which it incurs.\n\nI've also updated the supported Mesos version to 0.28.2 for support of the required protobufs.\n\nThis is blocked on: https://github.com/apache/spark/pull/14167\n\n## How was this patch tested?\n\n- manually testing jobs submitted with both \"mesos\" and \"docker\" settings for the new config var.\n- spark/mesos integration test suite\n\nAuthor: Michael Gummelt <mgummelt@mesosphere.io>\n\nCloses #14275 from mgummelt/unified-containerizer.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/266b92faffb66af24d8ed2725beb80770a2d91f8"},{"sha":"2c15323ad026da64caa68787c5d103a8595f63a0","author":{"name":"Sun Dapeng","email":"475e951cfcbbfda22e3cd6b3aae5dd506f1c7328@apache.org"},"message":"[SPARK-16761][DOC][ML] Fix doc link in docs/ml-guide.md\n\n## What changes were proposed in this pull request?\n\nFix the link at http://spark.apache.org/docs/latest/ml-guide.html.\n\n## How was this patch tested?\n\nNone\n\nAuthor: Sun Dapeng <sdp@apache.org>\n\nCloses #14386 from sundapeng/doclink.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/2c15323ad026da64caa68787c5d103a8595f63a0"},{"sha":"2182e4322da6ba732f99ae75dce00f76f1cdc4d9","author":{"name":"Nicholas Chammas","email":"3a05fc0c001acadf8f7c4ad23a983345f7ce437d@gmail.com"},"message":"[SPARK-16772][PYTHON][DOCS] Restore \"datatype string\" to Python API docstrings\n\n## What changes were proposed in this pull request?\n\nThis PR corrects [an error made in an earlier PR](https://github.com/apache/spark/pull/14393/files#r72843069).\n\n## How was this patch tested?\n\n```sh\n$ ./dev/lint-python\nPEP8 checks passed.\nrm -rf _build/*\npydoc checks passed.\n```\n\nI also built the docs and confirmed that they looked good in my browser.\n\nAuthor: Nicholas Chammas <nicholas.chammas@gmail.com>\n\nCloses #14408 from nchammas/SPARK-16772.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/2182e4322da6ba732f99ae75dce00f76f1cdc4d9"},{"sha":"bbc247548ac6faeca15afc05c266cee37ef13416","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-16748][SQL] SparkExceptions during planning should not wrapped in TreeNodeException\n\n## What changes were proposed in this pull request?\nWe do not want SparkExceptions from job failures in the planning phase to create TreeNodeException. Hence do not wrap SparkException in TreeNodeException.\n\n## How was this patch tested?\nNew unit test\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14395 from tdas/SPARK-16748.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/bbc247548ac6faeca15afc05c266cee37ef13416"},{"sha":"0dc4310b470c7e4355c0da67ca3373c3013cc9dd","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16694][CORE] Use for/foreach rather than map for Unit expressions whose side effects are required\n\n## What changes were proposed in this pull request?\n\nUse foreach/for instead of map where operation requires execution of body, not actually defining a transformation\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14332 from srowen/SPARK-16694.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/0dc4310b470c7e4355c0da67ca3373c3013cc9dd"},{"sha":"bce354c1d4e2b97b1159913085e9883a26bc605a","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16696][ML][MLLIB] destroy KMeans bcNewCenters when loop finished and update code where should release unused broadcast/RDD in proper time\n\n## What changes were proposed in this pull request?\n\nupdate unused broadcast in KMeans/Word2Vec,\nuse destroy(false) to release memory in time.\n\nand several place destroy() update to destroy(false) so that it will be async-called,\nit will better than blocking called.\n\nand update bcNewCenters in KMeans to make it destroy in correct time.\nI use a list to store all historical `bcNewCenters` generated in each loop iteration and delay them to release at the end of loop.\n\nfix TODO in `BisectingKMeans.run` \"unpersist old indices\",\nImplements the pattern \"persist current step RDD, and unpersist previous one\" in the loop iteration.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14333 from WeichenXu123/broadvar_unpersist_to_destroy.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/bce354c1d4e2b97b1159913085e9883a26bc605a"},{"sha":"a6290e51e402e8434d6207d553db1f551e714fde","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-16800][EXAMPLES][ML] Fix Java examples that fail to run due to exception\n\n## What changes were proposed in this pull request?\nSome Java examples are using mllib.linalg.Vectors instead of ml.linalg.Vectors and causes an exception when run.  Also there are some Java examples that incorrectly specify data types in the schema, also causing an exception.\n\n## How was this patch tested?\nRan corrected examples locally\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #14405 from BryanCutler/java-examples-ml.Vectors-fix-SPARK-16800.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/a6290e51e402e8434d6207d553db1f551e714fde"},{"sha":"957a8ab3743521850fb1c0106c37c5d3997b9e56","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-16818] Exchange reuse incorrectly reuses scans over different sets of partitions\n\n## What changes were proposed in this pull request?\n\nThis fixes a bug wherethe file scan operator does not take into account partition pruning in its implementation of `sameResult()`. As a result, executions may be incorrect on self-joins over the same base file relation.\n\nThe patch here is minimal, but we should reconsider relying on `metadata` for implementing sameResult() in the future, as string representations may not be uniquely identifying.\n\ncc rxin\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #14425 from ericl/spark-16818.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/957a8ab3743521850fb1c0106c37c5d3997b9e56"},{"sha":"7c27d075c39ebaf3e762284e2536fe7be0e3da87","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16812] Open up SparkILoop.getAddedJars\n\n## What changes were proposed in this pull request?\nThis patch makes SparkILoop.getAddedJars a public developer API. It is a useful function to get the list of jars added.\n\n## How was this patch tested?\nN/A - this is a simple visibility change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14417 from rxin/SPARK-16812.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/7c27d075c39ebaf3e762284e2536fe7be0e3da87"},{"sha":"064d91ff7342002414d3274694a8e2e37f154986","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16813][SQL] Remove private[sql] and private[spark] from catalyst package\n\n## What changes were proposed in this pull request?\nThe catalyst package is meant to be internal, and as a result it does not make sense to mark things as private[sql] or private[spark]. It simply makes debugging harder when Spark developers need to inspect the plans at runtime.\n\nThis patch removes all private[sql] and private[spark] visibility modifiers in org.apache.spark.sql.catalyst.\n\n## How was this patch tested?\nN/A - just visibility changes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14418 from rxin/SPARK-16813.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/064d91ff7342002414d3274694a8e2e37f154986"},{"sha":"301fb0d7236eb55d53c9cd60804a2d755b4ad3b2","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16731][SQL] use StructType in CatalogTable and remove CatalogColumn\n\n## What changes were proposed in this pull request?\n\n`StructField` has very similar semantic with `CatalogColumn`, except that `CatalogColumn` use string to express data type. I think it's reasonable to use `StructType` as the `CatalogTable.schema` and remove `CatalogColumn`.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14363 from cloud-fan/column.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/301fb0d7236eb55d53c9cd60804a2d755b4ad3b2"},{"sha":"579fbcf3bd9717003025caecc0c0b85bcff7ac7f","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16805][SQL] Log timezone when query result does not match\n\n## What changes were proposed in this pull request?\nIt is useful to log the timezone when query result does not match, especially on build machines that have different timezone from AMPLab Jenkins.\n\n## How was this patch tested?\nThis is a test-only change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14413 from rxin/SPARK-16805.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/579fbcf3bd9717003025caecc0c0b85bcff7ac7f"},{"sha":"64d8f37c717cbc9c1c3649cae4c7cc4e628cd72d","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16726][SQL] Improve `Union/Intersect/Except` error messages on incompatible types\n\n## What changes were proposed in this pull request?\n\nCurrently, `UNION` queries on incompatible types show misleading error messages, i.e., `unresolved operator Union`. We had better show a more correct message. This will help users in the situation of [SPARK-16704](https://issues.apache.org/jira/browse/SPARK-16704).\n\n**Before**\n```scala\nscala> sql(\"select 1,2,3 union (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: unresolved operator 'Union;\nscala> sql(\"select 1,2,3 intersect (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: unresolved operator 'Intersect;\nscala> sql(\"select 1,2,3 except (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: unresolved operator 'Except;\n```\n\n**After**\n```scala\nscala> sql(\"select 1,2,3 union (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. ArrayType(IntegerType,false) <> IntegerType at the second column of the second table;\nscala> sql(\"select 1,2,3 intersect (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: Intersect can only be performed on tables with the compatible column types. ArrayType(IntegerType,false) <> IntegerType at the second column of the second table;\nscala> sql(\"select 1,2,3 except (select array(1),array(2),3)\")\norg.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. ArrayType(IntegerType,false) <> IntegerType at the first column of the second table;\n```\n\n## How was this patch tested?\n\nPass the Jenkins test with a new test case.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14355 from dongjoon-hyun/SPARK-16726.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/64d8f37c717cbc9c1c3649cae4c7cc4e628cd72d"},{"sha":"2a0de7dc995844984d7dbb4238418967c6bbac70","author":{"name":"Shuai Lin","email":"7a96e9fba54b1a616c0c679715dfd11c0089fe85@gmail.com"},"message":"[SPARK-16485][DOC][ML] Remove useless latex in a log messge.\n\n## What changes were proposed in this pull request?\n\nRemoved useless latex in a log messge.\n\n## How was this patch tested?\n\nCheck generated scaladoc.\n\nAuthor: Shuai Lin <linshuai2012@gmail.com>\n\nCloses #14380 from lins05/fix-docs-formatting.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/2a0de7dc995844984d7dbb4238418967c6bbac70"},{"sha":"1e9b59b73bdb8aacf5a85e0eed29efc6485a3bc3","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with SQLContext\n\n## What changes were proposed in this pull request?\n\nChange to non-deprecated constructor for SQLContext.\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #14406 from holdenk/SPARK-16778-fix-use-of-deprecated-SQLContext-constructor.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/1e9b59b73bdb8aacf5a85e0eed29efc6485a3bc3"},{"sha":"f93ad4fe7c9728c8dd67a8095de3d39fad21d03f","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16776][STREAMING] Replace deprecated API in KafkaTestUtils for 0.10.0.\n\n## What changes were proposed in this pull request?\n\nThis PR replaces the old Kafka API to 0.10.0 ones in `KafkaTestUtils`.\n\nThe change include:\n\n - `Producer` to `KafkaProducer`\n - Change configurations to equalvant ones. (I referred [here](http://kafka.apache.org/documentation.html#producerconfigs) for 0.10.0 and [here](http://kafka.apache.org/082/documentation.html#producerconfigs\n) for old, 0.8.2).\n\nThis PR will remove the build warning as below:\n\n```scala\n[WARNING] .../spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala:71: class Producer in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.KafkaProducer instead.\n[WARNING]   private var producer: Producer[String, String] = _\n[WARNING]                         ^\n[WARNING] .../spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala:181: class Producer in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.KafkaProducer instead.\n[WARNING]     producer = new Producer[String, String](new ProducerConfig(producerConfiguration))\n[WARNING]                    ^\n[WARNING] .../spark/streaming/kafka010/KafkaTestUtils.scala:181: class ProducerConfig in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.ProducerConfig instead.\n[WARNING]     producer = new Producer[String, String](new ProducerConfig(producerConfiguration))\n[WARNING]                                                 ^\n[WARNING] .../spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala:182: class KeyedMessage in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.ProducerRecord instead.\n[WARNING]     producer.send(messages.map { new KeyedMessage[String, String](topic, _ ) }: _*)\n[WARNING]                                      ^\n[WARNING] four warnings found\n[WARNING] warning: [options] bootstrap class path not set in conjunction with -source 1.7\n[WARNING] 1 warning\n```\n\n## How was this patch tested?\n\nExisting tests that use `KafkaTestUtils` should cover this.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14416 from HyukjinKwon/SPARK-16776.","distinct":true,"url":"https://api.github.com/repos/aramesh117/spark/commits/f93ad4fe7c9728c8dd67a8095de3d39fad21d03f"}]}
{"actor":{"display_login":"aramesh117"}}