{"push_id":924892634,"size":21,"distinct_size":2,"ref":"refs/heads/spark12149","head":"5bcc2983a62c8b3c311a3df65c87c35b5ad1f150","before":"5e27bf017ac336cdaf9d27495816d97b373cdd44","commits":[{"sha":"d202ad2fc24b54de38ad7bfb646bf7703069e9f7","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12439][SQL] Fix toCatalystArray and MapObjects\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12439\n\nIn toCatalystArray, we should look at the data type returned by dataTypeFor instead of silentSchemaFor, to determine if the element is native type. An obvious problem is when the element is Option[Int] class, catalsilentSchemaFor will return Int, then we will wrongly recognize the element is native type.\n\nThere is another problem when using Option as array element. When we encode data like Seq(Some(1), Some(2), None) with encoder, we will use MapObjects to construct an array for it later. But in MapObjects, we don't check if the return value of lambdaFunction is null or not. That causes a bug that the decoded data for Seq(Some(1), Some(2), None) would be Seq(1, 2, -1), instead of Seq(1, 2, null).\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10391 from viirya/fix-catalystarray.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/d202ad2fc24b54de38ad7bfb646bf7703069e9f7"},{"sha":"047a31bb1042867b20132b347b1e08feab4562eb","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12617] [PYSPARK] Clean up the leak sockets of Py4J\n\nThis patch added Py4jCallbackConnectionCleaner to clean the leak sockets of Py4J every 30 seconds. This is a workaround before Py4J fixes the leak issue https://github.com/bartdag/py4j/issues/187\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10579 from zsxwing/SPARK-12617.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/047a31bb1042867b20132b347b1e08feab4562eb"},{"sha":"13a3b636d9425c5713cd1381203ee1b60f71b8c8","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-6724][MLLIB] Support model save/load for FPGrowthModel\n\nSupport model save/load for FPGrowthModel\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9267 from yanboliang/spark-6724.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/13a3b636d9425c5713cd1381203ee1b60f71b8c8"},{"sha":"c26d174265f6b4682210fcc406e6603b4f7dc784","author":{"email":"02057f328ddc9cc29160cbc5529070da8f16f5ee@gmail.com","name":"Nong"},"message":"[SPARK-12636] [SQL] Update UnsafeRowParquetRecordReader to support reading files directly.\n\nAs noted in the code, this change is to make this component easier to test in isolation.\n\nAuthor: Nong <nongli@gmail.com>\n\nCloses #10581 from nongli/spark-12636.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/c26d174265f6b4682210fcc406e6603b4f7dc784"},{"sha":"6cfe341ee89baa952929e91d33b9ecbca73a3ea0","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12511] [PYSPARK] [STREAMING] Make sure PythonDStream.registerSerializer is called only once\n\nThere is an issue that Py4J's PythonProxyHandler.finalize blocks forever. (https://github.com/bartdag/py4j/pull/184)\n\nPy4j will create a PythonProxyHandler in Java for \"transformer_serializer\" when calling \"registerSerializer\". If we call \"registerSerializer\" twice, the second PythonProxyHandler will override the first one, then the first one will be GCed and trigger \"PythonProxyHandler.finalize\". To avoid that, we should not call\"registerSerializer\" more than once, so that \"PythonProxyHandler\" in Java side won't be GCed.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10514 from zsxwing/SPARK-12511.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/6cfe341ee89baa952929e91d33b9ecbca73a3ea0"},{"sha":"1c6cf1a5639bf5111324e44d93a8c6462958750a","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12570][ML][DOC] DecisionTreeRegressor: provide variance of prediction: user guide update\n\nUpdate user guide doc for ```DecisionTreeRegressor``` providing variance of prediction.\n\ncc jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10594 from yanboliang/spark-12570.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/1c6cf1a5639bf5111324e44d93a8c6462958750a"},{"sha":"78015a8b7cc316343e302eeed6fe30af9f2961e8","author":{"email":"3270cab15a4ae4af5d46c40db43f5b35eac294d2@gmail.com","name":"RJ Nowling"},"message":"[SPARK-12450][MLLIB] Un-persist broadcasted variables in KMeans\n\nSPARK-12450 . Un-persist broadcasted variables in KMeans.\n\nAuthor: RJ Nowling <rnowling@gmail.com>\n\nCloses #10415 from rnowling/spark-12450.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/78015a8b7cc316343e302eeed6fe30af9f2961e8"},{"sha":"ff89975543b153d0d235c0cac615d45b34aa8fe7","author":{"email":"760e7dab2836853c63805033e514668301fa9c47@seatgeek.com","name":"BrianLondon"},"message":"[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk\n\nSuccessfully ran kinesis demo on a live, aws hosted kinesis stream against master and 1.6 branches.  For reasons I don't entirely understand it required a manual merge to 1.5 which I did as shown here: https://github.com/BrianLondon/spark/commit/075c22e89bc99d5e99be21f40e0d72154a1e23a2\n\nThe demo ran successfully on the 1.5 branch as well.\n\nAccording to `mvn dependency:tree` it is still pulling a fairly old version of the aws-java-sdk (1.9.37), but this appears to have fixed the kinesis regression in 1.5.2.\n\nAuthor: BrianLondon <brian@seatgeek.com>\n\nCloses #10492 from BrianLondon/remove-only.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/ff89975543b153d0d235c0cac615d45b34aa8fe7"},{"sha":"1537e55604cafafa49a8b7f3ce915f9745392bc0","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"Kai Jiang"},"message":"[SPARK-12041][ML][PYSPARK] Add columnSimilarities to IndexedRowMatrix\n\nAdd `columnSimilarities` to IndexedRowMatrix for PySpark spark.mllib.linalg.\n\nAuthor: Kai Jiang <jiangkai@gmail.com>\n\nCloses #10158 from vectorijk/spark-12041.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/1537e55604cafafa49a8b7f3ce915f9745392bc0"},{"sha":"df8bd97520fc67dad95141c5a8cf2e0d5332e693","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][SQL] Import ordering fixes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10573 from vanzin/SPARK-3873-sql.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/df8bd97520fc67dad95141c5a8cf2e0d5332e693"},{"sha":"0d42292f6a2dbe626e8f6a50e6c61dd79533f235","author":{"email":"d5493c25651f198266b02bc9287959eaadab29fc@gmail.com","name":"sureshthalamati"},"message":"[SPARK-12504][SQL] Masking credentials in the sql plan explain output for JDBC data sources.\n\nThis fix masks JDBC  credentials in the explain output.  URL patterns to specify credential seems to be vary between different databases. Added a new method to dialect to mask the credentials according to the database specific URL pattern.\n\nWhile adding tests I noticed explain output includes array variable for partitions ([Lorg.apache.spark.Partition;3ff74546,).  Modified the code to include the first, and last partition information.\n\nAuthor: sureshthalamati <suresh.thalamati@gmail.com>\n\nCloses #10452 from sureshthalamati/mask_jdbc_credentials_spark-12504.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/0d42292f6a2dbe626e8f6a50e6c61dd79533f235"},{"sha":"70fe6ce52f26904aa53bd20409db69b52bccf315","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12659] fix NPE in UnsafeExternalSorter (used by cartesian product)\n\nCartesian product use UnsafeExternalSorter without comparator to do spilling, it will NPE if spilling happens.\n\nThis bug also hitted by #10605\n\ncc JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10606 from davies/fix_spilling.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/70fe6ce52f26904aa53bd20409db69b52bccf315"},{"sha":"7a375bb87a8df56d9dde0c484e725e5c497a9876","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][CORE] Import ordering fixes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10578 from vanzin/SPARK-3873-core.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/7a375bb87a8df56d9dde0c484e725e5c497a9876"},{"sha":"b3ba1be3b77e42120145252b2730a56f1d55fd21","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][TESTS] Import ordering fixes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10582 from vanzin/SPARK-3873-tests.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/b3ba1be3b77e42120145252b2730a56f1d55fd21"},{"sha":"d1fea41363c175a67b97cb7b3fe89f9043708739","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12393][SPARKR] Add read.text and write.text for SparkR\n\nAdd ```read.text``` and ```write.text``` for SparkR.\ncc sun-rui felixcheung shivaram\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10348 from yanboliang/spark-12393.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/d1fea41363c175a67b97cb7b3fe89f9043708739"},{"sha":"b2467b381096804b862990d9ecda554f67e07ee1","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12578][SQL] Distinct should not be silently ignored when used in an aggregate function with OVER clause\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12578\n\nSlightly update to Hive parser. We should keep the distinct keyword when used in an aggregate function with OVER clause. So the CheckAnalysis will detect it and throw exception later.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10557 from viirya/keep-distinct-hivesql.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/b2467b381096804b862990d9ecda554f67e07ee1"},{"sha":"5d871ea43efdde59e05896a50d57021388412d30","author":{"email":"a71868187df990d49943173493ce6d1621f222ba@gmail.com","name":"QiangCai"},"message":"[SPARK-12340][SQL] fix Int overflow in the SparkPlan.executeTake, RDD.take and AsyncRDDActions.takeAsync\n\nI have closed pull request https://github.com/apache/spark/pull/10487. And I create this pull request to resolve the problem.\n\nspark jira\nhttps://issues.apache.org/jira/browse/SPARK-12340\n\nAuthor: QiangCai <david.caiq@gmail.com>\n\nCloses #10562 from QiangCai/bugfix.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/5d871ea43efdde59e05896a50d57021388412d30"},{"sha":"94c202c7d2c613d00a0a56f870a0cddbff599be3","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-12665][CORE][GRAPHX] Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used\n\nWhole code of Vector.scala, VectorSuite.scala and GraphKryoRegistrator.scala  are no longer used so it's time to remove them in Spark 2.0.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #10613 from sarutak/SPARK-12665.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/94c202c7d2c613d00a0a56f870a0cddbff599be3"},{"sha":"9061e777fdcd5767718808e325e8953d484aa761","author":{"email":"808d31295a27ded55339ed818ba0b1912ca15edb@guavus.com","name":"Yash Datta"},"message":"[SPARK-11878][SQL] Eliminate distribute by in case group by is present with exactly the same grouping expressi\n\nFor queries like :\nselect <> from table group by a distribute by a\nwe can eliminate distribute by ; since group by will anyways do a hash partitioning\nAlso applicable when user uses Dataframe API\n\nAuthor: Yash Datta <Yash.Datta@guavus.com>\n\nCloses #9858 from saucam/eliminatedistribute.","distinct":false,"url":"https://api.github.com/repos/ajbozarth/spark/commits/9061e777fdcd5767718808e325e8953d484aa761"},{"sha":"96a3899dc1868b598fe96e4c4b91596473c3d286","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"Merge branch 'master' into spark12149","distinct":true,"url":"https://api.github.com/repos/ajbozarth/spark/commits/96a3899dc1868b598fe96e4c4b91596473c3d286"}]}