{"push_id":929575410,"size":111,"distinct_size":111,"ref":"refs/heads/anti_join2","head":"5b3815ac6a457da5ac4cf4d90073aed8af4aa060","before":"7d9bf05b00aa52524d1423befa4b1fb6ff675c2d","commits":[{"sha":"c3d505602de2fd2361633f90e4fff7e041849e28","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-12327][SPARKR] fix code for lintr warning for commented code\n\nshivaram\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #10408 from felixcheung/rcodecomment.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/c3d505602de2fd2361633f90e4fff7e041849e28"},{"sha":"c82924d564c07e6e6f635b9e263994dedf06268a","author":{"email":"4d9d93d024b8bb6f54163a9abd3ede029ef7e21e@tcs.com","name":"thomastechs"},"message":"[SPARK-12533][SQL] hiveContext.table() throws the wrong exception\n\nAvoiding the the No such table exception and throwing analysis exception as per the bug: SPARK-12533\n\nAuthor: thomastechs <thomas.sebastian@tcs.com>\n\nCloses #10529 from thomastechs/topic-branch.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/c82924d564c07e6e6f635b9e263994dedf06268a"},{"sha":"7b92922f7f7ba4ff398dcbd734e8305ba03da87b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Update MimaExcludes now Spark 1.6 is in Maven.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10561 from rxin/update-mima.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/7b92922f7f7ba4ff398dcbd734e8305ba03da87b"},{"sha":"b8410ff9ce8cef7159a7364272e4c4234c5b474f","author":{"email":"1ac01174d25a69dd02ea3a466537bebdc934f2cf@korea.com","name":"Cazen"},"message":"[SPARK-12537][SQL] Add option to accept quoting of all character backslash quoting mechanism\n\nWe can provides the option to choose JSON parser can be enabled to accept quoting of all character or not.\n\nAuthor: Cazen <Cazen@korea.com>\nAuthor: Cazen Lee <cazen.lee@samsung.com>\nAuthor: Cazen Lee <Cazen@korea.com>\nAuthor: cazen.lee <cazen.lee@samsung.com>\n\nCloses #10497 from Cazen/master.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/b8410ff9ce8cef7159a7364272e4c4234c5b474f"},{"sha":"13dab9c3862cc454094cd9ba7b4504a2d095028f","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-12611][SQL][PYSPARK][TESTS] Fix test_infer_schema_to_local\n\nPreviously (when the PR was first created) not specifying b= explicitly was fine (and treated as default null) - instead be explicit about b being None in the test.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #10564 from holdenk/SPARK-12611-fix-test-infer-schema-local.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/13dab9c3862cc454094cd9ba7b4504a2d095028f"},{"sha":"84f8492c1555bf8ab44c9818752278f61768eb16","author":{"email":"122021f0743769ca0bda88e68286824599e80160@gmail.com","name":"Xiu Guo"},"message":"[SPARK-12562][SQL] DataFrame.write.format(text) requires the column name to be called value\n\nAuthor: Xiu Guo <xguo27@gmail.com>\n\nCloses #10515 from xguo27/SPARK-12562.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/84f8492c1555bf8ab44c9818752278f61768eb16"},{"sha":"0d165ec2050aa06aa545d74c8d7c2ff197fa02de","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12612][PROJECT-INFRA] Add missing Hadoop profiles to dev/run-tests-*.py scripts and dev/deps\n\nThere are a couple of places in the `dev/run-tests-*.py` scripts which deal with Hadoop profiles, but the set of profiles that they handle does not include all Hadoop profiles defined in our POM. Similarly, the `hadoop-2.2` and `hadoop-2.6` profiles were missing from `dev/deps`.\n\nThis patch updates these scripts to include all four Hadoop profiles defined in our POM.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10565 from JoshRosen/add-missing-hadoop-profiles-in-test-scripts.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/0d165ec2050aa06aa545d74c8d7c2ff197fa02de"},{"sha":"9fd7a2f0247ed6cea0e8dbcdd2b24f41200b3e24","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching\n\nThis patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script.\n\npwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead.\n\nI also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10558 from JoshRosen/build-dep-tests-round-3.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/9fd7a2f0247ed6cea0e8dbcdd2b24f41200b3e24"},{"sha":"962aac4db99f3988c07ccb23439327c18ec178f1","author":{"email":"d8e999f5f3d9d93b8c582fe1e51a2c254ea02f30@gmail.com","name":"guoxu1231"},"message":"[SPARK-12513][STREAMING] SocketReceiver hang in Netcat example\n\nExplicitly close client side socket connection before restart socket receiver.\n\nAuthor: guoxu1231 <guoxu1231@gmail.com>\nAuthor: Shawn Guo <guoxu1231@gmail.com>\n\nCloses #10464 from guoxu1231/SPARK-12513.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/962aac4db99f3988c07ccb23439327c18ec178f1"},{"sha":"8f659393b270c46e940c4e98af2d996bd4fd6442","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12486] Worker should kill the executors more forcefully if possible.\n\nThis patch updates the ExecutorRunner's terminate path to use the new java 8 API\nto terminate processes more forcefully if possible. If the executor is unhealthy,\nit would previously ignore the destroy() call. Presumably, the new java API was\nadded to handle cases like this.\n\nWe could update the termination path in the future to use OS specific commands\nfor older java versions.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10438 from nongli/spark-12486-executors.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/8f659393b270c46e940c4e98af2d996bd4fd6442"},{"sha":"6c83d938cc61bd5fabaf2157fcc3936364a83f02","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12579][SQL] Force user-specified JDBC driver to take precedence\n\nSpark SQL's JDBC data source allows users to specify an explicit JDBC driver to load (using the `driver` argument), but in the current code it's possible that the user-specified driver will not be used when it comes time to actually create a JDBC connection.\n\nIn a nutshell, the problem is that you might have multiple JDBC drivers on the classpath that claim to be able to handle the same subprotocol, so simply registering the user-provided driver class with the our `DriverRegistry` and JDBC's `DriverManager` is not sufficient to ensure that it's actually used when creating the JDBC connection.\n\nThis patch addresses this issue by first registering the user-specified driver with the DriverManager, then iterating over the driver manager's loaded drivers in order to obtain the correct driver and use it to create a connection (previously, we just called `DriverManager.getConnection()` directly).\n\nIf a user did not specify a JDBC driver to use, then we call `DriverManager.getDriver` to figure out the class of the driver to use, then pass that class's name to executors; this guards against corner-case bugs in situations where the driver and executor JVMs might have different sets of JDBC drivers on their classpaths (previously, there was the (rare) potential for `DriverManager.getConnection()` to use different drivers on the driver and executors if the user had not explicitly specified a JDBC driver class and the classpaths were different).\n\nThis patch is inspired by a similar patch that I made to the `spark-redshift` library (https://github.com/databricks/spark-redshift/pull/143), which contains its own modified fork of some of Spark's JDBC data source code (for cross-Spark-version compatibility reasons).\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10519 from JoshRosen/jdbc-driver-precedence.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/6c83d938cc61bd5fabaf2157fcc3936364a83f02"},{"sha":"b504b6a90a95a723210beb0031ed41a75d702f66","author":{"email":"73b9523a173a4d37b1985403702ac17bd8e586ea@gmail.com","name":"Pete Robbins"},"message":"[SPARK-12470] [SQL] Fix size reduction calculation\n\nalso only allocate required buffer size\n\nAuthor: Pete Robbins <robbinspg@gmail.com>\n\nCloses #10421 from robbinspg/master.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/b504b6a90a95a723210beb0031ed41a75d702f66"},{"sha":"43706bf8bdfe08010bb11848788e0718d15363b3","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12608][STREAMING] Remove submitJobThreadPool since submitJob doesn't create a separate thread to wait for the job result\n\nBefore #9264, submitJob would create a separate thread to wait for the job result. `submitJobThreadPool` was a workaround in `ReceiverTracker` to run these waiting-job-result threads. Now #9264 has been merged to master and resolved this blocking issue, `submitJobThreadPool` can be removed now.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10560 from zsxwing/remove-submitJobThreadPool.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/43706bf8bdfe08010bb11848788e0718d15363b3"},{"sha":"573ac55d7469ea2ea7a5979b4d3eea99c98f6560","author":{"email":"122021f0743769ca0bda88e68286824599e80160@gmail.com","name":"Xiu Guo"},"message":"[SPARK-12512][SQL] support column name with dot in withColumn()\n\nAuthor: Xiu Guo <xguo27@gmail.com>\n\nCloses #10500 from xguo27/SPARK-12512.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/573ac55d7469ea2ea7a5979b4d3eea99c98f6560"},{"sha":"40d03960d79debdff5cef21997417c4f8a8ce2e9","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[DOC] Adjust coverage for partitionBy()\n\nThis is the related thread: http://search-hadoop.com/m/q3RTtO3ReeJ1iF02&subj=Re+partitioning+json+data+in+spark\n\nMichael suggested fixing the doc.\n\nPlease review.\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #10499 from ted-yu/master.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/40d03960d79debdff5cef21997417c4f8a8ce2e9"},{"sha":"0171b71e9511cef512e96a759e407207037f3c49","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-12421][SQL] Prevent Internal/External row from exposing state.\n\nIt is currently possible to change the values of the supposedly immutable ```GenericRow``` and ```GenericInternalRow``` classes. This is caused by the fact that scala's ArrayOps ```toArray``` (returned by calling ```toSeq```) will return the backing array instead of a copy. This PR fixes this problem.\n\nThis PR was inspired by https://github.com/apache/spark/pull/10374 by apo1.\n\ncc apo1 sarutak marmbrus cloud-fan nongli (everyone in the previous conversation).\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #10553 from hvanhovell/SPARK-12421.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/0171b71e9511cef512e96a759e407207037f3c49"},{"sha":"ba5f81859d6ba37a228a1c43d26c47e64c0382cd","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11259][ML] Params.validateParams() should be called automatically\n\nSee JIRA: https://issues.apache.org/jira/browse/SPARK-11259\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9224 from yanboliang/spark-11259.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/ba5f81859d6ba37a228a1c43d26c47e64c0382cd"},{"sha":"93ef9b6a2aa1830170cb101f191022f2dda62c41","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-9622][ML] DecisionTreeRegressor: provide variance of prediction\n\nDecisionTreeRegressor will provide variance of prediction as a Double column.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #8866 from yanboliang/spark-9622.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/93ef9b6a2aa1830170cb101f191022f2dda62c41"},{"sha":"d084a2de3271fd8b0d29ee67e1e214e8b9d96d6d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12541] [SQL] support cube/rollup as function\n\nThis PR enable cube/rollup as function, so they can be used as this:\n```\nselect a, b, sum(c) from t group by rollup(a, b)\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10522 from davies/rollup.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/d084a2de3271fd8b0d29ee67e1e214e8b9d96d6d"},{"sha":"34de24abb518e95c4312b77aa107d061ce02c835","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set the row length.\n\nThe reader was previously not setting the row length meaning it was wrong if there were variable\nlength columns. This problem does not manifest usually, since the value in the column is correct and\nprojecting the row fixes the issue.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #10576 from nongli/spark-12589.","distinct":true,"url":"https://api.github.com/repos/chenghao-intel/spark/commits/34de24abb518e95c4312b77aa107d061ce02c835"}]}