{"push_id":1023849234,"size":83,"distinct_size":47,"ref":"refs/heads/infer-filters","head":"336a18e3e9f55514545a28f0bb32658dee2ff70b","before":"e3d84ce112f887666154b9090f65441fd32c9e75","commits":[{"sha":"d18276cb1d82790a402960835e112aebd0c55513","author":{"email":"21ef94aec54fae1a36abbd011347596ae66d9959@foxmail.com","name":"Zheng RuiFeng"},"message":"[SPARK-13672][ML] Add python examples of BisectingKMeans in ML and MLLIB\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13672\n\n## What changes were proposed in this pull request?\n\nadd two python examples of BisectingKMeans for ml and mllib\n\n## How was this patch tested?\n\nmanual tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #11515 from zhengruifeng/mllib_bkm_pe.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/d18276cb1d82790a402960835e112aebd0c55513"},{"sha":"6ca990fb366cf68cd9d5afb433725d28f07e51a0","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13294][PROJECT INFRA] Remove MiMa's dependency on spark-class / Spark assembly\n\nThis patch removes the need to build a full Spark assembly before running the `dev/mima` script.\n\n- I modified the `tools` project to remove a direct dependency on Spark, so `sbt/sbt tools/fullClasspath` will now return the classpath for the `GenerateMIMAIgnore` class itself plus its own dependencies.\n   - This required me to delete two classes full of dead code that we don't use anymore\n- `GenerateMIMAIgnore` now uses [ClassUtil](http://software.clapper.org/classutil/) to find all of the Spark classes rather than our homemade JAR traversal code. The problem in our own code was that it didn't handle folders of classes properly, which is necessary in order to generate excludes with an assembly-free Spark build.\n- `./dev/mima` no longer runs through `spark-class`, eliminating the need to reason about classpath ordering between `SPARK_CLASSPATH` and the assembly.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11178 from JoshRosen/remove-assembly-in-run-tests.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/6ca990fb366cf68cd9d5afb433725d28f07e51a0"},{"sha":"0b713e0455d01999d5a027ddc2ea8527eb085b34","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-13512][ML] add example and doc for MaxAbsScaler\n\n## What changes were proposed in this pull request?\n\njira: https://issues.apache.org/jira/browse/SPARK-13512\nAdd example and doc for ml.feature.MaxAbsScaler.\n\n## How was this patch tested?\n unit tests\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #11392 from hhbyyh/maxabsdoc.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/0b713e0455d01999d5a027ddc2ea8527eb085b34"},{"sha":"234f781ae19370ce1ab485e364a6fdab7ed2598c","author":{"email":"8b686eb7d2bd6ecbcaccba98aff1768585dde44b@gmail.com","name":"sethah"},"message":"[SPARK-13787][ML][PYSPARK] Pyspark feature importances for decision tree and random forest\n\n## What changes were proposed in this pull request?\n\nThis patch adds a `featureImportance` property to the Pyspark API for `DecisionTreeRegressionModel`, `DecisionTreeClassificationModel`, `RandomForestRegressionModel` and `RandomForestClassificationModel`.\n\n## How was this patch tested?\n\nPython doc tests for the affected classes were updated to check feature importances.\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #11622 from sethah/SPARK-13787.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/234f781ae19370ce1ab485e364a6fdab7ed2598c"},{"sha":"8fff0f92a4aca90b62c6e272eabcbb0257ba38d5","author":{"email":"25cbb71b4cb5d047244aacd790fc30e39dfd6a54@gmail.com","name":"Nick Pentreath"},"message":"[HOT-FIX][SQL][ML] Fix compile error from use of DataFrame in Java MaxAbsScaler example\n\n## What changes were proposed in this pull request?\n\nFix build failure introduced in #11392 (change `DataFrame` -> `Dataset<Row>`).\n\n## How was this patch tested?\n\nExisting build/unit tests\n\nAuthor: Nick Pentreath <nick.pentreath@gmail.com>\n\nCloses #11653 from MLnick/java-maxabs-example-fix.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/8fff0f92a4aca90b62c6e272eabcbb0257ba38d5"},{"sha":"07f1c5447753a3d593cd6ececfcb03c11b1cf8ff","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13577][YARN] Allow Spark jar to be multiple jars, archive.\n\nIn preparation for the demise of assemblies, this change allows the\nYARN backend to use multiple jars and globs as the \"Spark jar\". The\nconfig option has been renamed to \"spark.yarn.jars\" to reflect that.\n\nA second option \"spark.yarn.archive\" was also added; if set, this\ntakes precedence and uploads an archive expected to contain the jar\nfiles with the Spark code and its dependencies.\n\nExisting deployments should keep working, mostly. This change drops\nsupport for the \"SPARK_JAR\" environment variable, and also does not\nfall back to using \"jarOfClass\" if no configuration is set, falling\nback to finding files under SPARK_HOME instead. This should be fine\nsince \"jarOfClass\" probably wouldn't work unless you were using\nspark-submit anyway.\n\nTested with the unit tests, and trying the different config options\non a YARN cluster.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11500 from vanzin/SPARK-13577.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/07f1c5447753a3d593cd6ececfcb03c11b1cf8ff"},{"sha":"6d37e1eb90054cdb6323b75fb202f78ece604b15","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-13817][BUILD][SQL] Re-enable MiMA and removes object DataFrame\n\n## What changes were proposed in this pull request?\n\nPR #11443 temporarily disabled MiMA check, this PR re-enables it.\n\nOne extra change is that `object DataFrame` is also removed. The only purpose of introducing `object DataFrame` was to use it as an internal factory for creating `Dataset[Row]`. By replacing this internal factory with `Dataset.newDataFrame`, both `DataFrame` and `DataFrame$` are entirely removed from the API, so that we can simply put a `MissingClassProblem` filter in `MimaExcludes.scala` for most DataFrame API  changes.\n\n## How was this patch tested?\n\nTested by MiMA check triggered by Jenkins.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #11656 from liancheng/re-enable-mima.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/6d37e1eb90054cdb6323b75fb202f78ece604b15"},{"sha":"99b7187c2dce4c73829b9b32de80b02a053763cc","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13780][SQL] Add missing dependency to build.\n\nThis is needed to avoid odd compiler errors when building just the\nsql package with maven, because of odd interactions between scalac\nand shaded classes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11640 from vanzin/SPARK-13780.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/99b7187c2dce4c73829b9b32de80b02a053763cc"},{"sha":"eb650a81f14fa7bc665856397e19ddf1a92ca3c5","author":{"email":"0a5ae72d3567a55d9fadf2c4c5fb00f8ec263cad@gmail.com","name":"Liwei Lin"},"message":"[STREAMING][MINOR] Fix a duplicate \"be\" in comments\n\nAuthor: Liwei Lin <proflin.me@gmail.com>\n\nCloses #11650 from lw-lin/typo.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/eb650a81f14fa7bc665856397e19ddf1a92ca3c5"},{"sha":"ff776b2fc1cd4c571fd542dbf807e6fa3373cb34","author":{"email":"8baa999284405a44df6958c344ec0d51df537ce1@netflix.com","name":"Nezih Yigitbasi"},"message":"[SPARK-13328][CORE] Poor read performance for broadcast variables with dynamic resource allocation\n\nWhen dynamic resource allocation is enabled fetching broadcast variables from removed executors were causing job failures and SPARK-9591 fixed this problem by trying all locations of a block before giving up. However, the locations of a block is retrieved only once from the driver in this process and the locations in this list can be stale due to dynamic resource allocation. This situation gets worse when running on a large cluster as the size of this location list can be in the order of several hundreds out of which there may be tens of stale entries. What we have observed is with the default settings of 3 max retries and 5s between retries (that's 15s per location) the time it takes to read a broadcast variable can be as high as ~17m (70 failed attempts * 15s/attempt)\n\nAuthor: Nezih Yigitbasi <nyigitbasi@netflix.com>\n\nCloses #11241 from nezihyigitbasi/SPARK-13328.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/ff776b2fc1cd4c571fd542dbf807e6fa3373cb34"},{"sha":"073bf9d4d91e0242a813f3d227e52e76c26a2200","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13807] De-duplicate `Python*Helper` instantiation code in PySpark streaming\n\nThis patch de-duplicates code in PySpark streaming which loads the `Python*Helper` classes. I also changed a few `raise e` statements to simply `raise` in order to preserve the full exception stacktrace when re-throwing.\n\nHere's a link to the whitespace-change-free diff: https://github.com/apache/spark/compare/master...JoshRosen:pyspark-reflection-deduplication?w=0\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11641 from JoshRosen/pyspark-reflection-deduplication.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/073bf9d4d91e0242a813f3d227e52e76c26a2200"},{"sha":"42afd72c654318e9fb1f2a204198221e797c2485","author":{"email":"21ef94aec54fae1a36abbd011347596ae66d9959@foxmail.com","name":"Zheng RuiFeng"},"message":"[SPARK-13814] [PYSPARK] Delete unnecessary imports in python examples files\n\nJIRA:  https://issues.apache.org/jira/browse/SPARK-13814\n\n## What changes were proposed in this pull request?\n\ndelete unnecessary imports in python examples files\n\n## How was this patch tested?\n\nmanual tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #11651 from zhengruifeng/del_import_pe.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/42afd72c654318e9fb1f2a204198221e797c2485"},{"sha":"66d9d0edfef986895490bcdeacbc0ca38e091702","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13139][SQL] Parse Hive DDL commands ourselves\n\n## What changes were proposed in this pull request?\n\nThis patch is ported over from viirya's changes in #11048. Currently for most DDLs we just pass the query text directly to Hive. Instead, we should parse these commands ourselves and in the future (not part of this patch) use the `HiveCatalog` to process these DDLs. This is a pretext to merging `SQLContext` and `HiveContext`.\n\nNote: As of this patch we still pass the query text to Hive. The difference is that we now parse the commands ourselves so in the future we can just use our own catalog.\n\n## How was this patch tested?\n\nJenkins, new `DDLCommandSuite`, which comprises of about 40% of the changes here.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11573 from andrewor14/parser-plus-plus.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/66d9d0edfef986895490bcdeacbc0ca38e091702"},{"sha":"2ef4c5963bff3574fe17e669d703b25ddd064e5d","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13830] prefer block manager than direct result for large result\n\n## What changes were proposed in this pull request?\n\nThe current RPC can't handle large blocks very well, it's very slow to fetch 100M block (about 1 minute). Once switch to block manager to fetch that, it took about 10 seconds (still could be improved).\n\n## How was this patch tested?\n\nexisting unit tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11659 from davies/direct_result.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/2ef4c5963bff3574fe17e669d703b25ddd064e5d"},{"sha":"ba8c86d06f5968c1af4db8dd9a458005bc5f214c","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13671] [SPARK-13311] [SQL] Use different physical plans for RDD and data sources\n\n## What changes were proposed in this pull request?\n\nThis PR split the PhysicalRDD into two classes, PhysicalRDD and PhysicalScan. PhysicalRDD is used for DataFrames that is created from existing RDD. PhysicalScan is used for DataFrame that is created from data sources. This enable use to apply different optimization on both of them.\n\nAlso fix the problem for sameResult() on two DataSourceScan.\n\nAlso fix the equality check to toString for `In`. It's better to use Seq there, but we can't break this public API (sad).\n\n## How was this patch tested?\n\nExisting tests. Manually tested with TPCDS query Q59 and Q64, all those duplicated exchanges can be re-used now, also saw there are 40+% performance improvement (saving half of the scan).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11514 from davies/existing_rdd.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/ba8c86d06f5968c1af4db8dd9a458005bc5f214c"},{"sha":"4eace4d384f0e12b4934019d8654b5e3886ddaef","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-13828][SQL] Bring back stack trace of AnalysisException thrown from QueryExecution.assertAnalyzed\n\nPR #11443 added an extra `plan: Option[LogicalPlan]` argument to `AnalysisException` and attached partially analyzed plan to thrown `AnalysisException` in `QueryExecution.assertAnalyzed()`.  However, the original stack trace wasn't properly inherited.  This PR fixes this issue by inheriting the stack trace.\n\nA test case is added to verify that the first entry of `AnalysisException` stack trace isn't from `QueryExecution`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #11677 from liancheng/analysis-exception-stacktrace.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/4eace4d384f0e12b4934019d8654b5e3886ddaef"},{"sha":"c079420d7c55d8972db716a2695a5ddd606d11cd","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-13841][SQL] Removes Dataset.collectRows()/takeRows()\n\n## What changes were proposed in this pull request?\n\nThis PR removes two methods, `collectRows()` and `takeRows()`, from `Dataset[T]`. These methods were added in PR #11443, and were later considered not useful.\n\n## How was this patch tested?\n\nExisting tests should do the work.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #11678 from liancheng/remove-collect-rows-and-take-rows.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/c079420d7c55d8972db716a2695a5ddd606d11cd"},{"sha":"db88d0204e3a9a05dbe6e67e1abb942639c50a06","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[MINOR][DOCS] Replace `DataFrame` with `Dataset` in Javadoc.\n\n## What changes were proposed in this pull request?\n\nSPARK-13817 (PR #11656) replaces `DataFrame` with `Dataset` from Java. This PR fixes the remaining broken links and sample Java code in `package-info.java`. As a result, it will update the following Javadoc.\n\n* http://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/attribute/package-summary.html\n* http://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/feature/package-summary.html\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11675 from dongjoon-hyun/replace_dataframe_with_dataset_in_javadoc.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/db88d0204e3a9a05dbe6e67e1abb942639c50a06"},{"sha":"515e4afbc7ec957609451ea75772d6ef1b914908","author":{"email":"bb0eadddd335e82bba4e0686d2c318a2ab1db8a0@gmail.com","name":"Bjorn Jonsson"},"message":"[SPARK-13810][CORE] Add Port Configuration Suggestions on Bind Exceptions\n\n## What changes were proposed in this pull request?\nCurrently, when a java.net.BindException is thrown, it displays the following message:\n\njava.net.BindException: Address already in use: Service '$serviceName' failed after 16 retries!\n\nThis change adds port configuration suggestions to the BindException, for example, for the UI, it now displays\n\njava.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries! Consider explicitly setting the appropriate port for 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.\n\n## How was this patch tested?\nManual tests\n\nAuthor: Bjorn Jonsson <bjornjon@gmail.com>\n\nCloses #11644 from bjornjon/master.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/515e4afbc7ec957609451ea75772d6ef1b914908"},{"sha":"c7e68c3968357268f705dee1477c448472c21547","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-13812][SPARKR] Fix SparkR lint-r test errors.\n\n## What changes were proposed in this pull request?\n\nThis PR fixes all newly captured SparkR lint-r errors after the lintr package is updated from github.\n\n## How was this patch tested?\n\ndev/lint-r\nSparkR unit tests\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #11652 from sun-rui/SPARK-13812.","distinct":false,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/c7e68c3968357268f705dee1477c448472c21547"}]}