{"push_id":1068643448,"size":22,"distinct_size":22,"ref":"refs/heads/SPARK-14656","head":"e1e1cf8731c49e5bac41c5f2187555f8108d5f13","before":"b6ea54e0185413fc3e09cb364564b5a4c27a6f68","commits":[{"sha":"dac40b68dc52d5ab855dfde63f0872064aa3d999","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14619] Track internal accumulators (metrics) by stage attempt\n\n## What changes were proposed in this pull request?\nWhen there are multiple attempts for a stage, we currently only reset internal accumulator values if all the tasks are resubmitted. It would make more sense to reset the accumulator values for each stage attempt. This will allow us to eventually get rid of the internal flag in the Accumulator class. This is part of my bigger effort to simplify accumulators and task metrics.\n\n## How was this patch tested?\nCovered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12378 from rxin/SPARK-14619.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/dac40b68dc52d5ab855dfde63f0872064aa3d999"},{"sha":"a46f98d3f4ba6a79f4ef789806fec80a7d4f342d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14617] Remove deprecated APIs in TaskMetrics\n\n## What changes were proposed in this pull request?\nThis patch removes some of the deprecated APIs in TaskMetrics. This is part of my bigger effort to simplify accumulators and task metrics.\n\n## How was this patch tested?\nN/A - only removals\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12375 from rxin/SPARK-14617.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/a46f98d3f4ba6a79f4ef789806fec80a7d4f342d"},{"sha":"1d04c86fc575470e15f6667076377cea102552d7","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-14558][CORE] In ClosureCleaner, clean the outer pointer if it's a REPL line object\n\n## What changes were proposed in this pull request?\n\nWhen we clean a closure, if its outermost parent is not a closure, we won't clone and clean it as cloning user's objects is dangerous. However, if it's a REPL line object, which may carry a lot of unnecessary references(like hadoop conf, spark conf, etc.), we should clean it as it's not a user object.\n\nThis PR improves the check for user's objects to exclude REPL line object.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #12327 from cloud-fan/closure.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/1d04c86fc575470e15f6667076377cea102552d7"},{"sha":"c971aee40d806ed02d3d6a5cc478b63654052e54","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-14499][SQL][TEST] Drop Partition Does Not Delete Data of External Tables\n\n#### What changes were proposed in this pull request?\nThis PR is to add a test to ensure drop partitions of an external table will not delete data.\n\ncc yhuai andrewor14\n\n#### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nThis patch had conflicts when merged, resolved by\nCommitter: Andrew Or <andrew@databricks.com>\n\nCloses #12350 from gatorsmile/testDropPartition.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/c971aee40d806ed02d3d6a5cc478b63654052e54"},{"sha":"28efdd3fd789fa2ebed5be03b36ca0f682e37669","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-14592][SQL] Native support for CREATE TABLE LIKE DDL command\n\n## What changes were proposed in this pull request?\nJIRA: https://issues.apache.org/jira/browse/SPARK-14592\n\nThis patch adds native support for DDL command `CREATE TABLE LIKE`.\n\nThe SQL syntax is like:\n\n    CREATE TABLE table_name LIKE existing_table\n    CREATE TABLE IF NOT EXISTS table_name LIKE existing_table\n\n## How was this patch tested?\n`HiveDDLCommandSuite`. `HiveQuerySuite` already tests `CREATE TABLE LIKE`.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nThis patch had conflicts when merged, resolved by\nCommitter: Andrew Or <andrew@databricks.com>\n\nCloses #12362 from viirya/create-table-like.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/28efdd3fd789fa2ebed5be03b36ca0f682e37669"},{"sha":"c5172f8205beabe58c0b5392c0d83f9fb9c27f18","author":{"email":"060dd14c05a8ffa8f201883ef57d0676b7c9c04a@gmail.com","name":"Bryan Cutler"},"message":"[SPARK-13967][PYSPARK][ML] Added binary Param to Python CountVectorizer\n\nAdded binary toggle param to CountVectorizer feature transformer in PySpark.\n\nCreated a unit test for using CountVectorizer with the binary toggle on.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #12308 from BryanCutler/binary-param-python-CountVectorizer-SPARK-13967.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/c5172f8205beabe58c0b5392c0d83f9fb9c27f18"},{"sha":"bf65c87f706019d235d7093637341668a13b1be1","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-14618][ML][DOC] Updated RegressionEvaluator.metricName param doc\n\n## What changes were proposed in this pull request?\n\nIn Spark 1.4, we negated some metrics from RegressionEvaluator since CrossValidator always maximized metrics. This was fixed in 1.5, but the docs were not updated. This PR updates the docs.\n\n## How was this patch tested?\n\nno tests\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #12377 from jkbradley/regeval-doc.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/bf65c87f706019d235d7093637341668a13b1be1"},{"sha":"bc748b7b8f3b5aee28aff9ea078c216ca137a5b7","author":{"email":"765086fe2e0c1f980161f127fec596800f327f62@outlook.com","name":"Yong Tang"},"message":"[SPARK-14238][ML][MLLIB][PYSPARK] Add binary toggle Param to PySpark HashingTF in ML & MLlib\n\n## What changes were proposed in this pull request?\n\nThis fix tries to add binary toggle Param to PySpark HashingTF in ML & MLlib. If this toggle is set, then all non-zero counts will be set to 1.\n\nNote: This fix (SPARK-14238) is extended from SPARK-13963 where Scala implementation was done.\n\n## How was this patch tested?\n\nThis fix adds two tests to cover the code changes. One for HashingTF in PySpark's ML and one for HashingTF in PySpark's MLLib.\n\nAuthor: Yong Tang <yong.tang.github@outlook.com>\n\nCloses #12079 from yongtang/SPARK-14238.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/bc748b7b8f3b5aee28aff9ea078c216ca137a5b7"},{"sha":"d7e124edfe2578ecdf8e816a4dda3ce430a09172","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14545][SQL] Improve `LikeSimplification` by adding `a%b` rule\n\n## What changes were proposed in this pull request?\n\nCurrent `LikeSimplification` handles the following four rules.\n- 'a%' => expr.StartsWith(\"a\")\n- '%b' => expr.EndsWith(\"b\")\n- '%a%' => expr.Contains(\"a\")\n- 'a' => EqualTo(\"a\")\n\nThis PR adds the following rule.\n- 'a%b' => expr.Length() >= 2 && expr.StartsWith(\"a\") && expr.EndsWith(\"b\")\n\nHere, 2 is statically calculated from \"a\".size + \"b\".size.\n\n**Before**\n```\nscala> sql(\"select a from (select explode(array('abc','adc')) a) T where a like 'a%c'\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Filter a#5 LIKE a%c\n:     +- INPUT\n+- Generate explode([abc,adc]), false, false, [a#5]\n   +- Scan OneRowRelation[]\n```\n\n**After**\n```\nscala> sql(\"select a from (select explode(array('abc','adc')) a) T where a like 'a%c'\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Filter ((length(a#5) >= 2) && (StartsWith(a#5, a) && EndsWith(a#5, c)))\n:     +- INPUT\n+- Generate explode([abc,adc]), false, false, [a#5]\n   +- Scan OneRowRelation[]\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests (including new testcase).\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12312 from dongjoon-hyun/SPARK-14545.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/d7e124edfe2578ecdf8e816a4dda3ce430a09172"},{"sha":"01dd1f5c07f5c9ba91389c1556f911b028475cd3","author":{"email":"765086fe2e0c1f980161f127fec596800f327f62@outlook.com","name":"Yong Tang"},"message":"[SPARK-14565][ML] RandomForest should use parseInt and parseDouble for feature subset size instead of regexes\n\n## What changes were proposed in this pull request?\n\nThis fix tries to change RandomForest's supported strategies from using regexes to using parseInt and\nparseDouble, for the purpose of robustness and maintainability.\n\n## How was this patch tested?\n\nExisting tests passed.\n\nAuthor: Yong Tang <yong.tang.github@outlook.com>\n\nCloses #12360 from yongtang/SPARK-14565.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/01dd1f5c07f5c9ba91389c1556f911b028475cd3"},{"sha":"c80586d9e820d19fc328b3e4c6f1c1439f5583a7","author":{"email":"334ef69af843dec0ae865197c8d7a398eb89e557@catawiki.nl","name":"Fokko Driesprong"},"message":"[SPARK-12869] Implemented an improved version of the toIndexedRowMatrix\n\nHi guys,\n\nI've implemented an improved version of the `toIndexedRowMatrix` function on the `BlockMatrix`. I needed this for a project, but would like to share it with the rest of the community. In the case of dense matrices, it can increase performance up to 19 times:\nhttps://github.com/Fokko/BlockMatrixToIndexedRowMatrix\n\nIf there are any questions or suggestions, please let me know. Keep up the good work! Cheers.\n\nAuthor: Fokko Driesprong <f.driesprong@catawiki.nl>\nAuthor: Fokko Driesprong <fokko@driesprongen.nl>\n\nCloses #10839 from Fokko/master.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/c80586d9e820d19fc328b3e4c6f1c1439f5583a7"},{"sha":"ff9ae61a3b7bbbfc2aac93a99c05a9e1ea9c08bc","author":{"email":"f1b5a91d4d6ad523f2610114591c007e75d15084@apache.org","name":"Mark Grover"},"message":"[SPARK-14601][DOC] Minor doc/usage changes related to removal of Spark assembly\n\n## What changes were proposed in this pull request?\n\nRemoving references to assembly jar in documentation.\nAdding an additional (previously undocumented) usage of spark-submit to run examples.\n\n## How was this patch tested?\n\nRan spark-submit usage to ensure formatting was fine. Ran examples using SparkSubmit.\n\nAuthor: Mark Grover <mark@apache.org>\n\nCloses #12365 from markgrover/spark-14601.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/ff9ae61a3b7bbbfc2aac93a99c05a9e1ea9c08bc"},{"sha":"b5c60bcdca3bcace607b204a6c196a5386e8a896","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-14447][SQL] Speed up TungstenAggregate w/ keys using VectorizedHashMap\n\n## What changes were proposed in this pull request?\n\nThis patch speeds up group-by aggregates by around 3-5x by leveraging an in-memory `AggregateHashMap` (please see https://github.com/apache/spark/pull/12161), an append-only aggregate hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the `BytesToBytesMap` if a given key isn't found).\n\nArchitecturally, it is backed by a power-of-2-sized array for index lookups and a columnar batch that stores the key-value pairs. The index lookups in the array rely on linear probing (with a small number of maximum tries) and use an inexpensive hash function which makes it really efficient for a majority of lookups. However, using linear probing and an inexpensive hash function also makes it less robust as compared to the `BytesToBytesMap` (especially for a large number of keys or even for certain distribution of keys) and requires us to fall back on the latter for correctness.\n\n## How was this patch tested?\n\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_73-b02 on Mac OS X 10.11.4\n    Intel(R) Core(TM) i7-4960HQ CPU  2.60GHz\n    Aggregate w keys:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n    -------------------------------------------------------------------------------------------\n    codegen = F                              2124 / 2204          9.9         101.3       1.0X\n    codegen = T hashmap = F                  1198 / 1364         17.5          57.1       1.8X\n    codegen = T hashmap = T                   369 /  600         56.8          17.6       5.8X\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #12345 from sameeragarwal/tungsten-aggregate-integration.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/b5c60bcdca3bcace607b204a6c196a5386e8a896"},{"sha":"297ba3f1b49cc37d9891a529142c553e0a5e2d62","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-14275][SQL] Reimplement TypedAggregateExpression to DeclarativeAggregate\n\n## What changes were proposed in this pull request?\n\n`ExpressionEncoder` is just a container for serialization and deserialization expressions, we can use these expressions to build `TypedAggregateExpression` directly, so that it can fit in `DeclarativeAggregate`, which is more efficient.\n\nOne trick is, for each buffer serializer expression, it will reference to the result object of serialization and function call. To avoid re-calculating this result object, we can serialize the buffer object to a single struct field, so that we can use a special `Expression` to only evaluate result object once.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #12067 from cloud-fan/typed_udaf.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/297ba3f1b49cc37d9891a529142c553e0a5e2d62"},{"sha":"b9613239d303bc0f451233852c1eb1219a69875e","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-14374][ML][PYSPARK] PySpark ml GBTClassifier, Regressor support export/import\n\n## What changes were proposed in this pull request?\nPySpark ml GBTClassifier, Regressor support export/import.\n\n## How was this patch tested?\nDoc test.\n\ncc jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #12383 from yanboliang/spark-14374.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/b9613239d303bc0f451233852c1eb1219a69875e"},{"sha":"a9324a06ef0e3646410dc9b3d4f21d66b9064303","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #12407\nCloses #12408\nCloses #12401","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/a9324a06ef0e3646410dc9b3d4f21d66b9064303"},{"sha":"96534aa47c39e0ec40bc38be566455d11e21adb2","author":{"email":"2a07432b1b7252977a4c2dfc8e6d6d5ac763a480@netflix.com","name":"DB Tsai"},"message":"[SPARK-14549][ML] Copy the Vector and Matrix classes from mllib to ml in mllib-local\n\n## What changes were proposed in this pull request?\n\nThis task will copy the Vector and Matrix classes from mllib to ml package in mllib-local jar. The UDTs and `since` annotation in ml vector and matrix will be removed from now. UDTs will be achieved by #SPARK-14487, and `since` will be replaced by /*  since 1.2.0 */\n\nThe BLAS implementation will be copied, and some of the test utilities will be copies as well.\n\nSummary of changes:\n\n1. In mllib-local/src/main/scala/org/apache/spark/**ml**/linalg/BLAS.scala\n  - Copied from mllib/src/main/scala/org/apache/spark/**mllib**/linalg/BLAS.scala\n  - logDebug(\"gemm: alpha is equal to 0 and beta is equal to 1. Returning C.\") is removed in ml version.\n2. In  mllib-local/src/main/scala/org/apache/spark/**ml**/linalg/Matrices.scala\n  - Copied from mllib/src/main/scala/org/apache/spark/**mllib**/linalg/Matrices.scala\n  - `Since` was removed, and we'll use standard `/* Since /*` Java doc. Will be in another PR.\n  - `UDT` related code was removed, and will use `SPARK-13944` https://github.com/apache/spark/pull/12259  to replace the annotation.\n3. In mllib-local/src/main/scala/org/apache/spark/**ml**/linalg/Vectors.scala\n  - Copied from mllib/src/main/scala/org/apache/spark/**mllib**/linalg/Vectors.scala\n  - `Since` was removed.\n  - `UDT` related code was removed.\n  - In `def parseNumeric`, it was throwing `throw new SparkException(s\"Cannot parse $other.\")`, and now it's throwing `throw new IllegalArgumentException(s\"Cannot parse $other.\")`\n4. In mllib/src/main/scala/org/apache/spark/**mllib**/linalg/Vectors.scala\n  - For consistency with ML version of vector, `def parseNumeric` is now throwing `throw new IllegalArgumentException(s\"Cannot parse $other.\")`\n5. mllib/src/main/scala/org/apache/spark/**mllib**/util/NumericParser.scala is moved to mllib-local/src/main/scala/org/apache/spark/**ml**/util/NumericParser.scala\n  - All the `throw new SparkException` were replaced by `throw new IllegalArgumentException`\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: DB Tsai <dbt@netflix.com>\n\nCloses #12317 from dbtsai/dbtsai-ml-vector.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/96534aa47c39e0ec40bc38be566455d11e21adb2"},{"sha":"e24923267f79e7fc03180095fcbb28a91f998f5d","author":{"email":"bc9570497728884c7f6686fa0fc06a8d320e124b@in.ibm.com","name":"Pravin Gadakh"},"message":"[SPARK-14370][MLLIB] removed duplicate generation of ids in OnlineLDAOptimizer\n\n## What changes were proposed in this pull request?\n\nRemoved duplicated generation of `ids` in OnlineLDAOptimizer.\n\n## How was this patch tested?\n\ntested with existing unit tests.\n\nAuthor: Pravin Gadakh <prgadakh@in.ibm.com>\n\nCloses #12176 from pravingadakh/SPARK-14370.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/e24923267f79e7fc03180095fcbb28a91f998f5d"},{"sha":"06b9d623e8f58d7bd450a50d938f83b4b3472a32","author":{"email":"3ff3bed8b5f1480faa2d0264bcbfec714c961108@cloudera.com","name":"Peter Ableda"},"message":"[SPARK-14633] Use more readable format to show memory bytes in Error Message\n\n## What changes were proposed in this pull request?\n\nRound memory bytes and convert it to Long to it’s original type. This change fixes the formatting issue in the Exception message.\n\n## How was this patch tested?\n\nManual tests were done in CDH cluster.\n\nAuthor: Peter Ableda <peter.ableda@cloudera.com>\n\nCloses #12392 from peterableda/SPARK-14633.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/06b9d623e8f58d7bd450a50d938f83b4b3472a32"},{"sha":"83af297ac42546580983f91079f74e3a4cf25050","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-13925][ML][SPARKR] Expose R-like summary statistics in SparkR::glm for more family and link functions\n\n## What changes were proposed in this pull request?\nExpose R-like summary statistics in SparkR::glm for more family and link functions.\nNote: Not all values in R [summary.glm](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/summary.glm.html) are exposed, we only provide the most commonly used statistics in this PR. More statistics can be added in the followup work.\n\n## How was this patch tested?\nUnit tests.\n\nSparkR Output:\n```\nDeviance Residuals:\n(Note: These are approximate quantiles with relative error <= 0.01)\n     Min        1Q    Median        3Q       Max\n-0.95096  -0.16585  -0.00232   0.17410   0.72918\n\nCoefficients:\n                    Estimate  Std. Error  t value  Pr(>|t|)\n(Intercept)         1.6765    0.23536     7.1231   4.4561e-11\nSepal_Length        0.34988   0.046301    7.5566   4.1873e-12\nSpecies_versicolor  -0.98339  0.072075    -13.644  0\nSpecies_virginica   -1.0075   0.093306    -10.798  0\n\n(Dispersion parameter for gaussian family taken to be 0.08351462)\n\n    Null deviance: 28.307  on 149  degrees of freedom\nResidual deviance: 12.193  on 146  degrees of freedom\nAIC: 59.22\n\nNumber of Fisher Scoring iterations: 1\n```\nR output:\n```\nDeviance Residuals:\n     Min        1Q    Median        3Q       Max\n-0.95096  -0.16522   0.00171   0.18416   0.72918\n\nCoefficients:\n                  Estimate Std. Error t value Pr(>|t|)\n(Intercept)        1.67650    0.23536   7.123 4.46e-11 ***\nSepal.Length       0.34988    0.04630   7.557 4.19e-12 ***\nSpeciesversicolor -0.98339    0.07207 -13.644  < 2e-16 ***\nSpeciesvirginica  -1.00751    0.09331 -10.798  < 2e-16 ***\n---\nSignif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n\n(Dispersion parameter for gaussian family taken to be 0.08351462)\n\n    Null deviance: 28.307  on 149  degrees of freedom\nResidual deviance: 12.193  on 146  degrees of freedom\nAIC: 59.217\n\nNumber of Fisher Scoring iterations: 2\n```\n\ncc mengxr\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #12393 from yanboliang/spark-13925.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/83af297ac42546580983f91079f74e3a4cf25050"}]}