{"push_id":1059226440,"size":28,"distinct_size":28,"ref":"refs/heads/master","head":"5989c85b535f7f623392d6456d8b37052487f24b","before":"73b56a3c6c5c590219b42884c8bbe88b0a236987","commits":[{"sha":"6447098013fad708769423ef108a2b071e0930d8","author":{"email":"9749d1492af3d43f9c09e04c5c43f27bb909af51@japila.pl","name":"Jacek Laskowski"},"message":"[SPARK-14402][HOTFIX] Fix ExpressionDescription annotation\n\n## What changes were proposed in this pull request?\n\nFix for the error introduced in https://github.com/apache/spark/commit/c59abad052b7beec4ef550049413e95578e545be:\n\n```\n/Users/jacek/dev/oss/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:626: error: annotation argument needs to be a constant; found: \"_FUNC_(str) - \".+(\"Returns str, with the first letter of each word in uppercase, all other letters in \").+(\"lowercase. Words are delimited by white space.\")\n    \"Returns str, with the first letter of each word in uppercase, all other letters in \" +\n                                                                                          ^\n```\n\n## How was this patch tested?\n\nLocal build\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #12192 from jaceklaskowski/SPARK-14402-HOTFIX.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/6447098013fad708769423ef108a2b071e0930d8"},{"sha":"583b5e05309adb73cdffd974a810d6bfb5f2ff95","author":{"email":"43d9fbd9264b5df0f5e418c33a17ebff54285d80@amazon.com","name":"Aaron Tokhy"},"message":"[SPARK-14470] Allow for overriding both httpclient and httpcore versions\n\n## What changes were proposed in this pull request?\n\nThis splits commons.httpclient.version from commons.httpcore.version, since these two versions do not necessarily have to be the same.  This change may follow up with an up-to-date version of the httpclient/httpcore libraries.\n\nThe latest 4.3.x httpclient version as of writing is 4.3.6 and the latest 4.3.x httpcore version as of writing is 4.3.3.  This change would be a prerequisite for potentially moving to this new bugfix version.\n\n## How was this patch tested?\nno version change was made for httpclient/httpcore versions\nmvn package\n\nAuthor: Aaron Tokhy <tokaaron@amazon.com>\n\nCloses #12245 from atokhy/pull-request.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/583b5e05309adb73cdffd974a810d6bfb5f2ff95"},{"sha":"a9b630f42ac0c6be3437f206beddaf0ef737f5c8","author":{"email":"f1b5a91d4d6ad523f2610114591c007e75d15084@apache.org","name":"Mark Grover"},"message":"[SPARK-14477][BUILD] Allow custom mirrors for downloading artifacts in build/mvn\n\n## What changes were proposed in this pull request?\n\nAllows to override locations for downloading Apache and Typesafe artifacts in build/mvn script.\n\n## How was this patch tested?\nBy running script like\n````\n# Remove all previously downloaded artifacts\nrm -rf build/apache-maven*\nrm -rf build/zinc-*\nrm -rf build/scala-*\n\n# Make sure path is clean and doesn't contain mvn, for example.\n...\n\n# Run a command without setting anything and make sure it succeeds\nbuild/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package\n\n# Run a command setting the default location as mirror and make sure it succeeds\nAPACHE_MIRROR=http://mirror.infra.cloudera.com/apache/ build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package\n\n# Do the same without the trailing slash this time and make sure it succeeds\nAPACHE_MIRROR=http://mirror.infra.cloudera.com/apache build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package\n\n# Do it with a bad URL and make sure it fails\nAPACHE_MIRROR=xyz build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 package\n````\n\nAuthor: Mark Grover <mark@apache.org>\n\nCloses #12250 from markgrover/spark-14477.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/a9b630f42ac0c6be3437f206beddaf0ef737f5c8"},{"sha":"e5d8d6e09cad304e353c96f9408fb9f799348827","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"Kai Jiang"},"message":"[SPARK-14373][PYSPARK] PySpark RandomForestClassifier, Regressor support export/import\n\n## What changes were proposed in this pull request?\nsupporting `RandomForest{Classifier, Regressor}` save/load for Python API.\n[JIRA](https://issues.apache.org/jira/browse/SPARK-14373)\n## How was this patch tested?\ndoctest\n\nAuthor: Kai Jiang <jiangkai@gmail.com>\n\nCloses #12238 from vectorijk/spark-14373.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/e5d8d6e09cad304e353c96f9408fb9f799348827"},{"sha":"e0ad75f2b55772efc82a6f8ebb1b2d80fe27d9b5","author":{"email":"d82d494ad12feccfb00e29073898c0c0951e931f@hotmail.com","name":"wm624@hotmail.com"},"message":"[SPARK-12569][PYSPARK][ML] DecisionTreeRegressor: provide variance of prediction: Python API\n\n## What changes were proposed in this pull request?\n\nA new column VarianceCol has been added to DecisionTreeRegressor in ML scala code.\n\nThis patch adds the corresponding Python API, HasVarianceCol, to class DecisionTreeRegressor.\n\n## How was this patch tested?\n./dev/lint-python\nPEP8 checks passed.\nrm -rf _build/*\npydoc checks passed.\n\n./python/run-tests --python-executables=python2.7 --modules=pyspark-ml\nRunning PySpark tests. Output is in /Users/mwang/spark_ws_0904/python/unit-tests.log\nWill test against the following Python executables: ['python2.7']\nWill test the following Python modules: ['pyspark-ml']\nFinished test(python2.7): pyspark.ml.evaluation (12s)\nFinished test(python2.7): pyspark.ml.clustering (18s)\nFinished test(python2.7): pyspark.ml.classification (30s)\nFinished test(python2.7): pyspark.ml.recommendation (28s)\nFinished test(python2.7): pyspark.ml.feature (43s)\nFinished test(python2.7): pyspark.ml.regression (31s)\nFinished test(python2.7): pyspark.ml.tuning (19s)\nFinished test(python2.7): pyspark.ml.tests (34s)\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #12116 from wangmiao1981/fix_api.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/e0ad75f2b55772efc82a6f8ebb1b2d80fe27d9b5"},{"sha":"94ac58b2a8ae83be670169062c8b83bf10e41d74","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[BUILD][HOTFIX] Download Maven from regular mirror network rather than archive.apache.org\n\n[archive.apache.org](https://archive.apache.org/) is undergoing maintenance, breaking our `build/mvn` script:\n\n> We are in the process of relocating this service. To save on the immense bandwidth that this service outputs, we have put it in maintenance mode, disabling all downloads for the next few days. We expect the maintenance to be complete no later than the morning of Monday the 11th of April, 2016.\n\nThis patch fixes this issue by updating the script to use the regular mirror network to download Maven.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #12262 from JoshRosen/fix-mvn-download.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/94ac58b2a8ae83be670169062c8b83bf10e41d74"},{"sha":"56af8e85cca056096fe4e765d8d287e0f9efc0d2","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-14298][ML][MLLIB] LDA should support disable checkpoint\n\n## What changes were proposed in this pull request?\nIn the doc of [```checkpointInterval```](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala#L241), we told users that they can disable checkpoint by setting ```checkpointInterval = -1```. But we did not handle this situation for LDA actually, we should fix this bug.\n## How was this patch tested?\nExisting tests.\n\ncc jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #12089 from yanboliang/spark-14298.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/56af8e85cca056096fe4e765d8d287e0f9efc0d2"},{"sha":"02757535b58069ce8258108d89d8172a53c358e5","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-14448] Improvements to ColumnVector\n\n## What changes were proposed in this pull request?\n\nIn this PR, two changes are proposed for ColumnVector :\n1. ColumnVector should be declared as implementing AutoCloseable - it already has close() method\n2. In OnHeapColumnVector#reserveInternal(), we only need to allocate new array when existing array is null or the length of existing array is shorter than the newCapacity.\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #12225 from tedyu/master.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/02757535b58069ce8258108d89d8172a53c358e5"},{"sha":"f8c9beca38f1f396eb3220b23db6d77112a50293","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-14394][SQL] Generate AggregateHashMap class for LongTypes during TungstenAggregate codegen\n\n## What changes were proposed in this pull request?\n\nThis PR adds support for generating the `AggregateHashMap` class in `TungstenAggregate` if the aggregate group by keys/value are of `LongType`. Note that currently this generate aggregate is not actually used.\n\nNB: This currently only supports `LongType` keys/values (please see `isAggregateHashMapSupported` in `TungstenAggregate`) and will be generalized to other data types in a subsequent PR.\n\n## How was this patch tested?\n\nManually inspected the generated code. This is what the generated map looks like for 2 keys:\n\n```java\n/* 068 */   public class agg_GeneratedAggregateHashMap {\n/* 069 */     private org.apache.spark.sql.execution.vectorized.ColumnarBatch batch;\n/* 070 */     private int[] buckets;\n/* 071 */     private int numBuckets;\n/* 072 */     private int maxSteps;\n/* 073 */     private int numRows = 0;\n/* 074 */     private org.apache.spark.sql.types.StructType schema =\n/* 075 */     new org.apache.spark.sql.types.StructType()\n/* 076 */     .add(\"k1\", org.apache.spark.sql.types.DataTypes.LongType)\n/* 077 */     .add(\"k2\", org.apache.spark.sql.types.DataTypes.LongType)\n/* 078 */     .add(\"sum\", org.apache.spark.sql.types.DataTypes.LongType);\n/* 079 */\n/* 080 */     public agg_GeneratedAggregateHashMap(int capacity, double loadFactor, int maxSteps) {\n/* 081 */       assert (capacity > 0 && ((capacity & (capacity - 1)) == 0));\n/* 082 */       this.maxSteps = maxSteps;\n/* 083 */       numBuckets = (int) (capacity / loadFactor);\n/* 084 */       batch = org.apache.spark.sql.execution.vectorized.ColumnarBatch.allocate(schema,\n/* 085 */         org.apache.spark.memory.MemoryMode.ON_HEAP, capacity);\n/* 086 */       buckets = new int[numBuckets];\n/* 087 */       java.util.Arrays.fill(buckets, -1);\n/* 088 */     }\n/* 089 */\n/* 090 */     public agg_GeneratedAggregateHashMap() {\n/* 091 */       new agg_GeneratedAggregateHashMap(1 << 16, 0.25, 5);\n/* 092 */     }\n/* 093 */\n/* 094 */     public org.apache.spark.sql.execution.vectorized.ColumnarBatch.Row findOrInsert(long agg_key, long agg_key1) {\n/* 095 */       long h = hash(agg_key, agg_key1);\n/* 096 */       int step = 0;\n/* 097 */       int idx = (int) h & (numBuckets - 1);\n/* 098 */       while (step < maxSteps) {\n/* 099 */         // Return bucket index if it's either an empty slot or already contains the key\n/* 100 */         if (buckets[idx] == -1) {\n/* 101 */           batch.column(0).putLong(numRows, agg_key);\n/* 102 */           batch.column(1).putLong(numRows, agg_key1);\n/* 103 */           batch.column(2).putLong(numRows, 0);\n/* 104 */           buckets[idx] = numRows++;\n/* 105 */           return batch.getRow(buckets[idx]);\n/* 106 */         } else if (equals(idx, agg_key, agg_key1)) {\n/* 107 */           return batch.getRow(buckets[idx]);\n/* 108 */         }\n/* 109 */         idx = (idx + 1) & (numBuckets - 1);\n/* 110 */         step++;\n/* 111 */       }\n/* 112 */       // Didn't find it\n/* 113 */       return null;\n/* 114 */     }\n/* 115 */\n/* 116 */     private boolean equals(int idx, long agg_key, long agg_key1) {\n/* 117 */       return batch.column(0).getLong(buckets[idx]) == agg_key && batch.column(1).getLong(buckets[idx]) == agg_key1;\n/* 118 */     }\n/* 119 */\n/* 120 */     // TODO: Improve this Hash Function\n/* 121 */     private long hash(long agg_key, long agg_key1) {\n/* 122 */       return agg_key ^ agg_key1;\n/* 123 */     }\n/* 124 */\n/* 125 */   }\n```\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #12161 from sameeragarwal/tungsten-aggregate.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/f8c9beca38f1f396eb3220b23db6d77112a50293"},{"sha":"464a3c1e02c665c7ad2709f8c47898b682526eb3","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-14435][BUILD] Shade Kryo in our custom Hive 1.2.1 fork\n\nThis patch updates our custom Hive 1.2.1 fork in order to shade Kryo in Hive. This is a blocker for upgrading Spark to use Kryo 3 (see #12076).\n\nThe source for this new fork of Hive can be found at https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2\n\nHere's the complete diff from the official Hive 1.2.1 release: https://github.com/apache/hive/compare/release-1.2.1...JoshRosen:release-1.2.1-spark2\n\nHere's the diff from the sources that pwendell used to publish the current `1.2.1.spark` release of Hive: https://github.com/pwendell/hive/compare/release-1.2.1-spark...JoshRosen:release-1.2.1-spark2. This diff looks large because his branch used a shell script to rewrite the groupId, whereas I had to commit the groupId changes in order to prevent the find-and-replace from affecting the package names in our relocated Kryo classes: https://github.com/pwendell/hive/compare/release-1.2.1-spark...JoshRosen:release-1.2.1-spark2#diff-6ada9aaec70e069df8f2c34c5519dd1e\n\nUsing these changes, I was able to publish a local version of Hive and verify that this change fixes the test failures which are blocking #12076. Note that this PR will not compile until we complete the review of the Hive POM changes and stage and publish a release.\n\n/cc vanzin, steveloughran, and pwendell for review.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #12215 from JoshRosen/shade-kryo-in-hive.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/464a3c1e02c665c7ad2709f8c47898b682526eb3"},{"sha":"906eef4c7a380419f2d089262afdcf39454fe31e","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11416][BUILD] Update to Chill 0.8.0 & Kryo 3.0.3\n\nThis patch upgrades Chill to 0.8.0 and Kryo to 3.0.3. While we'll likely need to bump these dependencies again before Spark 2.0 (due to SPARK-14221 / https://github.com/twitter/chill/issues/252), I wanted to get the bulk of the Kryo 2 -> Kryo 3 migration done now in order to figure out whether there are any unexpected surprises.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #12076 from JoshRosen/kryo3.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/906eef4c7a380419f2d089262afdcf39454fe31e"},{"sha":"4d7c35926371f9e016577987c037abcf984443d9","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-14437][CORE] Use the address that NettyBlockTransferService listens to create BlockManagerId\n\n## What changes were proposed in this pull request?\n\nHere is why SPARK-14437 happens:\nBlockManagerId is created using NettyBlockTransferService.hostName which comes from `customHostname`. And `Executor` will set `customHostname` to the hostname which is detected by the driver. However, the driver may not be able to detect the correct address in some complicated network (Netty's Channel.remoteAddress doesn't always return a connectable address). In such case, `BlockManagerId` will be created using a wrong hostname.\n\nTo fix this issue, this PR uses `hostname` provided by `SparkEnv.create` to create `NettyBlockTransferService` and set `NettyBlockTransferService.hostname` to this one directly. A bonus of this approach is NettyBlockTransferService won't bound to `0.0.0.0` which is much safer.\n\n## How was this patch tested?\n\nManually checked the bound address using local-cluster.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #12240 from zsxwing/SPARK-14437.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/4d7c35926371f9e016577987c037abcf984443d9"},{"sha":"813e96e6faee44079eb52acbdc6c8aa58fb8d191","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-14454] Better exception handling while marking tasks as failed\n\n## What changes were proposed in this pull request?\n\nThis patch adds support for better handling of exceptions inside catch blocks if the code within the block throws an exception. For instance here is the code in a catch block before this change in `WriterContainer.scala`:\n\n```scala\nlogError(\"Aborting task.\", cause)\n// call failure callbacks first, so we could have a chance to cleanup the writer.\nTaskContext.get().asInstanceOf[TaskContextImpl].markTaskFailed(cause)\nif (currentWriter != null) {\n  currentWriter.close()\n}\nabortTask()\nthrow new SparkException(\"Task failed while writing rows.\", cause)\n```\n\nIf `markTaskFailed` or `currentWriter.close` throws an exception, we currently lose the original cause. This PR fixes this problem by implementing a utility function `Utils.tryWithSafeCatch` that suppresses (`Throwable.addSuppressed`) the exception that are thrown within the catch block and rethrowing the original exception.\n\n## How was this patch tested?\n\nNo new functionality added\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #12234 from sameeragarwal/fix-exception.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/813e96e6faee44079eb52acbdc6c8aa58fb8d191"},{"sha":"d7af736b2cf6c392b87e7b45c2d2219ef06979eb","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-14498][ML][PYTHON][SQL] Many cleanups to ML and ML-related docs\n\n## What changes were proposed in this pull request?\n\nCleanups to documentation.  No changes to code.\n* GBT docs: Move Scala doc for private object GradientBoostedTrees to public docs for GBTClassifier,Regressor\n* GLM regParam: needs doc saying it is for L2 only\n* TrainValidationSplitModel: add .. versionadded:: 2.0.0\n* Rename “_transformer_params_from_java” to “_transfer_params_from_java”\n* LogReg Summary classes: “probability” col should not say “calibrated”\n* LR summaries: coefficientStandardErrors —> document that intercept stderr comes last.  Same for t,p-values\n* approxCountDistinct: Document meaning of “rsd\" argument.\n* LDA: note which params are for online LDA only\n\n## How was this patch tested?\n\nDoc build\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #12266 from jkbradley/ml-doc-cleanups.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/d7af736b2cf6c392b87e7b45c2d2219ef06979eb"},{"sha":"2f0b882e5c8787b09bedcc8208e6dcc5662dbbab","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14482][SQL] Change default Parquet codec from gzip to snappy\n\n## What changes were proposed in this pull request?\nBased on our tests, gzip decompression is very slow (< 100MB/s), making queries decompression bound. Snappy can decompress at ~ 500MB/s on a single core.\n\nThis patch changes the default compression codec for Parquet output from gzip to snappy, and also introduces a ParquetOptions class to be more consistent with other data sources (e.g. CSV, JSON).\n\n## How was this patch tested?\nShould be covered by existing unit tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12256 from rxin/SPARK-14482.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/2f0b882e5c8787b09bedcc8208e6dcc5662dbbab"},{"sha":"520dde48d0d52dbbbbe1710a3275fdd5355dd69d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14451][SQL] Move encoder definition into Aggregator interface\n\n## What changes were proposed in this pull request?\nWhen we first introduced Aggregators, we required the user of Aggregators to (implicitly) specify the encoders. It would actually make more sense to have the encoders be specified by the implementation of Aggregators, since each implementation should have the most state about how to encode its own data type.\n\nNote that this simplifies the Java API because Java users no longer need to explicitly specify encoders for aggregators.\n\n## How was this patch tested?\nUpdated unit tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12231 from rxin/SPARK-14451.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/520dde48d0d52dbbbbe1710a3275fdd5355dd69d"},{"sha":"90c0a04506a4972b7a2ac2b7dda0c5f8509a6e2f","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-14419] [SQL] Improve HashedRelation for key fit within Long\n\n## What changes were proposed in this pull request?\n\nCurrently, we use java HashMap for HashedRelation if the key could fit within a Long. The java HashMap and CompactBuffer are not memory efficient, the memory used by them is also accounted accurately.\n\nThis PR introduce a LongToUnsafeRowMap (similar to BytesToBytesMap) for better memory efficiency and performance.\n\n## How was this patch tested?\n\nUpdated existing tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #12190 from davies/long_map2.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/90c0a04506a4972b7a2ac2b7dda0c5f8509a6e2f"},{"sha":"a9b8b655b25f4ed519037faaf7601a3d9842547f","author":{"email":"d82d494ad12feccfb00e29073898c0c0951e931f@hotmail.com","name":"wm624@hotmail.com"},"message":"[SPARK-14392][ML] CountVectorizer Estimator should include binary toggle Param\n\n## What changes were proposed in this pull request?\n\nCountVectorizerModel has a binary toggle param. This PR is to add binary toggle param for estimator CountVectorizer. As discussed in the JIRA, instead of adding a param into CountVerctorizer, I moved the binary param to CountVectorizerParams. Therefore, the estimator inherits the binary param.\n\n## How was this patch tested?\n\nAdd a new test case, which fits the model with binary flag set to true and then check the trained model's all non-zero counts is set to 1.0.\n\nAll tests in CounterVectorizerSuite.scala are passed.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #12200 from wangmiao1981/binary_param.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/a9b8b655b25f4ed519037faaf7601a3d9842547f"},{"sha":"10a95781ee8685d1473b156715b905c83af2aab0","author":{"email":"1a1ab5abfb65135a4f49eb66e79d9e570e28807d@us.ibm.com","name":"bomeng"},"message":"[SPARK-14496][SQL] fix some javadoc typos\n\n## What changes were proposed in this pull request?\n\nMinor issues. Found 2 typos while browsing the code.\n\n## How was this patch tested?\nNone.\n\nAuthor: bomeng <bmeng@us.ibm.com>\n\nCloses #12264 from bomeng/SPARK-14496.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/10a95781ee8685d1473b156715b905c83af2aab0"},{"sha":"1598d11bb0248384872cf88bc2b16f3b238046ad","author":{"email":"2a07432b1b7252977a4c2dfc8e6d6d5ac763a480@netflix.com","name":"DB Tsai"},"message":"[SPARK-14462][ML][MLLIB] add the mllib-local build to maven pom\n\n## What changes were proposed in this pull request?\n\nIn order to separate the linear algebra, and vector matrix classes into a standalone jar, we need to setup the build first. This PR will create a new jar called mllib-local with minimal dependencies. The test scope will still depend on spark-core and spark-core-test in order to use the common utilities, but the runtime will avoid any platform dependency. Couple platform independent classes will be moved to this package to demonstrate how this work.\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: DB Tsai <dbt@netflix.com>\n\nCloses #12241 from dbtsai/dbtsai-mllib-local-build.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/1598d11bb0248384872cf88bc2b16f3b238046ad"}]}