{"push_id":1410876011,"size":868,"distinct_size":2,"ref":"refs/heads/SPARK-12113","head":"10dca0e30711277343d1031bdb949b2ddab827c9","before":"7ccd981bf6cd3bbfa93652e033f0a7bdecbc8132","commits":[{"sha":"287bea13050b8eedc3b8b6b3491f1b5e5bc24d7a","author":{"name":"sethah","email":"c41626c9d885a60b8db6c40a26d851740b23beef@gmail.com"},"message":"[SPARK-7159][ML] Add multiclass logistic regression to Spark ML\n\n## What changes were proposed in this pull request?\n\nThis patch adds a new estimator/transformer `MultinomialLogisticRegression` to spark ML.\n\nJIRA: [SPARK-7159](https://issues.apache.org/jira/browse/SPARK-7159)\n\n## How was this patch tested?\n\nAdded new test suite `MultinomialLogisticRegressionSuite`.\n\n## Approach\n\n### Do not use a \"pivot\" class in the algorithm formulation\n\nMany implementations of multinomial logistic regression treat the problem as K - 1 independent binary logistic regression models where K is the number of possible outcomes in the output variable. In this case, one outcome is chosen as a \"pivot\" and the other K - 1 outcomes are regressed against the pivot. This is somewhat undesirable since the coefficients returned will be different for different choices of pivot variables. An alternative approach to the problem models class conditional probabilites using the softmax function and will return uniquely identifiable coefficients (assuming regularization is applied). This second approach is used in R's glmnet and was also recommended by dbtsai.\n\n### Separate multinomial logistic regression and binary logistic regression\n\nThe initial design makes multinomial logistic regression a separate estimator/transformer than the existing LogisticRegression estimator/transformer. An alternative design would be to merge them into one.\n\n**Arguments for:**\n\n* The multinomial case without pivot is distinctly different than the current binary case since the binary case uses a pivot class.\n* The current logistic regression model in ML uses a vector of coefficients and a scalar intercept. In the multinomial case, we require a matrix of coefficients and a vector of intercepts. There are potential workarounds for this issue if we were to merge the two estimators, but none are particularly elegant.\n\n**Arguments against:**\n\n* It may be inconvenient for users to have to switch the estimator class when transitioning between binary and multiclass (although the new multinomial estimator can be used for two class outcomes).\n* Some portions of the code are repeated.\n\nThis is a major design point and warrants more discussion.\n\n### Mean centering\n\nWhen no regularization is applied, the coefficients will not be uniquely identifiable. This is not hard to show and is discussed in further detail [here](https://core.ac.uk/download/files/153/6287975.pdf). R's glmnet deals with this by choosing the minimum l2 regularized solution (i.e. mean centering). Additionally, the intercepts are never regularized so they are always mean centered. This is the approach taken in this PR as well.\n\n### Feature scaling\n\nIn current ML logistic regression, the features are always standardized when running the optimization algorithm. They are always returned to the user in the original feature space, however. This same approach is maintained in this patch as well, but the implementation details are different. In ML logistic regression, the unregularized feature values are divided by the column standard deviation in every gradient update iteration. In contrast, MLlib transforms the entire input dataset to the scaled space _before_ optimizaton. In ML, this means that `numFeatures * numClasses` extra scalar divisions are required in every iteration. Performance testing shows that this has significant (4x in some cases) slow downs in each iteration. This can be avoided by transforming the input to the scaled space ala MLlib once, before iteration begins. This does add some overhead initially, but can make significant time savings in some cases.\n\nOne issue with this approach is that if the input data is already cached, there may not be enough memory to cache the transformed data, which would make the algorithm _much_ slower. The tradeoffs here merit more discussion.\n\n### Specifying and inferring the number of outcome classes\n\nThe estimator checks the dataframe label column for metadata which specifies the number of values. If they are not specified, the length of the `histogram` variable is used, which is essentially the maximum value found in the column. The assumption then, is that the labels are zero-indexed when they are provided to the algorithm.\n\n## Performance\n\nBelow are some performance tests I have run so far. I am happy to add more cases or trials if we deem them necessary.\n\nTest cluster: 4 bare metal nodes, 128 GB RAM each, 48 cores each\n\nNotes:\n\n* Time in units of seconds\n* Metric is classification accuracy\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 0 | true           | 0.746415 |        30 |      100000 |            3 |        100000 | 327.923 | true              |          0 |\n| mllib  |                 0 | true           | 0.743785 |        30 |      100000 |            3 |        100000 | 390.217 | true              |          0 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 0 | true           | 0.973238 |        30 |     2000000 |            3 |         10000 | 385.476 | true              |          0 |\n| mllib  |                 0 | true           | 0.949828 |        30 |     2000000 |            3 |         10000 | 550.403 | true              |          0 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| mllib  |                 0 | true           | 0.864358 |        30 |     2000000 |            3 |         10000 | 543.359 | true              |        0.1 |\n| ml     |                 0 | true           | 0.867418 |        30 |     2000000 |            3 |         10000 | 401.955 | true              |        0.1 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 1 | true           | 0.807449 |        30 |     2000000 |            3 |         10000 | 334.892 | true              |       0.05 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 0 | true           | 0.602006 |        30 |     2000000 |          500 |           100 | 112.319 | true              |          0 |\n| mllib  |                 0 | true           | 0.567226 |        30 |     2000000 |          500 |           100 | 263.768 | true              |          0 |e           | 0.567226 |        30 |     2000000 |          500 |           100 | 263.768 | true              |          0 |\n\n## References\n\nFriedman, et al. [\"Regularization Paths for Generalized Linear Models via Coordinate Descent\"](https://core.ac.uk/download/files/153/6287975.pdf)\n[http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html](http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html)\n\n## Follow up items\n* Consider using level 2 BLAS routines in the gradient computations - [SPARK-17134](https://issues.apache.org/jira/browse/SPARK-17134)\n* Add model summary for MLOR - [SPARK-17139](https://issues.apache.org/jira/browse/SPARK-17139)\n* Add initial model to MLOR and add test for intercept priors - [SPARK-17140](https://issues.apache.org/jira/browse/SPARK-17140)\n* Python API - [SPARK-17138](https://issues.apache.org/jira/browse/SPARK-17138)\n* Consider changing the tree aggregation level for MLOR/BLOR or making it user configurable to avoid memory problems with high dimensional data - [SPARK-17090](https://issues.apache.org/jira/browse/SPARK-17090)\n* Refactor helper classes out of `LogisticRegression.scala` - [SPARK-17135](https://issues.apache.org/jira/browse/SPARK-17135)\n* Design optimizer interface for added flexibility in ML algos - [SPARK-17136](https://issues.apache.org/jira/browse/SPARK-17136)\n* Support compressing the coefficients and intercepts for MLOR models - [SPARK-17137](https://issues.apache.org/jira/browse/SPARK-17137)\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #13796 from sethah/SPARK-7159_M.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/287bea13050b8eedc3b8b6b3491f1b5e5bc24d7a"},{"sha":"5377fc62360d5e9b5c94078e41d10a96e0e8a535","author":{"name":"Nick Lavers","email":"70e9874901837fcad34c49a60e101b46bfd7e11b@videoamp.com"},"message":"[SPARK-16961][CORE] Fixed off-by-one error that biased randomizeInPlace\n\nJIRA issue link:\nhttps://issues.apache.org/jira/browse/SPARK-16961\n\nChanged one line of Utils.randomizeInPlace to allow elements to stay in place.\n\nCreated a unit test that runs a Pearson's chi squared test to determine whether the output diverges significantly from a uniform distribution.\n\nAuthor: Nick Lavers <nick.lavers@videoamp.com>\n\nCloses #14551 from nicklavers/SPARK-16961-randomizeInPlace.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/5377fc62360d5e9b5c94078e41d10a96e0e8a535"},{"sha":"864be9359ae2f8409e6dbc38a7a18593f9cc5692","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-17141][ML] MinMaxScaler should remain NaN value.\n\n## What changes were proposed in this pull request?\nIn the existing code, ```MinMaxScaler``` handle ```NaN``` value indeterminately.\n* If a column has identity value, that is ```max == min```, ```MinMaxScalerModel``` transformation will output ```0.5``` for all rows even the original value is ```NaN```.\n* Otherwise, it will remain ```NaN``` after transformation.\n\nI think we should unify the behavior by remaining ```NaN``` value at any condition, since we don't know how to transform a ```NaN``` value. In Python sklearn, it will throw exception when there is ```NaN``` in the dataset.\n\n## How was this patch tested?\nUnit tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14716 from yanboliang/spark-17141.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/864be9359ae2f8409e6dbc38a7a18593f9cc5692"},{"sha":"072acf5e1460d66d4b60b536d5b2ccddeee80794","author":{"name":"Jeff Zhang","email":"8540ccd5710ce8e6da3f6c1c8189a4ef2fa98c60@apache.org"},"message":"[SPARK-16965][MLLIB][PYSPARK] Fix bound checking for SparseVector.\n\n## What changes were proposed in this pull request?\n\n1. In scala, add negative low bound checking and put all the low/upper bound checking in one place\n2. In python, add low/upper bound checking of indices.\n\n## How was this patch tested?\n\nunit test added\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #14555 from zjffdu/SPARK-16965.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/072acf5e1460d66d4b60b536d5b2ccddeee80794"},{"sha":"67e59d464f782ff5f509234212aa072a7653d7bf","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16994][SQL] Whitelist operators for predicate pushdown\n\n## What changes were proposed in this pull request?\nThis patch changes predicate pushdown optimization rule (PushDownPredicate) from using a blacklist to a whitelist. That is to say, operators must be explicitly allowed. This approach is more future-proof: previously it was possible for us to introduce a new operator and then render the optimization rule incorrect.\n\nThis also fixes the bug that previously we allowed pushing filter beneath limit, which was incorrect. That is to say, before this patch, the optimizer would rewrite\n```\nselect * from (select * from range(10) limit 5) where id > 3\n\nto\n\nselect * from range(10) where id > 3 limit 5\n```\n\n## How was this patch tested?\n- a unit test case in FilterPushdownSuite\n- an end-to-end test in limit.sql\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14713 from rxin/SPARK-16994.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/67e59d464f782ff5f509234212aa072a7653d7bf"},{"sha":"e98eb2146f1363956bfc3e5adcc11c246182d617","author":{"name":"Alex Bozarth","email":"850363088eed533ef66882756a87f03f1fc98fde@us.ibm.com"},"message":"[SPARK-16673][WEB UI] New Executor Page removed conditional for Logs and Thread Dump columns\n\n## What changes were proposed in this pull request?\n\nWhen #13670 switched `ExecutorsPage` to use JQuery DataTables it incidentally removed the conditional for the Logs and Thread Dump columns. I reimplemented the conditional display of the Logs and Thread dump columns as it was before the switch.\n\n## How was this patch tested?\n\nManually tested and dev/run-tests\n\n![both](https://cloud.githubusercontent.com/assets/13952758/17186879/da8dd1a8-53eb-11e6-8b0c-d0ff0156a9a7.png)\n![dump](https://cloud.githubusercontent.com/assets/13952758/17186881/dab08a04-53eb-11e6-8b1c-50ffd0bf2ae8.png)\n![logs](https://cloud.githubusercontent.com/assets/13952758/17186880/dab04d00-53eb-11e6-8754-68dd64d6d9f4.png)\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #14382 from ajbozarth/spark16673.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/e98eb2146f1363956bfc3e5adcc11c246182d617"},{"sha":"071eaaf9d2b63589f2e66e5279a16a5a484de6f5","author":{"name":"Kousuke Saruta","email":"32f3571f572bf6ab8cff58dd25027d319813b149@oss.nttdata.co.jp"},"message":"[SPARK-11227][CORE] UnknownHostException can be thrown when NameNode HA is enabled.\n\n## What changes were proposed in this pull request?\n\nIf the following conditions are satisfied, executors don't load properties in `hdfs-site.xml` and UnknownHostException can be thrown.\n\n(1) NameNode HA is enabled\n(2) spark.eventLogging is disabled or logging path is NOT on HDFS\n(3) Using Standalone or Mesos for the cluster manager\n(4) There are no code to load `HdfsCondition` class in the driver regardless of directly or indirectly.\n(5) The tasks access to HDFS\n\n(There might be some more conditions...)\n\nFor example, following code causes UnknownHostException when the conditions above are satisfied.\n```\nsc.textFile(\"<path on HDFS>\").collect\n\n```\n\n```\njava.lang.IllegalArgumentException: java.net.UnknownHostException: hacluster\n\tat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)\n\tat org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)\n\tat org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)\n\tat org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)\n\tat org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:656)\n\tat org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:438)\n\tat org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:411)\n\tat org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$32.apply(SparkContext.scala:986)\n\tat org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$32.apply(SparkContext.scala:986)\n\tat org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:177)\n\tat org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:177)\n\tat scala.Option.map(Option.scala:146)\n\tat org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:177)\n\tat org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:213)\n\tat org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:209)\n\tat org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:85)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.net.UnknownHostException: hacluster\n```\n\nBut following code doesn't cause the Exception because `textFile` method loads `HdfsConfiguration` indirectly.\n\n```\nsc.textFile(\"<path on HDFS>\").collect\n```\n\nWhen a job includes some operations which access to HDFS, the object of `org.apache.hadoop.Configuration` is wrapped by `SerializableConfiguration`,  serialized and broadcasted from driver to executors and each executor deserialize the object with `loadDefaults` false so HDFS related properties should be set before broadcasted.\n\n## How was this patch tested?\nTested manually on my standalone cluster.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #13738 from sarutak/SPARK-11227.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/071eaaf9d2b63589f2e66e5279a16a5a484de6f5"},{"sha":"cf0cce90364d17afe780ff9a5426dfcefa298535","author":{"name":"Sital Kedia","email":"e4343f82e25883a7c046475fe9553138fac5aa32@fb.com"},"message":"[SPARK-17113] [SHUFFLE] Job failure due to Executor OOM in offheap mode\n\n## What changes were proposed in this pull request?\n\nThis PR fixes executor OOM in offheap mode due to bug in Cooperative Memory Management for UnsafeExternSorter.  UnsafeExternalSorter was checking if memory page is being used by upstream by comparing the base object address of the current page with the base object address of upstream. However, in case of offheap memory allocation, the base object addresses are always null, so there was no spilling happening and eventually the operator would OOM.\n\nFollowing is the stack trace this issue addresses -\njava.lang.OutOfMemoryError: Unable to acquire 1220 bytes of memory, got 0\n\tat org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:120)\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPageIfNecessary(UnsafeExternalSorter.java:341)\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:362)\n\tat org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:93)\n\tat org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:170)\n\n## How was this patch tested?\n\nTested by running the failing job.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #14693 from sitalkedia/fix_offheap_oom.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/cf0cce90364d17afe780ff9a5426dfcefa298535"},{"sha":"acac7a508a29d0f75d86ee2e4ca83ebf01a36cf8","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARK-16443][SPARKR] Alternating Least Squares (ALS) wrapper\n\n## What changes were proposed in this pull request?\n\nAdd Alternating Least Squares wrapper in SparkR. Unit tests have been updated.\n\n## How was this patch tested?\n\nSparkR unit tests.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\n![screen shot 2016-07-27 at 3 50 31 pm](https://cloud.githubusercontent.com/assets/15318264/17195347/f7a6352a-5411-11e6-8e21-61a48070192a.png)\n![screen shot 2016-07-27 at 3 50 46 pm](https://cloud.githubusercontent.com/assets/15318264/17195348/f7a7d452-5411-11e6-845f-6d292283bc28.png)\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #14384 from junyangq/SPARK-16443.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/acac7a508a29d0f75d86ee2e4ca83ebf01a36cf8"},{"sha":"a117afa7c2d94f943106542ec53d74ba2b5f1058","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17149][SQL] array.sql for testing array related functions\n\n## What changes were proposed in this pull request?\nThis patch creates array.sql in SQLQueryTestSuite for testing array related functions, including:\n\n- indexing\n- array creation\n- size\n- array_contains\n- sort_array\n\n## How was this patch tested?\nThe patch itself is about adding tests.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14708 from petermaxlee/SPARK-17149.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/a117afa7c2d94f943106542ec53d74ba2b5f1058"},{"sha":"ba1737c21aab91ff3f1a1737aa2d6b07575e36a3","author":{"name":"Srinath Shankar","email":"5aa555d6908953c2e3dc43d416a76755eeaef637@databricks.com"},"message":"[SPARK-17158][SQL] Change error message for out of range numeric literals\n\n## What changes were proposed in this pull request?\n\nModifies error message for numeric literals to\nNumeric literal <literal> does not fit in range [min, max] for type <T>\n\n## How was this patch tested?\n\nFixed up the error messages for literals.sql in  SqlQueryTestSuite and re-ran via sbt. Also fixed up error messages in ExpressionParserSuite\n\nAuthor: Srinath Shankar <srinath@databricks.com>\n\nCloses #14721 from srinathshankar/sc4296.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/ba1737c21aab91ff3f1a1737aa2d6b07575e36a3"},{"sha":"45d40d9f66c666eec6df926db23937589d67225d","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17150][SQL] Support SQL generation for inline tables\n\n## What changes were proposed in this pull request?\nThis patch adds support for SQL generation for inline tables. With this, it would be possible to create a view that depends on inline tables.\n\n## How was this patch tested?\nAdded a test case in LogicalPlanToSQLSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14709 from petermaxlee/SPARK-17150.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/45d40d9f66c666eec6df926db23937589d67225d"},{"sha":"39f328ba3519b01940a7d1cdee851ba4e75ef31f","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-15018][PYSPARK][ML] Improve handling of PySpark Pipeline when used without stages\n\n## What changes were proposed in this pull request?\n\nWhen fitting a PySpark Pipeline without the `stages` param set, a confusing NoneType error is raised as attempts to iterate over the pipeline stages.  A pipeline with no stages should act as an identity transform, however the `stages` param still needs to be set to an empty list.  This change improves the error output when the `stages` param is not set and adds a better description of what the API expects as input.  Also minor cleanup of related code.\n\n## How was this patch tested?\nAdded new unit tests to verify an empty Pipeline acts as an identity transformer\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #12790 from BryanCutler/pipeline-identity-SPARK-15018.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/39f328ba3519b01940a7d1cdee851ba4e75ef31f"},{"sha":"01401e965b58f7e8ab615764a452d7d18f1d4bf0","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARK-16508][SPARKR] Fix CRAN undocumented/duplicated arguments warnings.\n\n## What changes were proposed in this pull request?\n\nThis PR tries to fix all the remaining \"undocumented/duplicated arguments\" warnings given by CRAN-check.\n\nOne left is doc for R `stats::glm` exported in SparkR. To mute that warning, we have to also provide document for all arguments of that non-SparkR function.\n\nSome previous conversation is in #14558.\n\n## How was this patch tested?\n\nR unit test and `check-cran.sh` script (with no-test).\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #14705 from junyangq/SPARK-16508-master.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/01401e965b58f7e8ab615764a452d7d18f1d4bf0"},{"sha":"3e5fdeb3fb084cc9d25ce2f3f8cbf07a0aa2c573","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARKR][EXAMPLE] change example APP name\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\n\nFor R SQL example, appname is \"MyApp\". While examples in scala, Java and python, the appName is \"x Spark SQL basic example\".\n\nI made the R example consistent with other examples.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\nManual test\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #14703 from wangmiao1981/example.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/3e5fdeb3fb084cc9d25ce2f3f8cbf07a0aa2c573"},{"sha":"31a015572024046f4deaa6cec66bb6fab110f31d","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-17104][SQL] LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation\n\n## What changes were proposed in this pull request?\n\nCurrently `LogicalRelation.newInstance()` simply creates another `LogicalRelation` object with the same parameters. However, the `newInstance()` method inherited from `MultiInstanceRelation` should return a copy of object with unique expression ids. Current `LogicalRelation.newInstance()` can cause failure when doing self-join.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #14682 from viirya/fix-localrelation.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/31a015572024046f4deaa6cec66bb6fab110f31d"},{"sha":"9560c8d29542a5dcaaa07b7af9ef5ddcdbb5d14d","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17124][SQL] RelationalGroupedDataset.agg should preserve order and allow multiple aggregates per column\n\n## What changes were proposed in this pull request?\nThis patch fixes a longstanding issue with one of the RelationalGroupedDataset.agg function. Even though the signature accepts vararg of pairs, the underlying implementation turns the seq into a map, and thus not order preserving nor allowing multiple aggregates per column.\n\nThis change also allows users to use this function to run multiple different aggregations for a single column, e.g.\n```\nagg(\"age\" -> \"max\", \"age\" -> \"count\")\n```\n\n## How was this patch tested?\nAdded a test case in DataFrameAggregateSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14697 from petermaxlee/SPARK-17124.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/9560c8d29542a5dcaaa07b7af9ef5ddcdbb5d14d"},{"sha":"9f37d4eac28dd179dd523fa7d645be97bb52af9c","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-12666][CORE] SparkSubmit packages fix for when 'default' conf doesn't exist in dependent module\n\n## What changes were proposed in this pull request?\n\nAdding a \"(runtime)\" to the dependency configuration will set a fallback configuration to be used if the requested one is not found.  E.g. with the setting \"default(runtime)\", Ivy will look for the conf \"default\" in the module ivy file and if not found will look for the conf \"runtime\".  This can help with the case when using \"sbt publishLocal\" which does not write a \"default\" conf in the published ivy.xml file.\n\n## How was this patch tested?\nused spark-submit with --packages option for a package published locally with no default conf, and a package resolved from Maven central.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #13428 from BryanCutler/fallback-package-conf-SPARK-12666.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/9f37d4eac28dd179dd523fa7d645be97bb52af9c"},{"sha":"61ef74f2272faa7ce8f2badc7e00039908e3551f","author":{"name":"hqzizania","email":"c652b1bc837a8bbc470c9c1d84e1b5848d2197ae@gmail.com"},"message":"[SPARK-17090][ML] Make tree aggregation level in linear/logistic regression configurable\n\n## What changes were proposed in this pull request?\n\nLinear/logistic regression use treeAggregate with default depth (always = 2) for collecting coefficient gradient updates to the driver. For high dimensional problems, this can cause OOM error on the driver. This patch makes it configurable to avoid this problem if users' input data has many features. It adds a HasTreeDepth API in `sharedParams.scala`, and extends it to both Linear regression and logistic regression in .ml\n\nAuthor: hqzizania <hqzizania@gmail.com>\n\nCloses #14717 from hqzizania/SPARK-17090.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/61ef74f2272faa7ce8f2badc7e00039908e3551f"},{"sha":"7f08a60b6e9acb89482fa0e268b192250d9ba6e4","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16961][FOLLOW-UP][SPARKR] More robust test case for spark.gaussianMixture.\n\n## What changes were proposed in this pull request?\n#14551 fixed off-by-one bug in ```randomizeInPlace``` and some test failure caused by this fix.\nBut for SparkR ```spark.gaussianMixture``` test case, the fix is inappropriate. It only changed the output result of native R which should be compared by SparkR, however, it did not change the R code in annotation which is used for reproducing the result in native R. It will confuse users who can not reproduce the same result in native R. This PR sends a more robust test case which can produce same result between SparkR and native R.\n\n## How was this patch tested?\nUnit test update.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14730 from yanboliang/spark-16961-followup.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/7f08a60b6e9acb89482fa0e268b192250d9ba6e4"}]}
{"actor":{"display_login":"maropu"}}