{"push_id":1279251447,"size":167,"distinct_size":167,"ref":"refs/heads/spark-16980-lazy_partition_fetching","head":"920a1809c6b1196c9036ffad949639c0c649cc63","before":"de59d5f02ad9a9ddc73b1e8597ca748c320bfbc9","commits":[{"sha":"928ca1c6d12b23d84f9b6205e22d2e756311f072","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17102][SQL] bypass UserDefinedGenerator for json format check\n\n## What changes were proposed in this pull request?\n\nWe use reflection to convert `TreeNode` to json string, and currently don't support arbitrary object. `UserDefinedGenerator` takes a function object, so we should skip json format test for it, or the tests can be flacky, e.g. `DataFrameSuite.simple explode`, this test always fail with scala 2.10(branch 1.6 builds with scala 2.10 by default), but pass with scala 2.11(master branch builds with scala 2.11 by default).\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14679 from cloud-fan/json.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/928ca1c6d12b23d84f9b6205e22d2e756311f072"},{"sha":"e3fec51fa1ed161789ab7aa32ed36efe357b5d31","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-16930][YARN] Fix a couple of races in cluster app initialization.\n\nThere are two narrow races that could cause the ApplicationMaster to miss\nwhen the user application instantiates the SparkContext, which could cause\napp failures when nothing was wrong with the app. It was also possible for\na failing application to get stuck in the loop that waits for the context\nfor a long time, instead of failing quickly.\n\nThe change uses a promise to track the SparkContext instance, which gets\nrid of the races and allows for some simplification of the code.\n\nTested with existing unit tests, and a new one being added to test the\ntimeout code.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14542 from vanzin/SPARK-16930.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/e3fec51fa1ed161789ab7aa32ed36efe357b5d31"},{"sha":"4d92af310ad29ade039e4130f91f2a3d9180deef","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16446][SPARKR][ML] Gaussian Mixture Model wrapper in SparkR\n\n## What changes were proposed in this pull request?\nGaussian Mixture Model wrapper in SparkR, similarly to R's ```mvnormalmixEM```.\n\n## How was this patch tested?\nUnit test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14392 from yanboliang/spark-16446.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/4d92af310ad29ade039e4130f91f2a3d9180deef"},{"sha":"cc97ea188e1d5b8e851d1a8438b8af092783ec04","author":{"name":"Steve Loughran","email":"504d4a581a50fc7a86a0698985fb52745234a417@apache.org"},"message":"[SPARK-16736][CORE][SQL] purge superfluous fs calls\n\nA review of the code, working back from Hadoop's `FileSystem.exists()` and `FileSystem.isDirectory()` code, then removing uses of the calls when superfluous.\n\n1. delete is harmless if called on a nonexistent path, so don't do any checks before deletes\n1. any `FileSystem.exists()`  check before `getFileStatus()` or `open()` is superfluous as the operation itself does the check. Instead the `FileNotFoundException` is caught and triggers the downgraded path. When a `FileNotFoundException` was thrown before, the code still creates a new FNFE with the error messages. Though now the inner exceptions are nested, for easier diagnostics.\n\nInitially, relying on Jenkins test runs.\n\nOne troublespot here is that some of the codepaths are clearly error situations; it's not clear that they have coverage anyway. Trying to create the failure conditions in tests would be ideal, but it will also be hard.\n\nAuthor: Steve Loughran <stevel@apache.org>\n\nCloses #14371 from steveloughran/cloud/SPARK-16736-superfluous-fs-calls.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/cc97ea188e1d5b8e851d1a8438b8af092783ec04"},{"sha":"d60af8f6aa53373de1333cc642cf2a9d7b39d912","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-17096][SQL][STREAMING] Improve exception string reported through the StreamingQueryListener\n\n## What changes were proposed in this pull request?\n\nCurrently, the stackTrace (as `Array[StackTraceElements]`) reported through StreamingQueryListener.onQueryTerminated is useless as it has the stack trace of where StreamingQueryException is defined, not the stack trace of underlying exception.  For example, if a streaming query fails because of a / by zero exception in a task, the `QueryTerminated.stackTrace` will have\n```\norg.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:211)\norg.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:124)\n```\nThis is basically useless, as it is location where the StreamingQueryException was defined. What we want is\n\nHere is the right way to reason about what should be posted as through StreamingQueryListener.onQueryTerminated\n- The actual exception could either be a SparkException, or an arbitrary exception.\n  - SparkException reports the relevant executor stack trace of a failed task as a string in the the exception message. The `Array[StackTraceElements]` returned by `SparkException.stackTrace()` is mostly irrelevant.\n  - For any arbitrary exception, the `Array[StackTraceElements]` returned by `exception.stackTrace()` may be relevant.\n- When there is an error in a streaming query, it's hard to reason whether the `Array[StackTraceElements]` is useful or not. In fact, it is not clear whether it is even useful to report the stack trace as this array of Java objects. It may be sufficient to report the strack trace as a string, along with the message. This is how Spark reported executor stra\n- Hence, this PR simplifies the API by removing the array `stackTrace` from `QueryTerminated`. Instead the `exception` returns a string containing the message and the stack trace of the actual underlying exception that failed the streaming query (i.e. not that of the StreamingQueryException). If anyone is interested in the actual stack trace as an array, can always access them through `streamingQuery.exception` which returns the exception object.\n\nWith this change, if a streaming query fails because of a / by zero exception in a task, the `QueryTerminated.exception` will be\n```\norg.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost): java.lang.ArithmeticException: / by zero\n\tat org.apache.spark.sql.streaming.StreamingQueryListenerSuite$$anonfun$5$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply$mcII$sp(StreamingQueryListenerSuite.scala:153)\n\tat org.apache.spark.sql.streaming.StreamingQueryListenerSuite$$anonfun$5$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply(StreamingQueryListenerSuite.scala:153)\n\tat org.apache.spark.sql.streaming.StreamingQueryListenerSuite$$anonfun$5$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply(StreamingQueryListenerSuite.scala:153)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:232)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:226)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:283)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:86)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:744)\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1429)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1417)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1416)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1416)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)\n...\n```\nIt contains the relevant executor stack trace. In a case non-SparkException, if the streaming source MemoryStream throws an exception, exception message will have the relevant stack trace.\n```\njava.lang.RuntimeException: this is the exception message\n\tat org.apache.spark.sql.execution.streaming.MemoryStream.getBatch(memory.scala:103)\n\tat org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$5.apply(StreamExecution.scala:316)\n\tat org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$5.apply(StreamExecution.scala:313)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:893)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1336)\n\tat scala.collection.IterableLike$class.foreach(IterableLike.scala:72)\n\tat org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:25)\n\tat scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)\n\tat org.apache.spark.sql.execution.streaming.StreamProgress.flatMap(StreamProgress.scala:25)\n\tat org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatch(StreamExecution.scala:313)\n\tat org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches$1.apply$mcZ$sp(StreamExecution.scala:197)\n\tat org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:43)\n\tat org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:187)\n\tat org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:124)\n```\n\nNote that this change in the public `QueryTerminated` class is okay as the APIs are still experimental.\n\n## How was this patch tested?\nUnit tests that test whether the right information is present in the exception message reported through QueryTerminated object.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14675 from tdas/SPARK-17096.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/d60af8f6aa53373de1333cc642cf2a9d7b39d912"},{"sha":"e6bef7d52f0e19ec771fb0f3e96c7ddbd1a6a19b","author":{"name":"Xin Ren","email":"82e6f9780faa35cac922f524c8a765f69b3ff7ff@126.com"},"message":"[SPARK-17038][STREAMING] fix metrics retrieval source of 'lastReceivedBatch'\n\nhttps://issues.apache.org/jira/browse/SPARK-17038\n\n## What changes were proposed in this pull request?\n\nStreamingSource's lastReceivedBatch_submissionTime, lastReceivedBatch_processingTimeStart, and lastReceivedBatch_processingTimeEnd all use data from lastCompletedBatch instead of lastReceivedBatch.\n\nIn particular, this makes it impossible to match lastReceivedBatch_records with a batchID/submission time.\n\nThis is apparent when looking at StreamingSource.scala, lines 89-94.\n\n## How was this patch tested?\n\nManually running unit tests on local laptop\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #14681 from keypointt/SPARK-17038.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/e6bef7d52f0e19ec771fb0f3e96c7ddbd1a6a19b"},{"sha":"10204b9d29cd69895f5a606e75510dc64cf2e009","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-16995][SQL] TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr\n\n## What changes were proposed in this pull request?\n\nA TreeNodeException is thrown when executing the following minimal example in Spark 2.0.\n\n    import spark.implicits._\n    case class test (x: Int, q: Int)\n\n    val d = Seq(1).toDF(\"x\")\n    d.withColumn(\"q\", lit(0)).as[test].groupByKey(_.x).flatMapGroups{case (x, iter) => List[Int]()}.show\n    d.withColumn(\"q\", expr(\"0\")).as[test].groupByKey(_.x).flatMapGroups{case (x, iter) => List[Int]()}.show\n\nThe problem is at `FoldablePropagation`. The rule will do `transformExpressions` on `LogicalPlan`. The query above contains a `MapGroups` which has a parameter `dataAttributes:Seq[Attribute]`. One attributes in `dataAttributes` will be transformed to an `Alias(literal(0), _)` in `FoldablePropagation`. `Alias` is not an `Attribute` and causes the error.\n\nWe can't easily detect such type inconsistency during transforming expressions. A direct approach to this problem is to skip doing `FoldablePropagation` on object operators as they should not contain such expressions.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #14648 from viirya/flat-mapping.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/10204b9d29cd69895f5a606e75510dc64cf2e009"},{"sha":"3e6ef2e8a435a91b6a76876e9833917e5aa0945e","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17034][SQL] Minor code cleanup for UnresolvedOrdinal\n\n## What changes were proposed in this pull request?\nI was looking at the code for UnresolvedOrdinal and made a few small changes to make it slightly more clear:\n\n1. Rename the rule to SubstituteUnresolvedOrdinals which is more consistent with other rules that start with verbs. Note that this is still inconsistent with CTESubstitution and WindowsSubstitution.\n2. Broke the test suite down from a single test case to three test cases.\n\n## How was this patch tested?\nThis is a minor cleanup.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14672 from petermaxlee/SPARK-17034.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/3e6ef2e8a435a91b6a76876e9833917e5aa0945e"},{"sha":"1748f824101870b845dbbd118763c6885744f98a","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16391][SQL] Support partial aggregation for reduceGroups\n\n## What changes were proposed in this pull request?\nThis patch introduces a new private ReduceAggregator interface that is a subclass of Aggregator. ReduceAggregator only requires a single associative and commutative reduce function. ReduceAggregator is also used to implement KeyValueGroupedDataset.reduceGroups in order to support partial aggregation.\n\nNote that the pull request was initially done by viirya.\n\n## How was this patch tested?\nCovered by original tests for reduceGroups, as well as a new test suite for ReduceAggregator.\n\nAuthor: Reynold Xin <rxin@databricks.com>\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #14576 from rxin/reduceAggregator.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/1748f824101870b845dbbd118763c6885744f98a"},{"sha":"e82dbe600e0d36d76cd5607a77c3243a26777b77","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-17107][SQL] Remove redundant pushdown rule for Union\n\n## What changes were proposed in this pull request?\n\nThe `Optimizer` rules `PushThroughSetOperations` and `PushDownPredicate` have a redundant rule to push down `Filter` through `Union`. We should remove it.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #14687 from viirya/remove-extra-pushdown.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/e82dbe600e0d36d76cd5607a77c3243a26777b77"},{"sha":"b81421afb04959bb22b53653be0a09c1f1c5845f","author":{"name":"Stavros Kontopoulos","email":"42bafac589feb2200b51ee576d084d6a9af943fa@lightbend.com"},"message":"[SPARK-17087][MESOS] Documentation for Making Spark on Mesos honor port restrictions\n\n## What changes were proposed in this pull request?\n\n- adds documentation for https://issues.apache.org/jira/browse/SPARK-11714\n\n## How was this patch tested?\nDoc no test needed.\n\nAuthor: Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>\n\nCloses #14667 from skonto/add_doc.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/b81421afb04959bb22b53653be0a09c1f1c5845f"},{"sha":"412dba63b511474a6db3c43c8618d803e604bc6b","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-17069] Expose spark.range() as table-valued function in SQL\n\n## What changes were proposed in this pull request?\n\nThis adds analyzer rules for resolving table-valued functions, and adds one builtin implementation for range(). The arguments for range() are the same as those of `spark.range()`.\n\n## How was this patch tested?\n\nUnit tests.\n\ncc hvanhovell\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #14656 from ericl/sc-4309.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/412dba63b511474a6db3c43c8618d803e604bc6b"},{"sha":"68f5087d2107d6afec5d5745f0cb0e9e3bdd6a0b","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17117][SQL] 1 / NULL should not fail analysis\n\n## What changes were proposed in this pull request?\nThis patch fixes the problem described in SPARK-17117, i.e. \"SELECT 1 / NULL\" throws an analysis exception:\n\n```\norg.apache.spark.sql.AnalysisException: cannot resolve '(1 / NULL)' due to data type mismatch: differing types in '(1 / NULL)' (int and null).\n```\n\nThe problem is that division type coercion did not take null type into account.\n\n## How was this patch tested?\nA unit test for the type coercion, and a few end-to-end test cases using SQLQueryTestSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14695 from petermaxlee/SPARK-17117.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/68f5087d2107d6afec5d5745f0cb0e9e3bdd6a0b"},{"sha":"b72bb62d421840f82d663c6b8e3922bd14383fbb","author":{"name":"Xusen Yin","email":"b3af76889e6bb255e66b77722a2ef0a4409760be@gmail.com"},"message":"[SPARK-16447][ML][SPARKR] LDA wrapper in SparkR\n\n## What changes were proposed in this pull request?\n\nAdd LDA Wrapper in SparkR with the following interfaces:\n\n- spark.lda(data, ...)\n\n- spark.posterior(object, newData, ...)\n\n- spark.perplexity(object, ...)\n\n- summary(object)\n\n- write.ml(object)\n\n- read.ml(path)\n\n## How was this patch tested?\n\nTest with SparkR unit test.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #14229 from yinxusen/SPARK-16447.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/b72bb62d421840f82d663c6b8e3922bd14383fbb"},{"sha":"f5472dda51b980a726346587257c22873ff708e3","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16947][SQL] Support type coercion and foldable expression for inline tables\n\n## What changes were proposed in this pull request?\nThis patch improves inline table support with the following:\n\n1. Support type coercion.\n2. Support using foldable expressions. Previously only literals were supported.\n3. Improve error message handling.\n4. Improve test coverage.\n\n## How was this patch tested?\nAdded a new unit test suite ResolveInlineTablesSuite and a new file-based end-to-end test inline-table.sql.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14676 from petermaxlee/SPARK-16947.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/f5472dda51b980a726346587257c22873ff708e3"},{"sha":"b482c09fa22c5762a355f95820e4ba3e2517fb77","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"HOTFIX: compilation broken due to protected ctor.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/b482c09fa22c5762a355f95820e4ba3e2517fb77"},{"sha":"287bea13050b8eedc3b8b6b3491f1b5e5bc24d7a","author":{"name":"sethah","email":"c41626c9d885a60b8db6c40a26d851740b23beef@gmail.com"},"message":"[SPARK-7159][ML] Add multiclass logistic regression to Spark ML\n\n## What changes were proposed in this pull request?\n\nThis patch adds a new estimator/transformer `MultinomialLogisticRegression` to spark ML.\n\nJIRA: [SPARK-7159](https://issues.apache.org/jira/browse/SPARK-7159)\n\n## How was this patch tested?\n\nAdded new test suite `MultinomialLogisticRegressionSuite`.\n\n## Approach\n\n### Do not use a \"pivot\" class in the algorithm formulation\n\nMany implementations of multinomial logistic regression treat the problem as K - 1 independent binary logistic regression models where K is the number of possible outcomes in the output variable. In this case, one outcome is chosen as a \"pivot\" and the other K - 1 outcomes are regressed against the pivot. This is somewhat undesirable since the coefficients returned will be different for different choices of pivot variables. An alternative approach to the problem models class conditional probabilites using the softmax function and will return uniquely identifiable coefficients (assuming regularization is applied). This second approach is used in R's glmnet and was also recommended by dbtsai.\n\n### Separate multinomial logistic regression and binary logistic regression\n\nThe initial design makes multinomial logistic regression a separate estimator/transformer than the existing LogisticRegression estimator/transformer. An alternative design would be to merge them into one.\n\n**Arguments for:**\n\n* The multinomial case without pivot is distinctly different than the current binary case since the binary case uses a pivot class.\n* The current logistic regression model in ML uses a vector of coefficients and a scalar intercept. In the multinomial case, we require a matrix of coefficients and a vector of intercepts. There are potential workarounds for this issue if we were to merge the two estimators, but none are particularly elegant.\n\n**Arguments against:**\n\n* It may be inconvenient for users to have to switch the estimator class when transitioning between binary and multiclass (although the new multinomial estimator can be used for two class outcomes).\n* Some portions of the code are repeated.\n\nThis is a major design point and warrants more discussion.\n\n### Mean centering\n\nWhen no regularization is applied, the coefficients will not be uniquely identifiable. This is not hard to show and is discussed in further detail [here](https://core.ac.uk/download/files/153/6287975.pdf). R's glmnet deals with this by choosing the minimum l2 regularized solution (i.e. mean centering). Additionally, the intercepts are never regularized so they are always mean centered. This is the approach taken in this PR as well.\n\n### Feature scaling\n\nIn current ML logistic regression, the features are always standardized when running the optimization algorithm. They are always returned to the user in the original feature space, however. This same approach is maintained in this patch as well, but the implementation details are different. In ML logistic regression, the unregularized feature values are divided by the column standard deviation in every gradient update iteration. In contrast, MLlib transforms the entire input dataset to the scaled space _before_ optimizaton. In ML, this means that `numFeatures * numClasses` extra scalar divisions are required in every iteration. Performance testing shows that this has significant (4x in some cases) slow downs in each iteration. This can be avoided by transforming the input to the scaled space ala MLlib once, before iteration begins. This does add some overhead initially, but can make significant time savings in some cases.\n\nOne issue with this approach is that if the input data is already cached, there may not be enough memory to cache the transformed data, which would make the algorithm _much_ slower. The tradeoffs here merit more discussion.\n\n### Specifying and inferring the number of outcome classes\n\nThe estimator checks the dataframe label column for metadata which specifies the number of values. If they are not specified, the length of the `histogram` variable is used, which is essentially the maximum value found in the column. The assumption then, is that the labels are zero-indexed when they are provided to the algorithm.\n\n## Performance\n\nBelow are some performance tests I have run so far. I am happy to add more cases or trials if we deem them necessary.\n\nTest cluster: 4 bare metal nodes, 128 GB RAM each, 48 cores each\n\nNotes:\n\n* Time in units of seconds\n* Metric is classification accuracy\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 0 | true           | 0.746415 |        30 |      100000 |            3 |        100000 | 327.923 | true              |          0 |\n| mllib  |                 0 | true           | 0.743785 |        30 |      100000 |            3 |        100000 | 390.217 | true              |          0 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 0 | true           | 0.973238 |        30 |     2000000 |            3 |         10000 | 385.476 | true              |          0 |\n| mllib  |                 0 | true           | 0.949828 |        30 |     2000000 |            3 |         10000 | 550.403 | true              |          0 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| mllib  |                 0 | true           | 0.864358 |        30 |     2000000 |            3 |         10000 | 543.359 | true              |        0.1 |\n| ml     |                 0 | true           | 0.867418 |        30 |     2000000 |            3 |         10000 | 401.955 | true              |        0.1 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 1 | true           | 0.807449 |        30 |     2000000 |            3 |         10000 | 334.892 | true              |       0.05 |\n\n| algo   |   elasticNetParam | fitIntercept   |   metric |   maxIter |   numPoints |   numClasses |   numFeatures |    time | standardization   |   regParam |\n|--------|-------------------|----------------|----------|-----------|-------------|--------------|---------------|---------|-------------------|------------|\n| ml     |                 0 | true           | 0.602006 |        30 |     2000000 |          500 |           100 | 112.319 | true              |          0 |\n| mllib  |                 0 | true           | 0.567226 |        30 |     2000000 |          500 |           100 | 263.768 | true              |          0 |e           | 0.567226 |        30 |     2000000 |          500 |           100 | 263.768 | true              |          0 |\n\n## References\n\nFriedman, et al. [\"Regularization Paths for Generalized Linear Models via Coordinate Descent\"](https://core.ac.uk/download/files/153/6287975.pdf)\n[http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html](http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html)\n\n## Follow up items\n* Consider using level 2 BLAS routines in the gradient computations - [SPARK-17134](https://issues.apache.org/jira/browse/SPARK-17134)\n* Add model summary for MLOR - [SPARK-17139](https://issues.apache.org/jira/browse/SPARK-17139)\n* Add initial model to MLOR and add test for intercept priors - [SPARK-17140](https://issues.apache.org/jira/browse/SPARK-17140)\n* Python API - [SPARK-17138](https://issues.apache.org/jira/browse/SPARK-17138)\n* Consider changing the tree aggregation level for MLOR/BLOR or making it user configurable to avoid memory problems with high dimensional data - [SPARK-17090](https://issues.apache.org/jira/browse/SPARK-17090)\n* Refactor helper classes out of `LogisticRegression.scala` - [SPARK-17135](https://issues.apache.org/jira/browse/SPARK-17135)\n* Design optimizer interface for added flexibility in ML algos - [SPARK-17136](https://issues.apache.org/jira/browse/SPARK-17136)\n* Support compressing the coefficients and intercepts for MLOR models - [SPARK-17137](https://issues.apache.org/jira/browse/SPARK-17137)\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #13796 from sethah/SPARK-7159_M.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/287bea13050b8eedc3b8b6b3491f1b5e5bc24d7a"},{"sha":"5377fc62360d5e9b5c94078e41d10a96e0e8a535","author":{"name":"Nick Lavers","email":"70e9874901837fcad34c49a60e101b46bfd7e11b@videoamp.com"},"message":"[SPARK-16961][CORE] Fixed off-by-one error that biased randomizeInPlace\n\nJIRA issue link:\nhttps://issues.apache.org/jira/browse/SPARK-16961\n\nChanged one line of Utils.randomizeInPlace to allow elements to stay in place.\n\nCreated a unit test that runs a Pearson's chi squared test to determine whether the output diverges significantly from a uniform distribution.\n\nAuthor: Nick Lavers <nick.lavers@videoamp.com>\n\nCloses #14551 from nicklavers/SPARK-16961-randomizeInPlace.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/5377fc62360d5e9b5c94078e41d10a96e0e8a535"},{"sha":"864be9359ae2f8409e6dbc38a7a18593f9cc5692","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-17141][ML] MinMaxScaler should remain NaN value.\n\n## What changes were proposed in this pull request?\nIn the existing code, ```MinMaxScaler``` handle ```NaN``` value indeterminately.\n* If a column has identity value, that is ```max == min```, ```MinMaxScalerModel``` transformation will output ```0.5``` for all rows even the original value is ```NaN```.\n* Otherwise, it will remain ```NaN``` after transformation.\n\nI think we should unify the behavior by remaining ```NaN``` value at any condition, since we don't know how to transform a ```NaN``` value. In Python sklearn, it will throw exception when there is ```NaN``` in the dataset.\n\n## How was this patch tested?\nUnit tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14716 from yanboliang/spark-17141.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/864be9359ae2f8409e6dbc38a7a18593f9cc5692"},{"sha":"072acf5e1460d66d4b60b536d5b2ccddeee80794","author":{"name":"Jeff Zhang","email":"8540ccd5710ce8e6da3f6c1c8189a4ef2fa98c60@apache.org"},"message":"[SPARK-16965][MLLIB][PYSPARK] Fix bound checking for SparseVector.\n\n## What changes were proposed in this pull request?\n\n1. In scala, add negative low bound checking and put all the low/upper bound checking in one place\n2. In python, add low/upper bound checking of indices.\n\n## How was this patch tested?\n\nunit test added\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #14555 from zjffdu/SPARK-16965.","distinct":true,"url":"https://api.github.com/repos/VideoAmp/spark-public/commits/072acf5e1460d66d4b60b536d5b2ccddeee80794"}]}
{"actor":{"display_login":"mallman"}}