{"push_id":1078824033,"size":18,"distinct_size":0,"ref":"refs/heads/master","head":"284b15d2fbff7c0c3ffe8737838071d366ea5742","before":"8e1bb0456db1ad60afa24aa033b574c4a79b9c09","commits":[{"sha":"a2e8d4fddd1446df946b3c05223e8b8ac6312c3c","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13643][SQL] Implement SparkSession\n\n## What changes were proposed in this pull request?\n\nAfter removing most of `HiveContext` in 8fc267ab3322e46db81e725a5cb1adb5a71b2b4d we can now move existing functionality in `SQLContext` to `SparkSession`. As of this PR `SQLContext` becomes a simple wrapper that has a `SparkSession` and delegates all functionality to it.\n\n## How was this patch tested?\n\nJenkins.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #12553 from andrewor14/implement-spark-session.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/a2e8d4fddd1446df946b3c05223e8b8ac6312c3c"},{"sha":"ef6be7bedd9918c3398d510bb1eb1584908581c8","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[MINOR] Comment whitespace changes in #12553","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/ef6be7bedd9918c3398d510bb1eb1584908581c8"},{"sha":"1a95397bb6a4e7e7a06ac450bf556fa3aa47b8cd","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14798][SQL] Move native command and script transformation parsing into SparkSqlAstBuilder\n\n## What changes were proposed in this pull request?\nThis patch moves native command and script transformation into SparkSqlAstBuilder. This builds on #12561. See the last commit for diff.\n\n## How was this patch tested?\nUpdated test cases to reflect this.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12564 from rxin/SPARK-14798.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/1a95397bb6a4e7e7a06ac450bf556fa3aa47b8cd"},{"sha":"f82aa82480d95451510ee7a74c52e83e98c8b794","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-14774][SQL] Write unscaled values in ColumnVector.putDecimal\n\n## What changes were proposed in this pull request?\n\nWe recently made `ColumnarBatch.row` mutable and added a new `ColumnVector.putDecimal` method to support putting `Decimal` values in the `ColumnarBatch`. This unfortunately introduced a bug wherein we were not updating the vector with the proper unscaled values.\n\n## How was this patch tested?\n\nThis codepath is hit only when the vectorized aggregate hashmap is enabled. https://github.com/apache/spark/pull/12440 makes sure that a number of regression tests/benchmarks test this bugfix.\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #12541 from sameeragarwal/fix-bigdecimal.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/f82aa82480d95451510ee7a74c52e83e98c8b794"},{"sha":"411454475a031869eb7dc0c5fd84f41b3fdfa295","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14780] [R] Add `setLogLevel` to SparkR\n\n## What changes were proposed in this pull request?\n\nThis PR aims to add `setLogLevel` function to SparkR shell.\n\n**Spark Shell**\n```scala\nscala> sc.setLogLevel(\"ERROR\")\n```\n\n**PySpark**\n```python\n>>> sc.setLogLevel(\"ERROR\")\n```\n\n**SparkR (this PR)**\n```r\n> setLogLevel(sc, \"ERROR\")\nNULL\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests including a new R testcase.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12547 from dongjoon-hyun/SPARK-14780.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/411454475a031869eb7dc0c5fd84f41b3fdfa295"},{"sha":"6d1e4c4a65541cbf78284005de1776dc49efa9f4","author":{"email":"623dda1cf42194a966de8b161dee064823eb61d3@126.com","name":"Xin Ren"},"message":"[SPARK-14569][ML] Log instrumentation in KMeans\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-14569\n\nLog instrumentation in KMeans:\n\n- featuresCol\n- predictionCol\n- k\n- initMode\n- initSteps\n- maxIter\n- seed\n- tol\n- summary\n\n## How was this patch tested?\n\nManually test on local machine, by running and checking output of org.apache.spark.examples.ml.KMeansExample\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #12432 from keypointt/SPARK-14569.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/6d1e4c4a65541cbf78284005de1776dc49efa9f4"},{"sha":"e2b5647ab92eb478b3f7b36a0ce6faf83e24c0e5","author":{"email":"cc463b26613f4ba480f0f94ca5cb2cab678c4fcb@databricks.com","name":"Eric Liang"},"message":"[SPARK-14724] Use radix sort for shuffles and sort operator when possible\n\n## What changes were proposed in this pull request?\n\nSpark currently uses TimSort for all in-memory sorts, including sorts done for shuffle. One low-hanging fruit is to use radix sort when possible (e.g. sorting by integer keys). This PR adds a radix sort implementation to the unsafe sort package and switches shuffles and sorts to use it when possible.\n\nThe current implementation does not have special support for null values, so we cannot radix-sort `LongType`. I will address this in a follow-up PR.\n\n## How was this patch tested?\n\nUnit tests, enabling radix sort on existing tests. Microbenchmark results:\n\n```\nRunning benchmark: radix sort 25000000\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 on Linux 3.13.0-44-generic\nIntel(R) Core(TM) i7-4600U CPU  2.10GHz\n\nradix sort 25000000:                Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n-------------------------------------------------------------------------------------------\nreference TimSort key prefix array     15546 / 15859          1.6         621.9       1.0X\nreference Arrays.sort                    2416 / 2446         10.3          96.6       6.4X\nradix sort one byte                       133 /  137        188.4           5.3     117.2X\nradix sort two bytes                      255 /  258         98.2          10.2      61.1X\nradix sort eight bytes                    991 /  997         25.2          39.6      15.7X\nradix sort key prefix array              1540 / 1563         16.2          61.6      10.1X\n```\n\nI also ran a mix of the supported TPCDS queries and compared TimSort vs RadixSort metrics. The overall benchmark ran ~10% faster with radix sort on. In the breakdown below, the radix-enabled sort phases averaged about 20x faster than TimSort, however sorting is only a small fraction of the overall runtime. About half of the TPCDS queries were able to take advantage of radix sort.\n\n```\nTPCDS on master: 2499s real time, 8185s executor\n    - 1171s in TimSort, avg 267 MB/s\n(note the /s accounting is weird here since dataSize counts the record sizes too)\n\nTPCDS with radix enabled: 2294s real time, 7391s executor\n    - 596s in TimSort, avg 254 MB/s\n    - 26s in radix sort, avg 4.2 GB/s\n```\n\ncc davies rxin\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #12490 from ericl/sort-benchmark.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/e2b5647ab92eb478b3f7b36a0ce6faf83e24c0e5"},{"sha":"f25a3ea8d3ee6972efb925826981918549deacaa","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-14734][ML][MLLIB] Added asML, fromML methods for all spark.mllib Vector, Matrix types\n\n## What changes were proposed in this pull request?\n\nFor maintaining wrappers around spark.mllib algorithms in spark.ml, it will be useful to have ```private[spark]``` methods for converting from one linear algebra representation to another.\nThis PR adds toNew, fromNew methods for all spark.mllib Vector and Matrix types.\n\n## How was this patch tested?\n\nUnit tests for all conversions\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #12504 from jkbradley/linalg-conversions.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/f25a3ea8d3ee6972efb925826981918549deacaa"},{"sha":"4e726227a3e68c776ea30b78b7db8d01d00b44d6","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-14479][ML] GLM supports output link prediction\n\n## What changes were proposed in this pull request?\nGLM supports output link prediction.\n## How was this patch tested?\nunit test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #12287 from yanboliang/spark-14479.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/4e726227a3e68c776ea30b78b7db8d01d00b44d6"},{"sha":"f181aee07c0ee105b2a34581105eeeada7d42363","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14821][SQL] Implement AnalyzeTable in sql/core and remove HiveSqlAstBuilder\n\n## What changes were proposed in this pull request?\nThis patch moves analyze table parsing into SparkSqlAstBuilder and removes HiveSqlAstBuilder.\n\nIn order to avoid extensive refactoring, I created a common trait for CatalogRelation and MetastoreRelation, and match on that. In the future we should probably just consolidate the two into a single thing so we don't need this common trait.\n\n## How was this patch tested?\nUpdated unit tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12584 from rxin/SPARK-14821.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/f181aee07c0ee105b2a34581105eeeada7d42363"},{"sha":"0bf8df250e0aeae306e2ef33e612ca27187447ed","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[HOTFIX] Fix Java 7 compilation break","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/0bf8df250e0aeae306e2ef33e612ca27187447ed"},{"sha":"df1953f0df8b43136157a18bea05fd6750906f68","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-14824][SQL] Rename HiveContext object to HiveUtils\n\n## What changes were proposed in this pull request?\n\nJust a rename so we can get rid of `HiveContext.scala`. Note that this will conflict with #12585.\n\n## How was this patch tested?\n\nNo change in functionality.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #12586 from andrewor14/rename-hc-object.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/df1953f0df8b43136157a18bea05fd6750906f68"},{"sha":"f1fdb23821b89623b592bfb3ef73d61afbe93b0a","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-14793] [SQL] Code generation for large complex type exceeds JVM size limit.\n\n## What changes were proposed in this pull request?\n\nCode generation for complex type, `CreateArray`, `CreateMap`, `CreateStruct`, `CreateNamedStruct`, exceeds JVM size limit for large elements.\n\nWe should split generated code into multiple `apply` functions if the complex types have large elements,  like `UnsafeProjection` or others for large expressions.\n\n## How was this patch tested?\n\nI added some tests to check if the generated codes for the expressions exceed or not.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #12559 from ueshin/issues/SPARK-14793.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/f1fdb23821b89623b592bfb3ef73d61afbe93b0a"},{"sha":"b29bc3f51518806ef7827b35df7c8aada329f961","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-14680] [SQL] Support all datatypes to use VectorizedHashmap in TungstenAggregate\n\n## What changes were proposed in this pull request?\n\nThis PR adds support for all primitive datatypes, decimal types and stringtypes in the VectorizedHashmap during aggregation.\n\n## How was this patch tested?\n\nExisting tests for group-by aggregates should already test for all these datatypes. Additionally, manually inspected the generated code for all supported datatypes (details below).\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #12440 from sameeragarwal/all-datatypes.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/b29bc3f51518806ef7827b35df7c8aada329f961"},{"sha":"145433f1aaf4a58f484f98c2f1d32abd8cc95b48","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-14369] [SQL] Locality support for FileScanRDD\n\n(This PR is a rebased version of PR #12153.)\n\n## What changes were proposed in this pull request?\n\nThis PR adds preliminary locality support for `FileFormat` data sources by overriding `FileScanRDD.preferredLocations()`. The strategy can be divided into two parts:\n\n1.  Block location lookup\n\n    Unlike `HadoopRDD` or `NewHadoopRDD`, `FileScanRDD` doesn't have access to the underlying `InputFormat` or `InputSplit`, and thus can't rely on `InputSplit.getLocations()` to gather locality information. Instead, this PR queries block locations using `FileSystem.getBlockLocations()` after listing all `FileStatus`es in `HDFSFileCatalog` and convert all `FileStatus`es into `LocatedFileStatus`es.\n\n    Note that although S3/S3A/S3N file systems don't provide valid locality information, their `getLocatedStatus()` implementations don't actually issue remote calls either. So there's no need to special case these file systems.\n\n2.  Selecting preferred locations\n\n    For each `FilePartition`, we pick up top 3 locations that containing the most data to be retrieved. This isn't necessarily the best algorithm out there. Further improvements may be brought up in follow-up PRs.\n\n## How was this patch tested?\n\nTested by overriding default `FileSystem` implementation for `file:///` with a mocked one, which returns mocked block locations.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #12527 from liancheng/spark-14369-locality-rebased.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/145433f1aaf4a58f484f98c2f1d32abd8cc95b48"},{"sha":"3405cc775843a3a80d009d4f9079ba9daa2220e7","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14835][SQL] Remove MetastoreRelation dependency from SQLBuilder\n\n## What changes were proposed in this pull request?\nThis patch removes SQLBuilder's dependency on MetastoreRelation. We should be able to move SQLBuilder into the sql/core package after this change.\n\n## How was this patch tested?\nN/A - covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12594 from rxin/SPARK-14835.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/3405cc775843a3a80d009d4f9079ba9daa2220e7"},{"sha":"80127935df06a829b734cafc2447aa1f3df40288","author":{"email":"6c68cabc7124e43c9b4d50f86510600f69e74c99@odersky.com","name":"Jakob Odersky"},"message":"[SPARK-10001] [CORE] Interrupt tasks in repl with Ctrl+C\n\n## What changes were proposed in this pull request?\n\nImprove signal handling to allow interrupting running tasks from the REPL (with Ctrl+C).\nIf no tasks are running or Ctrl+C is pressed twice, the signal is forwarded to the default handler resulting in the usual termination of the application.\n\nThis PR is a rewrite of -- and therefore closes #8216 -- as per piaozhexiu's request\n\n## How was this patch tested?\nSignal handling is not easily testable therefore no unit tests were added. Nevertheless, the new functionality is implemented in a best-effort approach, soft-failing in case signals aren't available on a specific OS.\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #12557 from jodersky/SPARK-10001-sigint.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/80127935df06a829b734cafc2447aa1f3df40288"},{"sha":"284b15d2fbff7c0c3ffe8737838071d366ea5742","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14826][SQL] Remove HiveQueryExecution\n\n## What changes were proposed in this pull request?\nThis patch removes HiveQueryExecution. As part of this, I consolidated all the describe commands into DescribeTableCommand.\n\n## How was this patch tested?\nShould be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12588 from rxin/SPARK-14826.","distinct":false,"url":"https://api.github.com/repos/bomeng/spark/commits/284b15d2fbff7c0c3ffe8737838071d366ea5742"}]}