{"push_id":1324874919,"size":39,"distinct_size":24,"ref":"refs/heads/global-temp-view","head":"952f13cd92e5e448622ad80377845d8448399b86","before":"88e684e4e69b9acfc3e9c395ec5ffdc4dfdf1610","commits":[{"sha":"6ee28423ad1b2e6089b82af64a31d77d3552bb38","author":{"name":"Ding Fei","email":"28c99dbb5fbb785cd6cc281220ad0b70074fd026@danix"},"message":"Fix two comments since Actor is not used anymore.\n\n## What changes were proposed in this pull request?\n\nFix two comments since Actor is not used anymore.\n\nAuthor: Ding Fei <danis@danix>\n\nCloses #15251 from danix800/comment-fixing.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/6ee28423ad1b2e6089b82af64a31d77d3552bb38"},{"sha":"85b0a157543201895557d66306b38b3ca52f2151","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-15962][SQL] Introduce implementation with a dense format for UnsafeArrayData\n\n## What changes were proposed in this pull request?\n\nThis PR introduces more compact representation for ```UnsafeArrayData```.\n\n```UnsafeArrayData``` needs to accept ```null``` value in each entry of an array. In the current version, it has three parts\n```\n[numElements] [offsets] [values]\n```\n`Offsets` has the number of `numElements`, and represents `null` if its value is negative. It may increase memory footprint, and introduces an indirection for accessing each of `values`.\n\nThis PR uses bitvectors to represent nullability for each element like `UnsafeRow`, and eliminates an indirection for accessing each element. The new ```UnsafeArrayData``` has four parts.\n```\n[numElements][null bits][values or offset&length][variable length portion]\n```\nIn the `null bits` region, we store 1 bit per element, represents whether an element is null. Its total size is ceil(numElements / 8) bytes, and it is aligned to 8-byte boundaries.\nIn the `values or offset&length` region, we store the content of elements. For fields that hold fixed-length primitive types, such as long, double, or int, we store the value directly in the field. For fields with non-primitive or variable-length values, we store a relative offset (w.r.t. the base address of the array) that points to the beginning of the variable-length field and length (they are combined into a long). Each is word-aligned. For `variable length portion`, each is aligned to 8-byte boundaries.\n\nThe new format can reduce memory footprint and improve performance of accessing each element. An example of memory foot comparison:\n1024x1024 elements integer array\nSize of ```baseObject``` for ```UnsafeArrayData```: 8 + 1024x1024 + 1024x1024 = 2M bytes\nSize of ```baseObject``` for ```UnsafeArrayData```: 8 + 1024x1024/8 + 1024x1024 = 1.25M bytes\n\nIn summary, we got 1.0-2.6x performance improvements over the code before applying this PR.\nHere are performance results of [benchmark programs](https://github.com/kiszk/spark/blob/04d2e4b6dbdc4eff43ce18b3c9b776e0129257c7/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/UnsafeArrayDataBenchmark.scala):\n\n**Read UnsafeArrayData**: 1.7x and 1.6x performance improvements over the code before applying this PR\n````\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.4.11-200.fc22.x86_64\nIntel Xeon E3-12xx v2 (Ivy Bridge)\n\nWithout SPARK-15962\nRead UnsafeArrayData:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            430 /  436        390.0           2.6       1.0X\nDouble                                         456 /  485        367.8           2.7       0.9X\n\nWith SPARK-15962\nRead UnsafeArrayData:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            252 /  260        666.1           1.5       1.0X\nDouble                                         281 /  292        597.7           1.7       0.9X\n````\n**Write UnsafeArrayData**: 1.0x and 1.1x performance improvements over the code before applying this PR\n````\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\nIntel Xeon E3-12xx v2 (Ivy Bridge)\n\nWithout SPARK-15962\nWrite UnsafeArrayData:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            203 /  273        103.4           9.7       1.0X\nDouble                                         239 /  356         87.9          11.4       0.8X\n\nWith SPARK-15962\nWrite UnsafeArrayData:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            196 /  249        107.0           9.3       1.0X\nDouble                                         227 /  367         92.3          10.8       0.9X\n````\n\n**Get primitive array from UnsafeArrayData**: 2.6x and 1.6x performance improvements over the code before applying this PR\n````\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\nIntel Xeon E3-12xx v2 (Ivy Bridge)\n\nWithout SPARK-15962\nGet primitive array from UnsafeArrayData: Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            207 /  217        304.2           3.3       1.0X\nDouble                                         257 /  363        245.2           4.1       0.8X\n\nWith SPARK-15962\nGet primitive array from UnsafeArrayData: Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            151 /  198        415.8           2.4       1.0X\nDouble                                         214 /  394        293.6           3.4       0.7X\n````\n\n**Create UnsafeArrayData from primitive array**: 1.7x and 2.1x performance improvements over the code before applying this PR\n````\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\nIntel Xeon E3-12xx v2 (Ivy Bridge)\n\nWithout SPARK-15962\nCreate UnsafeArrayData from primitive array: Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            340 /  385        185.1           5.4       1.0X\nDouble                                         479 /  705        131.3           7.6       0.7X\n\nWith SPARK-15962\nCreate UnsafeArrayData from primitive array: Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nInt                                            206 /  211        306.0           3.3       1.0X\nDouble                                         232 /  406        271.6           3.7       0.9X\n````\n\n1.7x and 1.4x performance improvements in [```UDTSerializationBenchmark```](https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala)  over the code before applying this PR\n````\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.4.11-200.fc22.x86_64\nIntel Xeon E3-12xx v2 (Ivy Bridge)\n\nWithout SPARK-15962\nVectorUDT de/serialization:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nserialize                                      442 /  533          0.0      441927.1       1.0X\ndeserialize                                    217 /  274          0.0      217087.6       2.0X\n\nWith SPARK-15962\nVectorUDT de/serialization:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nserialize                                      265 /  318          0.0      265138.5       1.0X\ndeserialize                                    155 /  197          0.0      154611.4       1.7X\n````\n\n## How was this patch tested?\n\nAdded unit tests into ```UnsafeArraySuite```\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #13680 from kiszk/SPARK-15962.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/85b0a157543201895557d66306b38b3ca52f2151"},{"sha":"7f16affa262b059580ed2775a7b05a767aa72315","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-17138][ML][MLIB] Add Python API for multinomial logistic regression\n\n## What changes were proposed in this pull request?\n\nAdd Python API for multinomial logistic regression.\n\n- add `family` param in python api.\n- expose `coefficientMatrix` and `interceptVector` for `LogisticRegressionModel`\n- add python-side testcase for multinomial logistic regression\n- update python doc.\n\n## How was this patch tested?\n\nexisting and added doc tests.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14852 from WeichenXu123/add_MLOR_python.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7f16affa262b059580ed2775a7b05a767aa72315"},{"sha":"6a68c5d7b4eb07e4ed6b702dd1536cd08d9bba7d","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[SPARK-16757] Set up Spark caller context to HDFS and YARN\n\n## What changes were proposed in this pull request?\n\n1. Pass `jobId` to Task.\n2. Invoke Hadoop APIs.\n    * A new function `setCallerContext` is added in `Utils`. `setCallerContext` function invokes APIs of   `org.apache.hadoop.ipc.CallerContext` to set up spark caller contexts, which will be written into `hdfs-audit.log` and Yarn RM audit log.\n    * For HDFS: Spark sets up its caller context by invoking`org.apache.hadoop.ipc.CallerContext` in `Task` and Yarn `Client` and `ApplicationMaster`.\n    * For Yarn: Spark sets up its caller context by invoking `org.apache.hadoop.ipc.CallerContext` in Yarn `Client`.\n\n## How was this patch tested?\nManual Tests against some Spark applications in Yarn client mode and Yarn cluster mode. Need to check if spark caller contexts are written into HDFS hdfs-audit.log and Yarn RM audit log successfully.\n\nFor example, run SparkKmeans in Yarn client mode:\n```\n./bin/spark-submit --verbose --executor-cores 3 --num-executors 1 --master yarn --deploy-mode client --class org.apache.spark.examples.SparkKMeans examples/target/original-spark-examples_2.11-2.1.0-SNAPSHOT.jar hdfs://localhost:9000/lr_big.txt 2 5\n```\n\n**Before**:\nThere will be no Spark caller context in records of `hdfs-audit.log` and Yarn RM audit log.\n\n**After**:\nSpark caller contexts will be written in records of `hdfs-audit.log` and Yarn RM audit log.\n\nThese are records in `hdfs-audit.log`:\n```\n2016-09-20 11:54:24,116 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_CLIENT_AppId_application_1474394339641_0005\n2016-09-20 11:54:28,164 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0005_JobId_0_StageId_0_AttemptId_0_TaskId_2_AttemptNum_0\n2016-09-20 11:54:28,164 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0005_JobId_0_StageId_0_AttemptId_0_TaskId_1_AttemptNum_0\n2016-09-20 11:54:28,164 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0005_JobId_0_StageId_0_AttemptId_0_TaskId_0_AttemptNum_0\n```\n```\n2016-09-20 11:59:33,868 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=mkdirs\tsrc=/private/tmp/hadoop-wyang/nm-local-dir/usercache/wyang/appcache/application_1474394339641_0006/container_1474394339641_0006_01_000001/spark-warehouse\tdst=null\tperm=wyang:supergroup:rwxr-xr-x\tproto=rpc\tcallerContext=SPARK_APPLICATION_MASTER_AppId_application_1474394339641_0006_AttemptId_1\n2016-09-20 11:59:37,214 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_1_AttemptNum_0\n2016-09-20 11:59:37,215 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_2_AttemptNum_0\n2016-09-20 11:59:37,215 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_0_AttemptNum_0\n2016-09-20 11:59:42,391 INFO FSNamesystem.audit: allowed=true\tugi=wyang (auth:SIMPLE)\tip=/127.0.0.1\tcmd=open\tsrc=/lr_big.txt\tdst=null\tperm=null\tproto=rpc\tcallerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_3_AttemptNum_0\n```\nThis is a record in Yarn RM log:\n```\n2016-09-20 11:59:24,050 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=wyang\tIP=127.0.0.1\tOPERATION=Submit Application Request\tTARGET=ClientRMService\tRESULT=SUCCESS\tAPPID=application_1474394339641_0006\tCALLERCONTEXT=SPARK_CLIENT_AppId_application_1474394339641_0006\n```\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #14659 from Sherry302/callercontextSubmit.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/6a68c5d7b4eb07e4ed6b702dd1536cd08d9bba7d"},{"sha":"5de1737b02710e36f6804d2ae243d1aeb30a0b32","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16777][SQL] Do not use deprecated listType API in ParquetSchemaConverter\n\n## What changes were proposed in this pull request?\n\nThis PR removes build waning as below.\n\n```scala\n[WARNING] .../spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:448: method listType in object ConversionPatterns is deprecated: see corresponding Javadoc for more information.\n[WARNING]         ConversionPatterns.listType(\n[WARNING]                            ^\n[WARNING] .../spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:464: method listType in object ConversionPatterns is deprecated: see corresponding Javadoc for more information.\n[WARNING]         ConversionPatterns.listType(\n[WARNING]                            ^\n```\n\nThis should not use `listOfElements` (recommended to be replaced from `listType`) instead because the new method checks if the name of elements in Parquet's `LIST` is `element` in Parquet schema and throws an exception if not. However, It seems Spark prior to 1.4.x writes `ArrayType` with Parquet's `LIST` but with `array` as its element name.\n\nTherefore, this PR avoids to use both `listOfElements` and `listType` but just use the existing schema builder to construct the same `GroupType`.\n\n## How was this patch tested?\n\nExisting tests should cover this.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14399 from HyukjinKwon/SPARK-16777.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/5de1737b02710e36f6804d2ae243d1aeb30a0b32"},{"sha":"2cac3b2d4a4a4f3d0d45af4defc23bb0ba53484b","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16516][SQL] Support for pushing down filters for decimal and timestamp types in ORC\n\n## What changes were proposed in this pull request?\n\nIt seems ORC supports all the types in  ([`PredicateLeaf.Type`](https://github.com/apache/hive/blob/e085b7e9bd059d91aaf013df0db4d71dca90ec6f/storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java#L50-L56)) which includes timestamp type and decimal type.\n\nIn more details, the types listed in [`SearchArgumentImpl.boxLiteral()`](https://github.com/apache/hive/blob/branch-1.2/ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java#L1068-L1093) can be used as a filter value.\n\nFYI, inital `case` caluse for supported types was introduced in https://github.com/apache/spark/commit/65d71bd9fbfe6fe1b741c80fed72d6ae3d22b028 and this was not changed overtime. At that time, Hive version was, 0.13 which supports only some types for filter-push down (See [SearchArgumentImpl.java#L945-L965](https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java#L945-L965) at 0.13).\n\nHowever, the version was upgraded into 1.2.x and now it supports more types (See [SearchArgumentImpl.java#L1068-L1093](https://github.com/apache/hive/blob/branch-1.2/ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java#L1068-L1093) at 1.2.0)\n\n## How was this patch tested?\n\nUnit tests in `OrcFilterSuite` and `OrcQuerySuite`\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14172 from HyukjinKwon/SPARK-16516.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2cac3b2d4a4a4f3d0d45af4defc23bb0ba53484b"},{"sha":"120723f934dc386a46a043d2833bfcee60d14e74","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17682][SQL] Mark children as final for unary, binary, leaf expressions and plan nodes\n\n## What changes were proposed in this pull request?\nThis patch marks the children method as final in unary, binary, and leaf expressions and plan nodes (both logical plan and physical plan), as brought up in http://apache-spark-developers-list.1001551.n3.nabble.com/Should-LeafExpression-have-children-final-override-like-Nondeterministic-td19104.html\n\n## How was this patch tested?\nThis is a simple modifier change and has no impact on test coverage.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15256 from rxin/SPARK-17682.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/120723f934dc386a46a043d2833bfcee60d14e74"},{"sha":"2ab24a7bf6687ec238306772c4c7ddef6ac93e99","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-17660][SQL] DESC FORMATTED for VIEW Lacks View Definition\n\n### What changes were proposed in this pull request?\nBefore this PR, `DESC FORMATTED` does not have a section for the view definition. We should add it for permanent views, like what Hive does.\n\n```\n+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------+-------+\n|col_name                    |data_type                                                                                                                            |comment|\n+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------+-------+\n|a                           |int                                                                                                                                  |null   |\n|                            |                                                                                                                                     |       |\n|# Detailed Table Information|                                                                                                                                     |       |\n|Database:                   |default                                                                                                                              |       |\n|Owner:                      |xiaoli                                                                                                                               |       |\n|Create Time:                |Sat Sep 24 21:46:19 PDT 2016                                                                                                         |       |\n|Last Access Time:           |Wed Dec 31 16:00:00 PST 1969                                                                                                         |       |\n|Location:                   |                                                                                                                                     |       |\n|Table Type:                 |VIEW                                                                                                                                 |       |\n|Table Parameters:           |                                                                                                                                     |       |\n|  transient_lastDdlTime     |1474778779                                                                                                                           |       |\n|                            |                                                                                                                                     |       |\n|# Storage Information       |                                                                                                                                     |       |\n|SerDe Library:              |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe                                                                                   |       |\n|InputFormat:                |org.apache.hadoop.mapred.SequenceFileInputFormat                                                                                     |       |\n|OutputFormat:               |org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat                                                                            |       |\n|Compressed:                 |No                                                                                                                                   |       |\n|Storage Desc Parameters:    |                                                                                                                                     |       |\n|  serialization.format      |1                                                                                                                                    |       |\n|                            |                                                                                                                                     |       |\n|# View Information          |                                                                                                                                     |       |\n|View Original Text:         |SELECT * FROM tbl                                                                                                                    |       |\n|View Expanded Text:         |SELECT `gen_attr_0` AS `a` FROM (SELECT `gen_attr_0` FROM (SELECT `a` AS `gen_attr_0` FROM `default`.`tbl`) AS gen_subquery_0) AS tbl|       |\n+----------------------------+-------------------------------------------------------------------------------------------------------------------------------------+-------+\n```\n\n### How was this patch tested?\nAdded a test case\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #15234 from gatorsmile/descFormattedView.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2ab24a7bf6687ec238306772c4c7ddef6ac93e99"},{"sha":"67c73052b877a8709ae6fa22b844a45f114b1f7e","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17677][SQL] Break WindowExec.scala into multiple files\n\n## What changes were proposed in this pull request?\nAs of Spark 2.0, all the window function execution code are in WindowExec.scala. This file is pretty large (over 1k loc) and has a lot of different abstractions in them. This patch creates a new package sql.execution.window, moves WindowExec.scala in it, and breaks WindowExec.scala into multiple, more maintainable pieces:\n\n- AggregateProcessor.scala\n- BoundOrdering.scala\n- RowBuffer.scala\n- WindowExec\n- WindowFunctionFrame.scala\n\n## How was this patch tested?\nThis patch mostly moves code around, and should not change any existing test coverage.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15252 from rxin/SPARK-17677.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/67c73052b877a8709ae6fa22b844a45f114b1f7e"},{"sha":"2f84a686604b298537bfd4d087b41594d2aa7ec6","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17618] Guard against invalid comparisons between UnsafeRow and other formats\n\nThis patch ports changes from #15185 to Spark 2.x. In that patch, a  correctness bug in Spark 1.6.x which was caused by an invalid `equals()` comparison between an `UnsafeRow` and another row of a different format. Spark 2.x is not affected by that specific correctness bug but it can still reap the error-prevention benefits of that patch's changes, which modify  ``UnsafeRow.equals()` to throw an IllegalArgumentException if it is called with an object that is not an `UnsafeRow`.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #15265 from JoshRosen/SPARK-17618-master.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2f84a686604b298537bfd4d087b41594d2aa7ec6"},{"sha":"e7bce9e1876de6ee975ccc89351db58119674aef","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-17056][CORE] Fix a wrong assert regarding unroll memory in MemoryStore\n\n## What changes were proposed in this pull request?\n\nThere is an assert in MemoryStore's putIteratorAsValues method which is used to check if unroll memory is not released too much. This assert looks wrong.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #14642 from viirya/fix-unroll-memory.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e7bce9e1876de6ee975ccc89351db58119674aef"},{"sha":"b03b4adf6d8f4c6d92575c0947540cb474bf7de1","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17666] Ensure that RecordReaders are closed by data source file scans\n\n## What changes were proposed in this pull request?\n\nThis patch addresses a potential cause of resource leaks in data source file scans. As reported in [SPARK-17666](https://issues.apache.org/jira/browse/SPARK-17666), tasks which do not fully-consume their input may cause file handles / network connections (e.g. S3 connections) to be leaked. Spark's `NewHadoopRDD` uses a TaskContext callback to [close its record readers](https://github.com/apache/spark/blame/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L208), but the new data source file scans will only close record readers once their iterators are fully-consumed.\n\nThis patch modifies `RecordReaderIterator` and `HadoopFileLinesReader` to add `close()` methods and modifies all six implementations of `FileFormat.buildReader()` to register TaskContext task completion callbacks to guarantee that cleanup is eventually performed.\n\n## How was this patch tested?\n\nTested manually for now.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #15245 from JoshRosen/SPARK-17666-close-recordreader.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b03b4adf6d8f4c6d92575c0947540cb474bf7de1"},{"sha":"4a83395681e0bca356363a6cfb25c952f235560d","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-17499][SPARKR][FOLLOWUP] Check null first for layers in spark.mlp to avoid warnings in test results\n\n## What changes were proposed in this pull request?\n\nSome tests in `test_mllib.r` are as below:\n\n```r\nexpect_error(spark.mlp(df, layers = NULL), \"layers must be a integer vector with length > 1.\")\nexpect_error(spark.mlp(df, layers = c()), \"layers must be a integer vector with length > 1.\")\n```\n\nThe problem is, `is.na` is internally called via `na.omit` in `spark.mlp` which causes warnings as below:\n\n```\nWarnings -----------------------------------------------------------------------\n1. spark.mlp (test_mllib.R#400) - is.na() applied to non-(list or vector) of type 'NULL'\n\n2. spark.mlp (test_mllib.R#401) - is.na() applied to non-(list or vector) of type 'NULL'\n```\n\n## How was this patch tested?\n\nManually tested. Also, Jenkins tests and AppVeyor.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15232 from HyukjinKwon/remove-warnnings.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4a83395681e0bca356363a6cfb25c952f235560d"},{"sha":"b2a7eedcddf0e682ff46afd1b764d0b81ccdf395","author":{"name":"Shuai Lin","email":"7a96e9fba54b1a616c0c679715dfd11c0089fe85@gmail.com"},"message":"[SPARK-17017][ML][MLLIB][ML][DOC] Updated the ml/mllib feature selection docs for ChiSqSelector\n\n## What changes were proposed in this pull request?\n\nA follow up for #14597 to update feature selection docs about ChiSqSelector.\n\n## How was this patch tested?\n\nGenerated html docs. It can be previewed at:\n\n* ml: http://sparkdocs.lins05.pw/spark-17017/ml-features.html#chisqselector\n* mllib: http://sparkdocs.lins05.pw/spark-17017/mllib-feature-extraction.html#chisqselector\n\nAuthor: Shuai Lin <linshuai2012@gmail.com>\n\nCloses #15236 from lins05/spark-17017-update-docs-for-chisq-selector-fpr.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b2a7eedcddf0e682ff46afd1b764d0b81ccdf395"},{"sha":"2190037757a81d3172f75227f7891d968e1f0d90","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[MINOR][PYSPARK][DOCS] Fix examples in PySpark documentation\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to fix wrongly indented examples in PySpark documentation\n\n```\n-        >>> json_sdf = spark.readStream.format(\"json\")\\\n-                                       .schema(sdf_schema)\\\n-                                       .load(tempfile.mkdtemp())\n+        >>> json_sdf = spark.readStream.format(\"json\") \\\\\n+        ...     .schema(sdf_schema) \\\\\n+        ...     .load(tempfile.mkdtemp())\n```\n\n```\n-        people.filter(people.age > 30).join(department, people.deptId == department.id)\\\n+        people.filter(people.age > 30).join(department, people.deptId == department.id) \\\\\n```\n\n```\n-        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, 1.23), (2, 4.56)])), \\\n-                        LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]\n+        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, 1.23), (2, 4.56)])),\n+        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]\n```\n\n```\n-        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])), \\\n-                        LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]\n+        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])),\n+        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]\n```\n\n```\n-        ...      for x in iterator:\n-        ...           print(x)\n+        ...     for x in iterator:\n+        ...          print(x)\n```\n\n## How was this patch tested?\n\nManually tested.\n\n**Before**\n\n![2016-09-26 8 36 02](https://cloud.githubusercontent.com/assets/6477701/18834471/05c7a478-8431-11e6-94bb-09aa37b12ddb.png)\n\n![2016-09-26 9 22 16](https://cloud.githubusercontent.com/assets/6477701/18834472/06c8735c-8431-11e6-8775-78631eab0411.png)\n\n<img width=\"601\" alt=\"2016-09-27 2 29 27\" src=\"https://cloud.githubusercontent.com/assets/6477701/18861294/29c0d5b4-84bf-11e6-99c5-3c9d913c125d.png\">\n\n<img width=\"1056\" alt=\"2016-09-27 2 29 58\" src=\"https://cloud.githubusercontent.com/assets/6477701/18861298/31694cd8-84bf-11e6-9e61-9888cb8c2089.png\">\n\n<img width=\"1079\" alt=\"2016-09-27 2 30 05\" src=\"https://cloud.githubusercontent.com/assets/6477701/18861301/359722da-84bf-11e6-97f9-5f5365582d14.png\">\n\n**After**\n\n![2016-09-26 9 29 47](https://cloud.githubusercontent.com/assets/6477701/18834467/0367f9da-8431-11e6-86d9-a490d3297339.png)\n\n![2016-09-26 9 30 24](https://cloud.githubusercontent.com/assets/6477701/18834463/f870fae0-8430-11e6-9482-01fc47898492.png)\n\n<img width=\"515\" alt=\"2016-09-27 2 28 19\" src=\"https://cloud.githubusercontent.com/assets/6477701/18861305/3ff88b88-84bf-11e6-902c-9f725e8a8b10.png\">\n\n<img width=\"652\" alt=\"2016-09-27 3 50 59\" src=\"https://cloud.githubusercontent.com/assets/6477701/18863053/592fbc74-84ca-11e6-8dbf-99cf57947de8.png\">\n\n<img width=\"709\" alt=\"2016-09-27 3 51 03\" src=\"https://cloud.githubusercontent.com/assets/6477701/18863060/601607be-84ca-11e6-80aa-a401df41c321.png\">\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15242 from HyukjinKwon/minor-example-pyspark.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/2190037757a81d3172f75227f7891d968e1f0d90"},{"sha":"46d1203bf2d01b219c4efc7e0e77a844c0c664da","author":{"name":"w00228970","email":"b7f66cd45f78ebd8439b28a88058866c6c775d1c@huawei.com"},"message":"[SPARK-17644][CORE] Do not add failedStages when abortStage for fetch failure\n\n## What changes were proposed in this pull request?\n| Time        |Thread 1 ,  Job1          | Thread 2 ,  Job2  |\n|:-------------:|:-------------:|:-----:|\n| 1 | abort stage due to FetchFailed |  |\n| 2 | failedStages += failedStage |    |\n| 3 |      |  task failed due to  FetchFailed |\n| 4 |      |  can not post ResubmitFailedStages because failedStages is not empty |\n\nThen job2 of thread2 never resubmit the failed stage and hang.\n\nWe should not add the failedStages when abortStage for fetch failure\n\n## How was this patch tested?\n\nadded unit test\n\nAuthor: w00228970 <wangfei1@huawei.com>\nAuthor: wangfei <wangfei_hello@126.com>\n\nCloses #15213 from scwf/dag-resubmit.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/46d1203bf2d01b219c4efc7e0e77a844c0c664da"},{"sha":"a6cfa3f38bcf6ba154d5ed2a53748fbc90c8872a","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-17673][SQL] Incorrect exchange reuse with RowDataSourceScan\n\n## What changes were proposed in this pull request?\n\nIt seems the equality check for reuse of `RowDataSourceScanExec` nodes doesn't respect the output schema. This can cause self-joins or unions over the same underlying data source to return incorrect results if they select different fields.\n\n## How was this patch tested?\n\nNew unit test passes after the fix.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #15273 from ericl/spark-17673.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a6cfa3f38bcf6ba154d5ed2a53748fbc90c8872a"},{"sha":"557d6e32272dee4eaa0f426cc3e2f82ea361c3da","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-17713][SQL] Move row-datasource related tests out of JDBCSuite\n\n## What changes were proposed in this pull request?\n\nAs a followup for https://github.com/apache/spark/pull/15273 we should move non-JDBC specific tests out of that suite.\n\n## How was this patch tested?\n\nRan the test.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #15287 from ericl/spark-17713.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/557d6e32272dee4eaa0f426cc3e2f82ea361c3da"},{"sha":"7d09232028967978d9db314ec041a762599f636b","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-17641][SQL] Collect_list/Collect_set should not collect null values.\n\n## What changes were proposed in this pull request?\nWe added native versions of `collect_set` and `collect_list` in Spark 2.0. These currently also (try to) collect null values, this is different from the original Hive implementation. This PR fixes this by adding a null check to the `Collect.update` method.\n\n## How was this patch tested?\nAdded a regression test to `DataFrameAggregateSuite`.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #15208 from hvanhovell/SPARK-17641.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7d09232028967978d9db314ec041a762599f636b"},{"sha":"7dfad4b132bc46263ef788ced4a935862f5c8756","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[SPARK-17710][HOTFIX] Fix ClassCircularityError in ReplSuite tests in Maven build: use 'Class.forName' instead of 'Utils.classForName'\n\n## What changes were proposed in this pull request?\nFix ClassCircularityError in ReplSuite tests when Spark is built by Maven build.\n\n## How was this patch tested?\n(1)\n```\nbuild/mvn -DskipTests -Phadoop-2.3 -Pyarn -Phive -Phive-thriftserver -Pkinesis-asl -Pmesos clean package\n```\nThen test:\n```\nbuild/mvn -Dtest=none -DwildcardSuites=org.apache.spark.repl.ReplSuite test\n```\nReplSuite tests passed\n\n(2)\nManual Tests against some Spark applications in Yarn client mode and Yarn cluster mode. Need to check if spark caller contexts are written into HDFS hdfs-audit.log and Yarn RM audit log successfully.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #15286 from Sherry302/SPARK-16757.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7dfad4b132bc46263ef788ced4a935862f5c8756"}]}
{"actor":{"display_login":"cloud-fan"}}