{"push_id":3636349876,"size":1000,"distinct_size":1,"ref":"refs/heads/write","head":"e9e629a2b6351b342d98709d47e2a3013b2cff64","before":"b91a3af185fa8c953648080c24c71664b1ebe646","commits":[{"sha":"c8ac6ae84c8a3cc7ed787155a84cbeb56c78a048","author":{"name":"Darcy Shen","email":"a9a536cfcb71f58b94de5571e1607dbc58071d16@zoho.com"},"message":"[SPARK-26319][SQL][TEST] Add appendReadColumns Unit Test for HiveShimSuite\n\n## What changes were proposed in this pull request?\n\nAdd appendReadColumns Unit Test for HiveShimSuite.\n\n## How was this patch tested?\n```\n$ build/sbt\n> project hive\n> testOnly *HiveShimSuite\n```\n\nCloses #23268 from sadhen/refactor/hiveshim.\n\nAuthored-by: Darcy Shen <sadhen@zoho.com>\nSigned-off-by: Hyukjin Kwon <gurwls223@apache.org>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/c8ac6ae84c8a3cc7ed787155a84cbeb56c78a048"},{"sha":"42e8c381b15dd48c2f00c088c897ebdd25405aef","author":{"name":"10087686","email":"3e895f885f2c011068158da255d17f478efae322@zte.com.cn"},"message":"[SPARK-26286][TEST] Add MAXIMUM_PAGE_SIZE_BYTES exception bound unit test\n\n## What changes were proposed in this pull request?\nAdd MAXIMUM_PAGE_SIZE_BYTES Exception test\n\n## How was this patch tested?\nExisting tests\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nCloses #23226 from wangjiaochun/BytesToBytesMapSuite.\n\nAuthored-by: 10087686 <wang.jiaochun@zte.com.cn>\nSigned-off-by: Hyukjin Kwon <gurwls223@apache.org>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/42e8c381b15dd48c2f00c088c897ebdd25405aef"},{"sha":"9794923272c26ee5ba760a57718a368c33d09f04","author":{"name":"liuxian","email":"b507fe1e602c016180e481ef5e61b475642f6f2d@zte.com.cn"},"message":"[MINOR][DOC] Update the condition description of serialized shuffle\n\n## What changes were proposed in this pull request?\n`1. The shuffle dependency specifies no aggregation or output ordering.`\nIf the shuffle dependency specifies aggregation, but it only aggregates at the reduce-side, serialized shuffle can still be used.\n`3. The shuffle produces fewer than 16777216 output partitions.`\nIf the number of output partitions is 16777216 , we can use serialized shuffle.\n\nWe can see this mothod: `canUseSerializedShuffle`\n## How was this patch tested?\nN/A\n\nCloses #23228 from 10110346/SerializedShuffle_doc.\n\nAuthored-by: liuxian <liu.xian3@zte.com.cn>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/9794923272c26ee5ba760a57718a368c33d09f04"},{"sha":"0bf6c77141e40cc636351c5e77194bb75144bb12","author":{"name":"Sean Owen","email":"3f9d48d7c3f24f7009479c5fa630bd1131ffc724@databricks.com"},"message":"This tests pushing to gitbox","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/0bf6c77141e40cc636351c5e77194bb75144bb12"},{"sha":"b1a724b468d5c1c4aee2a22ffc6d8edac537c3d6","author":{"name":"Sean Owen","email":"3f9d48d7c3f24f7009479c5fa630bd1131ffc724@databricks.com"},"message":"This tests pushing directly to github","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b1a724b468d5c1c4aee2a22ffc6d8edac537c3d6"},{"sha":"90c77ea3132d0b7a12c316bd42fb8d0f59bee253","author":{"name":"Reza Safi","email":"4e275f87153feb05bcceb3af025d6c1b1e2cd4ab@cloudera.com"},"message":"[SPARK-24958][CORE] Add memory from procfs to executor metrics.\n\nThis adds the entire memory used by spark’s executor (as measured by procfs) to the executor metrics.  The memory usage is collected from the entire process tree under the executor.  The metrics are subdivided into memory used by java, by python, and by other processes, to aid users in diagnosing the source of high memory usage.\nThe additional metrics are sent to the driver in heartbeats, using the mechanism introduced by SPARK-23429.  This also slightly extends that approach to allow one ExecutorMetricType to collect multiple metrics.\n\nAdded unit tests and also tested on a live cluster.\n\nCloses #22612 from rezasafi/ptreememory2.\n\nAuthored-by: Reza Safi <rezasafi@cloudera.com>\nSigned-off-by: Imran Rashid <irashid@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/90c77ea3132d0b7a12c316bd42fb8d0f59bee253"},{"sha":"0a37da68e1cbca0e0120beab916309898022ba85","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-26317][BUILD] Upgrade SBT to 0.13.18\n\n## What changes were proposed in this pull request?\n\nSBT 0.13.14 ~ 1.1.1 has a bug on accessing `java.util.Base64.getDecoder` with JDK9+. It's fixed at 1.1.2 and backported to [0.13.18 (released on Nov 28th)](https://github.com/sbt/sbt/releases/tag/v0.13.18). This PR aims to update SBT.\n\n## How was this patch tested?\n\nPass the Jenkins with the building and existing tests.\n\nCloses #23270 from dongjoon-hyun/SPARK-26317.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/0a37da68e1cbca0e0120beab916309898022ba85"},{"sha":"82c1ac48a37bcc929db86515bffd602c381415be","author":{"name":"韩田田00222924","email":"6eddf312d79578d14ddc99a1f594d131db4eb623@zte.com.cn"},"message":"[SPARK-25696] The storage memory displayed on spark Application UI is…\n\n… incorrect.\n\n## What changes were proposed in this pull request?\nIn the reported heartbeat information, the unit of the memory data is bytes, which is converted by the formatBytes() function in the utils.js file before being displayed in the interface. The cardinality of the unit conversion in the formatBytes function is 1000, which should be 1024.\nChange the cardinality of the unit conversion in the formatBytes function to 1024.\n\n## How was this patch tested?\n manual tests\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nCloses #22683 from httfighter/SPARK-25696.\n\nLead-authored-by: 韩田田00222924 <han.tiantian@zte.com.cn>\nCo-authored-by: han.tiantian@zte.com.cn <han.tiantian@zte.com.cn>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/82c1ac48a37bcc929db86515bffd602c381415be"},{"sha":"05cf81e6de3d61ddb0af81cd179665693f23351f","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-19827][R] spark.ml R API for PIC\n\n## What changes were proposed in this pull request?\n\nAdd PowerIterationCluster (PIC) in R\n## How was this patch tested?\nAdd test case\n\nCloses #23072 from huaxingao/spark-19827.\n\nAuthored-by: Huaxin Gao <huaxing@us.ibm.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/05cf81e6de3d61ddb0af81cd179665693f23351f"},{"sha":"cbe92305cd4f80725a251cf74353e8985d28306c","author":{"name":"10129659","email":"9a534c61f89b71653a31fec32bc55ad88fd05426@zte.com.cn"},"message":"[SPARK-26312][SQL] Replace RDDConversions.rowToRowRdd with RowEncoder to improve its conversion performance\n\n## What changes were proposed in this pull request?\n\n`RDDConversions` would get disproportionately slower as the number of columns in the query increased,\nfor the type of `converters` before is `scala.collection.immutable.::` which is a subtype of list.\nThis PR removing `RDDConversions` and using `RowEncoder` to convert the Row to InternalRow.\n\nThe test of `PrunedScanSuite` for 2000 columns and 20k rows takes 409 seconds before this PR, and 361 seconds after.\n\n## How was this patch tested?\n\nTest case of `PrunedScanSuite`\n\nCloses #23262 from eatoncys/toarray.\n\nAuthored-by: 10129659 <chen.yanshan@zte.com.cn>\nSigned-off-by: Hyukjin Kwon <gurwls223@apache.org>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/cbe92305cd4f80725a251cf74353e8985d28306c"},{"sha":"7d5f6e8c493b96898ba01edede1522121fe945fc","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[SPARK-26293][SQL] Cast exception when having python udf in subquery\n\n## What changes were proposed in this pull request?\n\nThis is a regression introduced by https://github.com/apache/spark/pull/22104 at Spark 2.4.0.\n\nWhen we have Python UDF in subquery, we will hit an exception\n```\nCaused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.AttributeReference cannot be cast to org.apache.spark.sql.catalyst.expressions.PythonUDF\n\tat scala.collection.immutable.Stream.map(Stream.scala:414)\n\tat org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$2(EvalPythonExec.scala:98)\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:815)\n...\n```\n\nhttps://github.com/apache/spark/pull/22104 turned `ExtractPythonUDFs` from a physical rule to optimizer rule. However, there is a difference between a physical rule and optimizer rule. A physical rule always runs once, an optimizer rule may be applied twice on a query tree even the rule is located in a batch that only runs once.\n\nFor a subquery, the `OptimizeSubqueries` rule will execute the entire optimizer on the query plan inside subquery. Later on subquery will be turned to joins, and the optimizer rules will be applied to it again.\n\nUnfortunately, the `ExtractPythonUDFs` rule is not idempotent. When it's applied twice on a query plan inside subquery, it will produce a malformed plan. It extracts Python UDF from Python exec plans.\n\nThis PR proposes 2 changes to be double safe:\n1. `ExtractPythonUDFs` should skip python exec plans, to make the rule idempotent\n2. `ExtractPythonUDFs` should skip subquery\n\n## How was this patch tested?\n\na new test.\n\nCloses #23248 from cloud-fan/python.\n\nAuthored-by: Wenchen Fan <wenchen@databricks.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7d5f6e8c493b96898ba01edede1522121fe945fc"},{"sha":"4e1d859c19d3bfdfcb8acf915a97c68633b9ca95","author":{"name":"Maxim Gekk","email":"b91a34c712a29cd91cde3972985e114f1bb54d88@gmail.com"},"message":"[SPARK-26303][SQL] Return partial results for bad JSON records\n\n## What changes were proposed in this pull request?\n\nIn the PR, I propose to return partial results from JSON datasource and JSON functions in the PERMISSIVE mode if some of JSON fields are parsed and converted to desired types successfully. The changes are made only for `StructType`. Whole bad JSON records are placed into the corrupt column specified by the `columnNameOfCorruptRecord` option or SQL config.\n\nPartial results are not returned for malformed JSON input.\n\n## How was this patch tested?\n\nAdded new UT which checks converting JSON strings with one invalid and one valid field at the end of the string.\n\nCloses #23253 from MaxGekk/json-bad-record.\n\nLead-authored-by: Maxim Gekk <max.gekk@gmail.com>\nCo-authored-by: Maxim Gekk <maxim.gekk@databricks.com>\nSigned-off-by: Hyukjin Kwon <gurwls223@apache.org>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4e1d859c19d3bfdfcb8acf915a97c68633b9ca95"},{"sha":"bd7df6b1e129741136d09a3d29f9ffcc32ce1de3","author":{"name":"Yuanjian Li","email":"2d53d9b773e88f84fee6a6dbdff6a5bb5ed7b612@gmail.com"},"message":"[SPARK-26327][SQL] Bug fix for `FileSourceScanExec` metrics update and name changing\n\n## What changes were proposed in this pull request?\n\nAs the description in [SPARK-26327](https://issues.apache.org/jira/browse/SPARK-26327), `postDriverMetricUpdates` was called on wrong place cause this bug, fix this by split the initializing of `selectedPartitions` and metrics updating logic. Add the updating logic in `inputRDD` initializing which can take effect in both code generation node and normal node. Also rename `metadataTime` to `fileListingTime` for clearer meaning.\n## How was this patch tested?\n\nNew test case in `SQLMetricsSuite`.\nManual test:\n\n|         | Before | After |\n|---------|:--------:|:-------:|\n| CodeGen |![image](https://user-images.githubusercontent.com/4833765/49741753-13c7e800-fcd2-11e8-97a8-8057b657aa3c.png)|![image](https://user-images.githubusercontent.com/4833765/49741774-1f1b1380-fcd2-11e8-98d9-78b950f4e43a.png)|\n| Normal  |![image](https://user-images.githubusercontent.com/4833765/49741836-378b2e00-fcd2-11e8-80c3-ab462a6a3184.png)|![image](https://user-images.githubusercontent.com/4833765/49741860-4a056780-fcd2-11e8-9ef1-863de217f183.png)|\n\nCloses #23277 from xuanyuanking/SPARK-26327.\n\nAuthored-by: Yuanjian Li <xyliyuanjian@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/bd7df6b1e129741136d09a3d29f9ffcc32ce1de3"},{"sha":"a3bbca98d7d120f22727a55cdc448608e6bb9fad","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-26265][CORE] Fix deadlock in BytesToBytesMap.MapIterator when locking both BytesToBytesMap.MapIterator and TaskMemoryManager\n\n## What changes were proposed in this pull request?\n\nIn `BytesToBytesMap.MapIterator.advanceToNextPage`, We will first lock this `MapIterator` and then `TaskMemoryManager` when going to free a memory page by calling `freePage`. At the same time, it is possibly that another memory consumer first locks `TaskMemoryManager` and then this `MapIterator` when it acquires memory and causes spilling on this `MapIterator`.\n\nSo it ends with the `MapIterator` object holds lock to the `MapIterator` object and waits for lock on `TaskMemoryManager`, and the other consumer holds lock to `TaskMemoryManager` and waits for lock on the `MapIterator` object.\n\nTo avoid deadlock here, this patch proposes to keep reference to the page to free and free it after releasing the lock of `MapIterator`.\n\n## How was this patch tested?\n\nAdded test and manually test by running the test 100 times to make sure there is no deadlock.\n\nCloses #23272 from viirya/SPARK-26265.\n\nAuthored-by: Liang-Chi Hsieh <viirya@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a3bbca98d7d120f22727a55cdc448608e6bb9fad"},{"sha":"5c67a9a7fa29836fc825504bbcc3c3fc820009c6","author":{"name":"jiake","email":"c845d3ba3010d77ea83202b84ef0f366c78158aa@intel.com"},"message":"[SPARK-26316][SPARK-21052] Revert hash join metrics in that causes performance degradation\n\n## What changes were proposed in this pull request?\nThe wrong implementation in the hash join metrics in [spark 21052](https://issues.apache.org/jira/browse/SPARK-21052) caused significant performance degradation in TPC-DS. And the result is [here](https://docs.google.com/spreadsheets/d/18a5BdOlmm8euTaRodyeWum9yu92mbWWu6JbhGXtr7yE/edit#gid=0) in TPC-DS 1TB scale. So we currently partial revert 21052.\n**Cluster info:**\n\n  | Master Node | Worker Nodes\n-- | -- | --\nNode | 1x | 4x\nProcessor | Intel(R) Xeon(R) Platinum 8170 CPU  2.10GHz | Intel(R) Xeon(R) Platinum 8180 CPU  2.50GHz\nMemory | 192 GB | 384 GB\nStorage Main | 8 x 960G SSD | 8 x 960G SSD\nNetwork | 10Gbe |  \nRole | CM Management NameNodeSecondary NameNodeResource ManagerHive Metastore Server | DataNodeNodeManager\nOS Version | CentOS 7.2 | CentOS 7.2\nHadoop | Apache Hadoop 2.7.5 | Apache Hadoop 2.7.5\nHive | Apache Hive 2.2.0 |  \nSpark | Apache Spark 2.1.0  & Apache Spark2.3.0 |  \nJDK  version | 1.8.0_112 | 1.8.0_112\n\n**Related parameters setting:**\n\nComponent | Parameter | Value\n-- | -- | --\nYarn Resource Manager | yarn.scheduler.maximum-allocation-mb | 120GB\n  | yarn.scheduler.minimum-allocation-mb | 1GB\n  | yarn.scheduler.maximum-allocation-vcores | 121\n  | Yarn.resourcemanager.scheduler.class | Fair Scheduler\nYarn Node Manager | yarn.nodemanager.resource.memory-mb | 120GB\n  | yarn.nodemanager.resource.cpu-vcores | 121\nSpark | spark.executor.memory | 110GB\n  | spark.executor.cores | 50\n\n## How was this patch tested?\nN/A\n\nCloses #23269 from JkSelf/partial-revert-21052.\n\nAuthored-by: jiake <ke.a.jia@intel.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/5c67a9a7fa29836fc825504bbcc3c3fc820009c6"},{"sha":"d811369ce23186cbb3208ad665e15408e13fea87","author":{"name":"liuxian","email":"b507fe1e602c016180e481ef5e61b475642f6f2d@zte.com.cn"},"message":"[SPARK-26300][SS] Remove a redundant `checkForStreaming` call\n\n## What changes were proposed in this pull request?\nIf `checkForContinuous`  is called ( `checkForStreaming` is called in `checkForContinuous`  ), the `checkForStreaming`  mothod  will be called twice in `createQuery` , this is not necessary,  and the `checkForStreaming` method has a lot of statements,  so it's better to remove one of them.\n\n## How was this patch tested?\n\nExisting unit tests in `StreamingQueryManagerSuite` and `ContinuousAggregationSuite`\n\nCloses #23251 from 10110346/isUnsupportedOperationCheckEnabled.\n\nAuthored-by: liuxian <liu.xian3@zte.com.cn>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/d811369ce23186cbb3208ad665e15408e13fea87"},{"sha":"57d6fbfa8c803ce1791e7be36aba0219a1fcaa63","author":{"name":"mcheah","email":"2e9bf58ec2f93170404b3a28538fb27400f699db@palantir.com"},"message":"[SPARK-26239] File-based secret key loading for SASL.\n\nThis proposes an alternative way to load secret keys into a Spark application that is running on Kubernetes. Instead of automatically generating the secret, the secret key can reside in a file that is shared between both the driver and executor containers.\n\nUnit tests.\n\nCloses #23252 from mccheah/auth-secret-with-file.\n\nAuthored-by: mcheah <mcheah@palantir.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/57d6fbfa8c803ce1791e7be36aba0219a1fcaa63"},{"sha":"bd8da3799dd160771ebb3ea55b7678b644248425","author":{"name":"Yuanjian Li","email":"2d53d9b773e88f84fee6a6dbdff6a5bb5ed7b612@gmail.com"},"message":"[SPARK-26193][SQL][FOLLOW UP] Read metrics rename and display text changes\n\n## What changes were proposed in this pull request?\nFollow up pr for #23207, include following changes:\n\n- Rename `SQLShuffleMetricsReporter` to `SQLShuffleReadMetricsReporter` to make it match with write side naming.\n- Display text changes for read side for naming consistent.\n- Rename function in `ShuffleWriteProcessor`.\n- Delete `private[spark]` in execution package.\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #23286 from xuanyuanking/SPARK-26193-follow.\n\nAuthored-by: Yuanjian Li <xyliyuanjian@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/bd8da3799dd160771ebb3ea55b7678b644248425"},{"sha":"79e36e2c2ac01458b5baa3f3ee310fddd29e9c35","author":{"name":"Sean Owen","email":"3f9d48d7c3f24f7009479c5fa630bd1131ffc724@databricks.com"},"message":"[SPARK-19827][R][FOLLOWUP] spark.ml R API for PIC\n\n## What changes were proposed in this pull request?\n\nFollow up style fixes to PIC in R; see #23072\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #23292 from srowen/SPARK-19827.2.\n\nAuthored-by: Sean Owen <sean.owen@databricks.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/79e36e2c2ac01458b5baa3f3ee310fddd29e9c35"},{"sha":"570b8f3d45ad8d6649ed633251a8194d910f1ab5","author":{"name":"Ilya Matiach","email":"41e5560f18035fdbdee9a5f6e19b09be707be423@microsoft.com"},"message":"[SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator\n\n## What changes were proposed in this pull request?\n\nThe evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data.\n\nI've closed the PR: https://github.com/apache/spark/pull/16557\n as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update.\n\nThe updates to the regression metrics were based on (and updated with new changes based on comments):\nhttps://issues.apache.org/jira/browse/SPARK-11520\n (\"RegressionMetrics should support instance weights\")\n but the pull request was closed as the changes were never checked in.\n\n## How was this patch tested?\n\nI added tests to the metrics class.\n\nCloses #17085 from imatiach-msft/ilmat/regression-evaluate.\n\nAuthored-by: Ilya Matiach <ilmat@microsoft.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/570b8f3d45ad8d6649ed633251a8194d910f1ab5"}]}
{"actor":{"display_login":"cloud-fan"}}