{"push_id":3033834362,"size":23,"distinct_size":23,"ref":"refs/heads/master","head":"0558d021cc0aeae37ef0e043d244fd0300a57cd5","before":"3ed91c9b8998f2512716f906cd1cba25578111ff","commits":[{"sha":"fdd3bace1da01e5958fe0345c38e889e740ce25e","author":{"name":"Dhruve Ashar","email":"56556026bd87097401a2aa2d3035b4c3c9d670a7@gmail.com"},"message":"[SPARK-22148][SPARK-15815][SCHEDULER] Acquire new executors to avoid hang because of blacklisting\n\n## What changes were proposed in this pull request?\nEvery time a task is unschedulable because of the condition where no. of task failures < no. of executors available, we currently abort the taskSet - failing the job. This change tries to acquire new executors so that we can complete the job successfully. We try to acquire a new executor only when we can kill an existing idle executor. We fallback to the older implementation where we abort the job if we cannot find an idle executor.\n\n## How was this patch tested?\n\nI performed some manual tests to check and validate the behavior.\n\n```scala\nval rdd = sc.parallelize(Seq(1 to 10), 3)\n\nimport org.apache.spark.TaskContext\n\nval mapped = rdd.mapPartitionsWithIndex ( (index, iterator) => { if (index == 2) { Thread.sleep(30 * 1000); val attemptNum = TaskContext.get.attemptNumber; if (attemptNum < 3) throw new Exception(\"Fail for blacklisting\")};  iterator.toList.map (x => x + \" -> \" + index).iterator } )\n\nmapped.collect\n```\n\nCloses #22288 from dhruve/bug/SPARK-22148.\n\nLead-authored-by: Dhruve Ashar <dhruveashar@gmail.com>\nCo-authored-by: Dhruve Ashar <dhruve@users.noreply.github.com>\nCo-authored-by: Tom Graves <tgraves@apache.org>\nSigned-off-by: Thomas Graves <tgraves@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/fdd3bace1da01e5958fe0345c38e889e740ce25e"},{"sha":"6b425874d311146d8fbf7685c1b5d8e97d73b101","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-25866][ML] Update KMeans formatVersion\n\n## What changes were proposed in this pull request?\n\nWhen we added the `distanceMeasure`, we didn't update the `formatVersion` for `KMeans`. Despite this is not a big issue, as that information is used nowhere, we are returning a wrong information.\n\n## How was this patch tested?\n\nNA\n\nCloses #22873 from mgaido91/SPARK-25866.\n\nAuthored-by: Marco Gaido <marcogaido91@gmail.com>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/6b425874d311146d8fbf7685c1b5d8e97d73b101"},{"sha":"cee230160ba2c3a210892f71e019190b02e34071","author":{"name":"Imran Rashid","email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com"},"message":"[SPARK-25871][STREAMING] Don't use EC for streaming WAL\n\nThe write ahead log expects to be able to call hflush, but that is a no-op when writing to a file with hdfs erasure coding.  So ensure that file is always written with replication instead, regardless of filesystem defaults.\n\nNone yet.  I'm posting this mostly to make it visible.\n\nCloses #22882 from squito/SPARK-25871.\n\nAuthored-by: Imran Rashid <irashid@cloudera.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/cee230160ba2c3a210892f71e019190b02e34071"},{"sha":"a241a150d52b24ce952efab0830af4c0c9343c1b","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[MINOR] update known_translations\n\n## What changes were proposed in this pull request?\n\nupdate known_translations after running `translate-contributors.py` during 2.4.0 release\n\n## How was this patch tested?\n\nN/A\n\nCloses #22949 from cloud-fan/contributors.\n\nAuthored-by: Wenchen Fan <wenchen@databricks.com>\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/a241a150d52b24ce952efab0830af4c0c9343c1b"},{"sha":"63ca4bbe792718029f6d6196e8a6bb11d1f20fca","author":{"name":"yucai","email":"053c978f3c27c11ff61cd92f4d337d976848b8ce@ebay.com"},"message":"[SPARK-25676][SQL][TEST] Rename and refactor BenchmarkWideTable to use main method\n\n## What changes were proposed in this pull request?\n\nRefactor BenchmarkWideTable to use main method.\nGenerate benchmark result:\n\n```\nSPARK_GENERATE_BENCHMARK_FILES=1 build/sbt \"sql/test:runMain org.apache.spark.sql.execution.benchmark.WideTableBenchmark\"\n```\n\n## How was this patch tested?\n\nmanual tests\n\nCloses #22823 from yucai/BenchmarkWideTable.\n\nLead-authored-by: yucai <yyu1@ebay.com>\nCo-authored-by: Yucai Yu <yucai.yu@foxmail.com>\nCo-authored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/63ca4bbe792718029f6d6196e8a6bb11d1f20fca"},{"sha":"76813cfa1e2607ea3b669a79e59b568e96395b2e","author":{"name":"Maxim Gekk","email":"b91a34c712a29cd91cde3972985e114f1bb54d88@gmail.com"},"message":"[SPARK-25950][SQL] from_csv should respect to spark.sql.columnNameOfCorruptRecord\n\n## What changes were proposed in this pull request?\n\nFix for `CsvToStructs` to take into account SQL config `spark.sql.columnNameOfCorruptRecord` similar to `from_json`.\n\n## How was this patch tested?\n\nAdded new test where `spark.sql.columnNameOfCorruptRecord` is set to corrupt column name different from default.\n\nCloses #22956 from MaxGekk/csv-tests.\n\nAuthored-by: Maxim Gekk <max.gekk@gmail.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/76813cfa1e2607ea3b669a79e59b568e96395b2e"},{"sha":"9e9fa2f69f3fd8be34c8e99efcf6cf9db70a4cd0","author":{"name":"Yuming Wang","email":"a045e9e381e8d6ed98f3af6ea924f96c8dd7fbad@ebay.com"},"message":"[SPARK-25098][SQL] Trim the string when cast stringToTimestamp and stringToDate\n\n## What changes were proposed in this pull request?\n\n**Hive** and **Oracle** trim the string when cast `stringToTimestamp` and `stringToDate`. this PR support this feature:\n![image](https://user-images.githubusercontent.com/5399861/47979721-793b1e80-e0ff-11e8-97c8-24b10950ee9e.png)\n![image](https://user-images.githubusercontent.com/5399861/47979725-7dffd280-e0ff-11e8-87d4-5767a00ed46e.png)\n\n## How was this patch tested?\n\nunit tests\n\nCloses https://github.com/apache/spark/pull/22089\n\nCloses #22943 from wangyum/SPARK-25098.\n\nAuthored-by: Yuming Wang <yumwang@ebay.com>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/9e9fa2f69f3fd8be34c8e99efcf6cf9db70a4cd0"},{"sha":"8fbc1830f962c446b915d0d8ff2b13c5c75d22fc","author":{"name":"Imran Rashid","email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com"},"message":"[SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue\n\nJVMs can't allocate arrays of length exactly Int.MaxValue, so ensure we never try to allocate an array that big.  This commit changes some defaults & configs to gracefully fallover to something that doesn't require one large array in some cases; in other cases it simply improves an error message for cases which will still fail.\n\nCloses #22818 from squito/SPARK-25827.\n\nAuthored-by: Imran Rashid <irashid@cloudera.com>\nSigned-off-by: Imran Rashid <irashid@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/8fbc1830f962c446b915d0d8ff2b13c5c75d22fc"},{"sha":"0a32238d034e4d73d56ec69b7069b9d61e05582b","author":{"name":"koraseg","email":"9868cd13b00cabaed3fa29584c74fb649d45d289@gmail.com"},"message":"[SPARK-25885][CORE][MINOR] HighlyCompressedMapStatus deserialization/construction optimization\n\n## What changes were proposed in this pull request?\n\nRemoval of intermediate structures in HighlyCompressedMapStatus will speed up its creation and deserialization time.\n\nhttps://issues.apache.org/jira/browse/SPARK-25885\n\n## How was this patch tested?\n\nAdditional tests are not necessary for the patch.\n\nCloses #22894 from Koraseg/mapStatusesOptimization.\n\nAuthored-by: koraseg <artem.kupchinsky@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/0a32238d034e4d73d56ec69b7069b9d61e05582b"},{"sha":"e4561e1c552cdd7291254dc3787a70aadfb05f0a","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[SPARK-25897][K8S] Hook up k8s integration tests to sbt build.\n\nThe integration tests can now be run in sbt if the right profile\nis enabled, using the \"test\" task under the respective project.\n\nThis avoids having to fall back to maven to run the tests, which\ninvalidates all your compiled stuff when you go back to sbt, making\ndevelopment way slower than it should.\n\nThere's also a task to run the tests directly without refreshing\nthe docker images, which is helpful if you just made a change to\nthe submission code which should not affect the code in the images.\n\nThe sbt tasks currently are not very customizable; there's some\nvery minor things you can set in the sbt shell itself, but otherwise\nit's hardcoded to run on minikube.\n\nI also had to make some slight adjustments to the IT code itself,\nmostly to remove assumptions about the existing harness.\n\nTested on sbt and maven.\n\nCloses #22909 from vanzin/SPARK-25897.\n\nAuthored-by: Marcelo Vanzin <vanzin@cloudera.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/e4561e1c552cdd7291254dc3787a70aadfb05f0a"},{"sha":"a8e1c9815fef0deb45c9a516d415cea6be511415","author":{"name":"hyukjinkwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[SPARK-25962][BUILD][PYTHON] Specify minimum versions for both pydocstyle and flake8 in 'lint-python' script\n\n## What changes were proposed in this pull request?\n\nThis PR explicitly specifies `flake8` and `pydocstyle` versions.\n\n- It checks flake8 binary executable\n- flake8 version check >= 3.5.0\n- pydocstyle >= 3.0.0 (previously it was == 3.0.0)\n\n## How was this patch tested?\n\nManually tested.\n\nCloses #22963 from HyukjinKwon/SPARK-25962.\n\nAuthored-by: hyukjinkwon <gurwls223@apache.org>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/a8e1c9815fef0deb45c9a516d415cea6be511415"},{"sha":"0025a8397f8723011917239fe47518457d4d6860","author":{"name":"Sean Owen","email":"3f9d48d7c3f24f7009479c5fa630bd1131ffc724@databricks.com"},"message":"[SPARK-25908][CORE][SQL] Remove old deprecated items in Spark 3\n\n## What changes were proposed in this pull request?\n\n- Remove some AccumulableInfo .apply() methods\n- Remove non-label-specific multiclass precision/recall/fScore in favor of accuracy\n- Remove toDegrees/toRadians in favor of degrees/radians (SparkR: only deprecated)\n- Remove approxCountDistinct in favor of approx_count_distinct (SparkR: only deprecated)\n- Remove unused Python StorageLevel constants\n- Remove Dataset unionAll in favor of union\n- Remove unused multiclass option in libsvm parsing\n- Remove references to deprecated spark configs like spark.yarn.am.port\n- Remove TaskContext.isRunningLocally\n- Remove ShuffleMetrics.shuffle* methods\n- Remove BaseReadWrite.context in favor of session\n- Remove Column.!== in favor of =!=\n- Remove Dataset.explode\n- Remove Dataset.registerTempTable\n- Remove SQLContext.getOrCreate, setActive, clearActive, constructors\n\nNot touched yet\n\n- everything else in MLLib\n- HiveContext\n- Anything deprecated more recently than 2.0.0, generally\n\n## How was this patch tested?\n\nExisting tests\n\nCloses #22921 from srowen/SPARK-25908.\n\nLead-authored-by: Sean Owen <sean.owen@databricks.com>\nCo-authored-by: hyukjinkwon <gurwls223@apache.org>\nCo-authored-by: Sean Owen <srowen@gmail.com>\nSigned-off-by: Sean Owen <sean.owen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/0025a8397f8723011917239fe47518457d4d6860"},{"sha":"d68f3a726ffb4280d85268ef5a13b408b123ff48","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-25676][FOLLOWUP][BUILD] Fix Scala 2.12 build error\n\n## What changes were proposed in this pull request?\n\nThis PR fixes the Scala-2.12 build.\n\n## How was this patch tested?\n\nManual build with Scala-2.12 profile.\n\nCloses #22970 from dongjoon-hyun/SPARK-25676-2.12.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: DB Tsai <d_tsai@apple.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/d68f3a726ffb4280d85268ef5a13b408b123ff48"},{"sha":"17449a2e6b28ecce7a273284eab037e8aceb3611","author":{"name":"Maxim Gekk","email":"b91a34c712a29cd91cde3972985e114f1bb54d88@gmail.com"},"message":"[SPARK-25952][SQL] Passing actual schema to JacksonParser\n\n## What changes were proposed in this pull request?\n\nThe PR fixes an issue when the corrupt record column specified via `spark.sql.columnNameOfCorruptRecord` or JSON options `columnNameOfCorruptRecord` is propagated to JacksonParser, and returned row breaks an assumption in `FailureSafeParser` that the row must contain only actual data. The issue is fixed by passing actual schema without the corrupt record field into `JacksonParser`.\n\n## How was this patch tested?\n\nAdded a test with the corrupt record column in the middle of user's schema.\n\nCloses #22958 from MaxGekk/from_json-corrupt-record-schema.\n\nAuthored-by: Maxim Gekk <max.gekk@gmail.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/17449a2e6b28ecce7a273284eab037e8aceb3611"},{"sha":"ee03f760b305e70a57c3b4409ec25897af348600","author":{"name":"Maxim Gekk","email":"b91a34c712a29cd91cde3972985e114f1bb54d88@gmail.com"},"message":"[SPARK-25955][TEST] Porting JSON tests for CSV functions\n\n## What changes were proposed in this pull request?\n\nIn the PR, I propose to port existing JSON tests from `JsonFunctionsSuite` that are applicable for CSV, and put them to `CsvFunctionsSuite`. In particular:\n- roundtrip `from_csv` to `to_csv`, and `to_csv` to `from_csv`\n- using `schema_of_csv` in `from_csv`\n- Java API `from_csv`\n- using `from_csv` and `to_csv` in exprs.\n\nCloses #22960 from MaxGekk/csv-additional-tests.\n\nAuthored-by: Maxim Gekk <max.gekk@gmail.com>\nSigned-off-by: hyukjinkwon <gurwls223@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/ee03f760b305e70a57c3b4409ec25897af348600"},{"sha":"0a2e45fdb8baadf7a57eb06f319e96f95eedf298","author":{"name":"hyukjinkwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"Revert \"[SPARK-23831][SQL] Add org.apache.derby to IsolatedClientLoader\"\n\nThis reverts commit a75571b46f813005a6d4b076ec39081ffab11844.","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/0a2e45fdb8baadf7a57eb06f319e96f95eedf298"},{"sha":"a3004d084c654237c60d02df1507333b92b860c6","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-25971][SQL] Ignore partition byte-size statistics in SQLQueryTestSuite\n\n## What changes were proposed in this pull request?\n\nCurrently, `SQLQueryTestSuite` is sensitive in terms of the bytes of parquet files in table partitions. If we change the default file format (from Parquet to ORC) or update the metadata of them, the test case should be changed accordingly. This PR aims to make `SQLQueryTestSuite` more robust by ignoring the partition byte statistics.\n```\n-Partition Statistics   1144 bytes, 2 rows\n+Partition Statistics   [not included in comparison] bytes, 2 rows\n```\n\n## How was this patch tested?\n\nPass the Jenkins with the newly updated test cases.\n\nCloses #22972 from dongjoon-hyun/SPARK-25971.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/a3004d084c654237c60d02df1507333b92b860c6"},{"sha":"0d7396f3af2d4348ae53e6a274df952b7f17c37c","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMemoryError` in `HashAggregateExec`, too.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up pr of #20014 which introduced `SparkOutOfMemoryError` to avoid killing the entire executor when an `OutOfMemoryError` is thrown.\nWe should throw `SparkOutOfMemoryError` in `HashAggregateExec`, too.\n\n## How was this patch tested?\n\nExisting tests.\n\nCloses #22969 from ueshin/issues/SPARK-22827/oome.\n\nAuthored-by: Takuya UESHIN <ueshin@databricks.com>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/0d7396f3af2d4348ae53e6a274df952b7f17c37c"},{"sha":"6abe90625efeb8140531a875700e87ed7e981044","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-25676][SQL][FOLLOWUP] Use 'foreach(_ => ())'\n\n## What changes were proposed in this pull request?\n\n#22970 fixed Scala 2.12 build error, and this PR updates the function according to the review comments.\n\n## How was this patch tested?\n\nThis is also manually tested with Scala 2.12 build.\n\nCloses #22978 from dongjoon-hyun/SPARK-25676-3.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/6abe90625efeb8140531a875700e87ed7e981044"},{"sha":"7bb901aa28d3000c2e18cc769fe5769abd650770","author":{"name":"Gengliang Wang","email":"bfe1a7b36425bd89859a70a36586361d28ec35eb@databricks.com"},"message":"[SPARK-25964][SQL][MINOR] Revise OrcReadBenchmark/DataSourceReadBenchmark case names and execution instructions\n\n## What changes were proposed in this pull request?\n\n1. OrcReadBenchmark is under hive module, so the way to run it should be\n```\nbuild/sbt \"hive/test:runMain <this class>\"\n```\n\n2. The benchmark \"String with Nulls Scan\" should be with case \"String with Nulls Scan(5%/50%/95%)\", not \"(0.05%/0.5%/0.95%)\"\n\n3. Add the null value percentages in the test case names of DataSourceReadBenchmark, for the benchmark \"String with Nulls Scan\" .\n\n## How was this patch tested?\n\nRe-run benchmarks\n\nCloses #22965 from gengliangwang/fixHiveOrcReadBenchmark.\n\nLead-authored-by: Gengliang Wang <gengliang.wang@databricks.com>\nCo-authored-by: Gengliang Wang <ltnwgl@gmail.com>\nCo-authored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Dongjoon Hyun <dongjoon@apache.org>","distinct":true,"url":"https://api.github.com/repos/xuanyuanking/spark/commits/7bb901aa28d3000c2e18cc769fe5769abd650770"}]}
{"actor":{"display_login":"xuanyuanking"}}