{"push_id":2669092698,"size":24,"distinct_size":24,"ref":"refs/heads/update-upstream","head":"68bc638d2fc059b474fc17a2ad1d6064421bee42","before":"42d8c08f081cd920a58687ab56e524d261504f45","commits":[{"sha":"de4aafd83365fd69c1fb7b49824e62c9cc1c4a49","author":{"name":"“attilapiros”","email":"4670cc73e44a0e3205b79a62530d1afa1c143320@gmail.com"},"message":"[SPARK-19181][CORE] Fixing flaky \"SparkListenerSuite.local metrics\"\n\n## What changes were proposed in this pull request?\n\nSometimes \"SparkListenerSuite.local metrics\" test fails because the average of executorDeserializeTime is too short. As squito suggested to avoid these situations in one of the task a reference introduced to an object implementing a custom Externalizable.readExternal which sleeps 1ms before returning.\n\n## How was this patch tested?\n\nWith unit tests (and checking the effect of this change to the average with a much larger sleep time).\n\nAuthor: “attilapiros” <piros.attila.zsolt@gmail.com>\nAuthor: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>\n\nCloses #21280 from attilapiros/SPARK-19181.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/de4aafd83365fd69c1fb7b49824e62c9cc1c4a49"},{"sha":"a6e0f43988f22bc3dce2e3ca6c5674be4144c7bf","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-10878][CORE] Fix race condition when multiple clients resolves artifacts at the same time\n\n## What changes were proposed in this pull request?\n\nWhen multiple clients attempt to resolve artifacts via the `--packages` parameter, they could run into race condition when they each attempt to modify the dummy `org.apache.spark-spark-submit-parent-default.xml` file created in the default ivy cache dir.\nThis PR changes the behavior to encode UUID in the dummy module descriptor so each client will operate on a different resolution file in the ivy cache dir. In addition, this patch changes the behavior of when and which resolution files are cleaned to prevent accumulation of resolution files in the default ivy cache dir.\n\nSince this PR is a successor of #18801, close #18801. Many codes were ported from #18801. **Many efforts were put here. I think this PR should credit to Victsm .**\n\n## How was this patch tested?\n\nadded UT into `SparkSubmitUtilsSuite`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #21251 from kiszk/SPARK-10878.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/a6e0f43988f22bc3dce2e3ca6c5674be4144c7bf"},{"sha":"e5fd1ba1050744f9e29f917698a0dfb2fd33b9e6","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[SPARK-22938][SQL][FOLLOWUP] Assert that SQLConf.get is accessed only on the driver\n\n## What changes were proposed in this pull request?\n\nThis is a followup of https://github.com/apache/spark/pull/20136 . #20136 didn't really work because in the test, we are using local backend, which shares the driver side `SparkEnv`, so `SparkEnv.get.executorId == SparkContext.DRIVER_IDENTIFIER` doesn't work.\n\nThis PR changes the check to `TaskContext.get != null`, and move the check to `SQLConf.get`, and fix all the places that violate this check:\n* `InMemoryTableScanExec#createAndDecompressColumn` is executed inside `rdd.map`, we can't access `conf.offHeapColumnVectorEnabled` there. https://github.com/apache/spark/pull/21223 merged\n* `DataType#sameType` may be executed in executor side, for things like json schema inference, so we can't call `conf.caseSensitiveAnalysis` there. This contributes to most of the code changes, as we need to add `caseSensitive` parameter to a lot of methods.\n* `ParquetFilters` is used in the file scan function, which is executed in executor side, so we can't can't call `conf.parquetFilterPushDownDate` there. https://github.com/apache/spark/pull/21224 merged\n* `WindowExec#createBoundOrdering` is called on executor side, so we can't use `conf.sessionLocalTimezone` there. https://github.com/apache/spark/pull/21225 merged\n* `JsonToStructs` can be serialized to executors and evaluate, we should not call `SQLConf.get.getConf(SQLConf.FROM_JSON_FORCE_NULLABLE_SCHEMA)` in the body. https://github.com/apache/spark/pull/21226 merged\n\n## How was this patch tested?\n\nexisting test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #21190 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/e5fd1ba1050744f9e29f917698a0dfb2fd33b9e6"},{"sha":"a4ba973429cd452dee37e12dc02749fc37606f58","author":{"name":"Marek Novotny","email":"51a58b3768f62be18290af0bb1b9e2d6d3d258c5@gmail.com"},"message":"[SPARK-24197][SPARKR][SQL] Adding array_sort function to SparkR\n\n## What changes were proposed in this pull request?\n\nThe PR adds array_sort function to SparkR.\n\n## How was this patch tested?\n\nTests added into R/pkg/tests/fulltests/test_sparkSQL.R\n\n## Example\n```\n> df <- createDataFrame(list(list(list(2L, 1L, 3L, NA)), list(list(NA, 6L, 5L, NA, 4L))))\n> head(collect(select(df, array_sort(df[[1]]))))\n```\nResult:\n```\n   array_sort(_1)\n1     1, 2, 3, NA\n2 4, 5, 6, NA, NA\n```\n\nAuthor: Marek Novotny <mn.mikke@gmail.com>\n\nCloses #21294 from mn-mikke/SPARK-24197.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/a4ba973429cd452dee37e12dc02749fc37606f58"},{"sha":"3cc98c660efd61c8e137ea666cf351bd61f462c5","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[SPARK-24182][YARN] Improve error message when client AM fails.\n\nInstead of always throwing a generic exception when the AM fails,\nprint a generic error and throw the exception with the YARN\ndiagnostics containing the reason for the failure.\n\nThere was an issue with YARN sometimes providing a generic diagnostic\nmessage, even though the AM provides a failure reason when\nunregistering. That was happening because the AM was registering\ntoo late, and if errors happened before the registration, YARN would\njust create a generic \"ExitCodeException\" which wasn't very helpful.\n\nSince most errors in this path are a result of not being able to\nconnect to the driver, this change modifies the AM registration\na bit so that the AM is registered before the connection to the\ndriver is established. That way, errors are properly propagated\nthrough YARN back to the driver.\n\nAs part of that, I also removed the code that retried connections\nto the driver from the client AM. At that point, the driver should\nalready be up and waiting for connections, so it's unlikely that\nretrying would help - and in case it does, that means a flaky\nnetwork, which would mean problems would probably show up again.\nThe effect of that is that connection-related errors are reported\nback to the driver much faster now (through the YARN report).\n\nOne thing to note is that there seems to be a race on the YARN\nside that causes a report to be sent to the client without the\ncorresponding diagnostics string from the AM; the diagnostics are\navailable later from the RM web page. For that reason, the generic\nerror messages are kept in the Spark scheduler code, to help\nguide users to a way of debugging their failure.\n\nAlso of note is that if YARN's max attempts configuration is lower\nthan Spark's, Spark will not unregister the AM with a proper\ndiagnostics message. Unfortunately there seems to be no way to\nunregister the AM and still allow further re-attempts to happen.\n\nTesting:\n- existing unit tests\n- some of our integration tests\n- hardcoded an invalid driver address in the code and verified\n  the error in the shell. e.g.\n\n```\nscala> 18/05/04 15:09:34 ERROR cluster.YarnClientSchedulerBackend: YARN application has exited unexpectedly with state FAILED! Check the YARN application logs for more details.\n18/05/04 15:09:34 ERROR cluster.YarnClientSchedulerBackend: Diagnostics message: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult:\n  <AM stack trace>\nCaused by: java.io.IOException: Failed to connect to localhost/127.0.0.1:1234\n  <More stack trace>\n```\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #21243 from vanzin/SPARK-24182.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/3cc98c660efd61c8e137ea666cf351bd61f462c5"},{"sha":"8b8597b8a7ac98f09e31219177a0ef42db4c3e7d","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[SPARK-24172][SQL] we should not apply operator pushdown to data source v2 many times\n\n## What changes were proposed in this pull request?\n\nIn `PushDownOperatorsToDataSource`, we use `transformUp` to match `PhysicalOperation` and apply pushdown. This is problematic if we have multiple `Filter` and `Project` above the data source v2 relation.\n\ne.g. for a query\n```\nProject\n  Filter\n    DataSourceV2Relation\n```\n\nThe pattern match will be triggered twice and we will do operator pushdown twice. This is unnecessary, we can use `mapChildren` to only apply pushdown once.\n\n## How was this patch tested?\n\nexisting test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #21230 from cloud-fan/step2.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/8b8597b8a7ac98f09e31219177a0ef42db4c3e7d"},{"sha":"a46cf7b2d7fdc1a539761bc0b83e100839c8602a","author":{"name":"aditkumar","email":"4e7368981a976b2768936e85ac074dbe56d38368@gmail.com"},"message":"[MINOR][DOCS] Documenting months_between direction\n\n## What changes were proposed in this pull request?\n\nIt's useful to know what relationship between date1 and date2 results in a positive number.\n\nAuthor: aditkumar <aditkumar@gmail.com>\nAuthor: Adit Kumar <aditkumar@gmail.com>\n\nCloses #20787 from aditkumar/master.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/a46cf7b2d7fdc1a539761bc0b83e100839c8602a"},{"sha":"3a2a8599cbf49fe034b50db79ddda15077969690","author":{"name":"Shivaram Venkataraman","email":"ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu"},"message":"[SPARKR] Require Java 8 for SparkR\n\nThis change updates the SystemRequirements and also includes a runtime check if the JVM is being launched by R. The runtime check is done by querying `java -version`\n\n## How was this patch tested?\n\nTested on a Mac and Windows machine\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #21278 from shivaram/sparkr-skip-solaris.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/3a2a8599cbf49fe034b50db79ddda15077969690"},{"sha":"37b5550e272b33f09f8b1f3cbe8aa209c71662d1","author":{"name":"Reynold Xin","email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com"},"message":"[SPARK-23907] Removes regr_* functions in functions.scala\n\n## What changes were proposed in this pull request?\nThis patch removes the various regr_* functions in functions.scala. They are so uncommon that I don't think they deserve real estate in functions.scala. We can consider adding them later if more users need them.\n\n## How was this patch tested?\nRemoved the associated test case as well.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #21309 from rxin/SPARK-23907.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/37b5550e272b33f09f8b1f3cbe8aa209c71662d1"},{"sha":"0cfdbe7188d10a5317d54938e88a39849f37b8aa","author":{"name":"Marek Novotny","email":"51a58b3768f62be18290af0bb1b9e2d6d3d258c5@gmail.com"},"message":"[SPARK-24198][SPARKR][SQL] Adding slice function to SparkR\n\n## What changes were proposed in this pull request?\nThe PR adds the `slice` function to SparkR. The function returns a subset of consecutive elements from the given array.\n```\n> df <- createDataFrame(cbind(model = rownames(mtcars), mtcars))\n> tmp <- mutate(df, v1 = create_array(df$mpg, df$cyl, df$hp))\n> head(select(tmp, slice(tmp$v1, 2L, 2L)))\n```\n```\n  slice(v1, 2, 2)\n1          6, 110\n2          6, 110\n3           4, 93\n4          6, 110\n5          8, 175\n6          6, 105\n```\n\n## How was this patch tested?\n\nA test added into R/pkg/tests/fulltests/test_sparkSQL.R\n\nAuthor: Marek Novotny <mn.mikke@gmail.com>\n\nCloses #21298 from mn-mikke/SPARK-24198.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/0cfdbe7188d10a5317d54938e88a39849f37b8aa"},{"sha":"846af732d9732983803dbac19ead7e39be809598","author":{"name":"Cody Allen","email":"86c9c8dfe5f3706d9d6d490b9842f9e8f15b155b@gmail.com"},"message":"Improve implicitNotFound message for Encoder\n\nThe `implicitNotFound` message for `Encoder` doesn't mention the name of\nthe type for which it can't find an encoder. Furthermore, it covers up\nthe fact that `Encoder` is the name of the relevant type class.\nHopefully this new message provides a little more specific type detail\nwhile still giving the general message about which types are supported.\n\n## What changes were proposed in this pull request?\n\nAugment the existing message to mention that it's looking for an `Encoder` and what the type of the encoder is.\n\nFor example instead of:\n\n```\nUnable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.\n```\n\nreturn this message:\n\n```\nUnable to find encoder for type Exception. An implicit Encoder[Exception] is needed to store Exception instances in a Dataset. Primitive types (Int, String, etc) and Product types (ca\nse classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.\n```\n\n## How was this patch tested?\n\nIt was tested manually in the Scala REPL, since triggering this in a test would cause a compilation error.\n\n```\nscala> implicitly[Encoder[Exception]]\n<console>:51: error: Unable to find encoder for type Exception. An implicit Encoder[Exception] is needed to store Exception instances in a Dataset. Primitive types (Int, String, etc) and Product types (ca\nse classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.\n       implicitly[Encoder[Exception]]\n                 ^\n```\n\nAuthor: Cody Allen <ceedubs@gmail.com>\n\nCloses #20869 from ceedubs/encoder-implicit-msg.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/846af732d9732983803dbac19ead7e39be809598"},{"sha":"3d2afefbdb87ed58ccb0caae50528e1c7fd30deb","author":{"name":"Kelley Robinson","email":"4770b181c28cfa642c1b65896b4a747fd9ed905a@twilio.com"},"message":"[SPARK-24262][PYTHON] Fix typo in UDF type match error message\n\n## What changes were proposed in this pull request?\n\nUpdates `functon` to `function`. This was called out in holdenk's PyCon 2018 conference talk. Didn't see any existing PR's for this.\n\nholdenk happy to fix the Pandas.Series bug too but will need a bit more guidance.\n\nAuthor: Kelley Robinson <krobinson@twilio.com>\n\nCloses #21304 from robinske/master.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/3d2afefbdb87ed58ccb0caae50528e1c7fd30deb"},{"sha":"1b4756629f20122c35368e48d3685fc8056dca61","author":{"name":"Fan Donglai","email":"da5f8ca3406da1d61e001c098dd1d462db99ada5@163.com"},"message":"Update StreamingKMeans.scala\n\n## What changes were proposed in this pull request?\n\nI think the ‘n_t+t’ in the following code may be wrong, it shoud be ‘n_t+1’ that means  is the number of points to the cluster  after it finish the no.t+1 min-batch.\n\n * <blockquote>\n *    $$\n *    \\begin{align}\n *     c_t+1 &= [(c_t * n_t * a) + (x_t * m_t)] / [n_t + m_t] \\\\\n *     n_t+t &= n_t * a + m_t\n *    \\end{align}\n *    $$\n * </blockquote>\n\nAuthor: Fan Donglai <ddna_1022@163.com>\n\nCloses #21179 from ddna1021/master.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/1b4756629f20122c35368e48d3685fc8056dca61"},{"sha":"a11ec233f79a8cfc973788ff52de809dea09f1e7","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-24186][R][SQL] change reverse and concat to collection functions in R\n\n## What changes were proposed in this pull request?\n\nreverse and concat are already in functions.R as column string functions. Since now these two functions are categorized as collection functions in scala and python, we will do the same in R.\n\n## How was this patch tested?\n\nAdd test in test_sparkSQL.R\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #21307 from huaxingao/spark_24186.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/a11ec233f79a8cfc973788ff52de809dea09f1e7"},{"sha":"371ac5fa4fdc2f450932000a84c80eb7c8da7c13","author":{"name":"Maxim Gekk","email":"05b3389ec46e7c78a10351def567499e4feebe3f@databricks.com"},"message":"[SPARK-17916][SQL] Fix empty string being parsed as null when nullValue is set.\n\nI propose to bump version of uniVocity parser up to 2.6.3 where quoted empty strings are replaced by the empty value (passed to `setEmptyValue`) instead of `null` values as in the current version 2.5.9:\nhttps://github.com/uniVocity/univocity-parsers/blob/v2.6.3/src/main/java/com/univocity/parsers/csv/CsvParser.java#L125\n\nEmpty value for writer is set to `\"\"`. So, empty string in dataframe/dataset is stored as empty quoted string `\"\"`. Empty value for reader is set to empty string (zero size). In this way, saved empty quoted string will be read as just empty string. Please, look at the tests for more details.\n\nHere are main changes made in [2.6.0](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.6.0), [2.6.1](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.6.1), [2.6.2](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.6.2), [2.6.3](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.6.3):\n\n- CSV parser now parses quoted values ~30% faster\n- CSV format detection process has option provide a list of possible delimiters, in order of priority ( i.e. settings.detectFormatAutomatically( '-', '.');) - https://github.com/uniVocity/univocity-parsers/issues/214\n- Implemented trim quoted values support - https://github.com/uniVocity/univocity-parsers/issues/230\n- NullPointer when stopping parser when nothing is parsed - https://github.com/uniVocity/univocity-parsers/issues/219\n- Concurrency issue when calling stopParsing() - https://github.com/uniVocity/univocity-parsers/issues/231\n\nCloses #20068\n\nAdded tests from the PR https://github.com/apache/spark/pull/20068\n\nAuthor: Maxim Gekk <maxim.gekk@databricks.com>\n\nCloses #21273 from MaxGekk/univocity-2.6.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/371ac5fa4fdc2f450932000a84c80eb7c8da7c13"},{"sha":"69b9e3dc4e17fe1d33808270054b1a03d9358c17","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-24228][SQL] Fix Java lint errors\n\nThis PR fixes the following Java lint errors due to importing unimport classes\n\n```\n$ dev/lint-java\nUsing `mvn` from path: /usr/bin/mvn\nCheckstyle checks failed at following occurrences:\n[ERROR] src/main/java/org/apache/spark/sql/sources/v2/reader/partitioning/Distribution.java:[25] (sizes) LineLength: Line is longer than 100 characters (found 109).\n[ERROR] src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/ContinuousReader.java:[38] (sizes) LineLength: Line is longer than 100 characters (found 102).\n[ERROR] src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java:[21,8] (imports) UnusedImports: Unused import - java.io.ByteArrayInputStream.\n[ERROR] src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedPlainValuesReader.java:[29,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.Platform.\n[ERROR] src/test/java/test/org/apache/spark/sql/sources/v2/JavaAdvancedDataSourceV2.java:[110] (sizes) LineLength: Line is longer than 100 characters (found 101).\n```\n\nWith this PR\n```\n$ dev/lint-java\nUsing `mvn` from path: /usr/bin/mvn\nCheckstyle checks passed.\n```\n\nExisting UTs. Also manually run checkstyles against these two files.\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #21301 from kiszk/SPARK-24228.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/69b9e3dc4e17fe1d33808270054b1a03d9358c17"},{"sha":"ae935d820d349d1112aa97acaa941622bac81732","author":{"name":"Robert Kruszewski","email":"34094693d149b7b2fde74f56f1ac1425b68971e2@palantir.com"},"message":"resolve conflicts","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/ae935d820d349d1112aa97acaa941622bac81732"},{"sha":"37b727ae97aacd5d56d0a0f11c85e85265f53dcc","author":{"name":"mcheah","email":"2e9bf58ec2f93170404b3a28538fb27400f699db@palantir.com"},"message":"Increase the max total size of files that can be added in k8s","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/37b727ae97aacd5d56d0a0f11c85e85265f53dcc"},{"sha":"a21b1c2b5d4ad1a9747c1cf0de5f9af92cd227ce","author":{"name":"Robert Kruszewski","email":"12e9293ec6b30c7fa8a0926af42807e929c1684f@robert.sx"},"message":"Upgrade jackson to 2.9.5 (#359)","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/a21b1c2b5d4ad1a9747c1cf0de5f9af92cd227ce"},{"sha":"c3eb923e53439b85f83fe92ae0b4ce680d73b68b","author":{"name":"Ruben Fiszel","email":"7cea60f360bb713c3e6089aee0fcdf45dc9df165@gmail.com"},"message":"[SPARK-24345][SQL] Improve ParseError stop location when offending symbol is a token (#375)\n\nIn the case where the offending symbol is a CommonToken, this PR increases the accuracy of the start and stop origin by leveraging the start and stop index information from CommonToken.","distinct":true,"url":"https://api.github.com/repos/palantir/spark/commits/c3eb923e53439b85f83fe92ae0b4ce680d73b68b"}]}
{"actor":{"display_login":"robert3005"}}