{"push_id":2676307937,"size":54,"distinct_size":54,"ref":"refs/heads/SPARK-23914","head":"6f721f0f53361743e86980c8a45807d14a303666","before":"572c4b9bc92aec508417945f2b9bb7694b2284de","commits":[{"sha":"39dfaf2fd167cafc84ec9cc637c114ed54a331e3","author":{"name":"Hieu Huynh","email":"2e92adf85ffa4a9093a828fcda49bc22d6379f22@oath.com”"},"message":"[SPARK-24519] Make the threshold for highly compressed map status configurable\n\n**Problem**\nMapStatus uses hardcoded value of 2000 partitions to determine if it should use highly compressed map status. We should make it configurable to allow users to more easily tune their jobs with respect to this without having for them to modify their code to change the number of partitions.  Note we can leave this as an internal/undocumented config for now until we have more advise for the users on how to set this config.\nSome of my reasoning:\nThe config gives you a way to easily change something without the user having to change code, redeploy jar, and then run again. You can simply change the config and rerun. It also allows for easier experimentation. Changing the # of partitions has other side affects, whether good or bad is situation dependent. It can be worse are you could be increasing # of output files when you don't want to be, affects the # of tasks needs and thus executors to run in parallel, etc.\nThere have been various talks about this number at spark summits where people have told customers to increase it to be 2001 partitions. Note if you just do a search for spark 2000 partitions you will fine various things all talking about this number.  This shows that people are modifying their code to take this into account so it seems to me having this configurable would be better.\nOnce we have more advice for users we could expose this and document information on it.\n\n**What changes were proposed in this pull request?**\nI make the hardcoded value mentioned above to be configurable under the name _SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_COMPRESS_, which has default value to be 2000. Users can set it to the value they want by setting the property name _spark.shuffle.minNumPartitionsToHighlyCompress_\n\n**How was this patch tested?**\nI wrote a unit test to make sure that the default value is 2000, and  _IllegalArgumentException_ will be thrown if user set it to a non-positive value. The unit test also checks that highly compressed map status is correctly used when the number of partition is greater than _SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_COMPRESS_.\n\nAuthor: Hieu Huynh <“Hieu.huynh@oath.com”>\n\nCloses #21527 from hthuynh2/spark_branch_1.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/39dfaf2fd167cafc84ec9cc637c114ed54a331e3"},{"sha":"33e77fa89b5805ecb1066fc534723527f70d37c7","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[SPARK-24518][CORE] Using Hadoop credential provider API to store password\n\n## What changes were proposed in this pull request?\n\nIn our distribution,  because we don't do such fine-grained access control of config file, also configuration file is world readable shared between different components, so password may leak to different users.\n\nHadoop credential provider API support storing password in a secure way, in which Spark could read it in a secure way, so here propose to add support of using credential provider API to get password.\n\n## How was this patch tested?\n\nAdding tests and verified locally.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #21548 from jerryshao/SPARK-24518.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/33e77fa89b5805ecb1066fc534723527f70d37c7"},{"sha":"4e7d8678a3d9b12797d07f5497e0ed9e471428dd","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[SPARK-24372][BUILD] Add scripts to help with preparing releases.\n\nThe \"do-release.sh\" script asks questions about the RC being prepared,\ntrying to find out as much as possible automatically, and then executes\nthe existing scripts with proper arguments to prepare the release. This\nscript was used to prepare the 2.3.1 release candidates, so was tested\nin that context.\n\nThe docker version runs that same script inside a docker image especially\ncrafted for building Spark releases. That image is based on the work\nby Felix C. linked in the bug. At this point is has been only midly\ntested.\n\nI also added a template for the vote e-mail, with placeholders for\nthings that need to be replaced, although there is no automation around\nthat for the moment. It shouldn't be hard to hook up certain things like\nversion and tags to this, or to figure out certain things like the\nrepo URL from the output of the release scripts.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #21515 from vanzin/SPARK-24372.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/4e7d8678a3d9b12797d07f5497e0ed9e471428dd"},{"sha":"c7e2742f9bce2fcb7c717df80761939272beff54","author":{"name":"Maxim Gekk","email":"05b3389ec46e7c78a10351def567499e4feebe3f@databricks.com"},"message":"[SPARK-24190][SQL] Allow saving of JSON files in UTF-16 and UTF-32\n\n## What changes were proposed in this pull request?\n\nCurrently, restrictions in JSONOptions for `encoding` and `lineSep` are the same for read and for write. For example, a requirement for `lineSep` in the code:\n\n```\ndf.write.option(\"encoding\", \"UTF-32BE\").json(file)\n```\ndoesn't allow to skip `lineSep` and use its default value `\\n` because it throws the exception:\n```\nequirement failed: The lineSep option must be specified for the UTF-32BE encoding\njava.lang.IllegalArgumentException: requirement failed: The lineSep option must be specified for the UTF-32BE encoding\n```\n\nIn the PR, I propose to separate JSONOptions in read and write, and make JSONOptions in write less restrictive.\n\n## How was this patch tested?\n\nAdded new test for blacklisted encodings in read. And the `lineSep` option was removed in write for some tests.\n\nAuthor: Maxim Gekk <maxim.gekk@databricks.com>\nAuthor: Maxim Gekk <max.gekk@gmail.com>\n\nCloses #21247 from MaxGekk/json-options-in-write.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/c7e2742f9bce2fcb7c717df80761939272beff54"},{"sha":"98f363b77488792009f5b97bf831ede280f232e2","author":{"name":"Takeshi Yamamuro","email":"9770f4f4b25a32d3dfa34670a10767defeb974f3@apache.org"},"message":"[SPARK-24206][SQL] Improve FilterPushdownBenchmark benchmark code\n\n## What changes were proposed in this pull request?\nThis pr added benchmark code `FilterPushdownBenchmark` for string pushdown and updated performance results on the AWS `r3.xlarge`.\n\n## How was this patch tested?\nN/A\n\nAuthor: Takeshi Yamamuro <yamamuro@apache.org>\n\nCloses #21288 from maropu/UpdateParquetBenchmark.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/98f363b77488792009f5b97bf831ede280f232e2"},{"sha":"a5849ad9a3e5d41b5938faa7c592bcc6aec36044","author":{"name":"Bryan Cutler","email":"060dd14c05a8ffa8f201883ef57d0676b7c9c04a@gmail.com"},"message":"[SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name\n\n## What changes were proposed in this pull request?\n\nCurrently, a `pandas_udf` of type `PandasUDFType.GROUPED_MAP` will assign the resulting columns based on index of the return pandas.DataFrame.  If a new DataFrame is returned and constructed using a dict, then the order of the columns could be arbitrary and be different than the defined schema for the UDF.  If the schema types still match, then no error will be raised and the user will see column names and column data mixed up.\n\nThis change will first try to assign columns using the return type field names.  If a KeyError occurs, then the column index is checked if it is string based. If so, then the error is raised as it is most likely a naming mistake, else it will fallback to assign columns by position and raise a TypeError if the field types do not match.\n\n## How was this patch tested?\n\nAdded a test that returns a new DataFrame with column order different than the schema.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #21427 from BryanCutler/arrow-grouped-map-mixesup-cols-SPARK-24324.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/a5849ad9a3e5d41b5938faa7c592bcc6aec36044"},{"sha":"f596ebe4d3170590b6fce34c179e51ee80c965d3","author":{"name":"Takeshi Yamamuro","email":"9770f4f4b25a32d3dfa34670a10767defeb974f3@apache.org"},"message":"[SPARK-24327][SQL] Verify and normalize a partition column name based on the JDBC resolved schema\n\n## What changes were proposed in this pull request?\nThis pr modified JDBC datasource code to verify and normalize a partition column based on the JDBC resolved schema before building `JDBCRelation`.\n\nCloses #20370\n\n## How was this patch tested?\nAdded tests in `JDBCSuite`.\n\nAuthor: Takeshi Yamamuro <yamamuro@apache.org>\n\nCloses #21379 from maropu/SPARK-24327.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/f596ebe4d3170590b6fce34c179e51ee80c965d3"},{"sha":"6e0596e2639117d7a0a58b644b0600086f45c7f9","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-23931][SQL][FOLLOW-UP] Make `arrays_zip` in function.scala `@scala.annotation.varargs`.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up pr of #21045 which added `arrays_zip`.\nThe `arrays_zip` in functions.scala should've been `scala.annotation.varargs`.\n\nThis pr makes it `scala.annotation.varargs`.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #21630 from ueshin/issues/SPARK-23931/fup1.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/6e0596e2639117d7a0a58b644b0600086f45c7f9"},{"sha":"8ab8ef7733b42e94f687a5520332814ac9caeda8","author":{"name":"Jim Kleckner","email":"1cd02e31b43620d7c664e038ca42a060d61727b9@cloudphysics.com"},"message":"Fix minor typo in docs/cloud-integration.md\n\n## What changes were proposed in this pull request?\n\nMinor typo in docs/cloud-integration.md\n\n## How was this patch tested?\n\nThis is trivial enough that it should not affect tests.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Jim Kleckner <jim@cloudphysics.com>\n\nCloses #21629 from jkleckner/fix-doc-typo.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/8ab8ef7733b42e94f687a5520332814ac9caeda8"},{"sha":"bac50aa37168a7612702a4503750a78ed5d59c78","author":{"name":"Maryann Xue","email":"a27c3be666bde3a2f598c62e1c810c9c07319551@apache.org"},"message":"[SPARK-24596][SQL] Non-cascading Cache Invalidation\n\n## What changes were proposed in this pull request?\n\n1. Add parameter 'cascade' in CacheManager.uncacheQuery(). Under 'cascade=false' mode, only invalidate the current cache, and for other dependent caches, rebuild execution plan and reuse cached buffer.\n2. Pass true/false from callers in different uncache scenarios:\n- Drop tables and regular (persistent) views: regular mode\n- Drop temporary views: non-cascading mode\n- Modify table contents (INSERT/UPDATE/MERGE/DELETE): regular mode\n- Call `DataSet.unpersist()`: non-cascading mode\n- Call `Catalog.uncacheTable()`: follow the same convention as drop tables/view, which is, use non-cascading mode for temporary views and regular mode for the rest\n\nNote that a regular (persistent) view is a database object just like a table, so after dropping a regular view (whether cached or not cached), any query referring to that view should no long be valid. Hence if a cached persistent view is dropped, we need to invalidate the all dependent caches so that exceptions will be thrown for any later reference. On the other hand, a temporary view is in fact equivalent to an unnamed DataSet, and dropping a temporary view should have no impact on queries referencing that view. Thus we should do non-cascading uncaching for temporary views, which also guarantees a consistent uncaching behavior between temporary views and unnamed DataSets.\n\n## How was this patch tested?\n\nNew tests in CachedTableSuite and DatasetCacheSuite.\n\nAuthor: Maryann Xue <maryannxue@apache.org>\n\nCloses #21594 from maryannxue/noncascading-cache.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/bac50aa37168a7612702a4503750a78ed5d59c78"},{"sha":"594ac4f7b816488091202918c409487058e6d8ac","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-24633][SQL] Fix codegen when split is required for arrays_zip\n\n## What changes were proposed in this pull request?\n\nIn function array_zip, when split is required by the high number of arguments, a codegen error can happen.\n\nThe PR fixes codegen for cases when splitting the code is required.\n\n## How was this patch tested?\n\nadded UT\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #21621 from mgaido91/SPARK-24633.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/594ac4f7b816488091202918c409487058e6d8ac"},{"sha":"5264164a67df498b73facae207eda12ee133be7d","author":{"name":"Stacy Kerkela","email":"b9eed989afa03314bda8893bdbc86061af52f24d@databricks.com"},"message":"[SPARK-24648][SQL] SqlMetrics should be threadsafe\n\nUse LongAdder to make SQLMetrics thread safe.\n\n## What changes were proposed in this pull request?\nReplace += with LongAdder.add() for concurrent counting\n\n## How was this patch tested?\nUnit tests with local threads\n\nAuthor: Stacy Kerkela <stacy.kerkela@databricks.com>\n\nCloses #21634 from dbkerkela/sqlmetrics-concurrency-stacy.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/5264164a67df498b73facae207eda12ee133be7d"},{"sha":"baa01c8ca9e8ea456f986fbb223c61ad541b52b0","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[INFRA] Close stale PR.\n\nCloses #21614","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/baa01c8ca9e8ea456f986fbb223c61ad541b52b0"},{"sha":"6d16b9885d6ad01e1cc56d5241b7ebad99487a0c","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[SPARK-24552][CORE][SQL] Use task ID instead of attempt number for writes.\n\nThis passes the unique task attempt id instead of attempt number to v2 data sources because attempt number is reused when stages are retried. When attempt numbers are reused, sources that track data by partition id and attempt number may incorrectly clean up data because the same attempt number can be both committed and aborted.\n\nFor v1 / Hadoop writes, generate a unique ID based on available attempt numbers to avoid a similar problem.\n\nCloses #21558\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\nAuthor: Ryan Blue <blue@apache.org>\n\nCloses #21606 from vanzin/SPARK-24552.2.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/6d16b9885d6ad01e1cc56d5241b7ebad99487a0c"},{"sha":"d48803bf64dc0fccd6f560738b4682f0c05e767a","author":{"name":"Bryan Cutler","email":"060dd14c05a8ffa8f201883ef57d0676b7c9c04a@gmail.com"},"message":"[SPARK-24324][PYTHON][FOLLOWUP] Grouped Map positional conf should have deprecation note\n\n## What changes were proposed in this pull request?\n\nFollowup to the discussion of the added conf in SPARK-24324 which allows assignment by column position only.  This conf is to preserve old behavior and will be removed in future releases, so it should have a note to indicate that.\n\n## How was this patch tested?\n\nNA\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #21637 from BryanCutler/arrow-groupedMap-conf-deprecate-followup-SPARK-24324.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/d48803bf64dc0fccd6f560738b4682f0c05e767a"},{"sha":"4c059ebc6008b4e78cbebc87a421cb87d1b800ed","author":{"name":"Bruce Robbins","email":"9fa1b8dc62c4e42f4b64eedaa587e59466a4067a@gmail.com"},"message":"[SPARK-23776][DOC] Update instructions for running PySpark after building with SBT\n\n## What changes were proposed in this pull request?\n\nThis update tells the reader how to build Spark with SBT such that pyspark-sql tests will succeed.\n\nIf you follow the current instructions for building Spark with SBT, pyspark/sql/udf.py fails with:\n<pre>\nAnalysisException: u'Can not load class test.org.apache.spark.sql.JavaStringLength, please make sure it is on the classpath;'\n</pre>\n\n## How was this patch tested?\n\nI ran the doc build command (SKIP_API=1 jekyll build) and eyeballed the result.\n\nAuthor: Bruce Robbins <bersprockets@gmail.com>\n\nCloses #21628 from bersprockets/SPARK-23776_doc.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/4c059ebc6008b4e78cbebc87a421cb87d1b800ed"},{"sha":"c7967c6049327a03b63ea7a3b0001a97d31e309d","author":{"name":"DB Tsai","email":"0430fe8116355e2ecfa7f2b79341c5ad5a243c66@apple.com"},"message":"[SPARK-24418][BUILD] Upgrade Scala to 2.11.12 and 2.12.6\n\n## What changes were proposed in this pull request?\n\nScala is upgraded to `2.11.12` and `2.12.6`.\n\nWe used `loadFIles()` in `ILoop` as a hook to initialize the Spark before REPL sees any files in Scala `2.11.8`. However, it was a hack, and it was not intended to be a public API, so it was removed in Scala `2.11.12`.\n\nFrom the discussion in Scala community, https://github.com/scala/bug/issues/10913 , we can use `initializeSynchronous` to initialize Spark instead. This PR implements the Spark initialization there.\n\nHowever, in Scala `2.11.12`'s `ILoop.scala`, in function `def startup()`, the first thing it calls is `printWelcome()`. As a result, Scala will call `printWelcome()` and `splash` before calling `initializeSynchronous`.\n\nThus, the Spark shell will allow users to type commends first, and then show the Spark UI URL. It's working, but it will change the Spark Shell interface as the following.\n\n```scala\n➜  apache-spark git:(scala-2.11.12) ✗ ./bin/spark-shell\nSetting default log level to \"WARN\".\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\nWelcome to\n      ____              __\n     / __/__  ___ _____/ /__\n    _\\ \\/ _ \\/ _ `/ __/  '_/\n   /___/ .__/\\_,_/_/ /_/\\_\\   version 2.4.0-SNAPSHOT\n      /_/\n\nUsing Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)\nType in expressions to have them evaluated.\nType :help for more information.\n\nscala> Spark context Web UI available at http://192.168.1.169:4040\nSpark context available as 'sc' (master = local[*], app id = local-1528180279528).\nSpark session available as 'spark'.\n\nscala>\n```\n\nIt seems there is no easy way to inject the Spark initialization code in the proper place as Scala doesn't provide a hook. Maybe som-snytt can comment on this.\n\nThe following command is used to update the dep files.\n```scala\n./dev/test-dependencies.sh --replace-manifest\n```\n## How was this patch tested?\n\nExisting tests\n\nAuthor: DB Tsai <d_tsai@apple.com>\n\nCloses #21495 from dbtsai/scala-2.11.12.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/c7967c6049327a03b63ea7a3b0001a97d31e309d"},{"sha":"e07aee2165af4d301ae12005a6d9ffb030bc2650","author":{"name":"Marek Novotny","email":"51a58b3768f62be18290af0bb1b9e2d6d3d258c5@gmail.com"},"message":"[SPARK-24636][SQL] Type coercion of arrays for array_join function\n\n## What changes were proposed in this pull request?\nPresto's implementation accepts arbitrary arrays of primitive types as an input:\n\n```\npresto> SELECT array_join(ARRAY [1, 2, 3], ', ');\n_col0\n---------\n1, 2, 3\n(1 row)\n```\n\nThis PR proposes to implement a type coercion rule for ```array_join``` function that converts arrays of primitive as well as non-primitive types to arrays of string.\n\n## How was this patch tested?\n\nNew test cases add into:\n- sql-tests/inputs/typeCoercion/native/arrayJoin.sql\n- DataFrameFunctionsSuite.scala\n\nAuthor: Marek Novotny <mn.mikke@gmail.com>\n\nCloses #21620 from mn-mikke/SPARK-24636.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/e07aee2165af4d301ae12005a6d9ffb030bc2650"},{"sha":"dcaa49ff1edd7fcf0f000c6f93ae0e30bd5b6464","author":{"name":"Yuming Wang","email":"a045e9e381e8d6ed98f3af6ea924f96c8dd7fbad@ebay.com"},"message":"[SPARK-24658][SQL] Remove workaround for ANTLR bug\n\n## What changes were proposed in this pull request?\n\nIssue antlr/antlr4#781 has already been fixed, so the workaround of extracting the pattern into a separate rule is no longer needed. The presto already removed it: https://github.com/prestodb/presto/pull/10744.\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Yuming Wang <yumwang@ebay.com>\n\nCloses #21641 from wangyum/ANTLR-780.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/dcaa49ff1edd7fcf0f000c6f93ae0e30bd5b6464"},{"sha":"02f8781fa2649cf1d3a5cb932e1c8408790974ff","author":{"name":"Dilip Biswal","email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com"},"message":"[SPARK-24423][SQL] Add a new option  for JDBC sources\n\n## What changes were proposed in this pull request?\nHere is the description in the JIRA -\n\nCurrently, our JDBC connector provides the option `dbtable` for users to specify the to-be-loaded JDBC source table.\n\n ```SQL\n val jdbcDf = spark.read\n   .format(\"jdbc\")\n   .option(\"dbtable\", \"dbName.tableName\")\n   .options(jdbcCredentials: Map)\n   .load()\n ```\n\nNormally, users do not fetch the whole JDBC table due to the poor performance/throughput of JDBC. Thus, they normally just fetch a small set of tables. For advanced users, they can pass a subquery as the option.\n\n ```SQL\n val query = \"\"\" (select * from tableName limit 10) as tmp \"\"\"\n val jdbcDf = spark.read\n   .format(\"jdbc\")\n   .option(\"dbtable\", query)\n   .options(jdbcCredentials: Map)\n   .load()\n ```\nHowever, this is straightforward to end users. We should simply allow users to specify the query by a new option `query`. We will handle the complexity for them.\n\n ```SQL\n val query = \"\"\"select * from tableName limit 10\"\"\"\n val jdbcDf = spark.read\n   .format(\"jdbc\")\n   .option(\"query\", query)\n   .options(jdbcCredentials: Map)\n   .load()\n```\n\n## How was this patch tested?\nAdded tests in JDBCSuite and JDBCWriterSuite.\nAlso tested against MySQL, Postgress, Oracle, DB2 (using docker infrastructure) to make sure there are no syntax issues.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #21590 from dilipbiswal/SPARK-24423.","distinct":true,"url":"https://api.github.com/repos/kiszk/spark/commits/02f8781fa2649cf1d3a5cb932e1c8408790974ff"}]}
{"actor":{"display_login":"kiszk"}}