{"push_id":1429947394,"size":35,"distinct_size":1,"ref":"refs/heads/ENG-17719-lbfgs-only","head":"aab2422c0d9c55d380664d03f2285cb39b4f7291","before":"0a0571dc9557e94a909ab8d8ef62c7dca474a1c0","commits":[{"sha":"5ecdc7c5c019acc6b1f9c2e6c5b7d35957eadb88","author":{"name":"Zhenhua Wang","email":"e1bb7e32d2908d1daa3f64ed12aed842612a516d@163.com"},"message":"[SPARK-18559][SQL] Fix HLL++ with small relative error\n\n## What changes were proposed in this pull request?\n\nIn `HyperLogLogPlusPlus`, if the relative error is so small that p >= 19, it will cause ArrayIndexOutOfBoundsException in `THRESHOLDS(p-4)` . We should check `p` and when p >= 19, regress to the original HLL result and use the small range correction they use.\n\nThe pr also fixes the upper bound in the log info in `require()`.\nThe upper bound is computed by:\n```\nval relativeSD = 1.106d / Math.pow(Math.E, p * Math.log(2.0d) / 2.0d)\n```\nwhich is derived from the equation for computing `p`:\n```\nval p = 2.0d * Math.log(1.106d / relativeSD) / Math.log(2.0d)\n```\n\n## How was this patch tested?\n\nadd test cases for:\n1. checking validity of parameter relatvieSD\n2. estimation with smaller relative error so that p >= 19\n\nAuthor: Zhenhua Wang <wzh_zju@163.com>\nAuthor: wangzhenhua <wangzhenhua@huawei.com>\n\nCloses #15990 from wzhfy/hllppRsd.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/5ecdc7c5c019acc6b1f9c2e6c5b7d35957eadb88"},{"sha":"445d4d9e13ebaee9eceea6135fe7ee47812d97de","author":{"name":"Zakaria_Hili","email":"ed73de320d07e5a1a1cd4a62dafb038223244e5e@gmail.com"},"message":"[SPARK-18356][ML] Improve MLKmeans Performance\n\n## What changes were proposed in this pull request?\n\nSpark Kmeans fit() doesn't cache the RDD which generates a lot of warnings :\n WARN KMeans: The input data is not directly cached, which may hurt performance if its parent RDDs are also uncached.\nSo, Kmeans should cache the internal rdd before calling the Mllib.Kmeans algo, this helped to improve spark kmeans performance by 14%\n\nhttps://github.com/ZakariaHili/spark/commit/a9cf905cf7dbd50eeb9a8b4f891f2f41ea672472\n\nhhbyyh\n## How was this patch tested?\nPass Kmeans tests and existing tests\n\nAuthor: Zakaria_Hili <zakahili@gmail.com>\nAuthor: HILI Zakaria <zakahili@gmail.com>\n\nCloses #15965 from ZakariaHili/zakbranch.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/445d4d9e13ebaee9eceea6135fe7ee47812d97de"},{"sha":"fb07bbe575aabe68422fd3a31865101fb7fa1722","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instead of `maxConnections`\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up PR of #15868 to merge `maxConnections` option into `numPartitions` options.\n\n## How was this patch tested?\n\nPass the existing tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15966 from dongjoon-hyun/SPARK-18413-2.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/fb07bbe575aabe68422fd3a31865101fb7fa1722"},{"sha":"e2fb9fd365466da888ab8b3a2a0836049a65f8c8","author":{"name":"jiangxingbo","email":"5b9d05a325a3f85d4c824395342a210596784248@gmail.com"},"message":"[SPARK-18436][SQL] isin causing SQL syntax error with JDBC\n\n## What changes were proposed in this pull request?\n\nThe expression `in(empty seq)` is invalid in some data source. Since `in(empty seq)` is always false, we should generate `in(empty seq)` to false literal in optimizer.\nThe sql `SELECT * FROM t WHERE a IN ()` throws a `ParseException` which is consistent with Hive, don't need to change that behavior.\n\n## How was this patch tested?\nAdd new test case in `OptimizeInSuite`.\n\nAuthor: jiangxingbo <jiangxb1987@gmail.com>\n\nCloses #15977 from jiangxb1987/isin-empty.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/e2fb9fd365466da888ab8b3a2a0836049a65f8c8"},{"sha":"a88329d4553b40c45ebf9eacf229db7839d46769","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@happy-camper.st"},"message":"[SPARK-18583][SQL] Fix nullability of InputFileName.\n\n## What changes were proposed in this pull request?\n\nThe nullability of `InputFileName` should be `false`.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #16007 from ueshin/issues/SPARK-18583.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/a88329d4553b40c45ebf9eacf229db7839d46769"},{"sha":"c4a7eef0ce2d305c5c90a0a9a73b5a32eccfba95","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-18481][ML] ML 2.1 QA: Remove deprecated methods for ML\n\n## What changes were proposed in this pull request?\nRemove deprecated methods for ML.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15913 from yanboliang/spark-18481.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/c4a7eef0ce2d305c5c90a0a9a73b5a32eccfba95"},{"sha":"f4a98e421e14434fddc3f9f1018a17124d660ef0","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[WIP][SQL][DOC] Fix incorrect `code` tag\n\n## What changes were proposed in this pull request?\nThis PR is to fix incorrect `code` tag in `sql-programming-guide.md`\n\n## How was this patch tested?\nManually.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #15941 from weiqingy/fixtag.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/f4a98e421e14434fddc3f9f1018a17124d660ef0"},{"sha":"9c03c564605783d8e94f6795432bb59c33933e52","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-17251][SQL] Improve `OuterReference` to be `NamedExpression`\n\n## What changes were proposed in this pull request?\n\nCurrently, `OuterReference` is not `NamedExpression`. So, it raises 'ClassCastException` when it used in projection lists of IN correlated subqueries. This PR aims to support that by making `OuterReference` as `NamedExpression` to show correct error messages.\n\n```scala\nscala> sql(\"CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)\")\nscala> sql(\"CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)\")\nscala> sql(\"SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)\").show\njava.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression\n```\n\n## How was this patch tested?\n\nPass the Jenkins test with new test cases.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #16015 from dongjoon-hyun/SPARK-17251-2.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/9c03c564605783d8e94f6795432bb59c33933e52"},{"sha":"07f32c2283e26e86474ba8c9b50125831009a1ea","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-18594][SQL] Name Validation of Databases/Tables\n\n### What changes were proposed in this pull request?\nCurrently, the name validation checks are limited to table creation. It is enfored by Analyzer rule: `PreWriteCheck`.\n\nHowever, table renaming and database creation have the same issues. It makes more sense to do the checks in `SessionCatalog`. This PR is to add it into `SessionCatalog`.\n\n### How was this patch tested?\nAdded test cases\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #16018 from gatorsmile/nameValidate.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/07f32c2283e26e86474ba8c9b50125831009a1ea"},{"sha":"fc2c13bdf0be5e349539b2ab90087c34b2d3faab","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18482][SQL] make sure Spark can access the table metadata created by older version of spark\n\n## What changes were proposed in this pull request?\n\nIn Spark 2.1, we did a lot of refactor for `HiveExternalCatalog` and related code path. These refactor may introduce external behavior changes and break backward compatibility. e.g. http://issues.apache.org/jira/browse/SPARK-18464\n\nTo avoid future compatibility problems of `HiveExternalCatalog`, this PR dumps some typical table metadata from tables created by 2.0, and test if they can recognized by current version of Spark.\n\n## How was this patch tested?\n\ntest only change\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16003 from cloud-fan/test.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/fc2c13bdf0be5e349539b2ab90087c34b2d3faab"},{"sha":"87141622ee6b11ac177f68f58d0dc5f8b9a9f948","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@happy-camper.st"},"message":"[SPARK-18585][SQL] Use `ev.isNull = \"false\"` if possible for Janino to have a chance to optimize.\n\n## What changes were proposed in this pull request?\n\nJanino can optimize `true ? a : b` into `a` or `false ? a : b` into `b`, or if/else with literal condition, so we should use literal as `ev.isNull` if possible.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #16008 from ueshin/issues/SPARK-18585.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/87141622ee6b11ac177f68f58d0dc5f8b9a9f948"},{"sha":"454b8049916a0353772a0ea5cfe14b62cbd81df4","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-18604][SQL] Make sure CollapseWindow returns the attributes in the same order.\n\n## What changes were proposed in this pull request?\nThe `CollapseWindow` optimizer rule changes the order of output attributes. This modifies the output of the plan, which the optimizer cannot do. This also breaks things like `collect()` for which we use a `RowEncoder` that assumes that the output attributes of the executed plan are equal to those outputted by the logical plan.\n\n## How was this patch tested?\nI have updated an incorrect test in `CollapseWindowSuite`.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #16027 from hvanhovell/SPARK-18604.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/454b8049916a0353772a0ea5cfe14b62cbd81df4"},{"sha":"f075cd9cb7157819df9aec67baee8913c4ed5c53","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-18118][SQL] fix a compilation error due to nested JavaBeans\n\n## What changes were proposed in this pull request?\n\nThis PR avoids a compilation error due to more than 64KB Java byte code size. This error occur since generated java code `SpecificSafeProjection.apply()` for nested JavaBeans is too big. This PR avoids this compilation error by splitting a big code chunk into multiple methods by calling `CodegenContext.splitExpression` at `InitializeJavaBean.doGenCode`\nAn object reference for JavaBean is stored to an instance variable `javaBean...`. Then, the instance variable will be referenced in the split methods.\n\nGenerated code with this PR\n````\n/* 22098 */   private void apply130_0(InternalRow i) {\n...\n/* 22125 */     boolean isNull238 = i.isNullAt(2);\n/* 22126 */     InternalRow value238 = isNull238 ? null : (i.getStruct(2, 3));\n/* 22127 */     boolean isNull236 = false;\n/* 22128 */     test.org.apache.spark.sql.JavaDatasetSuite$Nesting1 value236 = null;\n/* 22129 */     if (!false && isNull238) {\n/* 22130 */\n/* 22131 */       final test.org.apache.spark.sql.JavaDatasetSuite$Nesting1 value239 = null;\n/* 22132 */       isNull236 = true;\n/* 22133 */       value236 = value239;\n/* 22134 */     } else {\n/* 22135 */\n/* 22136 */       final test.org.apache.spark.sql.JavaDatasetSuite$Nesting1 value241 = false ? null : new test.org.apache.spark.sql.JavaDatasetSuite$Nesting1();\n/* 22137 */       this.javaBean14 = value241;\n/* 22138 */       if (!false) {\n/* 22139 */         apply25_0(i);\n/* 22140 */         apply25_1(i);\n/* 22141 */         apply25_2(i);\n/* 22142 */       }\n/* 22143 */       isNull236 = false;\n/* 22144 */       value236 = value241;\n/* 22145 */     }\n/* 22146 */     this.javaBean.setField2(value236);\n/* 22147 */\n/* 22148 */   }\n...\n/* 22928 */   public java.lang.Object apply(java.lang.Object _i) {\n/* 22929 */     InternalRow i = (InternalRow) _i;\n/* 22930 */\n/* 22931 */     final test.org.apache.spark.sql.JavaDatasetSuite$NestedComplicatedJavaBean value1 = false ? null : new test.org.apache.spark.sql.JavaDatasetSuite$NestedComplicatedJavaBean();\n/* 22932 */     this.javaBean = value1;\n/* 22933 */     if (!false) {\n/* 22934 */       apply130_0(i);\n/* 22935 */       apply130_1(i);\n/* 22936 */       apply130_2(i);\n/* 22937 */       apply130_3(i);\n/* 22938 */       apply130_4(i);\n/* 22939 */     }\n/* 22940 */     if (false) {\n/* 22941 */       mutableRow.setNullAt(0);\n/* 22942 */     } else {\n/* 22943 */\n/* 22944 */       mutableRow.update(0, value1);\n/* 22945 */     }\n/* 22946 */\n/* 22947 */     return mutableRow;\n/* 22948 */   }\n````\n\n## How was this patch tested?\n\nadded a test suite into `JavaDatasetSuite.java`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #16032 from kiszk/SPARK-18118.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/f075cd9cb7157819df9aec67baee8913c4ed5c53"},{"sha":"70dfdcbbf11c9c3174abc111afa2250236e31af2","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-18118][SQL] fix a compilation error due to nested JavaBeans\\nRemove this reference.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/70dfdcbbf11c9c3174abc111afa2250236e31af2"},{"sha":"9f273c5173c05017c3009faaf3e10f2f70a842d0","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-17783][SQL] Hide Credentials in CREATE and DESC FORMATTED/EXTENDED a PERSISTENT/TEMP Table for JDBC\n\n### What changes were proposed in this pull request?\n\nWe should never expose the Credentials in the EXPLAIN and DESC FORMATTED/EXTENDED command. However, below commands exposed the credentials.\n\nIn the related PR: https://github.com/apache/spark/pull/10452\n\n> URL patterns to specify credential seems to be vary between different databases.\n\nThus, we hide the whole `url` value if it contains the keyword `password`. We also hide the `password` property.\n\nBefore the fix, the command outputs look like:\n\n``` SQL\nCREATE TABLE tab1\nUSING org.apache.spark.sql.jdbc\nOPTIONS (\n url 'jdbc:h2:mem:testdb0;user=testUser;password=testPass',\n dbtable 'TEST.PEOPLE',\n user 'testUser',\n password '$password')\n\nDESC FORMATTED tab1\nDESC EXTENDED tab1\n```\n\nBefore the fix,\n- The output of SQL statement EXPLAIN\n```\n== Physical Plan ==\nExecutedCommand\n   +- CreateDataSourceTableCommand CatalogTable(\n\tTable: `tab1`\n\tCreated: Wed Nov 16 23:00:10 PST 2016\n\tLast Access: Wed Dec 31 15:59:59 PST 1969\n\tType: MANAGED\n\tProvider: org.apache.spark.sql.jdbc\n\tStorage(Properties: [url=jdbc:h2:mem:testdb0;user=testUser;password=testPass, dbtable=TEST.PEOPLE, user=testUser, password=testPass])), false\n```\n\n- The output of `DESC FORMATTED`\n```\n...\n|Storage Desc Parameters:    |                                                                  |       |\n|  url                       |jdbc:h2:mem:testdb0;user=testUser;password=testPass               |       |\n|  dbtable                   |TEST.PEOPLE                                                       |       |\n|  user                      |testUser                                                          |       |\n|  password                  |testPass                                                          |       |\n+----------------------------+------------------------------------------------------------------+-------+\n```\n\n- The output of `DESC EXTENDED`\n```\n|# Detailed Table Information|CatalogTable(\n\tTable: `default`.`tab1`\n\tCreated: Wed Nov 16 23:00:10 PST 2016\n\tLast Access: Wed Dec 31 15:59:59 PST 1969\n\tType: MANAGED\n\tSchema: [StructField(NAME,StringType,false), StructField(THEID,IntegerType,false)]\n\tProvider: org.apache.spark.sql.jdbc\n\tStorage(Location: file:/Users/xiaoli/IdeaProjects/sparkDelivery/spark-warehouse/tab1, Properties: [url=jdbc:h2:mem:testdb0;user=testUser;password=testPass, dbtable=TEST.PEOPLE, user=testUser, password=testPass]))|       |\n```\n\nAfter the fix,\n- The output of SQL statement EXPLAIN\n```\n== Physical Plan ==\nExecutedCommand\n   +- CreateDataSourceTableCommand CatalogTable(\n\tTable: `tab1`\n\tCreated: Wed Nov 16 22:43:49 PST 2016\n\tLast Access: Wed Dec 31 15:59:59 PST 1969\n\tType: MANAGED\n\tProvider: org.apache.spark.sql.jdbc\n\tStorage(Properties: [url=###, dbtable=TEST.PEOPLE, user=testUser, password=###])), false\n```\n- The output of `DESC FORMATTED`\n```\n...\n|Storage Desc Parameters:    |                                                                  |       |\n|  url                       |###                                                               |       |\n|  dbtable                   |TEST.PEOPLE                                                       |       |\n|  user                      |testUser                                                          |       |\n|  password                  |###                                                               |       |\n+----------------------------+------------------------------------------------------------------+-------+\n```\n\n- The output of `DESC EXTENDED`\n```\n|# Detailed Table Information|CatalogTable(\n\tTable: `default`.`tab1`\n\tCreated: Wed Nov 16 22:43:49 PST 2016\n\tLast Access: Wed Dec 31 15:59:59 PST 1969\n\tType: MANAGED\n\tSchema: [StructField(NAME,StringType,false), StructField(THEID,IntegerType,false)]\n\tProvider: org.apache.spark.sql.jdbc\n\tStorage(Location: file:/Users/xiaoli/IdeaProjects/sparkDelivery/spark-warehouse/tab1, Properties: [url=###, dbtable=TEST.PEOPLE, user=testUser, password=###]))|       |\n```\n\n### How was this patch tested?\n\nAdded test cases\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #15358 from gatorsmile/maskCredentials.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/9f273c5173c05017c3009faaf3e10f2f70a842d0"},{"sha":"38e29824d9a50464daa397c28e89610ed0aed4b6","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-18597][SQL] Do not push-down join conditions to the right side of a LEFT ANTI join\n\n## What changes were proposed in this pull request?\nWe currently push down join conditions of a Left Anti join to both sides of the join. This is similar to Inner, Left Semi and Existence (a specialized left semi) join. The problem is that this changes the semantics of the join; a left anti join filters out rows that matches the join condition.\n\nThis PR fixes this by only pushing down conditions to the left hand side of the join. This is similar to the behavior of left outer join.\n\n## How was this patch tested?\nAdded tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite file for left anti joins with a regression test.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #16026 from hvanhovell/SPARK-18597.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/38e29824d9a50464daa397c28e89610ed0aed4b6"},{"sha":"d31ff9b7caf4eba66724947b68f517072e6a011c","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17732][SQL] Revert ALTER TABLE DROP PARTITION should support comparators\n\n## What changes were proposed in this pull request?\n\nhttps://github.com/apache/spark/pull/15704 will fail if we use int literal in `DROP PARTITION`, and we have reverted it in branch-2.1.\n\nThis PR reverts it in master branch, and add a regression test for it, to make sure the master branch is healthy.\n\n## How was this patch tested?\n\nnew regression test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16036 from cloud-fan/revert.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/d31ff9b7caf4eba66724947b68f517072e6a011c"},{"sha":"237c3b9642a1a7c5e7884824b21877590d5d0b3b","author":{"name":"Mark Grover","email":"fd1ed290c28236090ee5c195ebfc58186b22ec62@apache.org"},"message":"[SPARK-18535][UI][YARN] Redact sensitive information from Spark logs and UI\n\n## What changes were proposed in this pull request?\n\nThis patch adds a new property called `spark.secret.redactionPattern` that\nallows users to specify a scala regex to decide which Spark configuration\nproperties and environment variables in driver and executor environments\ncontain sensitive information. When this regex matches the property or\nenvironment variable name, its value is redacted from the environment UI and\nvarious logs like YARN and event logs.\n\nThis change uses this property to redact information from event logs and YARN\nlogs. It also, updates the UI code to adhere to this property instead of\nhardcoding the logic to decipher which properties are sensitive.\n\nHere's an image of the UI post-redaction:\n![image](https://cloud.githubusercontent.com/assets/1709451/20506215/4cc30654-b007-11e6-8aee-4cde253fba2f.png)\n\nHere's the text in the YARN logs, post-redaction:\n``HADOOP_CREDSTORE_PASSWORD -> *********(redacted)``\n\nHere's the text in the event logs, post-redaction:\n``...,\"spark.executorEnv.HADOOP_CREDSTORE_PASSWORD\":\"*********(redacted)\",\"spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD\":\"*********(redacted)\",...``\n\n## How was this patch tested?\n1. Unit tests are added to ensure that redaction works.\n2. A YARN job reading data off of S3 with confidential information\n(hadoop credential provider password) being provided in the environment\nvariables of driver and executor. And, afterwards, logs were grepped to make\nsure that no mention of secret password was present. It was also ensure that\nthe job was able to read the data off of S3 correctly, thereby ensuring that\nthe sensitive information was being trickled down to the right places to read\nthe data.\n3. The event logs were checked to make sure no mention of secret password was\npresent.\n4. UI environment tab was checked to make sure there was no secret information\nbeing displayed.\n\nAuthor: Mark Grover <mark@apache.org>\n\nCloses #15971 from markgrover/master_redaction.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/237c3b9642a1a7c5e7884824b21877590d5d0b3b"},{"sha":"eba727757ed5dc23c635e1926795aea62ec0fc66","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-18602] Set the version of org.codehaus.janino:commons-compiler to 3.0.0 to match the version of org.codehaus.janino:janino\n\n## What changes were proposed in this pull request?\norg.codehaus.janino:janino depends on org.codehaus.janino:commons-compiler and we have been upgraded to org.codehaus.janino:janino 3.0.0.\n\nHowever, seems we are still pulling in org.codehaus.janino:commons-compiler 2.7.6 because of calcite. It looks like an accident because we exclude janino from calcite (see here https://github.com/apache/spark/blob/branch-2.1/pom.xml#L1759). So, this PR upgrades org.codehaus.janino:commons-compiler to 3.0.0.\n\n## How was this patch tested?\njenkins\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #16025 from yhuai/janino-commons-compile.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/eba727757ed5dc23c635e1926795aea62ec0fc66"},{"sha":"185642846e25fa812f9c7f398ab20bffc1e25273","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SQL][MINOR] DESC should use 'Catalog' as partition provider\n\n## What changes were proposed in this pull request?\n\n`CatalogTable` has a parameter named `tracksPartitionsInCatalog`, and in `CatalogTable.toString` we use `\"Partition Provider: Catalog\"` to represent it. This PR fixes `DESC TABLE` to make it consistent with `CatalogTable.toString`.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #16035 from cloud-fan/minor.","distinct":false,"url":"https://api.github.com/repos/criteo-forks/spark/commits/185642846e25fa812f9c7f398ab20bffc1e25273"}]}
{"actor":{"display_login":"AnthonyTruchet"}}