{"push_id":2379679931,"size":43,"distinct_size":43,"ref":"refs/heads/master","head":"ed7f65136ba398664f455ee857ddcf8c66285a96","before":"a8a69ebee6362fad2b682def88ac7489a290629b","commits":[{"sha":"649ed9c5732f85ef1306576fdd3a9278a2a6410c","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-23509][BUILD] Upgrade commons-net from 2.2 to 3.1\n\n## What changes were proposed in this pull request?\n\nThis PR avoids version conflicts of `commons-net` by upgrading commons-net from 2.2 to 3.1. We are seeing the following message during the build using sbt.\n\n```\n[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:\n...\n[warn] \t* commons-net:commons-net:3.1 is selected over 2.2\n[warn] \t    +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 3.1)\n[warn] \t    +- org.apache.spark:spark-core_2.11:2.4.0-SNAPSHOT    (depends on 2.2)\n[warn]\n```\n\n[Here](https://commons.apache.org/proper/commons-net/changes-report.html) is a release history.\n\n[Here](https://commons.apache.org/proper/commons-net/migration.html) is a migration guide from 2.x to 3.0.\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #20672 from kiszk/SPARK-23509.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/649ed9c5732f85ef1306576fdd3a9278a2a6410c"},{"sha":"eac0b067222a3dfa52be20360a453cb7bd420bf2","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets\n\n## What changes were proposed in this pull request?\n\nAdd a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to allow streaming jobs to proceed on compacted topics (or other situations involving gaps between offsets in the log).\n\n## How was this patch tested?\n\nAdded new unit test\n\njustinrmiller has been testing this branch in production for a few weeks\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #20572 from koeninger/SPARK-17147.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/eac0b067222a3dfa52be20360a453cb7bd420bf2"},{"sha":"414ee867ba0835b97aae2e8d4e489e1879c251dd","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-23523][SQL] Fix the incorrect result caused by the rule OptimizeMetadataOnlyQuery\n\n## What changes were proposed in this pull request?\n```Scala\nval tablePath = new File(s\"${path.getCanonicalPath}/cOl3=c/cOl1=a/cOl5=e\")\n Seq((\"a\", \"b\", \"c\", \"d\", \"e\")).toDF(\"cOl1\", \"cOl2\", \"cOl3\", \"cOl4\", \"cOl5\")\n .write.json(tablePath.getCanonicalPath)\n val df = spark.read.json(path.getCanonicalPath).select(\"CoL1\", \"CoL5\", \"CoL3\").distinct()\n df.show()\n```\n\nIt generates a wrong result.\n```\n[c,e,a]\n```\n\nWe have a bug in the rule `OptimizeMetadataOnlyQuery `. We should respect the attribute order in the original leaf node. This PR is to fix it.\n\n## How was this patch tested?\nAdded a test case\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #20684 from gatorsmile/optimizeMetadataOnly.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/414ee867ba0835b97aae2e8d4e489e1879c251dd"},{"sha":"ecb8b383af1cf1b67f3111c148229e00c9c17c40","author":{"name":"Imran Rashid","email":"63f21f1cb7b694d385ee53376e18c9c9be6c6f98@cloudera.com"},"message":"[SPARK-23365][CORE] Do not adjust num executors when killing idle executors.\n\nThe ExecutorAllocationManager should not adjust the target number of\nexecutors when killing idle executors, as it has already adjusted the\ntarget number down based on the task backlog.\n\nThe name `replace` was misleading with DynamicAllocation on, as the target number\nof executors is changed outside of the call to `killExecutors`, so I adjusted that name.  Also separated out the logic of `countFailures` as you don't always want that tied to `replace`.\n\nWhile I was there I made two changes that weren't directly related to this:\n1) Fixed `countFailures` in a couple cases where it was getting an incorrect value since it used to be tied to `replace`, eg. when killing executors on a blacklisted node.\n2) hard error if you call `sc.killExecutors` with dynamic allocation on, since that's another way the ExecutorAllocationManager and the CoarseGrainedSchedulerBackend would get out of sync.\n\nAdded a unit test case which verifies that the calls to ExecutorAllocationClient do not adjust the number of executors.\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #20604 from squito/SPARK-23365.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/ecb8b383af1cf1b67f3111c148229e00c9c17c40"},{"sha":"598446b74b61fee272d3aee3a2e9a3fc90a70d6a","author":{"name":"Marco Gaido","email":"8083472c630233ac7af8bdcbedddd2705bdb18b6@gmail.com"},"message":"[SPARK-23501][UI] Refactor AllStagesPage in order to avoid redundant code\n\nAs suggested in #20651, the code is very redundant in `AllStagesPage` and modifying it is a copy-and-paste work. We should avoid such a pattern, which is error prone, and have a cleaner solution which avoids code redundancy.\n\nexisting UTs\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #20663 from mgaido91/SPARK-23475_followup.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/598446b74b61fee272d3aee3a2e9a3fc90a70d6a"},{"sha":"23ac3aaba4a33bc3d31d01f21e93c4681ef6de03","author":{"name":"Bruce Robbins","email":"43eef5a966208ab0af98f7f914198b9392db7dc9@gmail.com"},"message":"[SPARK-23417][PYTHON] Fix the build instructions supplied by exception messages in python streaming tests\n\n## What changes were proposed in this pull request?\n\nFix the build instructions supplied by exception messages in python streaming tests.\n\nI also added -DskipTests to the maven instructions to avoid the 170 minutes of scala tests that occurs each time one wants to add a jar to the assembly directory.\n\n## How was this patch tested?\n\n- clone branch\n- run build/sbt package\n- run python/run-tests --modules \"pyspark-streaming\" , expect error message\n- follow instructions in error message. i.e., run build/sbt assembly/package streaming-kafka-0-8-assembly/assembly\n- rerun python tests, expect error message\n- follow instructions in error message. i.e run build/sbt -Pflume assembly/package streaming-flume-assembly/assembly\n- rerun python tests, see success.\n- repeated all of the above for mvn version of the process.\n\nAuthor: Bruce Robbins <bersprockets@gmail.com>\n\nCloses #20638 from bersprockets/SPARK-23417_propa.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/23ac3aaba4a33bc3d31d01f21e93c4681ef6de03"},{"sha":"b14993e1fcb68e1c946a671c6048605ab4afdf58","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-23448][SQL] Clarify JSON and CSV parser behavior in document\n\n## What changes were proposed in this pull request?\n\nClarify JSON and CSV reader behavior in document.\n\nJSON doesn't support partial results for corrupted records.\nCSV only supports partial results for the records with more or less tokens.\n\n## How was this patch tested?\n\nPass existing tests.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #20666 from viirya/SPARK-23448-2.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/b14993e1fcb68e1c946a671c6048605ab4afdf58"},{"sha":"6a8abe29ef3369b387d9bc2ee3459a6611246ab1","author":{"name":"zhoukang","email":"57146583828679e79d5137f9629c77d17d2b14a0@gmail.com"},"message":"[SPARK-23508][CORE] Fix BlockmanagerId in case blockManagerIdCache cause oom\n\n… cause oom\n\n## What changes were proposed in this pull request?\nblockManagerIdCache in BlockManagerId will not remove old values which may cause oom\n\n`val blockManagerIdCache = new ConcurrentHashMap[BlockManagerId, BlockManagerId]()`\nSince whenever we apply a new BlockManagerId, it will put into this map.\n\nThis patch will use guava cahce for  blockManagerIdCache instead.\n\nA heap dump show in [SPARK-23508](https://issues.apache.org/jira/browse/SPARK-23508)\n\n## How was this patch tested?\nExist tests.\n\nAuthor: zhoukang <zhoukang199191@gmail.com>\n\nCloses #20667 from caneGuy/zhoukang/fix-history.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/6a8abe29ef3369b387d9bc2ee3459a6611246ab1"},{"sha":"fab563b9bd1581112462c0fc0b299ad6510b6564","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-23517][PYTHON] Make `pyspark.util._exception_message` produce the trace from Java side by Py4JJavaError\n\n## What changes were proposed in this pull request?\n\nThis PR proposes for `pyspark.util._exception_message` to produce the trace from Java side by `Py4JJavaError`.\n\nCurrently, in Python 2, it uses `message` attribute which `Py4JJavaError` didn't happen to have:\n\n```python\n>>> from pyspark.util import _exception_message\n>>> try:\n...     sc._jvm.java.lang.String(None)\n... except Exception as e:\n...     pass\n...\n>>> e.message\n''\n```\n\nSeems we should use `str` instead for now:\n\n https://github.com/bartdag/py4j/blob/aa6c53b59027925a426eb09b58c453de02c21b7c/py4j-python/src/py4j/protocol.py#L412\n\nbut this doesn't address the problem with non-ascii string from Java side -\n `https://github.com/bartdag/py4j/issues/306`\n\nSo, we could directly call `__str__()`:\n\n```python\n>>> e.__str__()\nu'An error occurred while calling None.java.lang.String.\\n: java.lang.NullPointerException\\n\\tat java.lang.String.<init>(String.java:588)\\n\\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\\n\\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\\n\\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\\n\\tat java.lang.reflect.Constructor.newInstance(Constructor.java:422)\\n\\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)\\n\\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\\n\\tat py4j.Gateway.invoke(Gateway.java:238)\\n\\tat py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)\\n\\tat py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)\\n\\tat py4j.GatewayConnection.run(GatewayConnection.java:214)\\n\\tat java.lang.Thread.run(Thread.java:745)\\n'\n```\n\nwhich doesn't type coerce unicodes to `str` in Python 2.\n\nThis can be actually a problem:\n\n```python\nfrom pyspark.sql.functions import udf\nspark.conf.set(\"spark.sql.execution.arrow.enabled\", True)\nspark.range(1).select(udf(lambda x: [[]])()).toPandas()\n```\n\n**Before**\n\n```\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"/.../spark/python/pyspark/sql/dataframe.py\", line 2009, in toPandas\n    raise RuntimeError(\"%s\\n%s\" % (_exception_message(e), msg))\nRuntimeError:\nNote: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true. Please set it to false to disable this.\n```\n\n**After**\n\n```\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"/.../spark/python/pyspark/sql/dataframe.py\", line 2009, in toPandas\n    raise RuntimeError(\"%s\\n%s\" % (_exception_message(e), msg))\nRuntimeError: An error occurred while calling o47.collectAsArrowToPython.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 1 times, most recent failure: Lost task 7.0 in stage 0.0 (TID 7, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):\n  File \"/.../spark/python/pyspark/worker.py\", line 245, in main\n    process()\n  File \"/.../spark/python/pyspark/worker.py\", line 240, in process\n...\nNote: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true. Please set it to false to disable this.\n```\n\n## How was this patch tested?\n\nManually tested and unit tests were added.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #20680 from HyukjinKwon/SPARK-23517.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/fab563b9bd1581112462c0fc0b299ad6510b6564"},{"sha":"476a7f026bc45462067ebd39cd269147e84cd641","author":{"name":"Juliusz Sompolski","email":"ce8ee2bf39e2a6865d88ad58eae1e176ae0dce3c@databricks.com"},"message":"[SPARK-23514] Use SessionState.newHadoopConf() to propage hadoop configs set in SQLConf.\n\n## What changes were proposed in this pull request?\n\nA few places in `spark-sql` were using `sc.hadoopConfiguration` directly. They should be using `sessionState.newHadoopConf()` to blend in configs that were set through `SQLConf`.\n\nAlso, for better UX, for these configs blended in from `SQLConf`, we should consider removing the `spark.hadoop` prefix, so that the settings are recognized whether or not they were specified by the user.\n\n## How was this patch tested?\n\nTested that AlterTableRecoverPartitions now correctly recognizes settings that are passed in to the FileSystem through SQLConf.\n\nAuthor: Juliusz Sompolski <julek@databricks.com>\n\nCloses #20679 from juliuszsompolski/SPARK-23514.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/476a7f026bc45462067ebd39cd269147e84cd641"},{"sha":"25c2776dd9ae3f9792048c78be2cbd958fd99841","author":{"name":"Xingbo Jiang","email":"5eb8ac38ffd8c8a53b42dc017089ac416c0ba972@databricks.com"},"message":"[SPARK-23523][SQL][FOLLOWUP] Minor refactor of OptimizeMetadataOnlyQuery\n\n## What changes were proposed in this pull request?\n\nInside `OptimizeMetadataOnlyQuery.getPartitionAttrs`, avoid using `zip` to generate attribute map.\nAlso include other minor update of comments and format.\n\n## How was this patch tested?\n\nExisting test cases.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #20693 from jiangxb1987/SPARK-23523.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/25c2776dd9ae3f9792048c78be2cbd958fd99841"},{"sha":"22f3d3334c85c042c6e90f5a02f308d7cd1c1498","author":{"name":"liuxian","email":"d698299a02cb9d6095b897cb57e6057a3afd73b5@zte.com.cn"},"message":"[SPARK-23389][CORE] When the shuffle dependency specifies aggregation ,and `dependency.mapSideCombine =false`, we should be able to use serialized sorting.\n\n## What changes were proposed in this pull request?\nWhen the shuffle dependency specifies aggregation ,and `dependency.mapSideCombine=false`, in the map side,there is no need for aggregation and sorting, so we should be able to use serialized sorting.\n\n## How was this patch tested?\nExisting unit test\n\nAuthor: liuxian <liu.xian3@zte.com.cn>\n\nCloses #20576 from 10110346/mapsidecombine.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/22f3d3334c85c042c6e90f5a02f308d7cd1c1498"},{"sha":"ff1480189b827af0be38605d566a4ee71b4c36f6","author":{"name":"Yuming Wang","email":"29b28fa79ead4b6d70f8fcce482ebe6b1017e33b@ebay.com"},"message":"[SPARK-23510][SQL] Support Hive 2.2 and Hive 2.3 metastore\n\n## What changes were proposed in this pull request?\nThis is based on https://github.com/apache/spark/pull/20668 for supporting Hive 2.2 and Hive 2.3 metastore.\n\nWhen we merge the PR, we should give the major credit to wangyum\n\n## How was this patch tested?\nAdded the test cases\n\nAuthor: Yuming Wang <yumwang@ebay.com>\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #20671 from gatorsmile/pr-20668.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/ff1480189b827af0be38605d566a4ee71b4c36f6"},{"sha":"cdcccd7b41c43d79edff2fec7a84cd00e9524f75","author":{"name":"KaiXinXiaoLei","email":"08061e7fbb232b4863e075e15c22e8e9bd893146@qq.com"},"message":"[SPARK-23405] Generate additional constraints for Join's children\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\nI run a sql: `select ls.cs_order_number from ls left semi join catalog_sales cs on ls.cs_order_number = cs.cs_order_number`, The `ls` table is a small table ,and the number is one. The `catalog_sales` table is a big table,  and the number is 10 billion. The task will be hang up. And i find the many null values of `cs_order_number` in the `catalog_sales` table. I think the null value should be removed in the logical plan.\n\n>== Optimized Logical Plan ==\n>Join LeftSemi, (cs_order_number#1 = cs_order_number#22)\n>:- Project cs_order_number#1\n>   : +- Filter isnotnull(cs_order_number#1)\n>      : +- MetastoreRelation 100t, ls\n>+- Project cs_order_number#22\n>   +- MetastoreRelation 100t, catalog_sales\n\nNow, use this patch, the plan will be:\n>== Optimized Logical Plan ==\n>Join LeftSemi, (cs_order_number#1 = cs_order_number#22)\n>:- Project cs_order_number#1\n>   : +- Filter isnotnull(cs_order_number#1)\n>      : +- MetastoreRelation 100t, ls\n>+- Project cs_order_number#22\n>   : **+- Filter isnotnull(cs_order_number#22)**\n>     :+- MetastoreRelation 100t, catalog_sales\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: KaiXinXiaoLei <584620569@qq.com>\nAuthor: hanghang <584620569@qq.com>\n\nCloses #20670 from KaiXinXiaoLei/Spark-23405.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/cdcccd7b41c43d79edff2fec7a84cd00e9524f75"},{"sha":"34811e0b908449fd59bca476604612b1d200778d","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-23551][BUILD] Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`\n\n## What changes were proposed in this pull request?\n\nThis PR aims to prevent `orc-mapreduce` dependency from making IDEs and maven confused.\n\n**BEFORE**\nPlease note that `2.6.4` at `Spark Project SQL`.\n```\n$ mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core\n...\n[INFO] ------------------------------------------------------------------------\n[INFO] Building Spark Project Catalyst 2.4.0-SNAPSHOT\n[INFO] ------------------------------------------------------------------------\n[INFO]\n[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli)  spark-catalyst_2.11 ---\n[INFO] org.apache.spark:spark-catalyst_2.11:jar:2.4.0-SNAPSHOT\n[INFO] \\- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile\n[INFO]    \\- org.apache.hadoop:hadoop-client:jar:2.7.3:compile\n[INFO]       \\- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile\n[INFO]\n[INFO] ------------------------------------------------------------------------\n[INFO] Building Spark Project SQL 2.4.0-SNAPSHOT\n[INFO] ------------------------------------------------------------------------\n[INFO]\n[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli)  spark-sql_2.11 ---\n[INFO] org.apache.spark:spark-sql_2.11:jar:2.4.0-SNAPSHOT\n[INFO] \\- org.apache.orc:orc-mapreduce:jar:nohive:1.4.3:compile\n[INFO]    \\- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.4:compile\n```\n\n**AFTER**\n```\n$ mvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core\n...\n[INFO] ------------------------------------------------------------------------\n[INFO] Building Spark Project Catalyst 2.4.0-SNAPSHOT\n[INFO] ------------------------------------------------------------------------\n[INFO]\n[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli)  spark-catalyst_2.11 ---\n[INFO] org.apache.spark:spark-catalyst_2.11:jar:2.4.0-SNAPSHOT\n[INFO] \\- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile\n[INFO]    \\- org.apache.hadoop:hadoop-client:jar:2.7.3:compile\n[INFO]       \\- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile\n[INFO]\n[INFO] ------------------------------------------------------------------------\n[INFO] Building Spark Project SQL 2.4.0-SNAPSHOT\n[INFO] ------------------------------------------------------------------------\n[INFO]\n[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli)  spark-sql_2.11 ---\n[INFO] org.apache.spark:spark-sql_2.11:jar:2.4.0-SNAPSHOT\n[INFO] \\- org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile\n[INFO]    \\- org.apache.hadoop:hadoop-client:jar:2.7.3:compile\n[INFO]       \\- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3:compile\n```\n\n## How was this patch tested?\n\n1. Pass the Jenkins with `dev/test-dependencies.sh` with the existing dependencies.\n2. Manually do the following and see the change.\n```\nmvn dependency:tree -Phadoop-2.7 -Dincludes=org.apache.hadoop:hadoop-mapreduce-client-core\n```\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #20704 from dongjoon-hyun/SPARK-23551.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/34811e0b908449fd59bca476604612b1d200778d"},{"sha":"119f6a0e4729aa952e811d2047790a32ee90bf69","author":{"name":"Joseph K. Bradley","email":"9b6e24a661c030e39d84e80f958c2d82d5dc3d65@databricks.com"},"message":"[SPARK-22883][ML][TEST] Streaming tests for spark.ml.feature, from A to H\n\n## What changes were proposed in this pull request?\n\nAdds structured streaming tests using testTransformer for these suites:\n* BinarizerSuite\n* BucketedRandomProjectionLSHSuite\n* BucketizerSuite\n* ChiSqSelectorSuite\n* CountVectorizerSuite\n* DCTSuite.scala\n* ElementwiseProductSuite\n* FeatureHasherSuite\n* HashingTFSuite\n\n## How was this patch tested?\n\nIt tests itself because it is a bunch of tests!\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #20111 from jkbradley/SPARK-22883-streaming-featureAM.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/119f6a0e4729aa952e811d2047790a32ee90bf69"},{"sha":"0b6ceadeb563205cbd6bd03bc88e608086273b5b","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR][DOC] fix link in vignettes\n\n## What changes were proposed in this pull request?\n\nFix doc link that was changed in 2.3\n\nshivaram\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #20711 from felixcheung/rvigmean.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/0b6ceadeb563205cbd6bd03bc88e608086273b5b"},{"sha":"3a4d15e5d2b9ddbaeb2a6ab2d86d059ada6407b2","author":{"name":"Feng Liu","email":"053548b88f9b9bede03e83ce47a784e6fc89e414@databricks.com"},"message":"[SPARK-23518][SQL] Avoid metastore access when the users only want to read and write data frames\n\n## What changes were proposed in this pull request?\n\nhttps://github.com/apache/spark/pull/18944 added one patch, which allowed a spark session to be created when the hive metastore server is down. However, it did not allow running any commands with the spark session. This brings troubles to the user who only wants to read / write data frames without metastore setup.\n\n## How was this patch tested?\n\nAdded some unit tests to read and write data frames based on the original HiveMetastoreLazyInitializationSuite.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Feng Liu <fengliu@databricks.com>\n\nCloses #20681 from liufengdb/completely-lazy.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/3a4d15e5d2b9ddbaeb2a6ab2d86d059ada6407b2"},{"sha":"707e6506d0dbdb598a6c99d666f3c66746113b67","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-23097][SQL][SS] Migrate text socket source to V2\n\n## What changes were proposed in this pull request?\n\nThis PR moves structured streaming text socket source to V2.\n\nQuestions: do we need to remove old \"socket\" source?\n\n## How was this patch tested?\n\nUnit test and manual verification.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #20382 from jerryshao/SPARK-23097.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/707e6506d0dbdb598a6c99d666f3c66746113b67"},{"sha":"487377e693af65b2ff3d6b874ca7326c1ff0076c","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-23570][SQL] Add Spark 2.3.0 in HiveExternalCatalogVersionsSuite\n\n## What changes were proposed in this pull request?\nAdd Spark 2.3.0 in HiveExternalCatalogVersionsSuite since Spark 2.3.0 is released for ensuring backward compatibility.\n\n## How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #20720 from gatorsmile/add2.3.","distinct":true,"url":"https://api.github.com/repos/yanlin-Lynn/spark/commits/487377e693af65b2ff3d6b874ca7326c1ff0076c"}]}
{"actor":{"display_login":"yanlin-Lynn"}}