{"push_id":2447313333,"size":93,"distinct_size":93,"ref":"refs/heads/master","head":"a58f8c345b5590193595df8d76db44ba049359da","before":"7013eea11cb32b1e0038dc751c485da5c94a484b","commits":[{"sha":"92e7ecbbbd6817378abdbd56541a9c13dcea8659","author":{"name":"Marco Gaido","email":"8083472c630233ac7af8bdcbedddd2705bdb18b6@gmail.com"},"message":"[SPARK-23592][SQL] Add interpreted execution to DecodeUsingSerializer\n\n## What changes were proposed in this pull request?\n\nThe PR adds interpreted execution to DecodeUsingSerializer.\n\n## How was this patch tested?\nadded UT\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #20760 from mgaido91/SPARK-23592.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/92e7ecbbbd6817378abdbd56541a9c13dcea8659"},{"sha":"3be4adf6485ca19cdc5db23394c3f5a660d7dc6f","author":{"name":"lucio","email":"5e7f0a75146f8f4f5d5cbe1a5e9eb5002d8703b1@qq.com"},"message":"[SPARK-22751][ML] Improve ML RandomForest shuffle performance\n\n## What changes were proposed in this pull request?\n\nAs I mentioned in [SPARK-22751](https://issues.apache.org/jira/browse/SPARK-22751?jql=project%20%3D%20SPARK%20AND%20component%20%3D%20ML%20AND%20text%20~%20randomforest), there is a shuffle performance problem in ML Randomforest when train a RF in high dimensional data.\n\nThe reason is that, in _org.apache.spark.tree.impl.RandomForest_, the function _findSplitsBySorting_ will actually flatmap a sparse vector into a dense vector, then in groupByKey there will be a huge shuffle write size.\n\nTo avoid this, we can add a filter in flatmap, to filter out zero value. And in function _findSplitsForContinuousFeature_, we can infer the number of zero value by _metadata_.\n\nIn addition, if a feature only contains zero value, _continuousSplits_ will not has the key of feature id. So I add a check when using _continuousSplits_.\n\n## How was this patch tested?\nRan model locally using spark-submit.\n\nAuthor: lucio <576632108@qq.com>\n\nCloses #20472 from lucio-yz/master.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/3be4adf6485ca19cdc5db23394c3f5a660d7dc6f"},{"sha":"ea480990e726aed59750f1cea8d40adba56d991a","author":{"name":"Marco Gaido","email":"8083472c630233ac7af8bdcbedddd2705bdb18b6@gmail.com"},"message":"[SPARK-23628][SQL] calculateParamLength should not return 1 + num of epressions\n\n## What changes were proposed in this pull request?\n\nThere was a bug in `calculateParamLength` which caused it to return always 1 + the number of expressions. This could lead to Exceptions especially with expressions of type long.\n\n## How was this patch tested?\n\nadded UT + fixed previous UT\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #20772 from mgaido91/SPARK-23628.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/ea480990e726aed59750f1cea8d40adba56d991a"},{"sha":"e7bbca88964d95593fa15eb94643ba519801e352","author":{"name":"Marco Gaido","email":"8083472c630233ac7af8bdcbedddd2705bdb18b6@gmail.com"},"message":"[SPARK-23602][SQL] PrintToStderr prints value also in interpreted mode\n\n## What changes were proposed in this pull request?\n\n`PrintToStderr` was doing what is it supposed to only when code generation is enabled.\nThe PR adds the same behavior in interpreted mode too.\n\n## How was this patch tested?\n\nadded UT\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #20773 from mgaido91/SPARK-23602.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/e7bbca88964d95593fa15eb94643ba519801e352"},{"sha":"d90e77bd0ec19f8ba9198a24ec2ab3db7708eca8","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[SPARK-23271][SQL] Parquet output contains only _SUCCESS file after writing an empty dataframe\n\n## What changes were proposed in this pull request?\nBelow are the two cases.\n``` SQL\ncase 1\n\nscala> List.empty[String].toDF().rdd.partitions.length\nres18: Int = 1\n```\nWhen we write the above data frame as parquet, we create a parquet file containing\njust the schema of the data frame.\n\nCase 2\n``` SQL\n\nscala> val anySchema = StructType(StructField(\"anyName\", StringType, nullable = false) :: Nil)\nanySchema: org.apache.spark.sql.types.StructType = StructType(StructField(anyName,StringType,false))\nscala> spark.read.schema(anySchema).csv(\"/tmp/empty_folder\").rdd.partitions.length\nres22: Int = 0\n```\nFor the 2nd case, since number of partitions = 0, we don't call the write task (the task has logic to create the empty metadata only parquet file)\n\nThe fix is to create a dummy single partition RDD and set up the write task based on it to ensure\nthe metadata-only file.\n\n## How was this patch tested?\n\nA new test is added to DataframeReaderWriterSuite.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #20525 from dilipbiswal/spark-23271.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/d90e77bd0ec19f8ba9198a24ec2ab3db7708eca8"},{"sha":"2c3673680e16f88f1d1cd73a3f7445ded5b3daa8","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-23630][YARN] Allow user's hadoop conf customizations to take effect.\n\nThis change restores functionality that was inadvertently removed as part\nof the fix for SPARK-22372.\n\nAlso modified an existing unit test to make sure the feature works as intended.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #20776 from vanzin/SPARK-23630.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/2c3673680e16f88f1d1cd73a3f7445ded5b3daa8"},{"sha":"2ca9bb083c515511d2bfee271fc3e0269aceb9d5","author":{"name":"Michał Świtakowski","email":"83f6a27b67fa60ca2f1a1834601672bbb2b66e12@databricks.com"},"message":"[SPARK-23173][SQL] Avoid creating corrupt parquet files when loading data from JSON\n\n## What changes were proposed in this pull request?\n\nThe from_json() function accepts an additional parameter, where the user might specify the schema. The issue is that the specified schema might not be compatible with data. In particular, the JSON data might be missing data for fields declared as non-nullable in the schema. The from_json() function does not verify the data against such errors. When data with missing fields is sent to the parquet encoder, there is no verification either. The end results is a corrupt parquet file.\n\nTo avoid corruptions, make sure that all fields in the user-specified schema are set to be nullable.\nSince this changes the behavior of a public function, we need to include it in release notes.\nThe behavior can be reverted by setting `spark.sql.fromJsonForceNullableSchema=false`\n\n## How was this patch tested?\n\nAdded two new tests.\n\nAuthor: Michał Świtakowski <michal.switakowski@databricks.com>\n\nCloses #20694 from mswit-databricks/SPARK-23173.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/2ca9bb083c515511d2bfee271fc3e0269aceb9d5"},{"sha":"10b0657b035641ce735055bba2c8459e71bc2400","author":{"name":"Wang Gengliang","email":"c5a00a6d51e544313effefcc5eb48f36fadadf04@databricks.com"},"message":"[SPARK-23624][SQL] Revise doc of method pushFilters in Datasource V2\n\n## What changes were proposed in this pull request?\n\nRevise doc of method pushFilters in SupportsPushDownFilters/SupportsPushDownCatalystFilters\n\nIn `FileSourceStrategy`, except `partitionKeyFilters`(the references of which is subset of partition keys), all filters needs to be evaluated after scanning. Otherwise, Spark will get wrong result from data sources like Orc/Parquet.\n\nThis PR is to improve the doc.\n\nAuthor: Wang Gengliang <gengliang.wang@databricks.com>\n\nCloses #20769 from gengliangwang/revise_pushdown_doc.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/10b0657b035641ce735055bba2c8459e71bc2400"},{"sha":"1a54f48b6744032b16543594651ee6d5e3ad4bda","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-23510][SQL][FOLLOW-UP] Support Hive 2.2 and Hive 2.3 metastore\n\n## What changes were proposed in this pull request?\nIn the PR https://github.com/apache/spark/pull/20671, I forgot to update the doc about this new support.\n\n## How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #20789 from gatorsmile/docUpdate.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/1a54f48b6744032b16543594651ee6d5e3ad4bda"},{"sha":"b6f837c9d3cb0f76f0a52df37e34aea8944f6867","author":{"name":"DylanGuedes","email":"212f4187da83868657a53ae0e81d3411bbad32bc@gmail.com"},"message":"[PYTHON] Changes input variable to not conflict with built-in function\n\nSigned-off-by: DylanGuedes <djmgguedesgmail.com>\n\n## What changes were proposed in this pull request?\n\nChanges variable name conflict: [input is a built-in python function](https://stackoverflow.com/questions/20670732/is-input-a-keyword-in-python).\n\n## How was this patch tested?\n\nI runned the example and it works fine.\n\nAuthor: DylanGuedes <djmgguedes@gmail.com>\n\nCloses #20775 from DylanGuedes/input_variable.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/b6f837c9d3cb0f76f0a52df37e34aea8944f6867"},{"sha":"b304e07e0671faf96530f9d8f49c55a83b07fa15","author":{"name":"Xiayun Sun","email":"51e1e8b551e0e299c5efd69fd811cac9d0331ea6@gmail.com"},"message":"[SPARK-23462][SQL] improve missing field error message in `StructType`\n\n## What changes were proposed in this pull request?\n\nThe error message ```s\"\"\"Field \"$name\" does not exist.\"\"\"``` is thrown when looking up an unknown field in StructType. In the error message, we should also contain the information about which columns/fields exist in this struct.\n\n## How was this patch tested?\n\nAdded new unit tests.\n\nNote: I created a new `StructTypeSuite.scala` as I couldn't find an existing suite that's suitable to place these tests. I may be missing something so feel free to propose new locations.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Xiayun Sun <xiayunsun@gmail.com>\n\nCloses #20649 from xysun/SPARK-23462.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/b304e07e0671faf96530f9d8f49c55a83b07fa15"},{"sha":"d5b41aea62201cd5b1baad2f68f5fc7eb99c62c5","author":{"name":"Jooseong Kim","email":"9145d81d86a566fb07cc370cb9dca3495776f236@pinterest.com"},"message":"[SPARK-23618][K8S][BUILD] Initialize BUILD_ARGS in docker-image-tool.sh\n\n## What changes were proposed in this pull request?\n\nThis change initializes BUILD_ARGS to an empty array when $SPARK_HOME/RELEASE exists.\n\nIn function build, \"local BUILD_ARGS\" effectively creates an array of one element where the first and only element is an empty string, so \"${BUILD_ARGS[]}\" expands to \"\" and passes an extra argument to docker.\n\nSetting BUILD_ARGS to an empty array makes \"${BUILD_ARGS[]}\" expand to nothing.\n\n## How was this patch tested?\n\nManually tested.\n\n$ cat RELEASE\nSpark 2.3.0 (git revision a0d7949896) built for Hadoop 2.7.3\nBuild flags: -Phadoop-2.7 -Phive -Phive-thriftserver -Pkafka-0-8 -Pmesos -Pyarn -Pkubernetes -Pflume -Psparkr -DzincPort=3036\n$ ./bin/docker-image-tool.sh -m t testing build\nSending build context to Docker daemon  256.4MB\n...\n\nvanzin\n\nAuthor: Jooseong Kim <jooseong@pinterest.com>\n\nCloses #20791 from jooseong/SPARK-23618.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/d5b41aea62201cd5b1baad2f68f5fc7eb99c62c5"},{"sha":"567bd31e0ae8b632357baa93e1469b666fb06f3d","author":{"name":"Marco Gaido","email":"8083472c630233ac7af8bdcbedddd2705bdb18b6@gmail.com"},"message":"[SPARK-23412][ML] Add cosine distance to BisectingKMeans\n\n## What changes were proposed in this pull request?\n\nThe PR adds the option to specify a distance measure in BisectingKMeans. Moreover, it introduces the ability to use the cosine distance measure in it.\n\n## How was this patch tested?\n\nadded UTs + existing UTs\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #20600 from mgaido91/SPARK-23412.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/567bd31e0ae8b632357baa93e1469b666fb06f3d"},{"sha":"23370554d0f88b82154d4232744b874cc58c7848","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-23656][TEST] Perform assertions in XXH64Suite.testKnownByteArrayInputs() on big endian platform, too\n\n## What changes were proposed in this pull request?\n\nThis PR enables assertions in `XXH64Suite.testKnownByteArrayInputs()` on big endian platform, too. The current implementation performs them only on little endian platform. This PR increase test coverage of big endian platform.\n\n## How was this patch tested?\n\nUpdated `XXH64Suite`\nTested on big endian platform using JIT compiler or interpreter `-Xint`.\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #20804 from kiszk/SPARK-23656.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/23370554d0f88b82154d4232744b874cc58c7848"},{"sha":"9ddd1e2ceac8155b30beebb6bbfdcd32296fab2d","author":{"name":"Xingbo Jiang","email":"5eb8ac38ffd8c8a53b42dc017089ac416c0ba972@databricks.com"},"message":"[MINOR][SQL][TEST] Create table using `dataSourceName` in `HadoopFsRelationTest`\n\n## What changes were proposed in this pull request?\n\nThis PR fixes a minor issue in `HadoopFsRelationTest`, that you should create table using `dataSourceName` instead of `parquet`. The issue won't affect the correctness, but it will generate wrong error message in case the test fails.\n\n## How was this patch tested?\n\nExsiting tests.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #20780 from jiangxb1987/dataSourceName.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/9ddd1e2ceac8155b30beebb6bbfdcd32296fab2d"},{"sha":"918fb9beee6a2fd499b8f18dfe0d460f078f5290","author":{"name":"zuotingbing","email":"7d60a55152959e3b2588601b7961ee5470bd5c55@zte.com.cn"},"message":"[SPARK-23547][SQL] Cleanup the .pipeout file when the Hive Session closed\n\n## What changes were proposed in this pull request?\n\n![2018-03-07_121010](https://user-images.githubusercontent.com/24823338/37073232-922e10d2-2200-11e8-8172-6e03aa984b39.png)\n\nwhen the hive session closed, we should also cleanup the .pipeout file.\n\n## How was this patch tested?\n\nAdded test cases.\n\nAuthor: zuotingbing <zuo.tingbing9@zte.com.cn>\n\nCloses #20702 from zuotingbing/SPARK-23547.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/918fb9beee6a2fd499b8f18dfe0d460f078f5290"},{"sha":"1098933b0ac5cdb18101d3aebefa773c2ce05a50","author":{"name":"Kazuaki Ishizaki","email":"64981c6bd46971e76f308b8a384367244e8169bf@jp.ibm.com"},"message":"[SPARK-23598][SQL] Make methods in BufferedRowIterator public to avoid runtime error for a large query\n\n## What changes were proposed in this pull request?\n\nThis PR fixes runtime error regarding a large query when a generated code has split classes. The issue is `append()`, `stopEarly()`, and other methods are not accessible from split classes that are not subclasses of `BufferedRowIterator`.\nThis PR fixes this issue by making them `public`.\n\nBefore applying the PR, we see the following exception by running the attached program with `CodeGenerator.GENERATED_CLASS_SIZE_THRESHOLD=-1`.\n```\n  test(\"SPARK-23598\") {\n    // When set -1 to CodeGenerator.GENERATED_CLASS_SIZE_THRESHOLD, an exception is thrown\n    val df_pet_age = Seq((8, \"bat\"), (15, \"mouse\"), (5, \"horse\")).toDF(\"age\", \"name\")\n    df_pet_age.groupBy(\"name\").avg(\"age\").show()\n  }\n```\n\nException:\n```\n19:40:52.591 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n19:41:32.319 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)\njava.lang.IllegalAccessError: tried to access method org.apache.spark.sql.execution.BufferedRowIterator.shouldStop()Z from class org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1$agg_NestedClass1\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1$agg_NestedClass1.agg_doAggregateWithKeys$(generated.java:203)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(generated.java:160)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:616)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:109)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n...\n```\n\nGenerated code (line 195 calles `stopEarly()`).\n```\n/* 001 */ public Object generate(Object[] references) {\n/* 002 */   return new GeneratedIteratorForCodegenStage1(references);\n/* 003 */ }\n/* 004 */\n/* 005 */ // codegenStageId=1\n/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {\n/* 007 */   private Object[] references;\n/* 008 */   private scala.collection.Iterator[] inputs;\n/* 009 */   private boolean agg_initAgg;\n/* 010 */   private boolean agg_bufIsNull;\n/* 011 */   private double agg_bufValue;\n/* 012 */   private boolean agg_bufIsNull1;\n/* 013 */   private long agg_bufValue1;\n/* 014 */   private agg_FastHashMap agg_fastHashMap;\n/* 015 */   private org.apache.spark.unsafe.KVIterator<UnsafeRow, UnsafeRow> agg_fastHashMapIter;\n/* 016 */   private org.apache.spark.unsafe.KVIterator agg_mapIter;\n/* 017 */   private org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap agg_hashMap;\n/* 018 */   private org.apache.spark.sql.execution.UnsafeKVExternalSorter agg_sorter;\n/* 019 */   private scala.collection.Iterator inputadapter_input;\n/* 020 */   private boolean agg_agg_isNull11;\n/* 021 */   private boolean agg_agg_isNull25;\n/* 022 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder[] agg_mutableStateArray1 = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder[2];\n/* 023 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] agg_mutableStateArray2 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2];\n/* 024 */   private UnsafeRow[] agg_mutableStateArray = new UnsafeRow[2];\n/* 025 */\n/* 026 */   public GeneratedIteratorForCodegenStage1(Object[] references) {\n/* 027 */     this.references = references;\n/* 028 */   }\n/* 029 */\n/* 030 */   public void init(int index, scala.collection.Iterator[] inputs) {\n/* 031 */     partitionIndex = index;\n/* 032 */     this.inputs = inputs;\n/* 033 */\n/* 034 */     agg_fastHashMap = new agg_FastHashMap(((org.apache.spark.sql.execution.aggregate.HashAggregateExec) references[0] /* plan */).getTaskMemoryManager(), ((org.apache.spark.sql.execution.aggregate.HashAggregateExec) references[0] /* plan */).getEmptyAggregationBuffer());\n/* 035 */     agg_hashMap = ((org.apache.spark.sql.execution.aggregate.HashAggregateExec) references[0] /* plan */).createHashMap();\n/* 036 */     inputadapter_input = inputs[0];\n/* 037 */     agg_mutableStateArray[0] = new UnsafeRow(1);\n/* 038 */     agg_mutableStateArray1[0] = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_mutableStateArray[0], 32);\n/* 039 */     agg_mutableStateArray2[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_mutableStateArray1[0], 1);\n/* 040 */     agg_mutableStateArray[1] = new UnsafeRow(3);\n/* 041 */     agg_mutableStateArray1[1] = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_mutableStateArray[1], 32);\n/* 042 */     agg_mutableStateArray2[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_mutableStateArray1[1], 3);\n/* 043 */\n/* 044 */   }\n/* 045 */\n/* 046 */   public class agg_FastHashMap {\n/* 047 */     private org.apache.spark.sql.catalyst.expressions.RowBasedKeyValueBatch batch;\n/* 048 */     private int[] buckets;\n/* 049 */     private int capacity = 1 << 16;\n/* 050 */     private double loadFactor = 0.5;\n/* 051 */     private int numBuckets = (int) (capacity / loadFactor);\n/* 052 */     private int maxSteps = 2;\n/* 053 */     private int numRows = 0;\n/* 054 */     private org.apache.spark.sql.types.StructType keySchema = new org.apache.spark.sql.types.StructType().add(((java.lang.String) references[1] /* keyName */), org.apache.spark.sql.types.DataTypes.StringType);\n/* 055 */     private org.apache.spark.sql.types.StructType valueSchema = new org.apache.spark.sql.types.StructType().add(((java.lang.String) references[2] /* keyName */), org.apache.spark.sql.types.DataTypes.DoubleType)\n/* 056 */     .add(((java.lang.String) references[3] /* keyName */), org.apache.spark.sql.types.DataTypes.LongType);\n/* 057 */     private Object emptyVBase;\n/* 058 */     private long emptyVOff;\n/* 059 */     private int emptyVLen;\n/* 060 */     private boolean isBatchFull = false;\n/* 061 */\n/* 062 */     public agg_FastHashMap(\n/* 063 */       org.apache.spark.memory.TaskMemoryManager taskMemoryManager,\n/* 064 */       InternalRow emptyAggregationBuffer) {\n/* 065 */       batch = org.apache.spark.sql.catalyst.expressions.RowBasedKeyValueBatch\n/* 066 */       .allocate(keySchema, valueSchema, taskMemoryManager, capacity);\n/* 067 */\n/* 068 */       final UnsafeProjection valueProjection = UnsafeProjection.create(valueSchema);\n/* 069 */       final byte[] emptyBuffer = valueProjection.apply(emptyAggregationBuffer).getBytes();\n/* 070 */\n/* 071 */       emptyVBase = emptyBuffer;\n/* 072 */       emptyVOff = Platform.BYTE_ARRAY_OFFSET;\n/* 073 */       emptyVLen = emptyBuffer.length;\n/* 074 */\n/* 075 */       buckets = new int[numBuckets];\n/* 076 */       java.util.Arrays.fill(buckets, -1);\n/* 077 */     }\n/* 078 */\n/* 079 */     public org.apache.spark.sql.catalyst.expressions.UnsafeRow findOrInsert(UTF8String agg_key) {\n/* 080 */       long h = hash(agg_key);\n/* 081 */       int step = 0;\n/* 082 */       int idx = (int) h & (numBuckets - 1);\n/* 083 */       while (step < maxSteps) {\n/* 084 */         // Return bucket index if it's either an empty slot or already contains the key\n/* 085 */         if (buckets[idx] == -1) {\n/* 086 */           if (numRows < capacity && !isBatchFull) {\n/* 087 */             // creating the unsafe for new entry\n/* 088 */             UnsafeRow agg_result = new UnsafeRow(1);\n/* 089 */             org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder agg_holder\n/* 090 */             = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_result,\n/* 091 */               32);\n/* 092 */             org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter\n/* 093 */             = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(\n/* 094 */               agg_holder,\n/* 095 */               1);\n/* 096 */             agg_holder.reset(); //TODO: investigate if reset or zeroout are actually needed\n/* 097 */             agg_rowWriter.zeroOutNullBytes();\n/* 098 */             agg_rowWriter.write(0, agg_key);\n/* 099 */             agg_result.setTotalSize(agg_holder.totalSize());\n/* 100 */             Object kbase = agg_result.getBaseObject();\n/* 101 */             long koff = agg_result.getBaseOffset();\n/* 102 */             int klen = agg_result.getSizeInBytes();\n/* 103 */\n/* 104 */             UnsafeRow vRow\n/* 105 */             = batch.appendRow(kbase, koff, klen, emptyVBase, emptyVOff, emptyVLen);\n/* 106 */             if (vRow == null) {\n/* 107 */               isBatchFull = true;\n/* 108 */             } else {\n/* 109 */               buckets[idx] = numRows++;\n/* 110 */             }\n/* 111 */             return vRow;\n/* 112 */           } else {\n/* 113 */             // No more space\n/* 114 */             return null;\n/* 115 */           }\n/* 116 */         } else if (equals(idx, agg_key)) {\n/* 117 */           return batch.getValueRow(buckets[idx]);\n/* 118 */         }\n/* 119 */         idx = (idx + 1) & (numBuckets - 1);\n/* 120 */         step++;\n/* 121 */       }\n/* 122 */       // Didn't find it\n/* 123 */       return null;\n/* 124 */     }\n/* 125 */\n/* 126 */     private boolean equals(int idx, UTF8String agg_key) {\n/* 127 */       UnsafeRow row = batch.getKeyRow(buckets[idx]);\n/* 128 */       return (row.getUTF8String(0).equals(agg_key));\n/* 129 */     }\n/* 130 */\n/* 131 */     private long hash(UTF8String agg_key) {\n/* 132 */       long agg_hash = 0;\n/* 133 */\n/* 134 */       int agg_result = 0;\n/* 135 */       byte[] agg_bytes = agg_key.getBytes();\n/* 136 */       for (int i = 0; i < agg_bytes.length; i++) {\n/* 137 */         int agg_hash1 = agg_bytes[i];\n/* 138 */         agg_result = (agg_result ^ (0x9e3779b9)) + agg_hash1 + (agg_result << 6) + (agg_result >>> 2);\n/* 139 */       }\n/* 140 */\n/* 141 */       agg_hash = (agg_hash ^ (0x9e3779b9)) + agg_result + (agg_hash << 6) + (agg_hash >>> 2);\n/* 142 */\n/* 143 */       return agg_hash;\n/* 144 */     }\n/* 145 */\n/* 146 */     public org.apache.spark.unsafe.KVIterator<UnsafeRow, UnsafeRow> rowIterator() {\n/* 147 */       return batch.rowIterator();\n/* 148 */     }\n/* 149 */\n/* 150 */     public void close() {\n/* 151 */       batch.close();\n/* 152 */     }\n/* 153 */\n/* 154 */   }\n/* 155 */\n/* 156 */   protected void processNext() throws java.io.IOException {\n/* 157 */     if (!agg_initAgg) {\n/* 158 */       agg_initAgg = true;\n/* 159 */       long wholestagecodegen_beforeAgg = System.nanoTime();\n/* 160 */       agg_nestedClassInstance1.agg_doAggregateWithKeys();\n/* 161 */       ((org.apache.spark.sql.execution.metric.SQLMetric) references[8] /* aggTime */).add((System.nanoTime() - wholestagecodegen_beforeAgg) / 1000000);\n/* 162 */     }\n/* 163 */\n/* 164 */     // output the result\n/* 165 */\n/* 166 */     while (agg_fastHashMapIter.next()) {\n/* 167 */       UnsafeRow agg_aggKey = (UnsafeRow) agg_fastHashMapIter.getKey();\n/* 168 */       UnsafeRow agg_aggBuffer = (UnsafeRow) agg_fastHashMapIter.getValue();\n/* 169 */       wholestagecodegen_nestedClassInstance.agg_doAggregateWithKeysOutput(agg_aggKey, agg_aggBuffer);\n/* 170 */\n/* 171 */       if (shouldStop()) return;\n/* 172 */     }\n/* 173 */     agg_fastHashMap.close();\n/* 174 */\n/* 175 */     while (agg_mapIter.next()) {\n/* 176 */       UnsafeRow agg_aggKey = (UnsafeRow) agg_mapIter.getKey();\n/* 177 */       UnsafeRow agg_aggBuffer = (UnsafeRow) agg_mapIter.getValue();\n/* 178 */       wholestagecodegen_nestedClassInstance.agg_doAggregateWithKeysOutput(agg_aggKey, agg_aggBuffer);\n/* 179 */\n/* 180 */       if (shouldStop()) return;\n/* 181 */     }\n/* 182 */\n/* 183 */     agg_mapIter.close();\n/* 184 */     if (agg_sorter == null) {\n/* 185 */       agg_hashMap.free();\n/* 186 */     }\n/* 187 */   }\n/* 188 */\n/* 189 */   private wholestagecodegen_NestedClass wholestagecodegen_nestedClassInstance = new wholestagecodegen_NestedClass();\n/* 190 */   private agg_NestedClass1 agg_nestedClassInstance1 = new agg_NestedClass1();\n/* 191 */   private agg_NestedClass agg_nestedClassInstance = new agg_NestedClass();\n/* 192 */\n/* 193 */   private class agg_NestedClass1 {\n/* 194 */     private void agg_doAggregateWithKeys() throws java.io.IOException {\n/* 195 */       while (inputadapter_input.hasNext() && !stopEarly()) {\n/* 196 */         InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\n/* 197 */         int inputadapter_value = inputadapter_row.getInt(0);\n/* 198 */         boolean inputadapter_isNull1 = inputadapter_row.isNullAt(1);\n/* 199 */         UTF8String inputadapter_value1 = inputadapter_isNull1 ?\n/* 200 */         null : (inputadapter_row.getUTF8String(1));\n/* 201 */\n/* 202 */         agg_nestedClassInstance.agg_doConsume(inputadapter_row, inputadapter_value, inputadapter_value1, inputadapter_isNull1);\n/* 203 */         if (shouldStop()) return;\n/* 204 */       }\n/* 205 */\n/* 206 */       agg_fastHashMapIter = agg_fastHashMap.rowIterator();\n/* 207 */       agg_mapIter = ((org.apache.spark.sql.execution.aggregate.HashAggregateExec) references[0] /* plan */).finishAggregate(agg_hashMap, agg_sorter, ((org.apache.spark.sql.execution.metric.SQLMetric) references[4] /* peakMemory */), ((org.apache.spark.sql.execution.metric.SQLMetric) references[5] /* spillSize */), ((org.apache.spark.sql.execution.metric.SQLMetric) references[6] /* avgHashProbe */));\n/* 208 */\n/* 209 */     }\n/* 210 */\n/* 211 */   }\n/* 212 */\n/* 213 */   private class wholestagecodegen_NestedClass {\n/* 214 */     private void agg_doAggregateWithKeysOutput(UnsafeRow agg_keyTerm, UnsafeRow agg_bufferTerm)\n/* 215 */     throws java.io.IOException {\n/* 216 */       ((org.apache.spark.sql.execution.metric.SQLMetric) references[7] /* numOutputRows */).add(1);\n/* 217 */\n/* 218 */       boolean agg_isNull35 = agg_keyTerm.isNullAt(0);\n/* 219 */       UTF8String agg_value37 = agg_isNull35 ?\n/* 220 */       null : (agg_keyTerm.getUTF8String(0));\n/* 221 */       boolean agg_isNull36 = agg_bufferTerm.isNullAt(0);\n/* 222 */       double agg_value38 = agg_isNull36 ?\n/* 223 */       -1.0 : (agg_bufferTerm.getDouble(0));\n/* 224 */       boolean agg_isNull37 = agg_bufferTerm.isNullAt(1);\n/* 225 */       long agg_value39 = agg_isNull37 ?\n/* 226 */       -1L : (agg_bufferTerm.getLong(1));\n/* 227 */\n/* 228 */       agg_mutableStateArray1[1].reset();\n/* 229 */\n/* 230 */       agg_mutableStateArray2[1].zeroOutNullBytes();\n/* 231 */\n/* 232 */       if (agg_isNull35) {\n/* 233 */         agg_mutableStateArray2[1].setNullAt(0);\n/* 234 */       } else {\n/* 235 */         agg_mutableStateArray2[1].write(0, agg_value37);\n/* 236 */       }\n/* 237 */\n/* 238 */       if (agg_isNull36) {\n/* 239 */         agg_mutableStateArray2[1].setNullAt(1);\n/* 240 */       } else {\n/* 241 */         agg_mutableStateArray2[1].write(1, agg_value38);\n/* 242 */       }\n/* 243 */\n/* 244 */       if (agg_isNull37) {\n/* 245 */         agg_mutableStateArray2[1].setNullAt(2);\n/* 246 */       } else {\n/* 247 */         agg_mutableStateArray2[1].write(2, agg_value39);\n/* 248 */       }\n/* 249 */       agg_mutableStateArray[1].setTotalSize(agg_mutableStateArray1[1].totalSize());\n/* 250 */       append(agg_mutableStateArray[1]);\n/* 251 */\n/* 252 */     }\n/* 253 */\n/* 254 */   }\n/* 255 */\n/* 256 */   private class agg_NestedClass {\n/* 257 */     private void agg_doConsume(InternalRow inputadapter_row, int agg_expr_0, UTF8String agg_expr_1, boolean agg_exprIsNull_1) throws java.io.IOException {\n/* 258 */       UnsafeRow agg_unsafeRowAggBuffer = null;\n/* 259 */       UnsafeRow agg_fastAggBuffer = null;\n/* 260 */\n/* 261 */       if (true) {\n/* 262 */         if (!agg_exprIsNull_1) {\n/* 263 */           agg_fastAggBuffer = agg_fastHashMap.findOrInsert(\n/* 264 */             agg_expr_1);\n/* 265 */         }\n/* 266 */       }\n/* 267 */       // Cannot find the key in fast hash map, try regular hash map.\n/* 268 */       if (agg_fastAggBuffer == null) {\n/* 269 */         // generate grouping key\n/* 270 */         agg_mutableStateArray1[0].reset();\n/* 271 */\n/* 272 */         agg_mutableStateArray2[0].zeroOutNullBytes();\n/* 273 */\n/* 274 */         if (agg_exprIsNull_1) {\n/* 275 */           agg_mutableStateArray2[0].setNullAt(0);\n/* 276 */         } else {\n/* 277 */           agg_mutableStateArray2[0].write(0, agg_expr_1);\n/* 278 */         }\n/* 279 */         agg_mutableStateArray[0].setTotalSize(agg_mutableStateArray1[0].totalSize());\n/* 280 */         int agg_value7 = 42;\n/* 281 */\n/* 282 */         if (!agg_exprIsNull_1) {\n/* 283 */           agg_value7 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(agg_expr_1.getBaseObject(), agg_expr_1.getBaseOffset(), agg_expr_1.numBytes(), agg_value7);\n/* 284 */         }\n/* 285 */         if (true) {\n/* 286 */           // try to get the buffer from hash map\n/* 287 */           agg_unsafeRowAggBuffer =\n/* 288 */           agg_hashMap.getAggregationBufferFromUnsafeRow(agg_mutableStateArray[0], agg_value7);\n/* 289 */         }\n/* 290 */         // Can't allocate buffer from the hash map. Spill the map and fallback to sort-based\n/* 291 */         // aggregation after processing all input rows.\n/* 292 */         if (agg_unsafeRowAggBuffer == null) {\n/* 293 */           if (agg_sorter == null) {\n/* 294 */             agg_sorter = agg_hashMap.destructAndCreateExternalSorter();\n/* 295 */           } else {\n/* 296 */             agg_sorter.merge(agg_hashMap.destructAndCreateExternalSorter());\n/* 297 */           }\n/* 298 */\n/* 299 */           // the hash map had be spilled, it should have enough memory now,\n/* 300 */           // try to allocate buffer again.\n/* 301 */           agg_unsafeRowAggBuffer = agg_hashMap.getAggregationBufferFromUnsafeRow(\n/* 302 */             agg_mutableStateArray[0], agg_value7);\n/* 303 */           if (agg_unsafeRowAggBuffer == null) {\n/* 304 */             // failed to allocate the first page\n/* 305 */             throw new OutOfMemoryError(\"No enough memory for aggregation\");\n/* 306 */           }\n/* 307 */         }\n/* 308 */\n/* 309 */       }\n/* 310 */\n/* 311 */       if (agg_fastAggBuffer != null) {\n/* 312 */         // common sub-expressions\n/* 313 */         boolean agg_isNull21 = false;\n/* 314 */         long agg_value23 = -1L;\n/* 315 */         if (!false) {\n/* 316 */           agg_value23 = (long) agg_expr_0;\n/* 317 */         }\n/* 318 */         // evaluate aggregate function\n/* 319 */         boolean agg_isNull23 = true;\n/* 320 */         double agg_value25 = -1.0;\n/* 321 */\n/* 322 */         boolean agg_isNull24 = agg_fastAggBuffer.isNullAt(0);\n/* 323 */         double agg_value26 = agg_isNull24 ?\n/* 324 */         -1.0 : (agg_fastAggBuffer.getDouble(0));\n/* 325 */         if (!agg_isNull24) {\n/* 326 */           agg_agg_isNull25 = true;\n/* 327 */           double agg_value27 = -1.0;\n/* 328 */           do {\n/* 329 */             boolean agg_isNull26 = agg_isNull21;\n/* 330 */             double agg_value28 = -1.0;\n/* 331 */             if (!agg_isNull21) {\n/* 332 */               agg_value28 = (double) agg_value23;\n/* 333 */             }\n/* 334 */             if (!agg_isNull26) {\n/* 335 */               agg_agg_isNull25 = false;\n/* 336 */               agg_value27 = agg_value28;\n/* 337 */               continue;\n/* 338 */             }\n/* 339 */\n/* 340 */             boolean agg_isNull27 = false;\n/* 341 */             double agg_value29 = -1.0;\n/* 342 */             if (!false) {\n/* 343 */               agg_value29 = (double) 0;\n/* 344 */             }\n/* 345 */             if (!agg_isNull27) {\n/* 346 */               agg_agg_isNull25 = false;\n/* 347 */               agg_value27 = agg_value29;\n/* 348 */               continue;\n/* 349 */             }\n/* 350 */\n/* 351 */           } while (false);\n/* 352 */\n/* 353 */           agg_isNull23 = false; // resultCode could change nullability.\n/* 354 */           agg_value25 = agg_value26 + agg_value27;\n/* 355 */\n/* 356 */         }\n/* 357 */         boolean agg_isNull29 = false;\n/* 358 */         long agg_value31 = -1L;\n/* 359 */         if (!false && agg_isNull21) {\n/* 360 */           boolean agg_isNull31 = agg_fastAggBuffer.isNullAt(1);\n/* 361 */           long agg_value33 = agg_isNull31 ?\n/* 362 */           -1L : (agg_fastAggBuffer.getLong(1));\n/* 363 */           agg_isNull29 = agg_isNull31;\n/* 364 */           agg_value31 = agg_value33;\n/* 365 */         } else {\n/* 366 */           boolean agg_isNull32 = true;\n/* 367 */           long agg_value34 = -1L;\n/* 368 */\n/* 369 */           boolean agg_isNull33 = agg_fastAggBuffer.isNullAt(1);\n/* 370 */           long agg_value35 = agg_isNull33 ?\n/* 371 */           -1L : (agg_fastAggBuffer.getLong(1));\n/* 372 */           if (!agg_isNull33) {\n/* 373 */             agg_isNull32 = false; // resultCode could change nullability.\n/* 374 */             agg_value34 = agg_value35 + 1L;\n/* 375 */\n/* 376 */           }\n/* 377 */           agg_isNull29 = agg_isNull32;\n/* 378 */           agg_value31 = agg_value34;\n/* 379 */         }\n/* 380 */         // update fast row\n/* 381 */         if (!agg_isNull23) {\n/* 382 */           agg_fastAggBuffer.setDouble(0, agg_value25);\n/* 383 */         } else {\n/* 384 */           agg_fastAggBuffer.setNullAt(0);\n/* 385 */         }\n/* 386 */\n/* 387 */         if (!agg_isNull29) {\n/* 388 */           agg_fastAggBuffer.setLong(1, agg_value31);\n/* 389 */         } else {\n/* 390 */           agg_fastAggBuffer.setNullAt(1);\n/* 391 */         }\n/* 392 */       } else {\n/* 393 */         // common sub-expressions\n/* 394 */         boolean agg_isNull7 = false;\n/* 395 */         long agg_value9 = -1L;\n/* 396 */         if (!false) {\n/* 397 */           agg_value9 = (long) agg_expr_0;\n/* 398 */         }\n/* 399 */         // evaluate aggregate function\n/* 400 */         boolean agg_isNull9 = true;\n/* 401 */         double agg_value11 = -1.0;\n/* 402 */\n/* 403 */         boolean agg_isNull10 = agg_unsafeRowAggBuffer.isNullAt(0);\n/* 404 */         double agg_value12 = agg_isNull10 ?\n/* 405 */         -1.0 : (agg_unsafeRowAggBuffer.getDouble(0));\n/* 406 */         if (!agg_isNull10) {\n/* 407 */           agg_agg_isNull11 = true;\n/* 408 */           double agg_value13 = -1.0;\n/* 409 */           do {\n/* 410 */             boolean agg_isNull12 = agg_isNull7;\n/* 411 */             double agg_value14 = -1.0;\n/* 412 */             if (!agg_isNull7) {\n/* 413 */               agg_value14 = (double) agg_value9;\n/* 414 */             }\n/* 415 */             if (!agg_isNull12) {\n/* 416 */               agg_agg_isNull11 = false;\n/* 417 */               agg_value13 = agg_value14;\n/* 418 */               continue;\n/* 419 */             }\n/* 420 */\n/* 421 */             boolean agg_isNull13 = false;\n/* 422 */             double agg_value15 = -1.0;\n/* 423 */             if (!false) {\n/* 424 */               agg_value15 = (double) 0;\n/* 425 */             }\n/* 426 */             if (!agg_isNull13) {\n/* 427 */               agg_agg_isNull11 = false;\n/* 428 */               agg_value13 = agg_value15;\n/* 429 */               continue;\n/* 430 */             }\n/* 431 */\n/* 432 */           } while (false);\n/* 433 */\n/* 434 */           agg_isNull9 = false; // resultCode could change nullability.\n/* 435 */           agg_value11 = agg_value12 + agg_value13;\n/* 436 */\n/* 437 */         }\n/* 438 */         boolean agg_isNull15 = false;\n/* 439 */         long agg_value17 = -1L;\n/* 440 */         if (!false && agg_isNull7) {\n/* 441 */           boolean agg_isNull17 = agg_unsafeRowAggBuffer.isNullAt(1);\n/* 442 */           long agg_value19 = agg_isNull17 ?\n/* 443 */           -1L : (agg_unsafeRowAggBuffer.getLong(1));\n/* 444 */           agg_isNull15 = agg_isNull17;\n/* 445 */           agg_value17 = agg_value19;\n/* 446 */         } else {\n/* 447 */           boolean agg_isNull18 = true;\n/* 448 */           long agg_value20 = -1L;\n/* 449 */\n/* 450 */           boolean agg_isNull19 = agg_unsafeRowAggBuffer.isNullAt(1);\n/* 451 */           long agg_value21 = agg_isNull19 ?\n/* 452 */           -1L : (agg_unsafeRowAggBuffer.getLong(1));\n/* 453 */           if (!agg_isNull19) {\n/* 454 */             agg_isNull18 = false; // resultCode could change nullability.\n/* 455 */             agg_value20 = agg_value21 + 1L;\n/* 456 */\n/* 457 */           }\n/* 458 */           agg_isNull15 = agg_isNull18;\n/* 459 */           agg_value17 = agg_value20;\n/* 460 */         }\n/* 461 */         // update unsafe row buffer\n/* 462 */         if (!agg_isNull9) {\n/* 463 */           agg_unsafeRowAggBuffer.setDouble(0, agg_value11);\n/* 464 */         } else {\n/* 465 */           agg_unsafeRowAggBuffer.setNullAt(0);\n/* 466 */         }\n/* 467 */\n/* 468 */         if (!agg_isNull15) {\n/* 469 */           agg_unsafeRowAggBuffer.setLong(1, agg_value17);\n/* 470 */         } else {\n/* 471 */           agg_unsafeRowAggBuffer.setNullAt(1);\n/* 472 */         }\n/* 473 */\n/* 474 */       }\n/* 475 */\n/* 476 */     }\n/* 477 */\n/* 478 */   }\n/* 479 */\n/* 480 */ }\n```\n\n## How was this patch tested?\n\nAdded UT into `WholeStageCodegenSuite`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #20779 from kiszk/SPARK-23598.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/1098933b0ac5cdb18101d3aebefa773c2ce05a50"},{"sha":"279b3db8970809104c30941254e57e3d62da5041","author":{"name":"“attilapiros”","email":"a9f53fe51637dc7476f2d3337ce4be519453fe6d@gmail.com"},"message":"[SPARK-22915][MLLIB] Streaming tests for spark.ml.feature, from N to Z\n\n# What changes were proposed in this pull request?\n\nAdds structured streaming tests using testTransformer for these suites:\n\n- NGramSuite\n- NormalizerSuite\n- OneHotEncoderEstimatorSuite\n- OneHotEncoderSuite\n- PCASuite\n- PolynomialExpansionSuite\n- QuantileDiscretizerSuite\n- RFormulaSuite\n- SQLTransformerSuite\n- StandardScalerSuite\n- StopWordsRemoverSuite\n- StringIndexerSuite\n- TokenizerSuite\n- RegexTokenizerSuite\n- VectorAssemblerSuite\n- VectorIndexerSuite\n- VectorSizeHintSuite\n- VectorSlicerSuite\n- Word2VecSuite\n\n# How was this patch tested?\n\nThey are unit test.\n\nAuthor: “attilapiros” <piros.attila.zsolt@gmail.com>\n\nCloses #20686 from attilapiros/SPARK-22915.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/279b3db8970809104c30941254e57e3d62da5041"},{"sha":"4f5bad615b47d743b8932aea1071652293981604","author":{"name":"smallory","email":"e92abcaaef3f07b9b8b6032ad63a222de23a400c@gmail.com"},"message":"[SPARK-23642][DOCS] AccumulatorV2 subclass isZero scaladoc fix\n\nAdded/corrected scaladoc for isZero on the DoubleAccumulator, CollectionAccumulator, and LongAccumulator subclasses of AccumulatorV2, particularly noting where there are requirements in addition to having a value of zero in order to return true.\n\n## What changes were proposed in this pull request?\n\nThree scaladoc comments are updated in AccumulatorV2.scala\nNo changes outside of comment blocks were made.\n\n## How was this patch tested?\n\nRunning \"sbt unidoc\", fixing style errors found, and reviewing the resulting local scaladoc in firefox.\n\nAuthor: smallory <s.mallory@gmail.com>\n\nCloses #20790 from smallory/patch-1.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/4f5bad615b47d743b8932aea1071652293981604"},{"sha":"7c3e8995f18a1fb57c1f2c1b98a1d47590e28f38","author":{"name":"Yuanjian Li","email":"64e33dcf1d622ec8e079b74108dbaad04f19fbb4@gmail.com"},"message":"[SPARK-23533][SS] Add support for changing ContinuousDataReader's startOffset\n\n## What changes were proposed in this pull request?\n\nAs discussion in #20675, we need add a new interface `ContinuousDataReaderFactory` to support the requirements of setting start offset in Continuous Processing.\n\n## How was this patch tested?\n\nExisting UT.\n\nAuthor: Yuanjian Li <xyliyuanjian@gmail.com>\n\nCloses #20689 from xuanyuanking/SPARK-23533.","distinct":true,"url":"https://api.github.com/repos/wwjiang007/spark/commits/7c3e8995f18a1fb57c1f2c1b98a1d47590e28f38"}]}
{"actor":{"display_login":"wwjiang007"}}