{"push_id":2497339483,"size":83,"distinct_size":2,"ref":"refs/heads/SPARK-23584","head":"a3f0c70793b6f72f423c101fd2364705f4f6a59c","before":"3da287be685a018b1148bbc634e6462fc112b846","commits":[{"sha":"c5c8b544047a83cb6128a20d31f1d943a15f9260","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-23593][SQL] Add interpreted execution for InitializeJavaBean expression\n\n## What changes were proposed in this pull request?\n\nAdd interpreted execution for `InitializeJavaBean` expression.\n\n## How was this patch tested?\n\nAdded unit test.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #20756 from viirya/SPARK-23593.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/c5c8b544047a83cb6128a20d31f1d943a15f9260"},{"sha":"1822ecda51cc9e14bb18050e0b8c270fee47ced7","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23582][SQL] StaticInvoke should support interpreted execution\n\n## What changes were proposed in this pull request?\n\nThis pr added interpreted execution for `StaticInvoke`.\n\n## How was this patch tested?\n\nAdded tests in `ObjectExpressionsSuite`.\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #20753 from kiszk/SPARK-23582.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/1822ecda51cc9e14bb18050e0b8c270fee47ced7"},{"sha":"b2329fb1fcdc0e93c4bdc39d574cde7328ef6094","author":{"name":"Herman van Hovell","email":"679c4ded150c6b9a8052807f108ce836961fed8d@databricks.com"},"message":"Revert \"[SPARK-23593][SQL] Add interpreted execution for InitializeJavaBean expression\"\n\nThis reverts commit c5c8b544047a83cb6128a20d31f1d943a15f9260.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/b2329fb1fcdc0e93c4bdc39d574cde7328ef6094"},{"sha":"d9ca1c906bd0571802f2297c36b407e660fcdb64","author":{"name":"Liang-Chi Hsieh","email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com"},"message":"[SPARK-23593][SQL] Add interpreted execution for InitializeJavaBean expression\n\n## What changes were proposed in this pull request?\n\nAdd interpreted execution for `InitializeJavaBean` expression.\n\n## How was this patch tested?\n\nAdded unit test.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #20985 from viirya/SPARK-23593-2.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/d9ca1c906bd0571802f2297c36b407e660fcdb64"},{"sha":"4807d381bb113a5c61e6dad88202f23a8b6dd141","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks to choose several types of memory block\n\n## What changes were proposed in this pull request?\n\nThis PR allows us to use one of several types of `MemoryBlock`, such as byte array, int array, long array, or `java.nio.DirectByteBuffer`. To use `java.nio.DirectByteBuffer` allows to have off heap memory which is automatically deallocated by JVM. `MemoryBlock`  class has primitive accessors like `Platform.getInt()`, `Platform.putint()`, or `Platform.copyMemory()`.\n\nThis PR uses `MemoryBlock` for `OffHeapColumnVector`, `UTF8String`, and other places. This PR can improve performance of operations involving memory accesses (e.g. `UTF8String.trim`) by 1.8x.\n\nFor now, this PR does not use `MemoryBlock` for `BufferHolder` based on cloud-fan's [suggestion](https://github.com/apache/spark/pull/11494#issuecomment-309694290).\n\nSince this PR is a successor of #11494, close #11494. Many codes were ported from #11494. Many efforts were put here. **I think this PR should credit to yzotov.**\n\nThis PR can achieve **1.1-1.4x performance improvements** for  operations in `UTF8String` or `Murmur3_x86_32`. Other operations are almost comparable performances.\n\nWithout this PR\n```\nOpenJDK 64-Bit Server VM 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13 on Linux 4.4.0-22-generic\nIntel(R) Xeon(R) CPU E5-2667 v3  3.20GHz\nOpenJDK 64-Bit Server VM 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13 on Linux 4.4.0-22-generic\nIntel(R) Xeon(R) CPU E5-2667 v3  3.20GHz\nHash byte arrays with length 268435487:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nMurmur3_x86_32                                 526 /  536          0.0   131399881.5       1.0X\n\nUTF8String benchmark:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nhashCode                                       525 /  552       1022.6           1.0       1.0X\nsubstring                                      414 /  423       1298.0           0.8       1.3X\n```\n\nWith this PR\n```\nOpenJDK 64-Bit Server VM 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13 on Linux 4.4.0-22-generic\nIntel(R) Xeon(R) CPU E5-2667 v3  3.20GHz\nHash byte arrays with length 268435487:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nMurmur3_x86_32                                 474 /  488          0.0   118552232.0       1.0X\n\nUTF8String benchmark:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nhashCode                                       476 /  480       1127.3           0.9       1.0X\nsubstring                                      287 /  291       1869.9           0.5       1.7X\n```\n\nBenchmark program\n```\ntest(\"benchmark Murmur3_x86_32\") {\n  val length = 8192 * 32768 + 31\n  val seed = 42L\n  val iters = 1 << 2\n  val random = new Random(seed)\n  val arrays = Array.fill[MemoryBlock](numArrays) {\n    val bytes = new Array[Byte](length)\n    random.nextBytes(bytes)\n    new ByteArrayMemoryBlock(bytes, Platform.BYTE_ARRAY_OFFSET, length)\n  }\n\n  val benchmark = new Benchmark(\"Hash byte arrays with length \" + length,\n    iters * numArrays, minNumIters = 20)\n  benchmark.addCase(\"HiveHasher\") { _: Int =>\n    var sum = 0L\n    for (_ <- 0L until iters) {\n      sum += HiveHasher.hashUnsafeBytesBlock(\n        arrays(i), Platform.BYTE_ARRAY_OFFSET, length)\n    }\n  }\n  benchmark.run()\n}\n\ntest(\"benchmark UTF8String\") {\n  val N = 512 * 1024 * 1024\n  val iters = 2\n  val benchmark = new Benchmark(\"UTF8String benchmark\", N, minNumIters = 20)\n  val str0 = new java.io.StringWriter() { { for (i <- 0 until N) { write(\" \") } } }.toString\n  val s0 = UTF8String.fromString(str0)\n  benchmark.addCase(\"hashCode\") { _: Int =>\n    var h: Int = 0\n    for (_ <- 0L until iters) { h += s0.hashCode }\n  }\n  benchmark.addCase(\"substring\") { _: Int =>\n    var s: UTF8String = null\n    for (_ <- 0L until iters) { s = s0.substring(N / 2 - 5, N / 2 + 5) }\n  }\n  benchmark.run()\n}\n```\n\nI run [this benchmark program](https://gist.github.com/kiszk/94f75b506c93a663bbbc372ffe8f05de) using [the commit](https://github.com/apache/spark/pull/19222/commits/ee5a79861c18725fb1cd9b518cdfd2489c05b81d6). I got the following results:\n\n```\nOpenJDK 64-Bit Server VM 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12 on Linux 4.4.0-66-generic\nIntel(R) Xeon(R) CPU E5-2667 v3  3.20GHz\nMemory access benchmarks:                Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nByteArrayMemoryBlock get/putInt()              220 /  221        609.3           1.6       1.0X\nPlatform get/putInt(byte[])                    220 /  236        610.9           1.6       1.0X\nPlatform get/putInt(Object)                    492 /  494        272.8           3.7       0.4X\nOnHeapMemoryBlock get/putLong()                322 /  323        416.5           2.4       0.7X\nlong[]                                         221 /  221        608.0           1.6       1.0X\nPlatform get/putLong(long[])                   321 /  321        418.7           2.4       0.7X\nPlatform get/putLong(Object)                   561 /  563        239.2           4.2       0.4X\n```\n\nI also run [this benchmark program](https://gist.github.com/kiszk/5fdb4e03733a5d110421177e289d1fb5) for comparing performance of `Platform.copyMemory()`.\n```\nOpenJDK 64-Bit Server VM 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12 on Linux 4.4.0-66-generic\nIntel(R) Xeon(R) CPU E5-2667 v3  3.20GHz\nPlatform copyMemory:                     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nObject to Object                              1961 / 1967          8.6         116.9       1.0X\nSystem.arraycopy Object to Object             1917 / 1921          8.8         114.3       1.0X\nbyte array to byte array                      1961 / 1968          8.6         116.9       1.0X\nSystem.arraycopy byte array to byte array      1909 / 1937          8.8         113.8       1.0X\nint array to int array                        1921 / 1990          8.7         114.5       1.0X\ndouble array to double array                  1918 / 1923          8.7         114.3       1.0X\nObject to byte array                          1961 / 1967          8.6         116.9       1.0X\nObject to short array                         1965 / 1972          8.5         117.1       1.0X\nObject to int array                           1910 / 1915          8.8         113.9       1.0X\nObject to float array                         1971 / 1978          8.5         117.5       1.0X\nObject to double array                        1919 / 1944          8.7         114.4       1.0X\nbyte array to Object                          1959 / 1967          8.6         116.8       1.0X\nint array to Object                           1961 / 1970          8.6         116.9       1.0X\ndouble array to Object                        1917 / 1924          8.8         114.3       1.0X\n```\n\nThese results show three facts:\n1. According to the second/third or sixth/seventh results in the first experiment, if we use `Platform.get/putInt(Object)`, we achieve more than 2x worse performance than `Platform.get/putInt(byte[])` with concrete type (i.e. `byte[]`).\n2. According to the second/third or fourth/fifth/sixth results in the first experiment, the fastest way to access an array element on Java heap is `array[]`. **Cons of `array[]` is that it is not possible to support unaligned-8byte access.**\n3. According to the first/second/third or fourth/sixth/seventh results in the first experiment, `getInt()/putInt() or getLong()/putLong()` in subclasses of `MemoryBlock` can achieve comparable performance to `Platform.get/putInt()` or `Platform.get/putLong()` with concrete type (second or sixth result). There is no overhead regarding virtual call.\n4. According to results in the second experiment, for `Platform.copy()`, to pass `Object` can achieve the same performance as to pass any type of primitive array as source or destination.\n5. According to second/fourth results in the second experiment, `Platform.copy()` can achieve the same performance as `System.arrayCopy`. **It would be good to use `Platform.copy()` since `Platform.copy()` can take any types for src and dst.**\n\nWe are incrementally replace `Platform.get/putXXX` with `MemoryBlock.get/putXXX`. This is because we have two advantages.\n1) Achieve better performance due to having a concrete type for an array.\n2) Use simple OO design instead of passing `Object`\nIt is easy to use `MemoryBlock` in `InternalRow`, `BufferHolder`, `TaskMemoryManager`, and others that are already abstracted. It is not easy to use `MemoryBlock` in utility classes related to hashing or others.\n\nOther candidates are\n- UnsafeRow, UnsafeArrayData, UnsafeMapData, SpecificUnsafeRowJoiner\n- UTF8StringBuffer\n- BufferHolder\n- TaskMemoryManager\n- OnHeapColumnVector\n- BytesToBytesMap\n- CachedBatch\n- classes for hash\n- others.\n\n## How was this patch tested?\n\nAdded `UnsafeMemoryAllocator`\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #19222 from kiszk/SPARK-10399.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/4807d381bb113a5c61e6dad88202f23a8b6dd141"},{"sha":"f2ac0879561cde63ed4eb759f5efa0a5ce393a22","author":{"name":"Yogesh Garg","email":"7c1036458da30f3eb7cec2543056964b24449db4"},"message":"[SPARK-23870][ML] Forward RFormula handleInvalid Param to VectorAssembler to handle invalid values in non-string columns\n\n## What changes were proposed in this pull request?\n\n`handleInvalid` Param was forwarded to the VectorAssembler used by RFormula.\n\n## How was this patch tested?\n\nadded a test and ran all tests for RFormula and VectorAssembler\n\nAuthor: Yogesh Garg <yogesh(dot)garg()databricks(dot)com>\n\nCloses #20970 from yogeshg/spark_23562.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/f2ac0879561cde63ed4eb759f5efa0a5ce393a22"},{"sha":"d65e531b44a388fed25d3cbf28fdce5a2d0598e6","author":{"name":"JiahuiJiang","email":"4aef6a455c4637af7e425fce8c6dec77fa0d7996@palantir.com"},"message":"[SPARK-23823][SQL] Keep origin in transformExpression\n\nFixes https://issues.apache.org/jira/browse/SPARK-23823\n\nKeep origin for all the methods using transformExpression\n\n## What changes were proposed in this pull request?\n\nKeep origin in transformExpression\n\n## How was this patch tested?\n\nManually tested that this fixes https://issues.apache.org/jira/browse/SPARK-23823 and columns have correct origins after Analyzer.analyze\n\nAuthor: JiahuiJiang <jjiang@palantir.com>\nAuthor: Jiahui Jiang <jjiang@palantir.com>\n\nCloses #20961 from JiahuiJiang/jj/keep-origin.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/d65e531b44a388fed25d3cbf28fdce5a2d0598e6"},{"sha":"249007e37f51f00d14e596692aeac87fbc10b520","author":{"name":"Gengliang Wang","email":"bfe1a7b36425bd89859a70a36586361d28ec35eb@databricks.com"},"message":"[SPARK-19724][SQL] create a managed table with an existed default table should throw an exception\n\n## What changes were proposed in this pull request?\nThis PR is to finish https://github.com/apache/spark/pull/17272\n\nThis JIRA is a follow up work after SPARK-19583\n\nAs we discussed in that PR\n\nThe following DDL for a managed table with an existed default location should throw an exception:\n\nCREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...\nCREATE TABLE ... (PARTITIONED BY ...)\nCurrently there are some situations which are not consist with above logic:\n\nCREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location\nsituation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog)\n\nCREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...\nsituation: hive table succeed with an existed default location\n\nThis PR is going to make above two situations consist with the logic that it should throw an exception\nwith an existed default location.\n## How was this patch tested?\n\nunit test added\n\nAuthor: Gengliang Wang <gengliang.wang@databricks.com>\n\nCloses #20886 from gengliangwang/pr-17272.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/249007e37f51f00d14e596692aeac87fbc10b520"},{"sha":"6ade5cbb498f6c6ea38779b97f2325d5cf5013f2","author":{"name":"Daniel Sakuma","email":"276763d57141bb62272cbf17648b4fa6810acbf9@gmail.com"},"message":"[MINOR][DOC] Fix some typos and grammar issues\n\n## What changes were proposed in this pull request?\n\nEasy fix in the documentation.\n\n## How was this patch tested?\n\nN/A\n\nCloses #20948\n\nAuthor: Daniel Sakuma <dsakuma@gmail.com>\n\nCloses #20928 from dsakuma/fix_typo_configuration_docs.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/6ade5cbb498f6c6ea38779b97f2325d5cf5013f2"},{"sha":"94524019315ad463f9bc13c107131091d17c6af9","author":{"name":"Yuchen Huo","email":"c2fade40ae3cd03cb7abec04373cdda25c3b2be0@databricks.com"},"message":"[SPARK-23822][SQL] Improve error message for Parquet schema mismatches\n\n## What changes were proposed in this pull request?\n\nThis pull request tries to improve the error message for spark while reading parquet files with different schemas, e.g. One with a STRING column and the other with a INT column. A new ParquetSchemaColumnConvertNotSupportedException is added to replace the old UnsupportedOperationException. The Exception is again wrapped in FileScanRdd.scala to throw a more a general QueryExecutionException with the actual parquet file name which trigger the exception.\n\n## How was this patch tested?\n\nUnit tests added to check the new exception and verify the error messages.\n\nAlso manually tested with two parquet with different schema to check the error message.\n\n<img width=\"1125\" alt=\"screen shot 2018-03-30 at 4 03 04 pm\" src=\"https://user-images.githubusercontent.com/37087310/38156580-dd58a140-3433-11e8-973a-b816d859fbe1.png\">\n\nAuthor: Yuchen Huo <yuchen.huo@databricks.com>\n\nCloses #20953 from yuchenhuo/SPARK-23822.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/94524019315ad463f9bc13c107131091d17c6af9"},{"sha":"d766ea2ff2bf59afbd631d3cc2e43bebfccdebed","author":{"name":"Li Jin","email":"a3cb52d5143fd7b73dc39a807834f00d59edd35a@gmail.com"},"message":"[SPARK-23861][SQL][DOC] Clarify default window frame with and without orderBy clause\n\n## What changes were proposed in this pull request?\n\nAdd docstring to clarify default window frame boundaries with and without orderBy clause\n\n## How was this patch tested?\n\nManually generate doc and check.\n\nAuthor: Li Jin <ice.xelloss@gmail.com>\n\nCloses #20978 from icexelloss/SPARK-23861-window-doc.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/d766ea2ff2bf59afbd631d3cc2e43bebfccdebed"},{"sha":"c926acf719a6deb9d884a0f19bde075c312bfe5a","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23882][CORE] UTF8StringSuite.writeToOutputStreamUnderflow() is not expected to be supported\n\n## What changes were proposed in this pull request?\n\nThis PR excludes an existing UT [`writeToOutputStreamUnderflow()`](https://github.com/apache/spark/blob/master/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java#L519-L532) in `UTF8StringSuite`.\n\nAs discussed [here](https://github.com/apache/spark/pull/19222#discussion_r177692142), the behavior of this test looks surprising. This test seems to access metadata area of the JVM object where is reserved by `Platform.BYTE_ARRAY_OFFSET`.\n\nThis test is introduced thru #16089 by NathanHowell. More specifically, [the commit](https://github.com/apache/spark/pull/16089/commits/27c102deb1701fe62f776fe4da61dac959270b73) `Improve test coverage of UTFString.write` introduced this UT. However, I cannot find any discussion about this UT.\n\nI think that it would be good to exclude this UT.\n\n```java\n  public void writeToOutputStreamUnderflow() throws IOException {\n    // offset underflow is apparently supported?\n    final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();\n    final byte[] test = \"01234567\".getBytes(StandardCharsets.UTF_8);\n\n    for (int i = 1; i <= Platform.BYTE_ARRAY_OFFSET; ++i) {\n      new UTF8String(\n        new ByteArrayMemoryBlock(test, Platform.BYTE_ARRAY_OFFSET - i, test.length + i))\n          .writeTo(outputStream);\n      final ByteBuffer buffer = ByteBuffer.wrap(outputStream.toByteArray(), i, test.length);\n      assertEquals(\"01234567\", StandardCharsets.UTF_8.decode(buffer).toString());\n      outputStream.reset();\n    }\n  }\n```\n\n## How was this patch tested?\n\nExisting UTs\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #20995 from kiszk/SPARK-23882.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/c926acf719a6deb9d884a0f19bde075c312bfe5a"},{"sha":"d23a805f975f209f273db2b52de3f336be17d873","author":{"name":"Bago Amirbekian","email":"f9ab3b835b5fadeacae8b72242f24755535a123f@databricks.com"},"message":"[SPARK-23859][ML] Initial PR for Instrumentation improvements: UUID and logging levels\n\n## What changes were proposed in this pull request?\n\nInitial PR for Instrumentation improvements: UUID and logging levels.\nThis PR takes over #20837\n\nCloses #20837\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Bago Amirbekian <bago@databricks.com>\nAuthor: WeichenXu <weichen.xu@databricks.com>\n\nCloses #20982 from WeichenXu123/better-instrumentation.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/d23a805f975f209f273db2b52de3f336be17d873"},{"sha":"b6935ffb4dfb1d9fdf36ba402ac07bd02978c012","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-10399][SPARK-23879][HOTFIX] Fix Java lint errors\n\n## What changes were proposed in this pull request?\n\nThis PR fixes the following errors in [Java lint](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-lint/7717/console) after #19222 has been merged. These errors were pointed by ueshin .\n\n```\n[ERROR] src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java:[57] (sizes) LineLength: Line is longer than 100 characters (found 106).\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java:[26,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.Platform.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OffHeapMemoryBlock.java:[23,10] (modifier) ModifierOrder: 'public' modifier out of order with the JLS suggestions.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[64,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[69,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[74,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[79,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[84,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[89,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[94,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[99,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[104,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[109,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[114,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[119,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[124,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java:[129,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[60,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[65,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[70,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[75,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[80,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[85,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[90,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[95,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[100,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[105,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[110,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[115,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[120,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/ByteArrayMemoryBlock.java:[125,10] (modifier) RedundantModifier: Redundant 'final' modifier.\n[ERROR] src/main/java/org/apache/spark/unsafe/memory/MemoryBlock.java:[114,16] (modifier) ModifierOrder: 'static' modifier out of order with the JLS suggestions.\n[ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/HiveHasher.java:[20,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.Platform.\n[ERROR] src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java:[30,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.memory.MemoryBlock.\n[ERROR] src/test/java/org/apache/spark/unsafe/memory/MemoryBlockSuite.java:[126,15] (naming) MethodName: Method name 'ByteArrayMemoryBlockTest' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.\n[ERROR] src/test/java/org/apache/spark/unsafe/memory/MemoryBlockSuite.java:[143,15] (naming) MethodName: Method name 'OnHeapMemoryBlockTest' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.\n[ERROR] src/test/java/org/apache/spark/unsafe/memory/MemoryBlockSuite.java:[160,15] (naming) MethodName: Method name 'OffHeapArrayMemoryBlockTest' must match pattern '^[a-z][a-z0-9][a-zA-Z0-9_]*$'.\n[ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/XXH64.java:[19,8] (imports) UnusedImports: Unused import - com.google.common.primitives.Ints.\n[ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/XXH64.java:[21,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.Platform.\n[ERROR] src/test/java/org/apache/spark/sql/catalyst/expressions/HiveHasherSuite.java:[20,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.Platform.\n```\n\n## How was this patch tested?\n\nExisting UTs\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #20991 from kiszk/SPARK-10399-jlint.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/b6935ffb4dfb1d9fdf36ba402ac07bd02978c012"},{"sha":"e998250588de0df250e2800278da4d3e3705c259","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-23828][ML][PYTHON] PySpark StringIndexerModel should have constructor from labels\n\n## What changes were proposed in this pull request?\n\nThe Scala StringIndexerModel has an alternate constructor that will create the model from an array of label strings.  Add the corresponding Python API:\n\nmodel = StringIndexerModel.from_labels([\"a\", \"b\", \"c\"])\n\n## How was this patch tested?\n\nAdd doctest and unit test.\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #20968 from huaxingao/spark-23828.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/e998250588de0df250e2800278da4d3e3705c259"},{"sha":"6ab134ca7d8f7802a6d196929513cc02b9b4d35d","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-21898][ML][FOLLOWUP] Fix Scala 2.12 build.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up pr of #19108 which broke Scala 2.12 build.\n\n```\n[error] /Users/ueshin/workspace/apache-spark/spark/mllib/src/main/scala/org/apache/spark/ml/stat/KolmogorovSmirnovTest.scala:86: overloaded method value test with alternatives:\n[error]   (dataset: org.apache.spark.sql.DataFrame,sampleCol: String,cdf: org.apache.spark.api.java.function.Function[java.lang.Double,java.lang.Double])org.apache.spark.sql.DataFrame <and>\n[error]   (dataset: org.apache.spark.sql.DataFrame,sampleCol: String,cdf: scala.Double => scala.Double)org.apache.spark.sql.DataFrame\n[error]  cannot be applied to (org.apache.spark.sql.DataFrame, String, scala.Double => java.lang.Double)\n[error]     test(dataset, sampleCol, (x: Double) => cdf.call(x))\n[error]     ^\n[error] one error found\n```\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #20994 from ueshin/issues/SPARK-21898/fix_scala-2.12.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/6ab134ca7d8f7802a6d196929513cc02b9b4d35d"},{"sha":"2c1fe647575e97e28b2232478ca86847d113e185","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-23847][PYTHON][SQL] Add asc_nulls_first, asc_nulls_last to PySpark\n\n## What changes were proposed in this pull request?\n\nColumn.scala and Functions.scala have asc_nulls_first, asc_nulls_last,  desc_nulls_first and desc_nulls_last. Add the corresponding python APIs in column.py and functions.py\n\n## How was this patch tested?\nAdd doctest\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #20962 from huaxingao/spark-23847.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/2c1fe647575e97e28b2232478ca86847d113e185"},{"sha":"6a734575a80e6b4ec4963206254451f05d64b742","author":{"name":"Maxim Gekk","email":"05b3389ec46e7c78a10351def567499e4feebe3f@databricks.com"},"message":"[SPARK-23849][SQL] Tests for the samplingRatio option of JSON datasource\n\n## What changes were proposed in this pull request?\n\nProposed tests checks that only subset of input dataset is touched during schema inferring.\n\nAuthor: Maxim Gekk <maxim.gekk@databricks.com>\n\nCloses #20963 from MaxGekk/json-sampling-tests.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/6a734575a80e6b4ec4963206254451f05d64b742"},{"sha":"710a68cec27a94c2df10d8b4022a755a94a5443b","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23892][TEST] Improve converge and fix lint error in UTF8String-related tests\n\n## What changes were proposed in this pull request?\n\nThis PR improves test coverage in `UTF8StringSuite` and code efficiency in `UTF8StringPropertyCheckSuite`.\n\nThis PR also fixes lint-java issue in `UTF8StringSuite` reported at [here](https://github.com/apache/spark/pull/20995#issuecomment-379325527)\n\n```[ERROR] src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java:[28,8] (imports) UnusedImports: Unused import - org.apache.spark.unsafe.Platform.```\n\n## How was this patch tested?\n\nExisting UT\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #21000 from kiszk/SPARK-23892.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/710a68cec27a94c2df10d8b4022a755a94a5443b"},{"sha":"8d40a79a077a30024a8ef921781b68f6f7e542d1","author":{"name":"Kazuaki Ishizaki","email":"409019bab0e07036ad6abfa289bcfe098eb16317@jp.ibm.com"},"message":"[SPARK-23893][CORE][SQL] Avoid possible integer overflow in multiplication\n\n## What changes were proposed in this pull request?\n\nThis PR avoids possible overflow at an operation `long = (long)(int * int)`. The multiplication of large positive integer values may set one to MSB. This leads to a negative value in long while we expected a positive value (e.g. `0111_0000_0000_0000 * 0000_0000_0000_0010`).\n\nThis PR performs long cast before the multiplication to avoid this situation.\n\n## How was this patch tested?\n\nExisting UTs\n\nAuthor: Kazuaki Ishizaki <ishizaki@jp.ibm.com>\n\nCloses #21002 from kiszk/SPARK-23893.","distinct":false,"url":"https://api.github.com/repos/maropu/spark/commits/8d40a79a077a30024a8ef921781b68f6f7e542d1"}]}
{"actor":{"display_login":"maropu"}}