{"push_id":1329677076,"size":567,"distinct_size":118,"ref":"refs/heads/master","head":"1dd68d3827133d203e85294405400b04904879e0","before":"9d7a47406ed538f0005cdc7a62bc6e6f20634815","commits":[{"sha":"9d4e6212fa8d434089d32bff1217f39919abe44d","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16802] [SQL] fix overflow in LongToUnsafeRowMap\n\n## What changes were proposed in this pull request?\n\nThis patch fix the overflow in LongToUnsafeRowMap when the range of key is very wide (the key is much much smaller then minKey, for example, key is Long.MinValue, minKey is > 0).\n\n## How was this patch tested?\n\nAdded regression test (also for SPARK-16740)\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #14464 from davies/fix_overflow.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/9d4e6212fa8d434089d32bff1217f39919abe44d"},{"sha":"ac2a26d09e10c3f462ec773c3ebaa6eedae81ac0","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-16884] Move DataSourceScanExec out of ExistingRDD.scala file\n\n## What changes were proposed in this pull request?\n\nThis moves DataSourceScanExec out so it's more discoverable, and now that it doesn't necessarily depend on an existing RDD.  cc davies\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #14487 from ericl/split-scan.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/ac2a26d09e10c3f462ec773c3ebaa6eedae81ac0"},{"sha":"be8ea4b2f7ddf1196111acb61fe1a79866376003","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-16875][SQL] Add args checking for DataSet randomSplit and sample\n\n## What changes were proposed in this pull request?\n\nAdd the missing args-checking for randomSplit and sample\n\n## How was this patch tested?\nunit tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #14478 from zhengruifeng/fix_randomSplit.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/be8ea4b2f7ddf1196111acb61fe1a79866376003"},{"sha":"462784ffad77e43455dd0364064ce4994826a426","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16880][ML][MLLIB] make ann training data persisted if needed\n\n## What changes were proposed in this pull request?\n\nTo Make sure ANN layer input training data to be persisted,\nso that it can avoid overhead cost if the RDD need to be computed from lineage.\n\n## How was this patch tested?\n\nExisting Tests.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14483 from WeichenXu123/add_ann_persist_training_data.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/462784ffad77e43455dd0364064ce4994826a426"},{"sha":"1d781572e832058e2ef54bccd76ef71bc1fd548c","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16877][BUILD] Add rules for preventing to use Java annotations (Deprecated and Override)\n\n## What changes were proposed in this pull request?\n\nThis PR adds both rules for preventing to use `Deprecated` and `Override`.\n\n- Java's `Override`\n  It seems Scala compiler just ignores this. Apparently, `override` modifier is only mandatory for \" that override some other **concrete member definition** in a parent class\" but not for for **incomplete member definition** (such as ones from trait or abstract), see (http://www.scala-lang.org/files/archive/spec/2.11/05-classes-and-objects.html#override)\n\n  For a simple example,\n\n  - Normal class - needs `override` modifier\n\n  ```bash\n  scala> class A { def say = {}}\n  defined class A\n\n  scala> class B extends A { def say = {}}\n  <console>:8: error: overriding method say in class A of type => Unit;\n   method say needs `override' modifier\n         class B extends A { def say = {}}\n                                 ^\n  ```\n\n  - Trait - does not need `override` modifier\n\n  ```bash\n  scala> trait A { def say }\n  defined trait A\n\n  scala> class B extends A { def say = {}}\n  defined class B\n  ```\n\n  To cut this short, this case below is possible,\n\n  ```bash\n  scala> class B extends A {\n       |    Override\n       |    def say = {}\n       | }\n  defined class B\n  ```\n  we can write `Override` annotation (meaning nothing) which might confuse engineers that Java's annotation is working fine. It might be great if we prevent those potential confusion.\n\n- Java's `Deprecated`\n  When `Deprecated` is used,  it seems Scala compiler recognises this correctly but it seems we use Scala one `deprecated` across codebase.\n\n## How was this patch tested?\n\nManually tested, by inserting both `Override` and `Deprecated`. This will shows the error messages as below:\n\n```bash\nScalastyle checks failed at following occurrences:\n[error] ... : deprecated should be used instead of java.lang.Deprecated.\n```\n\n```basg\nScalastyle checks failed at following occurrences:\n[error] ... : override modifier should be used instead of java.lang.Override.\n```\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14490 from HyukjinKwon/SPARK-16877.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/1d781572e832058e2ef54bccd76ef71bc1fd548c"},{"sha":"0e2e5d7d0b42226c61c3200fd63d2831c558519d","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-16863][ML] ProbabilisticClassifier.fit check threshoulds' length\n\n## What changes were proposed in this pull request?\n\nAdd threshoulds' length checking for Classifiers which extends ProbabilisticClassifier\n\n## How was this patch tested?\n\nunit tests and manual tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #14470 from zhengruifeng/classifier_check_setThreshoulds_length.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/0e2e5d7d0b42226c61c3200fd63d2831c558519d"},{"sha":"9c15d079df2418a1412269a702f3a7861daee61c","author":{"name":"Sital Kedia","email":"e4343f82e25883a7c046475fe9553138fac5aa32@fb.com"},"message":"[SPARK-15074][SHUFFLE] Cache shuffle index file to speedup shuffle fetch\n\n## What changes were proposed in this pull request?\n\nShuffle fetch on large intermediate dataset is slow because the shuffle service open/close the index file for each shuffle fetch. This change introduces a cache for the index information so that we can avoid accessing the index files for each block fetch\n\n## How was this patch tested?\n\nTested by running a job on the cluster and the shuffle read time was reduced by 50%.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #12944 from sitalkedia/shuffle_service.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/9c15d079df2418a1412269a702f3a7861daee61c"},{"sha":"d91c6755ae46dfd1d9c8777830ab993b269e3051","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[HOTFIX] Remove unnecessary imports from #12944 that broke build\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14499 from JoshRosen/hotfix.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/d91c6755ae46dfd1d9c8777830ab993b269e3051"},{"sha":"53e766cfe2112265b606b68146a5798ccf7ec682","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"MAINTENANCE. Cleaning up stale PRs.\n\nClosing the following PRs due to requests or unresponsive users.\n\nCloses #13923\nCloses #14462\nCloses #13123\nCloses #14423 (requested by srowen)\nCloses #14424 (requested by srowen)\nCloses #14101 (requested by jkbradley)\nCloses #10676 (requested by srowen)\nCloses #10943 (requested by yhuai)\nCloses #9936\nCloses #10701\nCloses #10474\nCloses #13248\nCloses #14347\nCloses #10356\nCloses #9866\nCloses #14310 (requested by srowen)\nCloses #14390 (requested by srowen)\nCloses #14343 (requested by srowen)\nCloses #14402 (requested by srowen)\nCloses #14437 (requested by srowen)\nCloses #12000 (already merged)","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/53e766cfe2112265b606b68146a5798ccf7ec682"},{"sha":"1fa644497aed0a6d22f5fc7bf8e752508053b75b","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-16907][SQL] Fix performance regression for parquet table when vectorized parquet record reader is not being used\n\n## What changes were proposed in this pull request?\n\nFor non-partitioned parquet table, if the vectorized parquet record reader is not being used, Spark 2.0 adds an extra unnecessary memory copy to append partition values for each row.\n\nThere are several typical cases that vectorized parquet record reader is not being used:\n1. When the table schema is not flat, like containing nested fields.\n2. When `spark.sql.parquet.enableVectorizedReader = false`\n\nBy fixing this bug, we get about 20% - 30% performance gain in test case like this:\n\n```\n// Generates parquet table with nested columns\nspark.range(100000000).select(struct($\"id\").as(\"nc\")).write.parquet(\"/tmp/data4\")\n\ndef time[R](block: => R): Long = {\n    val t0 = System.nanoTime()\n    val result = block    // call-by-name\n    val t1 = System.nanoTime()\n    println(\"Elapsed time: \" + (t1 - t0)/1000000 + \"ms\")\n    (t1 - t0)/1000000\n}\n\nval x = ((0 until 20).toList.map(x => time(spark.read.parquet(\"/tmp/data4\").filter($\"nc.id\" < 100).collect()))).sum/20\n```\n\n## How was this patch tested?\n\nAfter a few times warm up, we get 26% performance improvement\n\nBefore fix:\n```\nAverage: 4584ms, raw data (10 tries): 4726ms 4509ms 4454ms 4879ms 4586ms 4733ms 4500ms 4361ms 4456ms 4640ms\n```\n\nAfter fix:\n```\nAverage: 3614ms, raw data(10 tries): 3554ms 3740ms 4019ms 3439ms 3460ms 3664ms 3557ms 3584ms 3612ms 3531ms\n```\n\nTest env: Intel(R) Core(TM) i7-6700 CPU  3.40GHz, Intel SSD SC2KW24\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #14445 from clockfly/fix_parquet_regression_2.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/1fa644497aed0a6d22f5fc7bf8e752508053b75b"},{"sha":"faaefab26ffea3a5edfeaff42db222c8cd3ff5f1","author":{"name":"Hiroshi Inoue","email":"3e15e65404b975101e094443b1e5ff2d3461d1d4@jp.ibm.com"},"message":"[SPARK-15726][SQL] Make DatasetBenchmark fairer among Dataset, DataFrame and RDD\n\n## What changes were proposed in this pull request?\n\nDatasetBenchmark compares the performances of RDD, DataFrame and Dataset while running the same operations. However, there are two problems that make the comparisons unfair.\n\n1) In backToBackMap test case, only DataFrame implementation executes less work compared to RDD or Dataset implementations. This test case processes Long+String pairs, but the output from the DataFrame implementation does not include String part while RDD or Dataset generates Long+String pairs as output. This difference significantly changes the performance characteristics due to the String manipulation and creation overheads.\n\n2) In back-to-back map and back-to-back filter test cases, `map` or `filter` operation is executed only once regardless of `numChains` parameter for RDD. Hence the execution times for RDD have been largely underestimated.\n\nOf course, these issues do not affect Spark users, but it may confuse Spark developers.\n\n## How was this patch tested?\nBy executing the DatasetBenchmark\n\nAuthor: Hiroshi Inoue <inouehrs@jp.ibm.com>\n\nCloses #13459 from inouehrs/fix_benchmark_fairness.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/faaefab26ffea3a5edfeaff42db222c8cd3ff5f1"},{"sha":"5effc016c893ce917d535cc1b5026d8e4c846721","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16879][SQL] unify logical plans for CREATE TABLE and CTAS\n\n## What changes were proposed in this pull request?\n\nwe have various logical plans for CREATE TABLE and CTAS: `CreateTableUsing`, `CreateTableUsingAsSelect`, `CreateHiveTableAsSelectLogicalPlan`. This PR unifies them to reduce the complexity and centralize the error handling.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14482 from cloud-fan/table.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/5effc016c893ce917d535cc1b5026d8e4c846721"},{"sha":"c9f2501af278241f780a38b9562e193755ed5af3","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-16312][STREAMING][KAFKA][DOC] Doc for Kafka 0.10 integration\n\n## What changes were proposed in this pull request?\nDoc for the Kafka 0.10 integration\n\n## How was this patch tested?\nScala code examples were taken from my example repo, so hopefully they compile.\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #14385 from koeninger/SPARK-16312.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/c9f2501af278241f780a38b9562e193755ed5af3"},{"sha":"e026064143367e4614cb866e321cc521fdde3170","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[MINOR] Update AccumulatorV2 doc to not mention \"+=\".\n\n## What changes were proposed in this pull request?\nAs reported by Bryan Cutler on the mailing list, AccumulatorV2 does not have a += method, yet the documentation still references it.\n\n## How was this patch tested?\nN/A\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14466 from petermaxlee/accumulator.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/e026064143367e4614cb866e321cc521fdde3170"},{"sha":"39a2b2ea74d420caa37019e3684f65b3a6fcb388","author":{"name":"Yuming Wang","email":"b4045a50547081d83941485c6596bd2032665522@gmail.com"},"message":"[SPARK-16625][SQL] General data types to be mapped to Oracle\n\n## What changes were proposed in this pull request?\n\nSpark will convert **BooleanType** to **BIT(1)**, **LongType** to **BIGINT**, **ByteType**  to **BYTE** when saving DataFrame to Oracle, but Oracle does not support BIT, BIGINT and BYTE types.\n\nThis PR is convert following _Spark Types_ to _Oracle types_ refer to [Oracle Developer's Guide](https://docs.oracle.com/cd/E19501-01/819-3659/gcmaz/)\n\nSpark Type | Oracle\n----|----\nBooleanType | NUMBER(1)\nIntegerType | NUMBER(10)\nLongType | NUMBER(19)\nFloatType | NUMBER(19, 4)\nDoubleType | NUMBER(19, 4)\nByteType | NUMBER(3)\nShortType | NUMBER(5)\n\n## How was this patch tested?\n\nAdd new tests in [JDBCSuite.scala](https://github.com/wangyum/spark/commit/22b0c2a4228cb8b5098ad741ddf4d1904e745ff6#diff-dc4b58851b084b274df6fe6b189db84d) and [OracleDialect.scala](https://github.com/wangyum/spark/commit/22b0c2a4228cb8b5098ad741ddf4d1904e745ff6#diff-5e0cadf526662f9281aa26315b3750ad)\n\nAuthor: Yuming Wang <wgyumg@gmail.com>\n\nCloses #14377 from wangyum/SPARK-16625.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/39a2b2ea74d420caa37019e3684f65b3a6fcb388"},{"sha":"2460f03ffe94154b73995e4f16dd799d1a0f56b8","author":{"name":"Sylvain Zimmer","email":"028d819198e8f89377b4eb9de11e35c0685e4939@sylvainzimmer.com"},"message":"[SPARK-16826][SQL] Switch to java.net.URI for parse_url()\n\n## What changes were proposed in this pull request?\nThe java.net.URL class has a globally synchronized Hashtable, which limits the throughput of any single executor doing lots of calls to parse_url(). Tests have shown that a 36-core machine can only get to 10% CPU use because the threads are locked most of the time.\n\nThis patch switches to java.net.URI which has less features than java.net.URL but focuses on URI parsing, which is enough for parse_url().\n\nNew tests were added to make sure a few common edge cases didn't change behaviour.\nhttps://issues.apache.org/jira/browse/SPARK-16826\n\n## How was this patch tested?\nI've kept the old URL code commented for now, so that people can verify that the new unit tests do pass with java.net.URL.\n\nThanks to srowen for the help!\n\nAuthor: Sylvain Zimmer <sylvain@sylvainzimmer.com>\n\nCloses #14488 from sylvinus/master.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/2460f03ffe94154b73995e4f16dd799d1a0f56b8"},{"sha":"180fd3e0a3426db200c97170926afb60751dfd0e","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-16421][EXAMPLES][ML] Improve ML Example Outputs\n\n## What changes were proposed in this pull request?\nImprove example outputs to better reflect the functionality that is being presented.  This mostly consisted of modifying what was printed at the end of the example, such as calling show() with truncate=False, but sometimes required minor tweaks in the example data to get relevant output.  Explicitly set parameters when they are used as part of the example.  Fixed Java examples that failed to run because of using old-style MLlib Vectors or problem with schema.  Synced examples between different APIs.\n\n## How was this patch tested?\nRan each example for Scala, Python, and Java and made sure output was legible on a terminal of width 100.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #14308 from BryanCutler/ml-examples-improve-output-SPARK-16260.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/180fd3e0a3426db200c97170926afb60751dfd0e"},{"sha":"1f96c97f2374a95140a0c72b1f4eae50ac21d84a","author":{"name":"Ekasit Kijsipongse","email":"7c201144b04aaa5174a4bea715db92e5131cc035@gmail.com"},"message":"[SPARK-13238][CORE] Add ganglia dmax parameter\n\nThe current ganglia reporter doesn't set metric expiration time (dmax). The metrics of all finished applications are indefinitely left displayed in ganglia web. The dmax parameter allows user to set the lifetime of the metrics. The default value is 0 for compatibility with previous versions.\n\nAuthor: Ekasit Kijsipongse <ekasitk@gmail.com>\n\nCloses #11127 from ekasitk/ganglia-dmax.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/1f96c97f2374a95140a0c72b1f4eae50ac21d84a"},{"sha":"6cbde337a539e5bb170d0eb81f715a95ee9c9af3","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16750][FOLLOW-UP][ML] Add transformSchema for StringIndexer/VectorAssembler and fix failed tests.\n\n## What changes were proposed in this pull request?\nThis is follow-up for #14378. When we add ```transformSchema``` for all estimators and transformers, I found there are tests failed for ```StringIndexer``` and ```VectorAssembler```. So I moved these parts of work separately in this PR, to make it more clear to review.\nThe corresponding tests should throw ```IllegalArgumentException``` at schema validation period after we add ```transformSchema```. It's efficient that to throw exception at the start of ```fit``` or ```transform``` rather than during the process.\n\n## How was this patch tested?\nModified unit tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14455 from yanboliang/transformSchema.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/6cbde337a539e5bb170d0eb81f715a95ee9c9af3"},{"sha":"e679bc3c1cd418ef0025d2ecbc547c9660cac433","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-16901] Hive settings in hive-site.xml may be overridden by Hive's default values\n\n## What changes were proposed in this pull request?\nWhen we create the HiveConf for metastore client, we use a Hadoop Conf as the base, which may contain Hive settings in hive-site.xml (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala#L49). However, HiveConf's initialize function basically ignores the base Hadoop Conf and always its default values (i.e. settings with non-null default values) as the base (https://github.com/apache/hive/blob/release-1.2.1/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2687). So, even a user put javax.jdo.option.ConnectionURL in hive-site.xml, it is not used and Hive will use its default, which is jdbc:derby:;databaseName=metastore_db;create=true.\n\nThis issue only shows up when `spark.sql.hive.metastore.jars` is not set to builtin.\n\n## How was this patch tested?\nNew test in HiveSparkSubmitSuite.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #14497 from yhuai/SPARK-16901.","distinct":false,"url":"https://api.github.com/repos/shubhamchopra/spark/commits/e679bc3c1cd418ef0025d2ecbc547c9660cac433"}]}
{"actor":{"display_login":"shubhamchopra"}}