{"push_id":956733958,"size":80,"distinct_size":80,"ref":"refs/heads/master","head":"7a90eaf71fc905182e3e259fd1abddd419ce8a8d","before":"e092deca6d8c3e3af660a0ae7dc3eba5710ef62b","commits":[{"sha":"f4004601b00044e1d5c57e72c45b815f36feac3e","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12971] Fix Hive tests which fail in Hadoop-2.3 SBT build\n\nErrorPositionSuite and one of the HiveComparisonTest tests have been consistently failing on the Hadoop 2.3 SBT build (but on no other builds). I believe that this is due to test isolation issues (e.g. tests sharing state via the sets of temporary tables that are registered to TestHive).\n\nThis patch attempts to improve the isolation of these tests in order to address this issue.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10884 from JoshRosen/fix-failing-hadoop-2.3-hive-tests.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/f4004601b00044e1d5c57e72c45b815f36feac3e"},{"sha":"a83400135d7506fd2e1c8971f372012fcb3cef23","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-10498][TOOLS][BUILD] Add requirements.txt file for dev python tools\n\nMinor since so few people use them, but it would probably be good to have a requirements file for our python release tools for easier setup (also version pinning).\n\ncc JoshRosen who looked at the original JIRA.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #10871 from holdenk/SPARK-10498-add-requirements-file-for-dev-python-tools.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/a83400135d7506fd2e1c8971f372012fcb3cef23"},{"sha":"e789b1d2c1eab6187f54424ed92697ca200c3101","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[SPARK-12120][PYSPARK] Improve exception message when failing to init…\n\n…ialize HiveContext in PySpark\n\ndavies Mind to review ?\n\nThis is the error message after this PR\n\n```\n15/12/03 16:59:53 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException\n/Users/jzhang/github/spark/python/pyspark/sql/context.py:689: UserWarning: You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly\n  warnings.warn(\"You must build Spark with Hive. \"\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"/Users/jzhang/github/spark/python/pyspark/sql/context.py\", line 663, in read\n    return DataFrameReader(self)\n  File \"/Users/jzhang/github/spark/python/pyspark/sql/readwriter.py\", line 56, in __init__\n    self._jreader = sqlContext._ssql_ctx.read()\n  File \"/Users/jzhang/github/spark/python/pyspark/sql/context.py\", line 692, in _ssql_ctx\n    raise e\npy4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.\n: java.lang.RuntimeException: java.net.ConnectException: Call From jzhangMBPr.local/127.0.0.1 to 0.0.0.0:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused\n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)\n\tat org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)\n\tat org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)\n\tat org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)\n\tat org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)\n\tat org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)\n\tat org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)\n\tat org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)\n\tat org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)\n\tat org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)\n\tat org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:526)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)\n\tat py4j.Gateway.invoke(Gateway.java:214)\n\tat py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)\n\tat py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:209)\n\tat java.lang.Thread.run(Thread.java:745)\n```\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #10126 from zjffdu/SPARK-12120.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/e789b1d2c1eab6187f54424ed92697ca200c3101"},{"sha":"3327fd28170b549516fee1972dc6f4c32541591b","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12624][PYSPARK] Checks row length when converting Java arrays to Python rows\n\nWhen actual row length doesn't conform to specified schema field length, we should give a better error message instead of throwing an unintuitive `ArrayOutOfBoundsException`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10886 from liancheng/spark-12624.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/3327fd28170b549516fee1972dc6f4c32541591b"},{"sha":"3adebfc9a37fdee5b7a4e891c4ee597b85f824c3","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-12901][SQL] Refactor options for JSON and CSV datasource (not case class and same format).\n\nhttps://issues.apache.org/jira/browse/SPARK-12901\nThis PR refactors the options in JSON and CSV datasources.\n\nIn more details,\n\n1. `JSONOptions` uses the same format as `CSVOptions`.\n2. Not case classes.\n3. `CSVRelation` that does not have to be serializable (it was `with Serializable` but I removed)\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #10895 from HyukjinKwon/SPARK-12901.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/3adebfc9a37fdee5b7a4e891c4ee597b85f824c3"},{"sha":"d8e480521e362bc6bc5d8ebcea9b2d50f72a71b9","author":{"email":"f68909d51919859739a1c5d24450a2cc1026e93b@gmail.com","name":"Andy Grove"},"message":"[SPARK-12932][JAVA API] improved error message for java type inference failure\n\nAuthor: Andy Grove <andygrove73@gmail.com>\n\nCloses #10865 from andygrove/SPARK-12932.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/d8e480521e362bc6bc5d8ebcea9b2d50f72a71b9"},{"sha":"4ee8191e57cb823a23ceca17908af86e70354554","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@videoamp.com","name":"Michael Allman"},"message":"[SPARK-12755][CORE] Stop the event logger before the DAG scheduler\n\n[SPARK-12755][CORE] Stop the event logger before the DAG scheduler to avoid a race condition where the standalone master attempts to build the app's history UI before the event log is stopped.\n\nThis contribution is my original work, and I license this work to the Spark project under the project's open source license.\n\nAuthor: Michael Allman <michael@videoamp.com>\n\nCloses #10700 from mallman/stop_event_logger_first.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/4ee8191e57cb823a23ceca17908af86e70354554"},{"sha":"dd2325d9a7de7bef9a6bc2f0d5f26e605545b52d","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11965][ML][DOC] Update user guide for RFormula feature interactions\n\nUpdate user guide for RFormula feature interactions. Meanwhile we also update other new features such as supporting string label in Spark 1.6.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10222 from yanboliang/spark-11965.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/dd2325d9a7de7bef9a6bc2f0d5f26e605545b52d"},{"sha":"ef8fb3612c7be1ac9058750be39ee28d88a148b4","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"Closes #10879\nCloses #9046\nCloses #8532\nCloses #10756\nCloses #8960\nCloses #10485\nCloses #10467","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/ef8fb3612c7be1ac9058750be39ee28d88a148b4"},{"sha":"c037d25482ea63430fb42bfd86124c268be5a4a4","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"[SPARK-12149][WEB UI] Executor UI improvement suggestions - Color UI\n\nAdded color coding to the Executors page for Active Tasks, Failed Tasks, Completed Tasks and Task Time.\n\nActive Tasks is shaded blue with it's range based on percentage of total cores used.\nFailed Tasks is shaded red ranging over the first 10% of total tasks failed\nCompleted Tasks is shaded green ranging over 10% of total tasks including failed and active tasks, but only when there are active or failed tasks on that executor.\nTask Time is shaded red when GC Time goes over 10% of total time with it's range directly corresponding to the percent of total time.\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #10154 from ajbozarth/spark12149.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/c037d25482ea63430fb42bfd86124c268be5a4a4"},{"sha":"7d877c3439d872ec2a9e07d245e9c96174c0cf00","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12902] [SQL] visualization for generated operators\n\nThis PR brings back visualization for generated operators, they looks like:\n\n![sql](https://cloud.githubusercontent.com/assets/40902/12460920/0dc7956a-bf6b-11e5-9c3f-8389f452526e.png)\n\n![stage](https://cloud.githubusercontent.com/assets/40902/12460923/11806ac4-bf6b-11e5-9c72-e84a62c5ea93.png)\n\nNote: SQL metrics are not supported right now, because they are very slow, will be supported once we have batch mode.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10828 from davies/viz_codegen.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/7d877c3439d872ec2a9e07d245e9c96174c0cf00"},{"sha":"00026fa9912ecee5637f1e7dd222f977f31f6766","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12901][SQL][HOT-FIX] Fix scala 2.11 compilation.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/00026fa9912ecee5637f1e7dd222f977f31f6766"},{"sha":"9348431da212ec3ab7be2b8e89a952a48b4e2a31","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12975][SQL] Throwing Exception when Bucketing Columns are part of Partitioning Columns\n\nWhen users are using `partitionBy` and `bucketBy` at the same time, some bucketing columns might be part of partitioning columns. For example,\n```\n        df.write\n          .format(source)\n          .partitionBy(\"i\")\n          .bucketBy(8, \"i\", \"k\")\n          .saveAsTable(\"bucketed_table\")\n```\nHowever, in the above case, adding column `i` into `bucketBy` is useless. It is just wasting extra CPU when reading or writing bucket tables. Thus, like Hive, we can issue an exception and let users do the change.\n\nAlso added a test case for checking if the information of `sortBy` and `bucketBy` columns are correctly saved in the metastore table.\n\nCould you check if my understanding is correct? cloud-fan rxin marmbrus Thanks!\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #10891 from gatorsmile/commonKeysInPartitionByBucketBy.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/9348431da212ec3ab7be2b8e89a952a48b4e2a31"},{"sha":"dcae355c64d7f6fdf61df2feefe464eb96c4cf5e","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12905][ML][PYSPARK] PCAModel return eigenvalues for PySpark\n\n```PCAModel```  can output ```explainedVariance``` at Python side.\n\ncc mengxr srowen\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10830 from yanboliang/spark-12905.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/dcae355c64d7f6fdf61df2feefe464eb96c4cf5e"},{"sha":"6f0f1d9e04a8db47e2f6f8fcfe9dea9de0f633da","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12934][SQL] Count-min sketch serialization\n\nThis PR adds serialization support for `CountMinSketch`.\n\nA version number is added to version the serialized binary format.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10893 from liancheng/cms-serialization.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/6f0f1d9e04a8db47e2f6f8fcfe9dea9de0f633da"},{"sha":"be375fcbd200fb0e210b8edcfceb5a1bcdbba94b","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12879] [SQL] improve the unsafe row writing framework\n\nAs we begin to use unsafe row writing framework(`BufferHolder` and `UnsafeRowWriter`) in more and more places(`UnsafeProjection`, `UnsafeRowParquetRecordReader`, `GenerateColumnAccessor`, etc.), we should add more doc to it and make it easier to use.\n\nThis PR abstract the technique used in `UnsafeRowParquetRecordReader`: avoid unnecessary operatition as more as possible. For example, do not always point the row to the buffer at the end, we only need to update the size of row. If all fields are of primitive type, we can even save the row size updating. Then we can apply this technique to more places easily.\n\na local benchmark shows `UnsafeProjection` is up to 1.7x faster after this PR:\n**old version**\n```\nIntel(R) Core(TM) i7-4960HQ CPU  2.60GHz\nunsafe projection:                 Avg Time(ms)    Avg Rate(M/s)  Relative Rate\n-------------------------------------------------------------------------------\nsingle long                             2616.04           102.61         1.00 X\nsingle nullable long                    3032.54            88.52         0.86 X\nprimitive types                         9121.05            29.43         0.29 X\nnullable primitive types               12410.60            21.63         0.21 X\n```\n\n**new version**\n```\nIntel(R) Core(TM) i7-4960HQ CPU  2.60GHz\nunsafe projection:                 Avg Time(ms)    Avg Rate(M/s)  Relative Rate\n-------------------------------------------------------------------------------\nsingle long                             1533.34           175.07         1.00 X\nsingle nullable long                    2306.73           116.37         0.66 X\nprimitive types                         8403.93            31.94         0.18 X\nnullable primitive types               12448.39            21.56         0.12 X\n```\n\nFor single non-nullable long(the best case), we can have about 1.7x speed up. Even it's nullable, we can still have 1.3x speed up. For other cases, it's not such a boost as the saved operations only take a little proportion of the whole process.  The benchmark code is included in this PR.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10809 from cloud-fan/unsafe-projection.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/be375fcbd200fb0e210b8edcfceb5a1bcdbba94b"},{"sha":"109061f7ad27225669cbe609ec38756b31d4e1b9","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12936][SQL] Initial bloom filter implementation\n\nThis PR adds an initial implementation of bloom filter in the newly added sketch module.  The implementation is based on the [`BloomFilter` class in guava](https://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/hash/BloomFilter.java).\n\nSome difference from the design doc:\n\n* expose `bitSize` instead of `sizeInBytes` to user.\n* always need the `expectedInsertions` parameter when create bloom filter.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10883 from cloud-fan/bloom-filter.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/109061f7ad27225669cbe609ec38756b31d4e1b9"},{"sha":"fdcc3512f7b45e5b067fc26cb05146f79c4a5177","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-12934] use try-with-resources for streams\n\nliancheng please take a look\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #10906 from tedyu/master.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/fdcc3512f7b45e5b067fc26cb05146f79c4a5177"},{"sha":"b66afdeb5253913d916dcf159aaed4ffdc15fd4b","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-11922][PYSPARK][ML] Python api for ml.feature.quantile discretizer\n\nAdd Python API for ml.feature.QuantileDiscretizer.\n\nOne open question: Do we want to do this stuff to re-use the java model, create a new model, or use a different wrapper around the java model.\ncc brkyvz & mengxr\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #10085 from holdenk/SPARK-11937-SPARK-11922-Python-API-for-ml.feature.QuantileDiscretizer.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/b66afdeb5253913d916dcf159aaed4ffdc15fd4b"},{"sha":"ae47ba718a280fc12720a71b981c38dbe647f35b","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-12834] Change ser/de of JavaArray and JavaList\n\nhttps://issues.apache.org/jira/browse/SPARK-12834\n\nWe use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #10772 from yinxusen/SPARK-12834.","distinct":true,"url":"https://api.github.com/repos/msjbear/spark/commits/ae47ba718a280fc12720a71b981c38dbe647f35b"}]}