{"push_id":1009864907,"size":465,"distinct_size":2,"ref":"refs/heads/SPARK-11171-SPARK-11237-Add-PMML-export-for-ML-KMeans","head":"4f693d1e5bb3356d9d922788e94beae5c94b06e4","before":"90b0e22674093fe39d5602d985b97180aee384fa","commits":[{"sha":"00026fa9912ecee5637f1e7dd222f977f31f6766","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12901][SQL][HOT-FIX] Fix scala 2.11 compilation.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/00026fa9912ecee5637f1e7dd222f977f31f6766"},{"sha":"9348431da212ec3ab7be2b8e89a952a48b4e2a31","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12975][SQL] Throwing Exception when Bucketing Columns are part of Partitioning Columns\n\nWhen users are using `partitionBy` and `bucketBy` at the same time, some bucketing columns might be part of partitioning columns. For example,\n```\n        df.write\n          .format(source)\n          .partitionBy(\"i\")\n          .bucketBy(8, \"i\", \"k\")\n          .saveAsTable(\"bucketed_table\")\n```\nHowever, in the above case, adding column `i` into `bucketBy` is useless. It is just wasting extra CPU when reading or writing bucket tables. Thus, like Hive, we can issue an exception and let users do the change.\n\nAlso added a test case for checking if the information of `sortBy` and `bucketBy` columns are correctly saved in the metastore table.\n\nCould you check if my understanding is correct? cloud-fan rxin marmbrus Thanks!\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #10891 from gatorsmile/commonKeysInPartitionByBucketBy.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/9348431da212ec3ab7be2b8e89a952a48b4e2a31"},{"sha":"dcae355c64d7f6fdf61df2feefe464eb96c4cf5e","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12905][ML][PYSPARK] PCAModel return eigenvalues for PySpark\n\n```PCAModel```  can output ```explainedVariance``` at Python side.\n\ncc mengxr srowen\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10830 from yanboliang/spark-12905.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/dcae355c64d7f6fdf61df2feefe464eb96c4cf5e"},{"sha":"6f0f1d9e04a8db47e2f6f8fcfe9dea9de0f633da","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-12934][SQL] Count-min sketch serialization\n\nThis PR adds serialization support for `CountMinSketch`.\n\nA version number is added to version the serialized binary format.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #10893 from liancheng/cms-serialization.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/6f0f1d9e04a8db47e2f6f8fcfe9dea9de0f633da"},{"sha":"be375fcbd200fb0e210b8edcfceb5a1bcdbba94b","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12879] [SQL] improve the unsafe row writing framework\n\nAs we begin to use unsafe row writing framework(`BufferHolder` and `UnsafeRowWriter`) in more and more places(`UnsafeProjection`, `UnsafeRowParquetRecordReader`, `GenerateColumnAccessor`, etc.), we should add more doc to it and make it easier to use.\n\nThis PR abstract the technique used in `UnsafeRowParquetRecordReader`: avoid unnecessary operatition as more as possible. For example, do not always point the row to the buffer at the end, we only need to update the size of row. If all fields are of primitive type, we can even save the row size updating. Then we can apply this technique to more places easily.\n\na local benchmark shows `UnsafeProjection` is up to 1.7x faster after this PR:\n**old version**\n```\nIntel(R) Core(TM) i7-4960HQ CPU  2.60GHz\nunsafe projection:                 Avg Time(ms)    Avg Rate(M/s)  Relative Rate\n-------------------------------------------------------------------------------\nsingle long                             2616.04           102.61         1.00 X\nsingle nullable long                    3032.54            88.52         0.86 X\nprimitive types                         9121.05            29.43         0.29 X\nnullable primitive types               12410.60            21.63         0.21 X\n```\n\n**new version**\n```\nIntel(R) Core(TM) i7-4960HQ CPU  2.60GHz\nunsafe projection:                 Avg Time(ms)    Avg Rate(M/s)  Relative Rate\n-------------------------------------------------------------------------------\nsingle long                             1533.34           175.07         1.00 X\nsingle nullable long                    2306.73           116.37         0.66 X\nprimitive types                         8403.93            31.94         0.18 X\nnullable primitive types               12448.39            21.56         0.12 X\n```\n\nFor single non-nullable long(the best case), we can have about 1.7x speed up. Even it's nullable, we can still have 1.3x speed up. For other cases, it's not such a boost as the saved operations only take a little proportion of the whole process.  The benchmark code is included in this PR.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10809 from cloud-fan/unsafe-projection.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/be375fcbd200fb0e210b8edcfceb5a1bcdbba94b"},{"sha":"109061f7ad27225669cbe609ec38756b31d4e1b9","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12936][SQL] Initial bloom filter implementation\n\nThis PR adds an initial implementation of bloom filter in the newly added sketch module.  The implementation is based on the [`BloomFilter` class in guava](https://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/hash/BloomFilter.java).\n\nSome difference from the design doc:\n\n* expose `bitSize` instead of `sizeInBytes` to user.\n* always need the `expectedInsertions` parameter when create bloom filter.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10883 from cloud-fan/bloom-filter.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/109061f7ad27225669cbe609ec38756b31d4e1b9"},{"sha":"fdcc3512f7b45e5b067fc26cb05146f79c4a5177","author":{"email":"22f3e725b314afeca4d45e12683c560674029d21@gmail.com","name":"tedyu"},"message":"[SPARK-12934] use try-with-resources for streams\n\nliancheng please take a look\n\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #10906 from tedyu/master.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/fdcc3512f7b45e5b067fc26cb05146f79c4a5177"},{"sha":"b66afdeb5253913d916dcf159aaed4ffdc15fd4b","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-11922][PYSPARK][ML] Python api for ml.feature.quantile discretizer\n\nAdd Python API for ml.feature.QuantileDiscretizer.\n\nOne open question: Do we want to do this stuff to re-use the java model, create a new model, or use a different wrapper around the java model.\ncc brkyvz & mengxr\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #10085 from holdenk/SPARK-11937-SPARK-11922-Python-API-for-ml.feature.QuantileDiscretizer.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/b66afdeb5253913d916dcf159aaed4ffdc15fd4b"},{"sha":"ae47ba718a280fc12720a71b981c38dbe647f35b","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-12834] Change ser/de of JavaArray and JavaList\n\nhttps://issues.apache.org/jira/browse/SPARK-12834\n\nWe use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #10772 from yinxusen/SPARK-12834.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/ae47ba718a280fc12720a71b981c38dbe647f35b"},{"sha":"27c910f7f29087d1ac216d4933d641d6515fd6ad","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-10086][MLLIB][STREAMING][PYSPARK] ignore StreamingKMeans test in PySpark for now\n\nI saw several failures from recent PR builds, e.g., https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50015/consoleFull. This PR marks the test as ignored and we will fix the flakyness in SPARK-10086.\n\ngliptak Do you know why the test failure didn't show up in the Jenkins \"Test Result\"?\n\ncc: jkbradley\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #10909 from mengxr/SPARK-10086.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/27c910f7f29087d1ac216d4933d641d6515fd6ad"},{"sha":"d54cfed5a6953a9ce2b9de2f31ee2d673cb5cc62","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SQL][MINOR] A few minor tweaks to CSV reader.\n\nThis pull request simply fixes a few minor coding style issues in csv, as I was reviewing the change post-hoc.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10919 from rxin/csv-minor.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/d54cfed5a6953a9ce2b9de2f31ee2d673cb5cc62"},{"sha":"6743de3a98e3f0d0e6064ca1872fa88c3aeaa143","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12937][SQL] bloom filter serialization\n\nThis PR adds serialization support for BloomFilter.\n\nA version number is added to version the serialized binary format.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10920 from cloud-fan/bloom-filter.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/6743de3a98e3f0d0e6064ca1872fa88c3aeaa143"},{"sha":"5936bf9fa85ccf7f0216145356140161c2801682","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12961][CORE] Prevent snappy-java memory leak\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12961\n\nTo prevent memory leak in snappy-java, just call the method once and cache the result. After the library releases new version, we can remove this object.\n\nJoshRosen\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10875 from viirya/prevent-snappy-memory-leak.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/5936bf9fa85ccf7f0216145356140161c2801682"},{"sha":"649e9d0f5b2d5fc13f2dd5be675331510525927f","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-3369][CORE][STREAMING] Java mapPartitions Iterator->Iterable is inconsistent with Scala's Iterator->Iterator\n\nFix Java function API methods for flatMap and mapPartitions to require producing only an Iterator, not Iterable. Also fix DStream.flatMap to require a function producing TraversableOnce only, not Traversable.\n\nCC rxin pwendell for API change; tdas since it also touches streaming.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #10413 from srowen/SPARK-3369.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/649e9d0f5b2d5fc13f2dd5be675331510525927f"},{"sha":"ae0309a8812a4fade3a0ea67d8986ca870aeb9eb","author":{"email":"02e10e341a2fea4868ba58fdfcfe400dcd4d1dd0@yahoo-inc.com","name":"zhuol"},"message":"[SPARK-10911] Executors should System.exit on clean shutdown.\n\nCall system.exit explicitly to make sure non-daemon user threads terminate. Without this, user applications might live forever if the cluster manager does not appropriately kill them. E.g., YARN had this bug: HADOOP-12441.\n\nAuthor: zhuol <zhuol@yahoo-inc.com>\n\nCloses #9946 from zhuoliu/10911.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/ae0309a8812a4fade3a0ea67d8986ca870aeb9eb"},{"sha":"08c781ca672820be9ba32838bbe40d2643c4bde4","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-12682][SQL] Add support for (optionally) not storing tables in hive metadata format\n\nThis PR adds a new table option (`skip_hive_metadata`) that'd allow the user to skip storing the table metadata in hive metadata format. While this could be useful in general, the specific use-case for this change is that Hive doesn't handle wide schemas well (see https://issues.apache.org/jira/browse/SPARK-12682 and https://issues.apache.org/jira/browse/SPARK-6024) which in turn prevents such tables from being queried in SparkSQL.\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #10826 from sameeragarwal/skip-hive-metadata.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/08c781ca672820be9ba32838bbe40d2643c4bde4"},{"sha":"cbd507d69cea24adfb335d8fe26ab5a13c053ffc","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-7799][STREAMING][DOCUMENT] Add the linking and deploying instructions for streaming-akka project\n\nSince `actorStream` is an external project, we should add the linking and deploying instructions for it.\n\nA follow up PR of #10744\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10856 from zsxwing/akka-link-instruction.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/cbd507d69cea24adfb335d8fe26ab5a13c053ffc"},{"sha":"8beab68152348c44cf2f89850f792f164b06470d","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11923][ML] Python API for ml.feature.ChiSqSelector\n\nhttps://issues.apache.org/jira/browse/SPARK-11923\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #10186 from yinxusen/SPARK-11923.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/8beab68152348c44cf2f89850f792f164b06470d"},{"sha":"fbf7623d49525e3aa6b08f482afd7ee8118d80cb","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-12952] EMLDAOptimizer initialize() should return EMLDAOptimizer other than its parent class\n\nhttps://issues.apache.org/jira/browse/SPARK-12952\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #10863 from yinxusen/SPARK-12952.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/fbf7623d49525e3aa6b08f482afd7ee8118d80cb"},{"sha":"ee74498de372b16fe6350e3617e9e6ec87c6ae7b","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-8725][PROJECT-INFRA] Test modules in topologically-sorted order in dev/run-tests\n\nThis patch improves our `dev/run-tests` script to test modules in a topologically-sorted order based on modules' dependencies.  This will help to ensure that bugs in upstream projects are not misattributed to downstream projects because those projects' tests were the first ones to exhibit the failure\n\nTopological sorting is also useful for shortening the feedback loop when testing pull requests: if I make a change in SQL then the SQL tests should run before MLlib, not after.\n\nIn addition, this patch also updates our test module definitions to split `sql` into `catalyst`, `sql`, and `hive` in order to allow more tests to be skipped when changing only `hive/` files.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10885 from JoshRosen/SPARK-8725.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/ee74498de372b16fe6350e3617e9e6ec87c6ae7b"}]}