{"push_id":920100350,"size":983,"distinct_size":19,"ref":"refs/heads/master","head":"0da7bd50ddf0fb9e0e8aeadb9c7fb3edf6f0ee6e","before":"82e9d9c81b7a8af94e312035328da872684f6e67","commits":[{"sha":"9f49895fefc294ef40b2e974f1f8b311087c54df","author":{"email":"dfaf75894ad3de5736b3eab77e7ff6b13128cd84@gmail.com","name":"Tijo Thomas"},"message":"[SPARK-10261][DOCUMENTATION, ML] Fixed @Since annotation to ml.evaluation\n\nAuthor: Tijo Thomas <tijoparacka@gmail.com>\nAuthor: tijo <tijo@ezzoft.com>\n\nCloses #8554 from tijoparacka/SPARK-10261-2.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/9f49895fefc294ef40b2e974f1f8b311087c54df"},{"sha":"04521ea067d6ed3c5398067f07904d27c77017ff","author":{"email":"1d6e1cf70ec6f9ab28d3ea4b27a49a77654d370e@noelsmith.com","name":"noelsmith"},"message":"[SPARK-10269][PYSPARK][MLLIB] Add @since annotation to pyspark.mllib.classification\n\nDuplicated the since decorator from pyspark.sql into pyspark (also tweaked to handle functions without docstrings).\n\nAdded since to methods + \"versionadded::\" to classes derived from the file history.\n\nNote - some methods are inherited from the regression module (i.e. LinearModel.intercept) so these won't have version numbers in the API docs until that model is updated.\n\nAuthor: noelsmith <mail@noelsmith.com>\n\nCloses #8626 from noel-smith/SPARK-10269-since-mlib-classification.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/04521ea067d6ed3c5398067f07904d27c77017ff"},{"sha":"da46b77afd13df8eb696e4612224ae29cc198c0b","author":{"email":"8757c4a255384de9fc2175f17a84f948e7081d30@gmail.com","name":"MechCoder"},"message":"[SPARK-10082][MLLIB] Validate i, j in apply DenseMatrices and SparseMatrices\n\nGiven row_ind should be less than the number of rows\nGiven col_ind should be less than the number of cols.\n\nThe current code in master gives unpredictable behavior for such cases.\n\nAuthor: MechCoder <manojkumarsivaraj334@gmail.com>\n\nCloses #8271 from MechCoder/hash_code_matrices.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/da46b77afd13df8eb696e4612224ae29cc198c0b"},{"sha":"aea7142c9802d1e855443c01621ebc8d57be8c5e","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@pigscanfly.ca","name":"Holden Karau"},"message":"[SPARK-10767][PYSPARK] Make pyspark shared params codegen more consistent\n\nNamely \".\" shows up in some places in the template when using the param docstring and not in others\n\nAuthor: Holden Karau <holden@pigscanfly.ca>\n\nCloses #9017 from holdenk/SPARK-10767-Make-pyspark-shared-params-codegen-more-consistent.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/aea7142c9802d1e855443c01621ebc8d57be8c5e"},{"sha":"89e6db6150704deab46232352d1986bc1449883b","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-11153][SQL] Disables Parquet filter push-down for string and binary columns\n\nDue to PARQUET-251, `BINARY` columns in existing Parquet files may be written with corrupted statistics information. This information is used by filter push-down optimization. Since Spark 1.5 turns on Parquet filter push-down by default, we may end up with wrong query results. PARQUET-251 has been fixed in parquet-mr 1.8.1, but Spark 1.5 is still using 1.7.0.\n\nThis affects all Spark SQL data types that can be mapped to Parquet {{BINARY}}, namely:\n\n- `StringType`\n\n- `BinaryType`\n\n- `DecimalType`\n\n  (But Spark SQL doesn't support pushing down filters involving `DecimalType` columns for now.)\n\nTo avoid wrong query results, we should disable filter push-down for columns of `StringType` and `BinaryType` until we upgrade to parquet-mr 1.8.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #9152 from liancheng/spark-11153.workaround-parquet-251.\n\n(cherry picked from commit 0887e5e87891e8e22f534ca6d0406daf86ec2dad)\nSigned-off-by: Cheng Lian <lian@databricks.com>","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/89e6db6150704deab46232352d1986bc1449883b"},{"sha":"45861693bef2619196f0fbdf5c166ad3f9b1e8d1","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-10082][MLLIB] minor style updates for matrix indexing after #8271\n\n* `>=0` => `>= 0`\n* print `i`, `j` in the log message\n\nMechCoder\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9189 from mengxr/SPARK-10082.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/45861693bef2619196f0fbdf5c166ad3f9b1e8d1"},{"sha":"135ade905089fe90d3cc9e84f8fedc7637e901d8","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[MINOR][ML] fix doc warnings\n\nWithout an empty line, sphinx will treat doctest as docstring. holdenk\n\n~~~\n/Users/meng/src/spark/python/pyspark/ml/feature.py:docstring of pyspark.ml.feature.CountVectorizer:3: ERROR: Undefined substitution referenced: \"label|raw |vectors | +-----+---------------+-------------------------+ |0 |[a, b, c] |(3,[0,1,2],[1.0,1.0,1.0])\".\n/Users/meng/src/spark/python/pyspark/ml/feature.py:docstring of pyspark.ml.feature.CountVectorizer:3: ERROR: Undefined substitution referenced: \"1 |[a, b, b, c, a]|(3,[0,1,2],[2.0,2.0,1.0])\".\n~~~\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #9188 from mengxr/py-count-vec-doc-fix.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/135ade905089fe90d3cc9e84f8fedc7637e901d8"},{"sha":"1107bd958aefc793d6e8e369ad6268b24a0f8510","author":{"email":"fd68b5404b021a07e5ec5abdeb1b0ffe8f04e45b@hotmail.com","name":"felixcheung"},"message":"[SPARK-11221][SPARKR] fix R doc for lit and add examples\n\nCurrently the documentation for `lit` is inconsistent with doc format, references \"Scala symbol\" and has no example. Fixing that.\nshivaram\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #9187 from felixcheung/rlit.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/1107bd958aefc793d6e8e369ad6268b24a0f8510"},{"sha":"5cdea7d1e54e751f928876ead33fc01e7e2f3437","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[SPARK-11205][PYSPARK] Delegate to scala DataFrame API rather than p…\n\n…rint in python\n\nNo test needed. Verify it manually in pyspark shell\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #9177 from zjffdu/SPARK-11205.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/5cdea7d1e54e751f928876ead33fc01e7e2f3437"},{"sha":"8e82e59834aefb74d49e45c9c9c926bb53143b4c","author":{"email":"bc9570497728884c7f6686fa0fc06a8d320e124b@in.ibm.com","name":"Pravin Gadakh"},"message":"[SPARK-11037][SQL] using Option instead of Some in JdbcDialects\n\nUsing Option instead of Some in getCatalystType method.\n\nAuthor: Pravin Gadakh <prgadakh@in.ibm.com>\n\nCloses #9195 from pravingadakh/master.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/8e82e59834aefb74d49e45c9c9c926bb53143b4c"},{"sha":"f62e3260889d67256d335fd0dd38f114ae4e3eca","author":{"email":"410f3a2b6cc1bf6fa9049367db5a8ed4fa3f6afc@guavus.com","name":"nitin goyal"},"message":"[SPARK-11179] [SQL] Push filters through aggregate\n\nPush conjunctive predicates though Aggregate operators when their references are a subset of the groupingExpressions.\n\nQuery plan before optimisation :-\nFilter ((c#138L = 2) && (a#0 = 3))\n Aggregate [a#0], [a#0,count(b#1) AS c#138L]\n  Project [a#0,b#1]\n   LocalRelation [a#0,b#1,c#2]\n\nQuery plan after optimisation :-\nFilter (c#138L = 2)\n Aggregate [a#0], [a#0,count(b#1) AS c#138L]\n  Filter (a#0 = 3)\n   Project [a#0,b#1]\n    LocalRelation [a#0,b#1,c#2]\n\nAuthor: nitin goyal <nitin.goyal@guavus.com>\nAuthor: nitin.goyal <nitin.goyal@guavus.com>\n\nCloses #9167 from nitin2goyal/master.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/f62e3260889d67256d335fd0dd38f114ae4e3eca"},{"sha":"ccf536f903ef1f81fb3e1b6ce781d5e40d0ae3e0","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11216] [SQL] add encoder/decoder for external row\n\nImplement encode/decode for external row based on `ClassEncoder`.\n\nTODO:\n* code cleanup\n* ~~fix corner cases~~\n* refactor the encoder interface\n* improve test for product codegen, to cover more corner cases.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9184 from cloud-fan/encoder.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/ccf536f903ef1f81fb3e1b6ce781d5e40d0ae3e0"},{"sha":"49ea0e9d7ce805d312d94a5b2936eec2053bc052","author":{"email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com","name":"Dilip Biswal"},"message":"[SPARK-10534] [SQL] ORDER BY clause allows only columns that are present in the select projection list\n\nFind out the missing attributes by recursively looking\nat the sort order expression and rest of the code\ntakes care of projecting them out.\n\nAdded description from cloud-fan\n\nI wanna explain a bit more about this bug.\n\nWhen we resolve sort ordering, we will use a special method, which only resolves UnresolvedAttributes and UnresolvedExtractValue. However, for something like Floor('a), even the 'a is resolved, the floor expression may still being unresolved as data type mismatch(for example, 'a is string type and Floor need double type), thus can't pass this filter, and we can't push down this missing attribute 'a\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #9123 from dilipbiswal/SPARK-10534.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/49ea0e9d7ce805d312d94a5b2936eec2053bc052"},{"sha":"7c74ebca05f40a2d8fe8f10f24a10486ce4f76c0","author":{"email":"ab56adc4e61f1b489818a700493ea97fb9f9e75e@163.com","name":"Wenchen Fan"},"message":"[SPARK-10743][SQL] keep the name of expression if possible when do cast\n\nAuthor: Wenchen Fan <cloud0fan@163.com>\n\nCloses #8859 from cloud-fan/cast.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/7c74ebca05f40a2d8fe8f10f24a10486ce4f76c0"},{"sha":"f8c6bec65784de89b47e96a367d3f9790c1b3115","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11197][SQL] run SQL on files directly\n\nThis PR introduce a new feature to run SQL directly on files without create a table, for example:\n\n```\nselect id from json.`path/to/json/files` as j\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9173 from davies/source.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/f8c6bec65784de89b47e96a367d3f9790c1b3115"},{"sha":"3afe448d39dc4877b2f2c62b3059aeb3ced0bd96","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9740][SPARK-9592][SPARK-9210][SQL] Change the default behavior of First/Last to RESPECT NULLS.\n\nI am changing the default behavior of `First`/`Last` to respect null values (the SQL standard default behavior).\n\nhttps://issues.apache.org/jira/browse/SPARK-9740\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #8113 from yhuai/firstLast.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/3afe448d39dc4877b2f2c62b3059aeb3ced0bd96"},{"sha":"a37cd870489d7ff85455bf9b57d9aa5329514058","author":{"email":"2636e2e8f2fb00edd3fc7f8e5f600f8384dd3893@datastax.com","name":"Artem Aliev"},"message":"[SPARK-11208][SQL] Filter out 'hive.metastore.rawstore.impl' from executionHive temporary config\n\nThe executionHive assumed to be a standard meta store located in temporary directory as a derby db. But hive.metastore.rawstore.impl was not filtered out so any custom implementation of the metastore with other storage properties (not JDO) will persist that temporary functions. CassandraHiveMetaStore from DataStax Enterprise is one of examples.\n\nAuthor: Artem Aliev <artem.aliev@datastax.com>\n\nCloses #9178 from artem-aliev/SPARK-11208.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/a37cd870489d7ff85455bf9b57d9aa5329514058"},{"sha":"19ad18638e27cc7b403ea98c4f9f40a940932e30","author":{"email":"5d7c20a9ebcf8b2e3fa111427e329c9b3fc41e59@gmail.com","name":"Shagun Sodhani"},"message":"[SPARK-11233][SQL] register cosh in function registry\n\nAuthor: Shagun Sodhani <sshagunsodhani@gmail.com>\n\nCloses #9199 from shagunsodhani/proposed-fix-#11233.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/19ad18638e27cc7b403ea98c4f9f40a940932e30"},{"sha":"dce2f8c9d74bda46f3c7a7ebe7372b04d6b17a61","author":{"email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com","name":"Dilip Biswal"},"message":"[SPARK-8654][SQL] Analysis exception when using NULL IN (...) : invalid cast\n\nIn the analysis phase , while processing the rules for IN predicate, we\ncompare the in-list types to the lhs expression type and generate\ncast operation if necessary. In the case of NULL [NOT] IN expr1 , we end up\ngenerating cast between in list types to NULL like cast (1 as NULL) which\nis not a valid cast.\n\nThe fix is to find a common type between LHS and RHS expressions and cast\nall the expression to the common type.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nThis patch had conflicts when merged, resolved by\nCommitter: Michael Armbrust <michael@databricks.com>\n\nCloses #9036 from dilipbiswal/spark_8654_new.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/dce2f8c9d74bda46f3c7a7ebe7372b04d6b17a61"},{"sha":"f481090a71940f06602a73f5bbd004980dea026f","author":{"email":"04ade2e3a4ce65c5ee0cbfa08dacc69e7d690176@apache.org","name":"navis.ryu"},"message":"[SPARK-10151][SQL] Support invocation of hive macro\n\nMacro in hive (which is GenericUDFMacro) contains real function inside of it but it's not conveyed to tasks, resulting null-pointer exception.\n\nAuthor: navis.ryu <navis@apache.org>\n\nCloses #8354 from navis/SPARK-10151.","distinct":false,"url":"https://api.github.com/repos/rxin/spark/commits/f481090a71940f06602a73f5bbd004980dea026f"}]}