{"push_id":937835869,"size":20,"distinct_size":20,"ref":"refs/heads/master","head":"86972fa52152d2149b88ba75be048a6986006285","before":"591c88c9e2a6c2e2ca84f1b66c635f198a16d112","commits":[{"sha":"fe7246fea67e1d71fba679dee3eb2c7386b4f4e2","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12830] Java style: disallow trailing whitespaces.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10764 from rxin/SPARK-12830.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/fe7246fea67e1d71fba679dee3eb2c7386b4f4e2"},{"sha":"d0a5c32bd05841f411a342a80c5da9f73f30d69a","author":{"email":"69d9c961933324ebb2a99682f1939032e0ff256e@us.ibm.com","name":"Jason Lee"},"message":"[SPARK-12655][GRAPHX] GraphX does not unpersist RDDs\n\nSome VertexRDD and EdgeRDD are created during the intermediate step of g.connectedComponents() but unnecessarily left cached after the method is done. The fix is to unpersist these RDDs once they are no longer in use.\n\nA test case is added to confirm the fix for the reported bug.\n\nAuthor: Jason Lee <cjlee@us.ibm.com>\n\nCloses #10713 from jasoncl/SPARK-12655.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/d0a5c32bd05841f411a342a80c5da9f73f30d69a"},{"sha":"96fb894d4b33e293625fa92bbeccbbf5e688015e","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@yahoo-inc.com","name":"Tom Graves"},"message":"[SPARK-2930] clarify docs on using webhdfs with spark.yarn.access.nam…\n\n…enodes\n\nAuthor: Tom Graves <tgraves@yahoo-inc.com>\n\nCloses #10699 from tgravescs/SPARK-2930.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/96fb894d4b33e293625fa92bbeccbbf5e688015e"},{"sha":"ba4a641902f95c5a9b3a6bebcaa56039eca2720d","author":{"email":"4aa8789978b9b9ff21ab5ed4497ed9d6a98ad6f4@oscars-mbp.usca.ibm.com","name":"Oscar D. Lara Yejas"},"message":"[SPARK-11031][SPARKR] Method str() on a DataFrame\n\nAuthor: Oscar D. Lara Yejas <odlaraye@oscars-mbp.usca.ibm.com>\nAuthor: Oscar D. Lara Yejas <olarayej@mail.usf.edu>\nAuthor: Oscar D. Lara Yejas <oscar.lara.yejas@us.ibm.com>\nAuthor: Oscar D. Lara Yejas <odlaraye@oscars-mbp.attlocal.net>\n\nCloses #9613 from olarayej/SPARK-11031.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/ba4a641902f95c5a9b3a6bebcaa56039eca2720d"},{"sha":"c5e7076da72657ea35a0aa388f8d2e6411d39280","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[MINOR] [SQL] GeneratedExpressionCode -> ExprCode\n\nGeneratedExpressionCode is too long\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10767 from davies/renaming.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/c5e7076da72657ea35a0aa388f8d2e6411d39280"},{"sha":"5f83c6991c95616ecbc2878f8860c69b2826f56c","author":{"email":"5ec9aca3de67ebdf671d9e5cddd02b8276856562@databricks.com","name":"Hossein"},"message":"[SPARK-12833][SQL] Initial import of spark-csv\n\nCSV is the most common data format in the \"small data\" world. It is often the first format people want to try when they see Spark on a single node. Having to rely on a 3rd party component for this leads to poor user experience for new users. This PR merges the popular spark-csv data source package (https://github.com/databricks/spark-csv) with SparkSQL.\n\nThis is a first PR to bring the functionality to spark 2.0 master. We will complete items outlines in the design document (see JIRA attachment) in follow up pull requests.\n\nAuthor: Hossein <hossein@databricks.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10766 from rxin/csv.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/5f83c6991c95616ecbc2878f8860c69b2826f56c"},{"sha":"ad1503f92e1f6e960a24f9f5d36b1735d1f5073a","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12667] Remove block manager's internal \"external block store\" API\n\nThis pull request removes the external block store API. This is rarely used, and the file system interface is actually a better, more standard way to interact with external storage systems.\n\nThere are some other things to remove also, as pointed out by JoshRosen. We will do those as follow-up pull requests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10752 from rxin/remove-offheap.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/ad1503f92e1f6e960a24f9f5d36b1735d1f5073a"},{"sha":"513266c0426faf8b9cc7576963bb9a57d2fdbc54","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12833][HOT-FIX] Fix scala 2.11 compilation.\n\nSeems https://github.com/apache/spark/commit/5f83c6991c95616ecbc2878f8860c69b2826f56c breaks scala 2.11 compilation.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10774 from yhuai/fixScala211Compile.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/513266c0426faf8b9cc7576963bb9a57d2fdbc54"},{"sha":"0bb73554a96088adca9dcd62a5a37b13772b02d8","author":{"email":"b82756e669c99980b9a86a0df23b80b425260060@gmail.com","name":"Julien Baley"},"message":"Fix typo\n\ndisvoered => discovered\n\nAuthor: Julien Baley <julien.baley@gmail.com>\n\nCloses #10773 from julienbaley/patch-1.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/0bb73554a96088adca9dcd62a5a37b13772b02d8"},{"sha":"61c45876fb532cdb7278dea48cc141208b63737c","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"[SPARK-12716][WEB UI] Add a TOTALS row to the Executors Web UI\n\nAdded a Totals table to the top of the page to display the totals of each applicable column in the executors table.\n\nOld Description:\n~~Created a TOTALS row containing the totals of each column in the executors UI. By default the TOTALS row appears at the top of the table. When a column is sorted the TOTALS row will always sort to either the top or bottom of the table.~~\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #10668 from ajbozarth/spark12716.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/61c45876fb532cdb7278dea48cc141208b63737c"},{"sha":"3f1c58d60b85625ab3abf16456ce27c820453ecf","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SQL][MINOR] BoundReference do not need to be NamedExpression\n\nWe made it a `NamedExpression` to workaroud some hacky cases long time ago, and now seems it's safe to remove it.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10765 from cloud-fan/minor.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/3f1c58d60b85625ab3abf16456ce27c820453ecf"},{"sha":"7cd7f2202547224593517b392f56e49e4c94cabc","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-12575][SQL] Grammar parity with existing SQL parser\n\nIn this PR the new CatalystQl parser stack reaches grammar parity with the old Parser-Combinator based SQL Parser. This PR also replaces all uses of the old Parser, and removes it from the code base.\n\nAlthough the existing Hive and SQL parser dialects were mostly the same, some kinks had to be worked out:\n- The SQL Parser allowed syntax like ```APPROXIMATE(0.01) COUNT(DISTINCT a)```. In order to make this work we needed to hardcode approximate operators in the parser, or we would have to create an approximate expression. ```APPROXIMATE_COUNT_DISTINCT(a, 0.01)``` would also do the job and is much easier to maintain. So, this PR **removes** this keyword.\n- The old SQL Parser supports ```LIMIT``` clauses in nested queries. This is **not supported** anymore. See https://github.com/apache/spark/pull/10689 for the rationale for this.\n- Hive has a charset name char set literal combination it supports, for instance the following expression ```_ISO-8859-1 0x4341464562616265``` would yield this string: ```CAFEbabe```. Hive will only allow charset names to start with an underscore. This is quite annoying in spark because as soon as you use a tuple names will start with an underscore. In this PR we **remove** this feature from the parser. It would be quite easy to implement such a feature as an Expression later on.\n- Hive and the SQL Parser treat decimal literals differently. Hive will turn any decimal into a ```Double``` whereas the SQL Parser would convert a non-scientific decimal into a ```BigDecimal```, and would turn a scientific decimal into a Double. We follow Hive's behavior here. The new parser supports a big decimal literal, for instance: ```81923801.42BD```, which can be used when a big decimal is needed.\n\ncc rxin viirya marmbrus yhuai cloud-fan\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #10745 from hvanhovell/SPARK-12575-2.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/7cd7f2202547224593517b392f56e49e4c94cabc"},{"sha":"5f843781e3e7581c61b7e235d4041d85e8e48c7e","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11925][ML][PYSPARK] Add PySpark missing methods for ml.feature during Spark 1.6 QA\n\nAdd PySpark missing methods and params for ml.feature:\n* ```RegexTokenizer``` should support setting ```toLowercase```.\n* ```MinMaxScalerModel``` should support output ```originalMin``` and ```originalMax```.\n* ```PCAModel``` should support output ```pc```.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9908 from yanboliang/spark-11925.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/5f843781e3e7581c61b7e235d4041d85e8e48c7e"},{"sha":"f6ddbb360ac6ac2778bbdbebbf2fcccabe73349b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12833][HOT-FIX] Reset the locale after we set it.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10778 from yhuai/resetLocale.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/f6ddbb360ac6ac2778bbdbebbf2fcccabe73349b"},{"sha":"8dbbf3e75e70e98391b4a1705472caddd129945a","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12842][TEST-HADOOP2.7] Add Hadoop 2.7 build profile\n\nThis patch adds a Hadoop 2.7 build profile in order to let us automate tests against that version.\n\n/cc rxin srowen\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10775 from JoshRosen/add-hadoop-2.7-profile.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/8dbbf3e75e70e98391b4a1705472caddd129945a"},{"sha":"3b5ccb12b8d33d99df0f206fecf00f51c2b88fdb","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12649][SQL] support reading bucketed table\n\nThis PR adds the support to read bucketed tables, and correctly populate `outputPartitioning`, so that we can avoid shuffle for some cases.\n\nTODO(follow-up PRs):\n\n* bucket pruning\n* avoid shuffle for bucketed table join when use any super-set of the bucketing key.\n (we should re-visit it after https://issues.apache.org/jira/browse/SPARK-12704 is fixed)\n* recognize hive bucketed table\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10604 from cloud-fan/bucket-read.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/3b5ccb12b8d33d99df0f206fecf00f51c2b88fdb"},{"sha":"9039333c0a0ce4bea32f012b81c1e82e31246fc1","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12644][SQL] Update parquet reader to be vectorized.\n\nThis inlines a few of the Parquet decoders and adds vectorized APIs to support decoding in batch.\nThere are a few particulars in the Parquet encodings that make this much more efficient. In\nparticular, RLE encodings are very well suited for batch decoding. The Parquet 2.0 encodings are\nalso very suited for this.\n\nThis is a work in progress and does not affect the current execution. In subsequent patches, we will\nsupport more encodings and types before enabling this.\n\nSimple benchmarks indicate this can decode single ints about > 3x faster.\n\nAuthor: Nong Li <nong@databricks.com>\nAuthor: Nong <nongli@gmail.com>\n\nCloses #10593 from nongli/spark-12644.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/9039333c0a0ce4bea32f012b81c1e82e31246fc1"},{"sha":"242efb7546084592a5e8122549a27117977303fb","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12840] [SQL] Support passing arbitrary objects (not just expressions) into code generated classes\n\nThis is a refactor to support codegen for aggregation and broadcast join.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10777 from davies/rename2.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/242efb7546084592a5e8122549a27117977303fb"},{"sha":"2f7d0b68a29de9755fc9fafd9a52c048981ad880","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12856] [SQL] speed up hashCode of unsafe array\n\nWe iterate the bytes to calculate hashCode before, but now we have `Murmur3_x86_32.hashUnsafeBytes` that don't require the bytes to be word algned, we should use that instead.\n\nA simple benchmark shows it's about 3 X faster, benchmark code: https://gist.github.com/cloud-fan/fa77713ccebf0823b2ab#file-arrayhashbenchmark-scala\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10784 from cloud-fan/array-hashcode.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/2f7d0b68a29de9755fc9fafd9a52c048981ad880"},{"sha":"86972fa52152d2149b88ba75be048a6986006285","author":{"email":"1880ad50d4044a0399fce6619a898bea749151dd@alumni.carnegiemellon.edu","name":"Jeff Lam"},"message":"[SPARK-12722][DOCS] Fixed typo in Pipeline example\n\nhttp://spark.apache.org/docs/latest/ml-guide.html#example-pipeline\n```\nval sameModel = Pipeline.load(\"/tmp/spark-logistic-regression-model\")\n```\nshould be\n```\nval sameModel = PipelineModel.load(\"/tmp/spark-logistic-regression-model\")\n```\ncc: jkbradley\n\nAuthor: Jeff Lam <sha0lin@alumni.carnegiemellon.edu>\n\nCloses #10769 from Agent007/SPARK-12722.","distinct":true,"url":"https://api.github.com/repos/Agent007/spark/commits/86972fa52152d2149b88ba75be048a6986006285"}]}