{"push_id":1164640605,"size":21,"distinct_size":21,"ref":"refs/heads/branch-2.0-mmx","head":"9b65b10a8b6d5bf8ec85a9c13439b79691efc7c8","before":"bdeb29ea0592fff3442994d0678cfe629993fe3b","commits":[{"sha":"7d8cddfb495d406b9f2fb5216edd14dea442ec73","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-15998][SQL] Verification of SQLConf HIVE_METASTORE_PARTITION_PRUNING\n\n#### What changes were proposed in this pull request?\n`HIVE_METASTORE_PARTITION_PRUNING` is a public `SQLConf`. When `true`, some predicates will be pushed down into the Hive metastore so that unmatching partitions can be eliminated earlier. The current default value is `false`. For performance improvement, users might turn this parameter on.\n\nSo far, the code base does not have such a test case to verify whether this `SQLConf` properly works. This PR is to improve the test case coverage for avoiding future regression.\n\n#### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13716 from gatorsmile/addTestMetastorePartitionPruning.","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/7d8cddfb495d406b9f2fb5216edd14dea442ec73"},{"sha":"fb0fab63cb005d9efc624aeb0ac85476a9ddc4f4","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[MINOR][DOCS][SQL] Fix some comments about types(TypeCoercion,Partition) and exceptions.\n\n## What changes were proposed in this pull request?\n\nThis PR contains a few changes on code comments.\n- `HiveTypeCoercion` is renamed into `TypeCoercion`.\n- `NoSuchDatabaseException` is only used for the absence of database.\n- For partition type inference, only `DoubleType` is considered.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13674 from dongjoon-hyun/minor_doc_types.\n\n(cherry picked from commit 2d27eb1e753daefbd311136fc7de1a3e8fb9dc63)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/fb0fab63cb005d9efc624aeb0ac85476a9ddc4f4"},{"sha":"27e274c3e8cad29fc684a1611cef19d60acdfbc0","author":{"name":"Alex Bozarth","email":"850363088eed533ef66882756a87f03f1fc98fde@us.ibm.com"},"message":"[SPARK-15868][WEB UI] Executors table in Executors tab should sort Executor IDs in numerical order\n\n## What changes were proposed in this pull request?\n\nCurrently the Executors table sorts by id using a string sort (since that's what it is stored as). Since  the id is a number (other than the driver) we should be sorting numerically. I have changed both the initial sort on page load as well as the table sort to sort on id numerically, treating non-numeric strings (like the driver) as \"-1\"\n\n## How was this patch tested?\n\nManually tested and dev/run-tests\n\n![pageload](https://cloud.githubusercontent.com/assets/13952758/16027882/d32edd0a-318e-11e6-9faf-fc972b7c36ab.png)\n![sorted](https://cloud.githubusercontent.com/assets/13952758/16027883/d34541c6-318e-11e6-9ed7-6bfc0cd4152e.png)\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #13654 from ajbozarth/spark15868.\n\n(cherry picked from commit e849285df03b1233d5f647f1b6c5a6dad0665855)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/27e274c3e8cad29fc684a1611cef19d60acdfbc0"},{"sha":"2280ad8a3ddfff0b7cc10de6eadb2cc93423bbcf","author":{"name":"Huaxin Gao","email":"bcd9b95fe24734cb14220adbd8abc2a182c19aa3@us.ibm.com"},"message":"[SPARK-15749][SQL] make the error message more meaningful\n\n## What changes were proposed in this pull request?\n\nFor table test1 (C1 varchar (10), C2 varchar (10)), when I insert a row using\n```\nsqlContext.sql(\"insert into test1 values ('abc', 'def', 1)\")\n```\nI got error message\n\n```\nException in thread \"main\" java.lang.RuntimeException: RelationC1#0,C2#1 JDBCRelation(test1)\nrequires that the query in the SELECT clause of the INSERT INTO/OVERWRITE statement\ngenerates the same number of columns as its schema.\n```\nThe error message is a little confusing. In my simple insert statement, it doesn't have a SELECT clause.\n\nI will change the error message to a more general one\n\n```\nException in thread \"main\" java.lang.RuntimeException: RelationC1#0,C2#1 JDBCRelation(test1)\nrequires that the data to be inserted have the same number of columns as the target table.\n```\n\n## How was this patch tested?\n\nI tested the patch using my simple unit test, but it's a very trivial change and I don't think I need to check in any test.\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #13492 from huaxingao/spark-15749.\n\n(cherry picked from commit 62d2fa5e996d428caaea005041b17ec115473762)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/2280ad8a3ddfff0b7cc10de6eadb2cc93423bbcf"},{"sha":"8f713885963a410571370faa6b147cd0ada3832b","author":{"name":"Dhruve Ashar","email":"27f589a372d341e853c1e390c399cb5de87102c3@gmail.com"},"message":"[SPARK-15966][DOC] Add closing tag to fix rendering issue for Spark monitoring\n\n## What changes were proposed in this pull request?\nAdds the missing closing tag for spark.ui.view.acls.groups\n\n## How was this patch tested?\nI built the docs locally and verified the changed in browser.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n**Before:**\n![image](https://cloud.githubusercontent.com/assets/7732317/16135005/49fc0724-33e6-11e6-9390-98711593fa5b.png)\n\n**After:**\n![image](https://cloud.githubusercontent.com/assets/7732317/16135021/62b5c4a8-33e6-11e6-8118-b22fda5c66eb.png)\n\nAuthor: Dhruve Ashar <dhruveashar@gmail.com>\n\nCloses #13719 from dhruve/doc/SPARK-15966.","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/8f713885963a410571370faa6b147cd0ada3832b"},{"sha":"b3678eb7e4ac6bb08ba8579867944ba42da99b81","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-15991] SparkContext.hadoopConfiguration should be always the base of hadoop conf created by SessionState\n\n## What changes were proposed in this pull request?\nBefore this patch, after a SparkSession has been created, hadoop conf set directly to SparkContext.hadoopConfiguration will not affect the hadoop conf created by SessionState. This patch makes the change to always use SparkContext.hadoopConfiguration  as the base.\n\nThis patch also changes the behavior of hive-site.xml support added in https://github.com/apache/spark/pull/12689/. With this patch, we will load hive-site.xml to SparkContext.hadoopConfiguration.\n\n## How was this patch tested?\nNew test in SparkSessionBuilderSuite.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #13711 from yhuai/SPARK-15991.\n\n(cherry picked from commit d9c6628c47de547dc537310e3c775c7f3e0e4a12)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/b3678eb7e4ac6bb08ba8579867944ba42da99b81"},{"sha":"68e7a25cc06cbfe357be8d224c117abaa7ba94f4","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-15608][ML][EXAMPLES][DOC] add examples and documents of ml.isotonic regression\n\n## What changes were proposed in this pull request?\n\nadd ml doc for ml isotonic regression\nadd scala example for ml isotonic regression\nadd java example for ml isotonic regression\nadd python example for ml isotonic regression\n\nmodify scala example for mllib isotonic regression\nmodify java example for mllib isotonic regression\nmodify python example for mllib isotonic regression\n\nadd data/mllib/sample_isotonic_regression_libsvm_data.txt\ndelete data/mllib/sample_isotonic_regression_data.txt\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #13381 from WeichenXu123/add_isotonic_regression_doc.\n\n(cherry picked from commit 9040d83bc2cdce06dab0e1bdee4f796da9a9a55c)\nSigned-off-by: Yanbo Liang <ybliang8@gmail.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/68e7a25cc06cbfe357be8d224c117abaa7ba94f4"},{"sha":"feaba979b30b27f661ae44ae3f12eabc3a6e55b3","author":{"name":"Nezih Yigitbasi","email":"e4abc6cedac890c04299a695aca24e4495ea8a5c@netflix.com"},"message":"[SPARK-15782][YARN] Fix spark.jars and spark.yarn.dist.jars handling\n\nWhen `--packages` is specified with spark-shell the classes from those packages cannot be found, which I think is due to some of the changes in SPARK-12343.\n\nTested manually with both scala 2.10 and 2.11 repls.\n\nvanzin davies can you guys please review?\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\nAuthor: Nezih Yigitbasi <nyigitbasi@netflix.com>\n\nCloses #13709 from nezihyigitbasi/SPARK-15782.\n\n(cherry picked from commit 63470afc997fb9d6b6f8a911c25964743556c9cc)\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/feaba979b30b27f661ae44ae3f12eabc3a6e55b3"},{"sha":"2127f99f2c8cf6d3f85e6408ce47b82e0c3cad4d","author":{"name":"Kai Jiang","email":"ef4ed51a712a0637a2f995790bc70ab93f1b05ee@gmail.com"},"message":"[SPARK-15490][R][DOC] SparkR 2.0 QA: New R APIs and API docs for non-MLib changes\n\n## What changes were proposed in this pull request?\nR Docs changes\ninclude typos, format, layout.\n## How was this patch tested?\nTest locally.\n\nAuthor: Kai Jiang <jiangkai@gmail.com>\n\nCloses #13394 from vectorijk/spark-15490.\n\n(cherry picked from commit 5fd20b66ffe18c05cf257af7f30d32464d2fe8e7)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/2127f99f2c8cf6d3f85e6408ce47b82e0c3cad4d"},{"sha":"f530331e6f8160f3fb2613722fae01ea589f0e99","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-15908][R] Add varargs-type dropDuplicates() function in SparkR\n\n## What changes were proposed in this pull request?\n\nThis PR adds varargs-type `dropDuplicates` function to SparkR for API parity.\nRefer to https://issues.apache.org/jira/browse/SPARK-15807, too.\n\n## How was this patch tested?\n\nPass the Jenkins tests with new testcases.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13684 from dongjoon-hyun/SPARK-15908.\n\n(cherry picked from commit 513a03e41e27d9c5f70911faccc5d3aecd8bdde9)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/f530331e6f8160f3fb2613722fae01ea589f0e99"},{"sha":"3994372f48eefe080ce7a80750ccf960e3a7968b","author":{"name":"Pete Robbins","email":"65c4535b59e9806d6fbe65e3ae9fd980a3fbb159@gmail.com"},"message":"[SPARK-15822] [SQL] Prevent byte array backed classes from referencing freed memory\n\n## What changes were proposed in this pull request?\n`UTF8String` and all `Unsafe*` classes are backed by either on-heap or off-heap byte arrays. The code generated version `SortMergeJoin` buffers the left hand side join keys during iteration. This was actually problematic in off-heap mode when one of the keys is a `UTF8String` (or any other 'Unsafe*` object) and the left hand side iterator was exhausted (and released its memory); the buffered keys would reference freed memory. This causes Seg-faults and all kinds of other undefined behavior when we would use one these buffered keys.\n\nThis PR fixes this problem by creating copies of the buffered variables. I have added a general method to the `CodeGenerator` for this. I have checked all places in which this could happen, and only `SortMergeJoin` had this problem.\n\nThis PR is largely based on the work of robbinspg and he should be credited for this.\n\ncloses https://github.com/apache/spark/pull/13707\n\n## How was this patch tested?\nManually tested on problematic workloads.\n\nAuthor: Pete Robbins <robbinspg@gmail.com>\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #13723 from hvanhovell/SPARK-15822-2.\n\n(cherry picked from commit 5ada606144c7bf38a797764619d7d1ff677802b3)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/3994372f48eefe080ce7a80750ccf960e3a7968b"},{"sha":"b82abde060d97bd95f4fba547545a830602a35fa","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-15706][SQL] Fix Wrong Answer when using IF NOT EXISTS in INSERT OVERWRITE for DYNAMIC PARTITION\n\n#### What changes were proposed in this pull request?\n`IF NOT EXISTS` in `INSERT OVERWRITE` should not support dynamic partitions. If we specify `IF NOT EXISTS`, the inserted statement is not shown in the table.\n\nThis PR is to issue an exception in this case, just like what Hive does. Also issue an exception if users specify `IF NOT EXISTS` if users do not specify any `PARTITION` specification.\n\n#### How was this patch tested?\nAdded test cases into `PlanParserSuite` and `InsertIntoHiveTableSuite`\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13447 from gatorsmile/insertIfNotExist.\n\n(cherry picked from commit e5d703bca85c65ce329b1e202283cfa35d109146)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/b82abde060d97bd95f4fba547545a830602a35fa"},{"sha":"3c3865d0b42413435f8cc9044da842cfb4458afc","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-15811][SQL] fix the Python UDF in Scala 2.10\n\n## What changes were proposed in this pull request?\n\nIterator can't be serialized in Scala 2.10, we should force it into a array to make sure that .\n\n## How was this patch tested?\n\nBuild with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13717 from davies/fix_udf_210.\n\n(cherry picked from commit ef43b4ed87894982678fcc6f2c61cf1487ee9e14)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/3c3865d0b42413435f8cc9044da842cfb4458afc"},{"sha":"de964e4194cac1ad9ed4ca7f16dbb809fc7c9471","author":{"name":"Sameer Agarwal","email":"73cebedf715c5415a2956d45e4fbb64de14b0dfd@databricks.com"},"message":"Remove non-obvious conf settings from TPCDS benchmark\n\n## What changes were proposed in this pull request?\n\nMy fault -- these 2 conf entries are mysteriously hidden inside the benchmark code and makes it non-obvious to disable whole stage codegen and/or the vectorized parquet reader.\n\nPS: Didn't attach a JIRA as this change should otherwise be a no-op (both these conf are enabled by default in Spark)\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #13726 from sameeragarwal/tpcds-conf.\n\n(cherry picked from commit 34d6c4cd113729fcc1d0bc1df8916d06b8854922)\nSigned-off-by: Herman van Hovell <hvanhovell@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/de964e4194cac1ad9ed4ca7f16dbb809fc7c9471"},{"sha":"269b715e478fea7cb5ec6237d6439c6642ebc5e6","author":{"name":"sethah","email":"c41626c9d885a60b8db6c40a26d851740b23beef@gmail.com"},"message":"[SPARK-16008][ML] Remove unnecessary serialization in logistic regression\n\nJIRA: [SPARK-16008](https://issues.apache.org/jira/browse/SPARK-16008)\n\n## What changes were proposed in this pull request?\n`LogisticAggregator` stores references to two arrays of dimension `numFeatures` which are serialized before the combine op, unnecessarily. This results in the shuffle write being ~3x (for multiclass logistic regression, this number will go up) larger than it should be (in MLlib, for instance, it is 3x smaller).\n\nThis patch modifies `LogisticAggregator.add` to accept the two arrays as method parameters which avoids the serialization.\n\n## How was this patch tested?\n\nI tested this locally and verified the serialization reduction.\n\n![image](https://cloud.githubusercontent.com/assets/7275795/16140387/d2974bac-3404-11e6-94f9-268860c931a2.png)\n\nAdditionally, I ran some tests of a 4 node cluster (4x48 cores, 4x128 GB RAM). Data set size of 2M rows and 10k features showed >2x iteration speedup.\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #13729 from sethah/lr_improvement.\n\n(cherry picked from commit 1f0a46958ef51a01560ada23665dccde89696e12)\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/269b715e478fea7cb5ec6237d6439c6642ebc5e6"},{"sha":"3457497e0257cd2ce8d87cccf258a5abe2339eb9","author":{"name":"Dhruve Ashar","email":"27f589a372d341e853c1e390c399cb5de87102c3@gmail.com"},"message":"[SPARK-16018][SHUFFLE] Shade netty to load shuffle jar in Nodemanger\n\n## What changes were proposed in this pull request?\nShade the netty.io namespace so that we can use it in shuffle independent of the dependencies being pulled by hadoop jars.\n\n## How was this patch tested?\nRan a decent job involving shuffle write/read and tested the new spark-x-yarn-shuffle jar. After shading netty.io namespace, the nodemanager loads and shuffle job completes successfully.\n\nAuthor: Dhruve Ashar <dhruveashar@gmail.com>\n\nCloses #13739 from dhruve/bug/SPARK-16018.\n\n(cherry picked from commit 298c4ae81520b6b39230a6b0bf733c2b7caea627)\nSigned-off-by: Tom Graves <tgraves@yahoo-inc.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/3457497e0257cd2ce8d87cccf258a5abe2339eb9"},{"sha":"0701b8d95caa79f220425b2a7376e88d69864663","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-16017][CORE] Send hostname from CoarseGrainedExecutorBackend to driver\n\n## What changes were proposed in this pull request?\n\n[SPARK-15395](https://issues.apache.org/jira/browse/SPARK-15395) changes the behavior that how the driver gets the executor host and the driver will get the executor IP address instead of the host name. This PR just sends the hostname from executors to driver so that driver can pass it to TaskScheduler.\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #13741 from zsxwing/SPARK-16017.\n\n(cherry picked from commit 62d8fe2089659e8212753a622708517e0f4a77bc)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/0701b8d95caa79f220425b2a7376e88d69864663"},{"sha":"2e5211ff9b141c7cc39a7fe05c278364d934fc52","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16014][SQL] Rename optimizer rules to be more consistent\n\n## What changes were proposed in this pull request?\nThis small patch renames a few optimizer rules to make the naming more consistent, e.g. class name start with a verb. The main important \"fix\" is probably SamplePushDown -> PushProjectThroughSample. SamplePushDown is actually the wrong name, since the rule is not about pushing Sample down.\n\n## How was this patch tested?\nUpdated test cases.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13732 from rxin/SPARK-16014.\n\n(cherry picked from commit 1a65e62a7ff961ba140b9189e5335736334edbd6)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/2e5211ff9b141c7cc39a7fe05c278364d934fc52"},{"sha":"d4bb9a3ff16bc1f51bcf7156abff783e901d19d2","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-15925][SPARKR] R DataFrame add back registerTempTable, add tests\n\n## What changes were proposed in this pull request?\n\nAdd registerTempTable to DataFrame with Deprecate\n\n## How was this patch tested?\n\nunit tests\nshivaram liancheng\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13722 from felixcheung/rregistertemptable.\n\n(cherry picked from commit ef3cc4fc096e831823d62af4fd2a12ae88d434b4)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/d4bb9a3ff16bc1f51bcf7156abff783e901d19d2"},{"sha":"ca0802fd55f42fdcdd98533ee515d40d9f04a4b3","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16005][R] Add `randomSplit` to SparkR\n\n## What changes were proposed in this pull request?\n\nThis PR adds `randomSplit` to SparkR for API parity.\n\n## How was this patch tested?\n\nPass the Jenkins tests (with new testcase.)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13721 from dongjoon-hyun/SPARK-16005.\n\n(cherry picked from commit 7d65a0db4a231882200513836f2720f59b35f364)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/metamx/spark/commits/ca0802fd55f42fdcdd98533ee515d40d9f04a4b3"}]}
{"actor":{"display_login":"drcrallen"}}