{"push_id":1386902408,"size":21,"distinct_size":21,"ref":"refs/heads/master","head":"a87471c83006ec11c372b4f915e17a0501f1f536","before":"27daf6bcde782ed3e0f0d951c90c8040fd47e985","commits":[{"sha":"b17057c0a69b9c56e503483d97f5dc209eef0884","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18244][SQL] Rename partitionProviderIsHive -> tracksPartitionsInCatalog\n\n## What changes were proposed in this pull request?\nThis patch renames partitionProviderIsHive to tracksPartitionsInCatalog, as the old name was too Hive specific.\n\n## How was this patch tested?\nShould be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15750 from rxin/SPARK-18244.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/b17057c0a69b9c56e503483d97f5dc209eef0884"},{"sha":"16293311cdb25a62733a9aae4355659b971a3ce1","author":{"name":"福星","email":"593dbf5d0ef3bc166e99e03426dfabbd1f951119@wacai.com"},"message":"[SPARK-18237][HIVE] hive.exec.stagingdir have no effect\n\nhive.exec.stagingdir have no effect in spark2.0.1，\nHive confs in hive-site.xml will be loaded in `hadoopConf`, so we should use `hadoopConf` in `InsertIntoHiveTable` instead of `SessionState.conf`\n\nAuthor: 福星 <fuxing@wacai.com>\n\nCloses #15744 from ClassNotFoundExp/master.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/16293311cdb25a62733a9aae4355659b971a3ce1"},{"sha":"098e4ca9c7af61e64839a50c65be449749af6482","author":{"name":"Kishor Patil","email":"53fdec9933651013cff2c5b00d76e765b1cc9456@yahoo-inc.com"},"message":"[SPARK-18099][YARN] Fail if same files added to distributed cache for --files and --archives\n\n## What changes were proposed in this pull request?\n\nDuring spark-submit, if yarn dist cache is instructed to add same file under --files and --archives, This code change ensures the spark yarn distributed cache behaviour is retained i.e. to warn and fail if same files is mentioned in both --files and --archives.\n## How was this patch tested?\n\nManually tested:\n1. if same jar is mentioned in --jars and --files it will continue to submit the job.\n- basically functionality [SPARK-14423] #12203 is unchanged\n  1. if same file is mentioned in --files and --archives it will fail to submit the job.\n\nPlease review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.\n\n… under archives and files\n\nAuthor: Kishor Patil <kpatil@yahoo-inc.com>\n\nCloses #15627 from kishorvpatil/spark18099.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/098e4ca9c7af61e64839a50c65be449749af6482"},{"sha":"67659c9afaeb2289e56fd87fafee953e8f050383","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-18212][SS][KAFKA] increase executor poll timeout\n\n## What changes were proposed in this pull request?\n\nIncrease poll timeout to try and address flaky test\n\n## How was this patch tested?\n\nRan existing unit tests\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #15737 from koeninger/SPARK-18212.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/67659c9afaeb2289e56fd87fafee953e8f050383"},{"sha":"e89202523bc2f5573bd736278c6b96c6e6759909","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARKR][TEST] remove unnecessary suppressWarnings\n\n## What changes were proposed in this pull request?\n\nIn test_mllib.R, there are two unnecessary suppressWarnings. This PR just removes them.\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #15697 from wangmiao1981/rtest.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/e89202523bc2f5573bd736278c6b96c6e6759909"},{"sha":"f22954ad49bf5a32c7b6d8487cd38ffe0da904ca","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18257][SS] Improve error reporting for FileStressSuite\n\n## What changes were proposed in this pull request?\nThis patch improves error reporting for FileStressSuite, when there is an error in Spark itself (not user code). This works by simply tightening the exception verification, and gets rid of the unnecessary thread for starting the stream.\n\nAlso renamed the class FileStreamStressSuite to make it more obvious it is a streaming suite.\n\n## How was this patch tested?\nThis is a test only change and I manually verified error reporting by injecting some bug in the addBatch code for FileStreamSink.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15757 from rxin/SPARK-18257.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/f22954ad49bf5a32c7b6d8487cd38ffe0da904ca"},{"sha":"dc4c60098641cf64007e2f0e36378f000ad5f6b1","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-18138][DOCS] Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0\n\n## What changes were proposed in this pull request?\n\nDocument that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0. This does not actually implement any of the change in SPARK-18138, just peppers the documentation with notices about it.\n\n## How was this patch tested?\n\nDoc build\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #15733 from srowen/SPARK-18138.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/dc4c60098641cf64007e2f0e36378f000ad5f6b1"},{"sha":"aa412c55e31e61419d3de57ef4b13e50f9b38af0","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-18259][SQL] Do not capture Throwable in QueryExecution\n\n## What changes were proposed in this pull request?\n`QueryExecution.toString` currently captures `java.lang.Throwable`s; this is far from a best practice and can lead to confusing situation or invalid application states. This PR fixes this by only capturing `AnalysisException`s.\n\n## How was this patch tested?\nAdded a `QueryExecutionSuite`.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #15760 from hvanhovell/SPARK-18259.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/aa412c55e31e61419d3de57ef4b13e50f9b38af0"},{"sha":"a08463b1d32348a81d0f148dfaf22741d5c23b1a","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-14393][SQL][DOC] update doc for python and R\n\n## What changes were proposed in this pull request?\n\nminor doc update that should go to master & branch-2.1\n\n## How was this patch tested?\n\nmanual\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #15747 from felixcheung/pySPARK-14393.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/a08463b1d32348a81d0f148dfaf22741d5c23b1a"},{"sha":"27602c33751cebf6cd173c0de103454608cf6625","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18200][GRAPHX][FOLLOW-UP] Support zero as an initial capacity in OpenHashSet\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up PR of #15741 in order to keep `nextPowerOf2` consistent.\n\n**Before**\n```\nnextPowerOf2(0) => 2\nnextPowerOf2(1) => 1\nnextPowerOf2(2) => 2\nnextPowerOf2(3) => 4\nnextPowerOf2(4) => 4\nnextPowerOf2(5) => 8\n```\n\n**After**\n```\nnextPowerOf2(0) => 1\nnextPowerOf2(1) => 1\nnextPowerOf2(2) => 2\nnextPowerOf2(3) => 4\nnextPowerOf2(4) => 4\nnextPowerOf2(5) => 8\n```\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15754 from dongjoon-hyun/SPARK-18200-2.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/27602c33751cebf6cd173c0de103454608cf6625"},{"sha":"14f235d5643bca75e270652c15154d86e57a7a70","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"Closing some stale/invalid pull requests\n\nCloses #15758\nCloses #15753\nCloses #12708","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/14f235d5643bca75e270652c15154d86e57a7a70"},{"sha":"a42d738c5de08bd395a7c220c487146173c6c163","author":{"name":"Adam Roberts","email":"ac6c9ba1da70de03b556cc2f8b9c0fac7b1e271e@uk.ibm.com"},"message":"[SPARK-18197][CORE] Optimise AppendOnlyMap implementation\n\n## What changes were proposed in this pull request?\nThis improvement works by using the fastest comparison test first and we observed a 1% throughput performance improvement on PageRank (HiBench large profile) with this change.\n\nWe used tprof and before the change in AppendOnlyMap.changeValue (where the optimisation occurs) this method was being used for 8053 profiling ticks representing 0.72% of the overall application time.\n\nAfter this change we observed this method only occurring for 2786 ticks and for 0.25% of the overall time.\n\n## How was this patch tested?\nExisting unit tests and for performance we used HiBench large, profiling with tprof and IBM Healthcenter.\n\nAuthor: Adam Roberts <aroberts@uk.ibm.com>\n\nCloses #15714 from a-roberts/patch-9.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/a42d738c5de08bd395a7c220c487146173c6c163"},{"sha":"550cd56e8b6addb26efe3ce16976c9c34fa0c832","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-17337][SQL] Do not pushdown predicates through filters with  predicate subqueries\n\n## What changes were proposed in this pull request?\nThe `PushDownPredicate` rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source).\n\nThe current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan.\n\n## How was this patch tested?\nAdded a test to `SubquerySuite`. nsyca has done previous work this. I have taken test from his initial PR.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #15761 from hvanhovell/SPARK-17337.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/550cd56e8b6addb26efe3ce16976c9c34fa0c832"},{"sha":"4cee2ce251110218e68c0f8f30363ec2f2498bea","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-18167] Re-enable the non-flaky parts of SQLQuerySuite\n\n## What changes were proposed in this pull request?\n\nIt seems the proximate cause of the test failures is that `cast(str as decimal)` in derby will raise an exception instead of returning NULL. This is a problem since Hive sometimes inserts `__HIVE_DEFAULT_PARTITION__` entries into the partition table as documented here: https://github.com/apache/hive/blob/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1034\n\nBasically, when these special default partitions are present, partition pruning pushdown using the SQL-direct mode will fail due this cast exception. As commented on in `MetaStoreDirectSql.java` above, this is normally fine since Hive falls back to JDO pruning, however when the pruning predicate contains an unsupported operator such as `>`, that will fail as well.\n\nThe only remaining question is why this behavior is nondeterministic. We know that when the test flakes, retries do not help, therefore the cause must be environmental. The current best hypothesis is that some config is different between different jenkins runs, which is why this PR prints out the Spark SQL and Hive confs for the test. The hope is that by comparing the config state for failure vs success we can isolate the root cause of the flakiness.\n\n**Update:** we could not isolate the issue. It does not seem to be due to configuration differences. As such, I'm going to enable the non-flaky parts of the test since we are fairly confident these issues only occur with Derby (which is not used in production).\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #15725 from ericl/print-confs-out.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/4cee2ce251110218e68c0f8f30363ec2f2498bea"},{"sha":"0e3312ee72c44f4c9acafbd80d0c8a14f3aff875","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-18256] Improve the performance of event log replay in HistoryServer\n\n## What changes were proposed in this pull request?\n\nThis patch significantly improves the performance of event log replay in the HistoryServer via two simple changes:\n\n- **Don't use `extractOpt`**: it turns out that `json4s`'s `extractOpt` method uses exceptions for control flow, causing huge performance bottlenecks due to the overhead of initializing exceptions. To avoid this overhead, we can simply use our own` Utils.jsonOption` method. This patch replaces all uses of `extractOpt` with `Utils.jsonOption` and adds a style checker rule to ban the use of the slow `extractOpt` method.\n- **Don't call `Utils.getFormattedClassName` for every event**: the old code called` Utils.getFormattedClassName` dozens of times per replayed event in order to match up class names in events with SparkListener event names. By simply storing the results of these calls in constants rather than recomputing them, we're able to eliminate a huge performance hotspot by removing thousands of expensive `Class.getSimpleName` calls.\n\n## How was this patch tested?\n\nTested by profiling the replay of a long event log using YourKit. For an event log containing 1000+ jobs, each of which had thousands of tasks, the changes in this patch cut the replay time in half:\n\n![image](https://cloud.githubusercontent.com/assets/50748/19980953/31154622-a1bd-11e6-9be4-21fbb9b3f9a7.png)\n\nPrior to this patch's changes, the two slowest methods in log replay were internal exceptions thrown by `Json4S` and calls to `Class.getSimpleName()`:\n\n![image](https://cloud.githubusercontent.com/assets/50748/19981052/87416cce-a1bd-11e6-9f25-06a7cd391822.png)\n\nAfter this patch, these hotspots are completely eliminated.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #15756 from JoshRosen/speed-up-jsonprotocol.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/0e3312ee72c44f4c9acafbd80d0c8a14f3aff875"},{"sha":"0f7c9e84e0d00813bf56712097677add5657f19f","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18189] [SQL] [Followup] Move test from ReplSuite to prevent java.lang.ClassCircularityError\n\ncloses #15774","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/0f7c9e84e0d00813bf56712097677add5657f19f"},{"sha":"8a9ca1924792d1a7c733bdfd757996b3ade0d63d","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[SPARK-17710][FOLLOW UP] Add comments to state why 'Utils.classForName' is not used\n\n## What changes were proposed in this pull request?\nAdd comments.\n\n## How was this patch tested?\nBuild passed.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #15776 from weiqingy/SPARK-17710.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/8a9ca1924792d1a7c733bdfd757996b3ade0d63d"},{"sha":"6e2701815761d5870111cb56300e30d3059b39ed","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-18260] Make from_json null safe\n\n## What changes were proposed in this pull request?\n\n`from_json` is currently not safe against `null` rows. This PR adds a fix and a regression test for it.\n\n## How was this patch tested?\n\nRegression test\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #15771 from brkyvz/json_fix.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/6e2701815761d5870111cb56300e30d3059b39ed"},{"sha":"95ec4e25bb65f37f80222ffe70a95993a9149f80","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17183][SPARK-17983][SPARK-18101][SQL] put hive serde table schema to table properties like data source table\n\n## What changes were proposed in this pull request?\n\nFor data source tables, we will put its table schema, partition columns, etc. to table properties, to work around some hive metastore issues, e.g. not case-preserving, bad decimal type support, etc.\n\nWe should also do this for hive serde tables, to reduce the difference between hive serde tables and data source tables, e.g. column names should be case preserving.\n## How was this patch tested?\n\nexisting tests, and a new test in `HiveExternalCatalog`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14750 from cloud-fan/minor1.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/95ec4e25bb65f37f80222ffe70a95993a9149f80"},{"sha":"e2648d35577c9664968cf6da5069277dbfb410d2","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18287][SQL] Move hash expressions from misc.scala into hash.scala\n\n## What changes were proposed in this pull request?\nAs the title suggests, this patch moves hash expressions from misc.scala into hash.scala, to make it easier to find the hash functions. I wanted to do this a while ago but decided to wait for the branch-2.1 cut so the chance of conflicts will be smaller.\n\n## How was this patch tested?\nTest cases were also moved out of MiscFunctionsSuite into HashExpressionsSuite.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15784 from rxin/SPARK-18287.","distinct":true,"url":"https://api.github.com/repos/techaddict/spark/commits/e2648d35577c9664968cf6da5069277dbfb410d2"}]}
{"actor":{"display_login":"techaddict"}}