{"push_id":1164788286,"size":14,"distinct_size":1,"ref":"refs/heads/SPARK-14995","head":"0092967df878bbaaf5ad2a007d56cc68146daea2","before":"df3e0396757bf3c2ceea1ae71f9de64c0d672e0c","commits":[{"sha":"c8809db5a5ae4111e193907ac35929a906ddac3e","author":{"name":"Kay Ousterhout","email":"c3633890c4773690888d598adeeb420b7a01d083@gmail.com"},"message":"[SPARK-15926] Improve readability of DAGScheduler stage creation methods\n\n## What changes were proposed in this pull request?\n\nThis pull request refactors parts of the DAGScheduler to improve readability, focusing on the code around stage creation.  One goal of this change it to make it clearer which functions may create new stages (as opposed to looking up stages that already exist).  There are no functionality changes in this pull request.  In more detail:\n\n* shuffleToMapStage was renamed to shuffleIdToMapStage (when reading the existing code I have sometimes struggled to remember what the key is -- is it a stage? A stage id? This change is intended to avoid that confusion)\n* Cleaned up the code to create shuffle map stages.  Previously, creating a shuffle map stage involved 3 different functions (newOrUsedShuffleStage, newShuffleMapStage, and getShuffleMapStage), and it wasn't clear what the purpose of each function was.  With the new code, a single function (getOrCreateShuffleMapStage) is responsible for getting a stage (if it already exists) or creating new shuffle map stages and any missing ancestor stages, and it delegates to createShuffleMapStage when new stages need to be created.  There's some remaining confusion here because the getOrCreateParentStages call in createShuffleMapStage may recursively create ancestor stages; this is an issue I plan to fix in a future pull request, because it's trickier to fix and involves a slight functionality change.\n* newResultStage was renamed to createResultStage, for consistency with naming around shuffle map stages.\n* getParentStages has been renamed to getOrCreateParentStages, to make it clear that this function will sometimes create missing ancestor stages.\n* The only *slight* functionality change is that on line 478, updateJobIdStageIdMaps now uses a stage's parents instance variable rather than re-calculating them (I couldn't see any reason why they'd need to be re-calculated, and suspect this is just leftover from older code).\n* getAncestorShuffleDependencies was renamed to getMissingAncestorShuffleDependencies, to make it clear that this only returns dependencies that have not yet been run.\n\ncc squito markhamstra JoshRosen (who requested more DAG scheduler commenting long ago -- an issue this pull request tries, in part, to address)\n\nFYI rxin\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #13677 from kayousterhout/SPARK-15926.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/c8809db5a5ae4111e193907ac35929a906ddac3e"},{"sha":"298c4ae81520b6b39230a6b0bf733c2b7caea627","author":{"name":"Dhruve Ashar","email":"27f589a372d341e853c1e390c399cb5de87102c3@gmail.com"},"message":"[SPARK-16018][SHUFFLE] Shade netty to load shuffle jar in Nodemanger\n\n## What changes were proposed in this pull request?\nShade the netty.io namespace so that we can use it in shuffle independent of the dependencies being pulled by hadoop jars.\n\n## How was this patch tested?\nRan a decent job involving shuffle write/read and tested the new spark-x-yarn-shuffle jar. After shading netty.io namespace, the nodemanager loads and shuffle job completes successfully.\n\nAuthor: Dhruve Ashar <dhruveashar@gmail.com>\n\nCloses #13739 from dhruve/bug/SPARK-16018.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/298c4ae81520b6b39230a6b0bf733c2b7caea627"},{"sha":"62d8fe2089659e8212753a622708517e0f4a77bc","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-16017][CORE] Send hostname from CoarseGrainedExecutorBackend to driver\n\n## What changes were proposed in this pull request?\n\n[SPARK-15395](https://issues.apache.org/jira/browse/SPARK-15395) changes the behavior that how the driver gets the executor host and the driver will get the executor IP address instead of the host name. This PR just sends the hostname from executors to driver so that driver can pass it to TaskScheduler.\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #13741 from zsxwing/SPARK-16017.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/62d8fe2089659e8212753a622708517e0f4a77bc"},{"sha":"1a65e62a7ff961ba140b9189e5335736334edbd6","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16014][SQL] Rename optimizer rules to be more consistent\n\n## What changes were proposed in this pull request?\nThis small patch renames a few optimizer rules to make the naming more consistent, e.g. class name start with a verb. The main important \"fix\" is probably SamplePushDown -> PushProjectThroughSample. SamplePushDown is actually the wrong name, since the rule is not about pushing Sample down.\n\n## How was this patch tested?\nUpdated test cases.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13732 from rxin/SPARK-16014.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/1a65e62a7ff961ba140b9189e5335736334edbd6"},{"sha":"ef3cc4fc096e831823d62af4fd2a12ae88d434b4","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-15925][SPARKR] R DataFrame add back registerTempTable, add tests\n\n## What changes were proposed in this pull request?\n\nAdd registerTempTable to DataFrame with Deprecate\n\n## How was this patch tested?\n\nunit tests\nshivaram liancheng\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13722 from felixcheung/rregistertemptable.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ef3cc4fc096e831823d62af4fd2a12ae88d434b4"},{"sha":"7d65a0db4a231882200513836f2720f59b35f364","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16005][R] Add `randomSplit` to SparkR\n\n## What changes were proposed in this pull request?\n\nThis PR adds `randomSplit` to SparkR for API parity.\n\n## How was this patch tested?\n\nPass the Jenkins tests (with new testcase.)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13721 from dongjoon-hyun/SPARK-16005.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/7d65a0db4a231882200513836f2720f59b35f364"},{"sha":"ebb9a3b6fd834e2c856a192b4455aab83e9c4dc8","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-15916][SQL] JDBC filter push down should respect operator precedence\n\n## What changes were proposed in this pull request?\n\nThis PR fixes the problem that the precedence order is messed when pushing where-clause expression to JDBC layer.\n\n**Case 1:**\n\nFor sql `select * from table where (a or b) and c`, the where-clause is wrongly converted to JDBC where-clause `a or (b and c)` after filter push down. The consequence is that JDBC may returns less or more rows than expected.\n\n**Case 2:**\n\nFor sql `select * from table where always_false_condition`, the result table may not be empty if the JDBC RDD is partitioned using where-clause:\n```\nspark.read.jdbc(url, table, predicates = Array(\"partition 1 where clause\", \"partition 2 where clause\"...)\n```\n\n## How was this patch tested?\n\nUnit test.\n\nThis PR also close #13640\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #13743 from clockfly/SPARK-15916.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ebb9a3b6fd834e2c856a192b4455aab83e9c4dc8"},{"sha":"10b671447bc04af250cbd8a7ea86f2769147a78a","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-16033][SQL] insertInto() can't be used together with partitionBy()\n\n## What changes were proposed in this pull request?\n\nWhen inserting into an existing partitioned table, partitioning columns should always be determined by catalog metadata of the existing table to be inserted. Extra `partitionBy()` calls don't make sense, and mess up existing data because newly inserted data may have wrong partitioning directory layout.\n\n## How was this patch tested?\n\nNew test case added in `InsertIntoHiveTableSuite`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #13747 from liancheng/spark-16033-insert-into-without-partition-by.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/10b671447bc04af250cbd8a7ea86f2769147a78a"},{"sha":"af2a4b0826b2358c0fe75c3e4d7fd8f7bccdd8e5","author":{"name":"GayathriMurali","email":"4ab130683b4c8acb132110628fb68dab8eafbf6a@intel.com"},"message":"[SPARK-15129][R][DOC] R API changes in ML\n\n## What changes were proposed in this pull request?\n\nMake user guide changes to SparkR documentation for all changes that happened in 2.0 to Machine Learning APIs\n\nAuthor: GayathriMurali <gayathri.m@intel.com>\n\nCloses #13285 from GayathriMurali/SPARK-15129.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/af2a4b0826b2358c0fe75c3e4d7fd8f7bccdd8e5"},{"sha":"edb23f9e47eecfe60992dde0e037ec1985c77e1d","author":{"name":"Xiangrui Meng","email":"e4720f5dc69a5967cefd679a46054c60c7e8d310@databricks.com"},"message":"[SPARK-15946][MLLIB] Conversion between old/new vector columns in a DataFrame (Python)\n\n## What changes were proposed in this pull request?\n\nThis PR implements python wrappers for #13662 to convert old/new vector columns in a DataFrame.\n\n## How was this patch tested?\n\ndoctest in Python\n\ncc: yanboliang\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #13731 from mengxr/SPARK-15946.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/edb23f9e47eecfe60992dde0e037ec1985c77e1d"},{"sha":"8c198e246d64b5779dc3a2625d06ec958553a20b","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-15159][SPARKR] SparkR SparkSession API\n\n## What changes were proposed in this pull request?\n\nThis PR introduces the new SparkSession API for SparkR.\n`sparkR.session.getOrCreate()` and `sparkR.session.stop()`\n\n\"getOrCreate\" is a bit unusual in R but it's important to name this clearly.\n\nSparkR implementation should\n- SparkSession is the main entrypoint (vs SparkContext; due to limited functionality supported with SparkContext in SparkR)\n- SparkSession replaces SQLContext and HiveContext (both a wrapper around SparkSession, and because of API changes, supporting all 3 would be a lot more work)\n- Changes to SparkSession is mostly transparent to users due to SPARK-10903\n- Full backward compatibility is expected - users should be able to initialize everything just in Spark 1.6.1 (`sparkR.init()`), but with deprecation warning\n- Mostly cosmetic changes to parameter list - users should be able to move to `sparkR.session.getOrCreate()` easily\n- An advanced syntax with named parameters (aka varargs aka \"...\") is supported; that should be closer to the Builder syntax that is in Scala/Python (which unfortunately does not work in R because it will look like this: `enableHiveSupport(config(config(master(appName(builder(), \"foo\"), \"local\"), \"first\", \"value\"), \"next, \"value\"))`\n- Updating config on an existing SparkSession is supported, the behavior is the same as Python, in which config is applied to both SparkContext and SparkSession\n- Some SparkSession changes are not matched in SparkR, mostly because it would be breaking API change: `catalog` object, `createOrReplaceTempView`\n- Other SQLContext workarounds are replicated in SparkR, eg. `tables`, `tableNames`\n- `sparkR` shell is updated to use the SparkSession entrypoint (`sqlContext` is removed, just like with Scale/Python)\n- All tests are updated to use the SparkSession entrypoint\n- A bug in `read.jdbc` is fixed\n\nTODO\n- [x] Add more tests\n- [ ] Separate PR - update all roxygen2 doc coding example\n- [ ] Separate PR - update SparkR programming guide\n\n## How was this patch tested?\n\nunit tests, manual tests\n\nshivaram sun-rui rxin\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #13635 from felixcheung/rsparksession.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/8c198e246d64b5779dc3a2625d06ec958553a20b"},{"sha":"d0ac0e6f433bfccf4ced3743a2526f67fdb5c38e","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-16020][SQL] Fix complete mode aggregation with console sink\n\n## What changes were proposed in this pull request?\n\nWe cannot use `limit` on DataFrame in ConsoleSink because it will use a wrong planner. This PR just collects `DataFrame` and calls `show` on a batch DataFrame based on the result. This is fine since ConsoleSink is only for debugging.\n\n## How was this patch tested?\n\nManually confirmed ConsoleSink now works with complete mode aggregation.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #13740 from zsxwing/complete-console.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/d0ac0e6f433bfccf4ced3743a2526f67fdb5c38e"},{"sha":"4c64e88d5ba4c36cbdbc903376492f0f43401e4e","author":{"name":"andreapasqua","email":"fe714fe7c07fd0f92a2400b512fa35e6342369e8@radius.com"},"message":"[SPARK-16035][PYSPARK] Fix SparseVector parser assertion for end parenthesis\n\n## What changes were proposed in this pull request?\nThe check on the end parenthesis of the expression to parse was using the wrong variable. I corrected that.\n## How was this patch tested?\nManual test\n\nAuthor: andreapasqua <andrea@radius.com>\n\nCloses #13750 from andreapasqua/sparse-vector-parser-assertion-fix.","distinct":false,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/4c64e88d5ba4c36cbdbc903376492f0f43401e4e"},{"sha":"0092967df878bbaaf5ad2a007d56cc68146daea2","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-14995][R] Add `since` tag in Roxygen documentation for SparkR API methods","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/0092967df878bbaaf5ad2a007d56cc68146daea2"}]}
{"actor":{"display_login":"dongjoon-hyun"}}