{"push_id":1346504334,"size":18,"distinct_size":1,"ref":"refs/heads/SPARK-17894","head":"6f79ff857fb3db0eb9baea390c96fec198b39f62","before":"c78631af776c232dd921c293c78eda4e0b755310","commits":[{"sha":"8880fd13ef2b581f9c7190e7e3e6d24bc11b4ef7","author":{"name":"Bijay Pathak","email":"43d697da37c7bf7fe3502262d1203789485c91be@mtu.edu"},"message":"[SPARK-14761][SQL] Reject invalid join methods when join columns are not specified in PySpark DataFrame join.\n\n## What changes were proposed in this pull request?\n\nIn PySpark, the invalid join type will not throw error for the following join:\n```df1.join(df2, how='not-a-valid-join-type')```\n\nThe signature of the join is:\n```def join(self, other, on=None, how=None):```\nThe existing code completely ignores the `how` parameter when `on` is `None`. This patch will process the arguments passed to join and pass in to JVM Spark SQL Analyzer, which will validate the join type passed.\n\n## How was this patch tested?\nUsed manual and existing test suites.\n\nAuthor: Bijay Pathak <bkpathak@mtu.edu>\n\nCloses #15409 from bkpathak/SPARK-14761.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/8880fd13ef2b581f9c7190e7e3e6d24bc11b4ef7"},{"sha":"d5580ebaa086b9feb72d5428f24c5b60cd7da745","author":{"name":"prigarg","email":"4c4991f804c188fcf2fbd8e1ce3d8224d687e073@adobe.com"},"message":"[SPARK-17884][SQL] To resolve Null pointer exception when casting from empty string to interval type.\n\n## What changes were proposed in this pull request?\nThis change adds a check in castToInterval method of Cast expression , such that if converted value is null , then isNull variable should be set to true.\n\nEarlier, the expression Cast(Literal(), CalendarIntervalType) was throwing NullPointerException because of the above mentioned reason.\n\n## How was this patch tested?\nAdded test case in CastSuite.scala\n\njira entry for detail: https://issues.apache.org/jira/browse/SPARK-17884\n\nAuthor: prigarg <prigarg@adobe.com>\n\nCloses #15449 from priyankagargnitk/SPARK-17884.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/d5580ebaa086b9feb72d5428f24c5b60cd7da745"},{"sha":"5cc503f4fe9737a4c7947a80eecac053780606df","author":{"name":"Hossein","email":"d6113056401dfad7afd27763bfa2c3f84d4619bc@databricks.com"},"message":"[SPARK-17790][SPARKR] Support for parallelizing R data.frame larger than 2GB\n\n## What changes were proposed in this pull request?\nIf the R data structure that is being parallelized is larger than `INT_MAX` we use files to transfer data to JVM. The serialization protocol mimics Python pickling. This allows us to simply call `PythonRDD.readRDDFromFile` to create the RDD.\n\nI tested this on my MacBook. Following code works with this patch:\n```R\nintMax <- .Machine$integer.max\nlargeVec <- 1:intMax\nrdd <- SparkR:::parallelize(sc, largeVec, 2)\n```\n\n## How was this patch tested?\n* [x] Unit tests\n\nAuthor: Hossein <hossein@databricks.com>\n\nCloses #15375 from falaki/SPARK-17790.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/5cc503f4fe9737a4c7947a80eecac053780606df"},{"sha":"f8062b63fc5e07a6bf4c153a74a966602865fa6e","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-17840][DOCS] Add some pointers for wiki/CONTRIBUTING.md in README.md and some warnings in PULL_REQUEST_TEMPLATE\n\n## What changes were proposed in this pull request?\n\nLink to contributing wiki in PR template, README.md\n\n## How was this patch tested?\n\nDoc-only change, tested by Jekyll\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #15429 from srowen/SPARK-17840.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/f8062b63fc5e07a6bf4c153a74a966602865fa6e"},{"sha":"eb69335cdbce54f943ae6168aed39687f40e53ed","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[BUILD] Closing stale PRs\n\nCloses #15303\nCloses #15078\nCloses #15080\nCloses #15135\nCloses #14565\nCloses #12355\nCloses #15404\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #15451 from srowen/CloseStalePRs.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/eb69335cdbce54f943ae6168aed39687f40e53ed"},{"sha":"47776e7c0c68590fe446cef910900b1aaead06f9","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17850][CORE] Add a flag to ignore corrupt files\n\n## What changes were proposed in this pull request?\n\nAdd a flag to ignore corrupt files. For Spark core, the configuration is `spark.files.ignoreCorruptFiles`. For Spark SQL, it's `spark.sql.files.ignoreCorruptFiles`.\n\n## How was this patch tested?\n\nThe added unit tests\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15422 from zsxwing/SPARK-17850.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/47776e7c0c68590fe446cef910900b1aaead06f9"},{"sha":"9ce7d3e542e786c62f047c13f3001e178f76e06a","author":{"name":"Imran Rashid","email":"63f21f1cb7b694d385ee53376e18c9c9be6c6f98@cloudera.com"},"message":"[SPARK-17675][CORE] Expand Blacklist for TaskSets\n\n## What changes were proposed in this pull request?\n\nThis is a step along the way to SPARK-8425.\n\nTo enable incremental review, the first step proposed here is to expand the blacklisting within tasksets. In particular, this will enable blacklisting for\n* (task, executor) pairs (this already exists via an undocumented config)\n* (task, node)\n* (taskset, executor)\n* (taskset, node)\n\nAdding (task, node) is critical to making spark fault-tolerant of one-bad disk in a cluster, without requiring careful tuning of \"spark.task.maxFailures\". The other additions are also important to avoid many misleading task failures and long scheduling delays when there is one bad node on a large cluster.\n\nNote that some of the code changes here aren't really required for just this -- they put pieces in place for SPARK-8425 even though they are not used yet (eg. the `BlacklistTracker` helper is a little out of place, `TaskSetBlacklist` holds onto a little more info than it needs to for just this change, and `ExecutorFailuresInTaskSet` is more complex than it needs to be).\n\n## How was this patch tested?\n\nAdded unit tests, run tests via jenkins.\n\nAuthor: Imran Rashid <irashid@cloudera.com>\nAuthor: mwws <wei.mao@intel.com>\n\nCloses #15249 from squito/taskset_blacklist_only.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/9ce7d3e542e786c62f047c13f3001e178f76e06a"},{"sha":"f9a56a153e0579283160519065c7f3620d12da3e","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-17782][STREAMING][KAFKA] alternative eliminate race condition of poll twice\n\n## What changes were proposed in this pull request?\n\nAlternative approach to https://github.com/apache/spark/pull/15387\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #15401 from koeninger/SPARK-17782-alt.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/f9a56a153e0579283160519065c7f3620d12da3e"},{"sha":"6f20a92ca30f9c367009c4556939ea4de4284cb9","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-17845] [SQL] More self-evident window function frame boundary API\n\n## What changes were proposed in this pull request?\nThis patch improves the window function frame boundary API to make it more obvious to read and to use. The two high level changes are:\n\n1. Create Window.currentRow, Window.unboundedPreceding, Window.unboundedFollowing to indicate the special values in frame boundaries. These methods map to the special integral values so we are not breaking backward compatibility here. This change makes the frame boundaries more self-evident (instead of Long.MinValue, it becomes Window.unboundedPreceding).\n\n2. In Python, for any value less than or equal to JVM's Long.MinValue, treat it as Window.unboundedPreceding. For any value larger than or equal to JVM's Long.MaxValue, treat it as Window.unboundedFollowing. Before this change, if the user specifies any value that is less than Long.MinValue but not -sys.maxsize (e.g. -sys.maxsize + 1), the number we pass over to the JVM would overflow, resulting in a frame that does not make sense.\n\nCode example required to specify a frame before this patch:\n```\nWindow.rowsBetween(-Long.MinValue, 0)\n```\n\nWhile the above code should still work, the new way is more obvious to read:\n```\nWindow.rowsBetween(Window.unboundedPreceding, Window.currentRow)\n```\n\n## How was this patch tested?\n- Updated DataFrameWindowSuite (for Scala/Java)\n- Updated test_window_functions_cumulative_sum (for Python)\n- Renamed DataFrameWindowSuite DataFrameWindowFunctionsSuite to better reflect its purpose\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15438 from rxin/SPARK-17845.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/6f20a92ca30f9c367009c4556939ea4de4284cb9"},{"sha":"0d4a695279c514c76aa0e9288c70ac7aaef91b03","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-17745][ML][PYSPARK] update NB python api - add weight col parameter\n\n## What changes were proposed in this pull request?\n\nupdate python api for NaiveBayes: add weight col parameter.\n\n## How was this patch tested?\n\ndoctests added.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #15406 from WeichenXu123/nb_python_update.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/0d4a695279c514c76aa0e9288c70ac7aaef91b03"},{"sha":"21cb59f1cd137d96b2596f1abe691b544581cf59","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-17835][ML][MLLIB] Optimize NaiveBayes mllib wrapper to eliminate extra pass on data\n\n## What changes were proposed in this pull request?\n[SPARK-14077](https://issues.apache.org/jira/browse/SPARK-14077) copied the ```NaiveBayes``` implementation from mllib to ml and left mllib as a wrapper. However, there are some difference between mllib and ml to handle labels:\n* mllib allow input labels as {-1, +1}, however, ml assumes the input labels in range [0, numClasses).\n* mllib ```NaiveBayesModel``` expose ```labels``` but ml did not due to the assumption mention above.\n\nDuring the copy in [SPARK-14077](https://issues.apache.org/jira/browse/SPARK-14077), we use\n```val labels = data.map(_.label).distinct().collect().sorted```\nto get the distinct labels firstly, and then encode the labels for training. It involves extra Spark job compared with the original implementation. Since ```NaiveBayes``` only do one pass aggregation during training, adding another one seems less efficient. We can get the labels in a single pass along with ```NaiveBayes``` training and send them to MLlib side.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15402 from yanboliang/spark-17835.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/21cb59f1cd137d96b2596f1abe691b544581cf59"},{"sha":"edeb51a39d76d64196d7635f52be1b42c7ec4341","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-17876] Write StructuredStreaming WAL to a stream instead of materializing all at once\n\n## What changes were proposed in this pull request?\n\nThe CompactibleFileStreamLog materializes the whole metadata log in memory as a String. This can cause issues when there are lots of files that are being committed, especially during a compaction batch.\nYou may come across stacktraces that look like:\n```\njava.lang.OutOfMemoryError: Requested array size exceeds VM limit\nat java.lang.StringCoding.encode(StringCoding.java:350)\nat java.lang.String.getBytes(String.java:941)\nat org.apache.spark.sql.execution.streaming.FileStreamSinkLog.serialize(FileStreamSinkLog.scala:127)\n\n```\nThe safer way is to write to an output stream so that we don't have to materialize a huge string.\n\n## How was this patch tested?\n\nExisting unit tests\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #15437 from brkyvz/ser-to-stream.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/edeb51a39d76d64196d7635f52be1b42c7ec4341"},{"sha":"064d6650e93ed6515a1309079c361e20404843cc","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-17866][SPARK-17867][SQL] Fix Dataset.dropduplicates\n\n## What changes were proposed in this pull request?\n\nTwo issues regarding Dataset.dropduplicates:\n\n1. Dataset.dropDuplicates should consider the columns with same column name\n\n    We find and get the first resolved attribute from output with the given column name in `Dataset.dropDuplicates`. When we have the more than one columns with the same name. Other columns are put into aggregation columns, instead of grouping columns.\n\n2. Dataset.dropDuplicates should not change the output of child plan\n\n    We create new `Alias` with new exprId in `Dataset.dropDuplicates` now. However it causes problem when we want to select the columns as follows:\n\n        val ds = Seq((\"a\", 1), (\"a\", 2), (\"b\", 1), (\"a\", 1)).toDS()\n        // ds(\"_2\") will cause analysis exception\n        ds.dropDuplicates(\"_1\").select(ds(\"_1\").as[String], ds(\"_2\").as[Int])\n\nBecause the two issues are both related to `Dataset.dropduplicates` and the code changes are not big, so submitting them together as one PR.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #15427 from viirya/fix-dropduplicates.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/064d6650e93ed6515a1309079c361e20404843cc"},{"sha":"7222a25a11790fa9d9d1428c84b6f827a785c9e8","author":{"name":"buzhihuojie","email":"3b7b32ff7a855ef90b9e6ed76eed407349c5fd01@gmail.com"},"message":"minor doc fix for Row.scala\n\n## What changes were proposed in this pull request?\n\nminor doc fix for \"getAnyValAs\" in class Row\n\n## How was this patch tested?\n\nNone.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: buzhihuojie <ren.weiluo@gmail.com>\n\nCloses #15452 from david-weiluo-ren/minorDocFixForRow.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/7222a25a11790fa9d9d1428c84b6f827a785c9e8"},{"sha":"6f2fa6c54a11caccd446d5560d2014c645fcf7cc","author":{"name":"Alex Bozarth","email":"850363088eed533ef66882756a87f03f1fc98fde@us.ibm.com"},"message":"[SPARK-11272][WEB UI] Add support for downloading event logs from HistoryServer UI\n\n## What changes were proposed in this pull request?\n\nThis is a reworked PR based on feedback in #9238 after it was closed and not reopened. As suggested in that PR I've only added the download feature. This functionality already exists in the api and this allows easier access to download event logs to share with others.\n\nI've attached a screenshot of the committed version, but I will also include alternate options with screen shots in the comments below. I'm personally not sure which option is best.\n\n## How was this patch tested?\n\nManual testing\n\n![screen shot 2016-10-07 at 6 11 12 pm](https://cloud.githubusercontent.com/assets/13952758/19209213/832fe48e-8cba-11e6-9840-749b1be4d399.png)\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #15400 from ajbozarth/spark11272.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/6f2fa6c54a11caccd446d5560d2014c645fcf7cc"},{"sha":"db8784feaa605adcbd37af4bc8b7146479b631f8","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17899][SQL] add a debug mode to keep raw table properties in HiveExternalCatalog\n\n## What changes were proposed in this pull request?\n\nCurrently `HiveExternalCatalog` will filter out the Spark SQL internal table properties, e.g. `spark.sql.sources.provider`, `spark.sql.sources.schema`, etc. This is reasonable for external users as they don't want to see these internal properties in `DESC TABLE`.\n\nHowever, as a Spark developer, sometimes we do wanna see the raw table properties. This PR adds a new internal SQL conf, `spark.sql.debug`, to enable debug mode and keep these raw table properties.\n\nThis config can also be used in similar places where we wanna retain debug information in the future.\n\n## How was this patch tested?\n\nnew test in MetastoreDataSourcesSuite\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15458 from cloud-fan/debug.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/db8784feaa605adcbd37af4bc8b7146479b631f8"},{"sha":"7bf8a4049866b2ec7fdf0406b1ad0c3a12488645","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-17686][CORE] Support printing out scala and java version with spark-submit --version command\n\n## What changes were proposed in this pull request?\n\nIn our universal gateway service we need to specify different jars to Spark according to scala version. For now only after launching Spark application can we know which version of Scala it depends on. It makes hard for us to support different Scala + Spark versions to pick the right jars.\n\nSo here propose to print out Scala version according to Spark version in \"spark-submit --version\", so that user could leverage this output to make the choice without needing to launching application.\n\n## How was this patch tested?\n\nManually verified in local environment.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #15456 from jerryshao/SPARK-17686.","distinct":false,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/7bf8a4049866b2ec7fdf0406b1ad0c3a12488645"},{"sha":"6f79ff857fb3db0eb9baea390c96fec198b39f62","author":{"name":"erenavsarogullari","email":"2b1f8430a64e745099cfadb513e73988eae74931@gmail.com"},"message":"Ensure uniqueness of TaskSetManager name.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/6f79ff857fb3db0eb9baea390c96fec198b39f62"}]}
{"actor":{"display_login":"erenavsarogullari"}}