{"push_id":1470737585,"size":567,"distinct_size":567,"ref":"refs/heads/master","head":"f252cb5d161e064d39cc1ed1d9299307a0636174","before":"ea3605e82545031a00235ee0f449e1e2418674e8","commits":[{"sha":"fb0a8a8dd7e8985676a846684b956e2d988875c6","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-17961][SPARKR][SQL] Add storageLevel to DataFrame for SparkR\n\n## What changes were proposed in this pull request?\n\nAdd storageLevel to DataFrame for SparkR.\nThis is similar to this RP:  https://github.com/apache/spark/pull/13780\n\nbut in R I do not make a class for `StorageLevel`\nbut add a method `storageToString`\n\n## How was this patch tested?\n\ntest added.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #15516 from WeichenXu123/storageLevel_df_r.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/fb0a8a8dd7e8985676a846684b956e2d988875c6"},{"sha":"dcdda19785a272969fb1e3ec18382403aaad6c91","author":{"name":"Xin Ren","email":"82e6f9780faa35cac922f524c8a765f69b3ff7ff@126.com"},"message":"[SPARK-14300][DOCS][MLLIB] Scala MLlib examples code merge and clean up\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-14300\n\nDuplicated code found in scala/examples/mllib, below all deleted in this PR:\n\n- DenseGaussianMixture.scala\n- StreamingLinearRegression.scala\n\n## delete reasons:\n\n#### delete: mllib/DenseGaussianMixture.scala\n\n- duplicate of mllib/GaussianMixtureExample\n\n#### delete: mllib/StreamingLinearRegression.scala\n\n- duplicate of mllib/StreamingLinearRegressionExample\n\nWhen merging and cleaning those code, be sure not disturb the previous example on and off blocks.\n\n## How was this patch tested?\n\nTest with `SKIP_API=1 jekyll` manually to make sure that works well.\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #12195 from keypointt/SPARK-14300.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/dcdda19785a272969fb1e3ec18382403aaad6c91"},{"sha":"5b7d403c1819c32a6a5b87d470f8de1a8ad7a987","author":{"name":"jiangxingbo","email":"5b9d05a325a3f85d4c824395342a210596784248@gmail.com"},"message":"[SPARK-18094][SQL][TESTS] Move group analytics test cases from `SQLQuerySuite` into a query file test.\n\n## What changes were proposed in this pull request?\n\nCurrently we have several test cases for group analytics(ROLLUP/CUBE/GROUPING SETS) in `SQLQuerySuite`, should better move them into a query file test.\nThe following test cases are moved to `group-analytics.sql`:\n```\ntest(\"rollup\")\ntest(\"grouping sets when aggregate functions containing groupBy columns\")\ntest(\"cube\")\ntest(\"grouping sets\")\ntest(\"grouping and grouping_id\")\ntest(\"grouping and grouping_id in having\")\ntest(\"grouping and grouping_id in sort\")\n```\n\nThis is followup work of #15582\n\n## How was this patch tested?\n\nModified query file `group-analytics.sql`, which will be tested by `SQLQueryTestSuite`.\n\nAuthor: jiangxingbo <jiangxb1987@gmail.com>\n\nCloses #15624 from jiangxb1987/group-analytics-test.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/5b7d403c1819c32a6a5b87d470f8de1a8ad7a987"},{"sha":"29cea8f332aa3750f8ff7c3b9e705d107278da4b","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-17157][SPARKR] Add multiclass logistic regression SparkR Wrapper\n\n## What changes were proposed in this pull request?\n\nAs we discussed in #14818, I added a separate R wrapper spark.logit for logistic regression.\n\nThis single interface supports both binary and multinomial logistic regression. It also has \"predict\" and \"summary\" for binary logistic regression.\n\n## How was this patch tested?\n\nNew unit tests are added.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #15365 from wangmiao1981/glm.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/29cea8f332aa3750f8ff7c3b9e705d107278da4b"},{"sha":"a76846cfb1c2d6c8f4d647426030b59de20d9433","author":{"name":"Miao Wang","email":"9c0c07792e7dd434f45925bb4d177a11ca1a89a7@Miaos-MacBook-Pro.local"},"message":"[SPARK-18126][SPARK-CORE] getIteratorZipWithIndex accepts negative value as index\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\n\n`Utils.getIteratorZipWithIndex` was added to deal with number of records > 2147483647 in one partition.\n\nmethod `getIteratorZipWithIndex` accepts `startIndex` < 0, which leads to negative index.\n\nThis PR just adds a defensive check on `startIndex` to make sure it is >= 0.\n\n## How was this patch tested?\n\nAdd a new unit test.\n\nAuthor: Miao Wang <miaowang@Miaos-MacBook-Pro.local>\n\nCloses #15639 from wangmiao1981/zip.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/a76846cfb1c2d6c8f4d647426030b59de20d9433"},{"sha":"5b27598ff50cb08e7570fade458da0a3d4d4eabc","author":{"name":"frreiss","email":"b2f409dba498dd996d8771bf1c667af870613edc@us.ibm.com"},"message":"[SPARK-16963][STREAMING][SQL] Changes to Source trait and related implementation classes\n\n## What changes were proposed in this pull request?\n\nThis PR contains changes to the Source trait such that the scheduler can notify data sources when it is safe to discard buffered data. Summary of changes:\n* Added a method `commit(end: Offset)` that tells the Source that is OK to discard all offsets up `end`, inclusive.\n* Changed the semantics of a `None` value for the `getBatch` method to mean \"from the very beginning of the stream\"; as opposed to \"all data present in the Source's buffer\".\n* Added notes that the upper layers of the system will never call `getBatch` with a start value less than the last value passed to `commit`.\n* Added a `lastCommittedOffset` method to allow the scheduler to query the status of each Source on restart. This addition is not strictly necessary, but it seemed like a good idea -- Sources will be maintaining their own persistent state, and there may be bugs in the checkpointing code.\n* The scheduler in `StreamExecution.scala` now calls `commit` on its stream sources after marking each batch as complete in its checkpoint.\n* `MemoryStream` now cleans committed batches out of its internal buffer.\n* `TextSocketSource` now cleans committed batches from its internal buffer.\n\n## How was this patch tested?\nExisting regression tests already exercise the new code.\n\nAuthor: frreiss <frreiss@us.ibm.com>\n\nCloses #14553 from frreiss/fred-16963.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/5b27598ff50cb08e7570fade458da0a3d4d4eabc"},{"sha":"f1aeed8b022e043de2eb38b30187dcc36ee8dcdb","author":{"name":"ALeksander Eskilson","email":"6f7b3a70b185c910c42902b968727445c382abaf@cerner.com"},"message":"[SPARK-17770][CATALYST] making ObjectType public\n\n## What changes were proposed in this pull request?\n\nIn order to facilitate the writing of additional Encoders, I proposed opening up the ObjectType SQL DataType. This DataType is used extensively in the JavaBean Encoder, but would also be useful in writing other custom encoders.\n\nAs mentioned by marmbrus, it is understood that the Expressions API is subject to potential change.\n\n## How was this patch tested?\n\nThe change only affects the visibility of the ObjectType class, and the existing SQL test suite still runs without error.\n\nAuthor: ALeksander Eskilson <alek.eskilson@cerner.com>\n\nCloses #15453 from bdrillard/master.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/f1aeed8b022e043de2eb38b30187dcc36ee8dcdb"},{"sha":"dd4f088c1df6abd728e5544a17ba85322bedfe4c","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[SPARK-18009][SQL] Fix ClassCastException while calling toLocalIterator() on dataframe produced by RunnableCommand\n\n## What changes were proposed in this pull request?\nA short code snippet that uses toLocalIterator() on a dataframe produced by a RunnableCommand\nreproduces the problem. toLocalIterator() is called by thriftserver when\n`spark.sql.thriftServer.incrementalCollect`is set to handle queries producing large result\nset.\n\n**Before**\n```SQL\nscala> spark.sql(\"show databases\")\nres0: org.apache.spark.sql.DataFrame = [databaseName: string]\n\nscala> res0.toLocalIterator()\n16/10/26 03:00:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)\njava.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow\n```\n\n**After**\n```SQL\nscala> spark.sql(\"drop database databases\")\nres30: org.apache.spark.sql.DataFrame = []\n\nscala> spark.sql(\"show databases\")\nres31: org.apache.spark.sql.DataFrame = [databaseName: string]\n\nscala> res31.toLocalIterator().asScala foreach println\n[default]\n[parquet]\n```\n## How was this patch tested?\nAdded a test in DDLSuite\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #15642 from dilipbiswal/SPARK-18009.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/dd4f088c1df6abd728e5544a17ba85322bedfe4c"},{"sha":"d3b4831d009905185ad74096ce3ecfa934bc191d","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-18132] Fix checkstyle\n\nThis PR fixes checkstyle.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #15656 from yhuai/fix-format.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/d3b4831d009905185ad74096ce3ecfa934bc191d"},{"sha":"1dbe9896b7f30538a5fad2f5d718d035c7906936","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-17157][SPARKR][FOLLOW-UP] doc fixes\n\n## What changes were proposed in this pull request?\n\na couple of small late finding fixes for doc\n\n## How was this patch tested?\n\nmanually\nwangmiao1981\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #15650 from felixcheung/logitfix.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/1dbe9896b7f30538a5fad2f5d718d035c7906936"},{"sha":"44c8bfda793b7655e2bd1da5e9915a09ed9d42ce","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SQL][DOC] updating doc for JSON source to link to jsonlines.org\n\n## What changes were proposed in this pull request?\n\nAPI and programming guide doc changes for Scala, Python and R.\n\n## How was this patch tested?\n\nmanual test\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #15629 from felixcheung/jsondoc.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/44c8bfda793b7655e2bd1da5e9915a09ed9d42ce"},{"sha":"701a9d361b3045a25c42b3c0e44e7755d45ff78c","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-CORE][TEST][MINOR] Fix the wrong comment in test\n\n## What changes were proposed in this pull request?\n\nWhile learning core scheduler code, I found two lines of wrong comments. This PR simply corrects the comments.\n\n## How was this patch tested?\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #15631 from wangmiao1981/Rbug.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/701a9d361b3045a25c42b3c0e44e7755d45ff78c"},{"sha":"104232580528c097a284d753adb5795f6de8b0a5","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-17813][SQL][KAFKA] Maximum data per trigger\n\n## What changes were proposed in this pull request?\n\nmaxOffsetsPerTrigger option for rate limiting, proportionally based on volume of different topicpartitions.\n\n## How was this patch tested?\n\nAdded unit test\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #15527 from koeninger/SPARK-17813.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/104232580528c097a284d753adb5795f6de8b0a5"},{"sha":"0b076d4cb6afde2946124e6411ed6a6ce7b8b1a7","author":{"name":"VinceShieh","email":"2445c9be3c24c11954fcdbf8bdec5f86733bb70d@intel.com"},"message":"[SPARK-17219][ML] enhanced NaN value handling in Bucketizer\n\n## What changes were proposed in this pull request?\n\nThis PR is an enhancement of PR with commit ID:57dc326bd00cf0a49da971e9c573c48ae28acaa2.\nNaN is a special type of value which is commonly seen as invalid. But We find that there are certain cases where NaN are also valuable, thus need special handling. We provided user when dealing NaN values with 3 options, to either reserve an extra bucket for NaN values, or remove the NaN values, or report an error, by setting handleNaN \"keep\", \"skip\", or \"error\"(default) respectively.\n\n'''Before:\nval bucketizer: Bucketizer = new Bucketizer()\n          .setInputCol(\"feature\")\n          .setOutputCol(\"result\")\n          .setSplits(splits)\n'''After:\nval bucketizer: Bucketizer = new Bucketizer()\n          .setInputCol(\"feature\")\n          .setOutputCol(\"result\")\n          .setSplits(splits)\n          .setHandleNaN(\"keep\")\n\n## How was this patch tested?\nTests added in QuantileDiscretizerSuite, BucketizerSuite and DataFrameStatSuite\n\nSigned-off-by: VinceShieh <vincent.xieintel.com>\n\nAuthor: VinceShieh <vincent.xie@intel.com>\nAuthor: Vincent Xie <vincent.xie@intel.com>\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #15428 from VinceShieh/spark-17219_followup.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/0b076d4cb6afde2946124e6411ed6a6ce7b8b1a7"},{"sha":"79fd0cc0584e48fb021c4237877b15abbffb319a","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-16963][SQL] Fix test \"StreamExecution metadata garbage collection\"\n\n## What changes were proposed in this pull request?\n\nA follow up PR for #14553 to fix the flaky test. It's flaky because the file list API doesn't guarantee any order of the return list.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15661 from zsxwing/fix-StreamingQuerySuite.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/79fd0cc0584e48fb021c4237877b15abbffb319a"},{"sha":"ccb11543048dccd4cc590a8db1df1d9d5847d112","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-17970][SQL] store partition spec in metastore for data source table\n\n## What changes were proposed in this pull request?\n\nWe should follow hive table and also store partition spec in metastore for data source table.\nThis brings 2 benefits:\n\n1. It's more flexible to manage the table data files, as users can use `ADD PARTITION`, `DROP PARTITION` and `RENAME PARTITION`\n2. We don't need to cache all file status for data source table anymore.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Eric Liang <ekl@databricks.com>\nAuthor: Michael Allman <michael@videoamp.com>\nAuthor: Eric Liang <ekhliang@gmail.com>\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15515 from cloud-fan/partition.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/ccb11543048dccd4cc590a8db1df1d9d5847d112"},{"sha":"ab5f938bc7c3c9b137d63e479fced2b7e9c9d75b","author":{"name":"Sunitha Kambhampati","email":"56dd3f83cb9a85c95bdcdec7d012dceb1fccf7d1@us.ibm.com"},"message":"[SPARK-18121][SQL] Unable to query global temp views when hive support is enabled\n\n## What changes were proposed in this pull request?\n\nIssue:\nQuerying on a global temp view throws Table or view not found exception.\n\nFix:\nUpdate the lookupRelation in HiveSessionCatalog to check for global temp views similar to the SessionCatalog.lookupRelation.\n\nBefore fix:\nQuerying on a global temp view ( for. e.g.:  select * from global_temp.v1)  throws Table or view not found exception\n\nAfter fix:\nQuery succeeds and returns the right result.\n\n## How was this patch tested?\n- Two unit tests are added to check for global temp view for the code path when hive support is enabled.\n- Regression unit tests were run successfully. ( build/sbt -Phive hive/test, build/sbt sql/test, build/sbt catalyst/test)\n\nAuthor: Sunitha Kambhampati <skambha@us.ibm.com>\n\nCloses #15649 from skambha/lookuprelationChanges.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/ab5f938bc7c3c9b137d63e479fced2b7e9c9d75b"},{"sha":"569788a55e4c6b218fb697e1e54c6138ffe657a6","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-18109][ML] Add instrumentation to GMM\n\n## What changes were proposed in this pull request?\n\nAdd instrumentation to GMM\n\n## How was this patch tested?\n\nTest in spark-shell\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #15636 from zhengruifeng/gmm_instr.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/569788a55e4c6b218fb697e1e54c6138ffe657a6"},{"sha":"e9746f87d0b553b8115948acb79f7e32c23dfd86","author":{"name":"Jagadeesan","email":"77eaebe535b25af84722e54783804ea15c5d230a@us.ibm.com"},"message":"[SPARK-18133][EXAMPLES][ML] Python ML Pipeline Example has syntax e…\n\n## What changes were proposed in this pull request?\n\nIn Python 3, there is only one integer type (i.e., int), which mostly behaves like the long type in Python 2. Since Python 3 won't accept \"L\", so removed \"L\" in all examples.\n\n## How was this patch tested?\n\nUnit tests.\n\n…rrors]\n\nAuthor: Jagadeesan <as2@us.ibm.com>\n\nCloses #15660 from jagadeesanas2/SPARK-18133.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/e9746f87d0b553b8115948acb79f7e32c23dfd86"},{"sha":"ac26e9cf27862fbfb97ae18d591606ecf2cd41cf","author":{"name":"Yunni","email":"b3e238c6ce2e13cf73c75da0184cc5711b19c090@gmail.com"},"message":"[SPARK-5992][ML] Locality Sensitive Hashing\n\n## What changes were proposed in this pull request?\n\nImplement Locality Sensitive Hashing along with approximate nearest neighbors and approximate similarity join based on the [design doc](https://docs.google.com/document/d/1D15DTDMF_UWTTyWqXfG7y76iZalky4QmifUYQ6lH5GM/edit).\n\nDetailed changes are as follows:\n(1) Implement abstract LSH, LSHModel classes as Estimator-Model\n(2) Implement approxNearestNeighbors and approxSimilarityJoin in the abstract LSHModel\n(3) Implement Random Projection as LSH subclass for Euclidean distance, Min Hash for Jaccard Distance\n(4) Implement unit test utility methods including checkLshProperty, checkNearestNeighbor and checkSimilarityJoin\n\nThings that will be implemented in a follow-up PR:\n - Bit Sampling for Hamming Distance, SignRandomProjection for Cosine Distance\n - PySpark Integration for the scala classes and methods.\n\n## How was this patch tested?\nUnit test is implemented for all the implemented classes and algorithms. A scalability test on Uber's dataset was performed internally.\n\nTested the methods on [WEX dataset](https://aws.amazon.com/items/2345) from AWS, with the steps and results [here](https://docs.google.com/document/d/19BXg-67U83NVB3M0I84HVBVg3baAVaESD_mrg_-vLro/edit).\n\n## References\nGionis, Aristides, Piotr Indyk, and Rajeev Motwani. \"Similarity search in high dimensions via hashing.\" VLDB 7 Sep. 1999: 518-529.\nWang, Jingdong et al. \"Hashing for similarity search: A survey.\" arXiv preprint arXiv:1408.2927 (2014).\n\nAuthor: Yunni <Euler57721@gmail.com>\nAuthor: Yun Ni <yunn@uber.com>\n\nCloses #15148 from Yunni/SPARK-5992-yunn-lsh.","distinct":true,"url":"https://api.github.com/repos/erenavsarogullari/spark/commits/ac26e9cf27862fbfb97ae18d591606ecf2cd41cf"}]}
{"actor":{"display_login":"erenavsarogullari"}}