{"push_id":3058921645,"size":33,"distinct_size":30,"ref":"refs/heads/master","head":"4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f","before":"ec8dffdaa2a2c2d2f348bd6fd5d583e812480907","commits":[{"sha":"6d2ffdd54273e7d4a75bde38527f75acc14d6ef3","author":{"name":"Alex Bozarth","email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com"},"message":"[MINOR] Addressed Docs dependency security vulnerability followup\n\nFollowup to 26428c56f20ba5ea608038ed8c2e11d8f04665d4\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #82 from ajbozarth/gem.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/6d2ffdd54273e7d4a75bde38527f75acc14d6ef3"},{"sha":"b9c2f10194f32532009690fe89a11959c4c083c0","author":{"name":"Meisam","email":"7da70d489d7ac82af3838e5429f8b5156b9b87b2@Hachinohe.local"},"message":"[LIVY-448] Assembly format is duplicated in pom.xml and assembly.xml\n\n## What changes were proposed in this pull request?\nThe assembly format for Livy is specified twice:\n1 in `assembly/pom.xml`:\n```xml\n              <formats>\n                <format>${assembly.format}</format>\n              </formats>\n```\n2 in `assembly/assembly.xml`\n```\n              <formats>\n                <format>zip</format>\n              </formats>\n```\nBut only one is enough. Also in `assembly/assembly.xml`, the format is hard coded as zip, but is should be based\n\n## How was this patch tested?\nAll the unit tests and ITs pass as usual. Moreover, I tested that  `targz` profile builds a tar.gz archive and the default profile builds a zip archive.\n\nFor example,\nthis builds the zip archive:\n```\nmvn -am -pl assembly package\n```\nAnd this builds the tar.gz artifact:\n```\nmvn -Ptargz -am -pl assembly package\n```\n\nTask-url: https://issues.apache.org/jira/browse/LIVY-448\n\nAuthor: Meisam <meisam@Hachinohe.local>\n\nCloses #83 from meisam/LIVY-448.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/b9c2f10194f32532009690fe89a11959c4c083c0"},{"sha":"10373b6e2edc1fa7af6ec363491eb25ff48d730f","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-446][BUILD] Livy to Support Spark 2.3\n\n## What changes were proposed in this pull request?\n\nThis PR aims to support Spark 2.3 in Livy, several changes due to internal changes of Spark 2.3\n\n## How was this patch tested?\n\nExisting UTs.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #81 from jerryshao/LIVY-446.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/10373b6e2edc1fa7af6ec363491eb25ff48d730f"},{"sha":"06cfa7bce2cde205895be31fa83adc006e7d4ec2","author":{"name":"Wieland Hoffmann","email":"5f18aad644584ef5e80c97b38b7d7f4dc0b30185@de.ibm.com"},"message":"[MINOR] Fix travis builds\n\nEach individual commit has a more detailed description of what's being changed and why.\n\nAt the moment, Travis builds don't work:\n\n* https://travis-ci.org/mineo/incubator-livy/builds/359324523 - the `sudo pip3 install --upgrade pip \"setuptools < 36\"` command fails with `sudo: pip3: command not found`.\n* fixing that, the `failing the org.apache.rat:apache-rat-plugin:0.12:check` maven goal fails because it sees `.pytest_cache` folders that it doesn't know about (https://github.com/pytest-dev/pytest/issues/3286, failure in https://travis-ci.org/mineo/incubator-livy/jobs/359326261, check the raw log).\n\nThis pull request works around the pip3 failures by just using pip as a callable module (possible since Python 3.4) and adding `.pytest_cache` to raw-excludes, as well as showing the contents of `rat.txt` files in the `after_failure` step. I concede that the pip change is more of a workaround, but I don't know how else to fix it and unblock testing.\n\nRunning the tests on travis.\n\nPlease review https://livy.incubator.apache.org/community/ before opening a pull request.\n\nAuthor: Wieland Hoffmann <WHoffman@de.ibm.com>\n\nCloses #84 from mineo/rat.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/06cfa7bce2cde205895be31fa83adc006e7d4ec2"},{"sha":"7e4bb3bd6888742e43aed7396ed490d343a063a2","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-455][REPL] Fix json4s doesn't support java.math.BigDecimal issue\n\n## What changes were proposed in this pull request?\n\nLivy's SQLInterpreter will throw exception when rows contain java.math.BigDecimal data. This is because current version of json4s doesn't treat java.math.BigDecimal type as primitive type. On the contrary, json4s supports Scala BigDecimal as primitive type. So the fix is to convert java BigDecimal to Scala BigDecimal.\n\n## How was this patch tested?\n\nUnit test is added.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #85 from jerryshao/LIVY-455.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/7e4bb3bd6888742e43aed7396ed490d343a063a2"},{"sha":"cd8b11261811abb4fe3afe0f19540cd4bdaeb5f7","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-457][REPL] Fix SQLContext is not initialized correctly issue\n\n## What changes were proposed in this pull request?\n\nThe signature of SQLContext's constructor is changed in Spark2, but we're still using the Spark1's signature, which will throw an exception when using this object.\n\n## How was this patch tested?\n\nUT and local verification.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #86 from jerryshao/LIVY-457.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/cd8b11261811abb4fe3afe0f19540cd4bdaeb5f7"},{"sha":"71c3feed9cbafef40805975ea7ce6fa2498bab72","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-458][BUILD] Upgrade jackson version to 2.9.5\n\n## What changes were proposed in this pull request?\n\nDue to several security issues of jackson databind module (CVE-2018-5968, CVE-2017-17485, CVE-2018-7489), here propose to upgrade jackson version 2.9.5.\n\n## How was this patch tested?\n\nExisting UTs.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #87 from jerryshao/LIVY-458.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/71c3feed9cbafef40805975ea7ce6fa2498bab72"},{"sha":"9d381bdf04704b596006f51de56dc3827054f96c","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-460][SERVER] Remove misleading log prefix\n\n## What changes were proposed in this pull request?\n\nRemoving misleading log prefix \"stdout: \" for the captured stdout/stderr of spark-submit process\n\n## How was this patch tested?\n\nN/A\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #89 from jerryshao/LIVY-460.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/9d381bdf04704b596006f51de56dc3827054f96c"},{"sha":"e3f45a057cc45bca5bceb04af8ea9218b35fa621","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown\n\n## What changes were proposed in this pull request?\n\nDuring RSCDriver's shutdown, it will first shutdown RPC server, and then all the RPC clients. When RPC client is closed, it will register a timeout to avoid orphaned RSCDriver, but this is not necessary during RSCDriver's shutdown, so here fixing this issue. The details can be seen in [JIRA](https://issues.apache.org/jira/browse/LIVY-466).\n\n## How was this patch tested?\n\nLocal verification.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #90 from jerryshao/LIVY-466.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/e3f45a057cc45bca5bceb04af8ea9218b35fa621"},{"sha":"551cc53095f0a4b5382602ba0c296f8cf8932e44","author":{"name":"Alexey Romanenko","email":"b7adc4af5b779e131ba4fd27519612b4f7aa8080@gmail.com"},"message":"[LIVY-452] Differentiate FAILED and KILLED states\n\nCurrently, it's not possible to distinguish between two states - SparkApp.State.KILLED and SparkApp.State.FAILED. In both cases the session state will be SessionState.Dead(). This patch adds new SessionState.Killed() which will be used when job was actually killed by user.\n\nhttps://issues.apache.org/jira/browse/LIVY-452\n\n- Corresponding unit test was adjusted\n- Tested manually on YARN cluster with `livy.spark.deploy-mode=cluster`\n\nAuthor: Alexey Romanenko <aromanenko.dev@gmail.com>\n\nCloses #92 from aromanenko-dev/LIVY-452-SessionState-Killed.\n\nChange-Id: Ia18808d7954e20653e152d8bf6748f43c2707b18","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/551cc53095f0a4b5382602ba0c296f8cf8932e44"},{"sha":"6f1756cffbde5c58075dc2d2dc8a7a8ea51516fe","author":{"name":"M Wcislo","email":"3480a8bcc6be67e34293b4de47a82afe07eef865@gmail.com"},"message":"[LIVY-468] Reverse proxy support\n\n## What changes were proposed in this pull request?\nImplements reverse proxy support, by adding basePath/URL concept, configurable using livy.conf\n\nResolves: [LIVY-468](https://issues.apache.org/jira/browse/LIVY-468)\n\n## How was this patch tested?\nTested manually with Kong 0.11 and Spark 2.1.0 in cluster mode (using spark dispatcher).\n\nAuthor: M Wcislo <mwcislo999@gmail.com>\n\nCloses #93 from m-wcislo/LIVY-468_reverse_proxy_support.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/6f1756cffbde5c58075dc2d2dc8a7a8ea51516fe"},{"sha":"ca4cad22968e1a2f88fa0ec262c1088812e3d251","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-472][SERVER] Improve the logs for fail-to-create session\n\n## What changes were proposed in this pull request?\n\nLivy currently doesn't give a very clear log about the fail-to-create session, it only says that session related app tag cannot be found in RM, but doesn't tell user how to search and get the true root cause. So here change the logs to make it more clear.\n\n## How was this patch tested?\n\nLocal verification.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #96 from jerryshao/LIVY-472.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/ca4cad22968e1a2f88fa0ec262c1088812e3d251"},{"sha":"8b3fcbc8d52a6e6e1628cf139c0b0f49be797cec","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-473][TEST] Minor refactor of integration test to remove legacy code\n\nIntegration test has some legacy codes related to different cluster type (mini or real). Since now we already use real Spark package to do test and partially removed that code, we should refactor to remove all the unused legacy code.\n\nExisting tests.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #97 from jerryshao/LIVY-473.\n\nChange-Id: I7ca6c69884f7ed11a34cde5a6e1fc9bfcd141cbb","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/8b3fcbc8d52a6e6e1628cf139c0b0f49be797cec"},{"sha":"02c5915f38a0cd19acfc8222233cba37d73393f9","author":{"name":"Ivan Dzikovsky","email":"d176693ddb64ef9126bc6332772b926cf030aafe@protonmail.com"},"message":"[LIVY-475] Support of Hadoop CredentialProvider API\n\nIn this PR I've added following option to livy.conf:\n```\n```\nto allow to specify path to Hadoop Credential Provider, which than used in `WebServer.scala` to set Keystore password and Key password to JKS that used to enable SSL encryption.\n\nAlso before trying to use Hadoop CredentialProvider API I'm checking if it available (as it was not available in Hadoop < 2.6) using the same [method that used in Oozie](https://github.com/apache/oozie/commit/6a731f9926158da38d1e3b518671ada95a544fe8#diff-800f95e605f21c5aaf5edef13039c9b9R124).\n\nTo use this, you will need to generate Credential Provider containing \"livy.keystore.password\" and/or \"livy.key-password\" in the common way:\n```bash\nhadoop credential create \"livy.keystore.password\" -value \"keystore_secret\" -provider jceks://hdfsnn1.example.com/my/path/livy_creds.jceks\nhadoop credential create \"livy.key-password\" -value \"key_secret\" -provider jceks://hdfsnn1.example.com/my/path/livy_creds.jceks\n```\n\nAuthor: Ivan Dzikovsky <idzikovsky@protonmail.com>\n\nCloses #99 from idzikovsky/LIVY-475.\n\nChange-Id: Iae60550900067e61a37a6f74acb5e299005f3397","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/02c5915f38a0cd19acfc8222233cba37d73393f9"},{"sha":"fe0283f864b850973276f3c857908ccb2e6370f2","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-477][BUILD] Upgrade Scala version to 2.11.12\n\n## What changes were proposed in this pull request?\n\nScala version below 2.11.12 has CVE (https://scala-lang.org/news/security-update-nov17.html), and Spark will also upgrade its supported version to 2.11.12.\n\nSo here upgrading Livy's Scala version also.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #100 from jerryshao/LIVY-477.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/fe0283f864b850973276f3c857908ccb2e6370f2"},{"sha":"2196302731590def9a8f8a25628dd302eac06260","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[Security] Update to support pyspark and sparkr changes in Spark 2.3.1","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/2196302731590def9a8f8a25628dd302eac06260"},{"sha":"551dd5c27f05c35c75015c7dae9153e2a7f23089","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[HOTFIX] Fix Base64 not existed in jdk7 issue","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/551dd5c27f05c35c75015c7dae9153e2a7f23089"},{"sha":"8027ca708fdc3df9a5b08d2d33d0436018154bcc","author":{"name":"jerryshao","email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com"},"message":"[LIVY-498][REPL] Fix Windows CRLF line ending issue in SparkR interpreter\n\n## What changes were proposed in this pull request?\n\nIf the issued query contains CRLF EOL, it will be failed to execute on *nix machine. This happens when submitting queries from Windows machine and executing on Linux machine.\n\nSo here propose to convert statement to match system's EOL.\n\n## How was this patch tested?\n\nNew UT added.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #105 from jerryshao/LIVY-498.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/8027ca708fdc3df9a5b08d2d33d0436018154bcc"},{"sha":"7d8fa69f1804025d95971bcfb389951075dc6b98","author":{"name":"Marco Gaido","email":"035f2fe0944149c518e8b1dfd76528ec1d89d784@hortonworks.com"},"message":"[LIVY-491][LIVY-492][LIVY-493] Add Thriftserver module and implementation\n\n## What changes were proposed in this pull request?\n\nThe PR contains an implementation of a JDBC API for Livy server based on the Hive Thriftserver. The implementation is based on the version 3.0 of Hive Thriftserver.\n\nThis initial PR contains the thriftserver module added to Livy and its implementation. It doesn't contain any binding for starting it, this will be added later.\n\nSome Hive classes have been ported here because they needed come modifications in order to works properly in Livy. Long term solution is to get of them all by re-implementing the needed parts in the Livy thriftserver itself, without relying on Hive code (other than the PRC interface) anymore. Those classes/changes can be summarized in three categories:\n\n 1. Changes to make the Hive classes easy to extend: for instance, some visibility modifiers were changes (moving from `private` or `package private` to `protected` or `public`);\n 2. Changes in order to reduce the dependencies on Hive modules/classes: for instance all the classes in the `operation` package were modified in order to get rid of the usage of `HiveSession` and the `HiveServer2` was changed in order not to use `CLIService`.\n 3. The UGI management which is currently performed in Livy is definitely very different from the Hive one, this required changes to the `HiveAuthFactory` in order not to interfere with the existing codebase.\n\n## How was this patch tested?\n\nadded integration tests as UTs\n\nAuthor: Marco Gaido <mgaido@hortonworks.com>\n\nCloses #104 from mgaido91/LIVY-491.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/7d8fa69f1804025d95971bcfb389951075dc6b98"},{"sha":"1ef6d2a7b6b10325854d710ec6f27c9aaf33f50d","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[MINOR] Remove unneded repos from root pom.\n\nAlso a minor indentation fix.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #109 from vanzin/pom-cleanup.","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/1ef6d2a7b6b10325854d710ec6f27c9aaf33f50d"}]}
{"actor":{"display_login":"risdenk"}}