{"push_id":3110288572,"size":7,"distinct_size":1,"ref":"refs/heads/doAs","head":"de4a794a320cfe90edaaedfb602de008080e499f","before":"1c612bd0e4254579c7868a08fc9d401f35bf8159","commits":[{"sha":"8504c641eae91473f0c112d7438def489484619d","author":{"name":"Alexey Romanenko","email":"b7adc4af5b779e131ba4fd27519612b4f7aa8080@gmail.com"},"message":"[LIVY-535] Fix non-atomic session creation\n\n## What changes were proposed in this pull request?\n\nAll steps, that include check for too many sessions and creating new\nsession, should be atomic operation to avoid concurrent access of\nseveral threads to different part of this code in the same time.\n\nhttps://issues.apache.org/jira/browse/LIVY-535\n\n## How was this patch tested?\nIf was tested manually on local virtual cluster with this different values of config option `livy.server.session.max-creation`: 1, 2 and 3\n\nThen I run the following command to make sure that only allowed number of sessions were created in the same time and others were rejected:\n`for i in {1..5}; do curl -X POST --data '{\"file\": \"/tmp/spark-examples-2.jar\", \"className\": \"org.apache.spark.examples.SparkPi\"}' -H \"Content-Type: application/json\" http://localhost:8999/batches; done`\n\nAuthor: Alexey Romanenko <aromanenko.dev@gmail.com>\n\nCloses #129 from aromanenko-dev/LIVY-535-session-max-creation.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/8504c641eae91473f0c112d7438def489484619d"},{"sha":"ae2228f568f97865eb502a2cb568d431eeda47c0","author":{"name":"Marco Gaido","email":"035f2fe0944149c518e8b1dfd76528ec1d89d784@hortonworks.com"},"message":"[LIVY-537] Add hiveconf for setting the num of executors in thrift sessions\n\n## What changes were proposed in this pull request?\n\nWe missed to add a parameter to set the number of executors when creating a session through the thrift API. See https://issues.apache.org/jira/browse/LIVY-537.\n\n## How was this patch tested?\n\nManual tests.\n\nAuthor: Marco Gaido <mgaido@hortonworks.com>\n\nCloses #130 from mgaido91/LIVY-537.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/ae2228f568f97865eb502a2cb568d431eeda47c0"},{"sha":"39fa887cfa482c5edaba2ad0c2af683d8e5c63c6","author":{"name":"Marcelo Vanzin","email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com"},"message":"[LIVY-503] Separate thrift server session code in separate module.\n\nThis change creates a new module (\"livy-thriftserver-session\") with\nthe code related to the Thrift server that is expected to run inside\na Spark session. The module contains a few Job implementations that\nare used by the Thrift server to run SQL statements on the session\nside, plus some internal bookkeeping code.\n\nThe new module is a re-implementation of the current Scala code in\nJava; Java is chosen because it becomes simpler to support different\nScala versions, for a couple of reasons:\n\n- no need to compile the module separately for different Scala versions,\n  for when that becomes an issue.\n- better control of serialization, instead of relying on Scala types that\n  may not be compatible across versions (imagine the Livy server and the\n  Spark session using different Scala versions).\n\nThe change also removes the \"jarLocation\" configuration; in a release\npackage, the session jars would be available with the rest of the RSC\njars; for development, some code was added to the RSC to add the\nthrift server jars to the session when a local build is detected.\n\nAlso included is a small change in the RSC pom so that the shaded jar\nis generated in the correct target directory and later copied into the\nstaging directory; this solves an issue with using the RSC in test code\nin other modules (such as the one being added).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #118 from vanzin/LIVY-503.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/39fa887cfa482c5edaba2ad0c2af683d8e5c63c6"},{"sha":"47d3ee6b6555a63d0b871788a53aab022fff518a","author":{"name":"Marco Gaido","email":"035f2fe0944149c518e8b1dfd76528ec1d89d784@hortonworks.com"},"message":"[LIVY-502] Remove dependency on hive-exec\n\n## What changes were proposed in this pull request?\n\nThis PR removes the dependency on `hive-exec`. Only modules of Hive which are used after this PR are `hive-service-rpc` and `hive-service`. This drastically reduces the amount of JARs needed by the thriftserver module.\n\nThe PR takes the Hive classes which we were using and adapts them when necessary (or simpify when we don't need something) in order to work in the Livy thriftserver.\n\nMost of the classes are just migrated replacing all occurrences of `HiveConf`, `HiveSession` and other Hive specific classes. Only one class has a quite different logic than Hive's and it is `AuthFactory`, as we are using a different UGI handling from Hive (as we are not running the TS in a standalone JVM).\n\nThe functionalities we are taking from Hive are:\n - the thrift protocol endpoints (the classes in the `cli` package), ie. the classes handling lower level details about the communication with the client;\n - the authentication layer (classes in the `auth` package, which are used in the ones in `cli`), despite this PR leaves LDAP and PAM as out of scope as they are not trivial to be ported and not needed for a working solution. We can add them later;\n  - the classes in the `operation` package which are used to answer to metadata queries by the JDBC driver.\n\n## How was this patch tested?\n\nexisting UTs + manual tests\n\nAuthor: Marco Gaido <mgaido@hortonworks.com>\n\nCloses #117 from mgaido91/LIVY-502.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/47d3ee6b6555a63d0b871788a53aab022fff518a"},{"sha":"dddb29b5de6aa045580776ef13c764f408fcf5a9","author":{"name":"Bjoern Lohrmann","email":"4c2dd73d86d48718910c266f398bbe70151f47bb@users.noreply.github.com"},"message":"[LIVY-533] Use setJobGroup/cancelJobGroup to cancel bypass jobs\n\n## What changes were proposed in this pull request?\n\nThis PR fixes [LIVY-533](https://issues.apache.org/jira/browse/LIVY-533).\n\nThe proposed fix invokes setJobGroup() in the worker thread before calling job code, and cancelJobGroup() when another thread tries to cancel job execution. synchronized blocks are employed to guard against race conditions.\n\nAlso, this PR removes some unused code from the JobWrapper class.\n\n## How was this patch tested?\n\nManual as well as Livy test suite. No new tests were added, since the code is covered by existing tests.\n\nAuthor: Bjoern Lohrmann <5310706+bjoernlohrmann@users.noreply.github.com>\n\nCloses #128 from bjoernlohrmann/master.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/dddb29b5de6aa045580776ef13c764f408fcf5a9"},{"sha":"1d05d18cf037121722750504050195249616019f","author":{"name":"Adrian Petrescu","email":"60c5d5f6ee4204ff95f9cf3cd211530080c3e877@gmail.com"},"message":"[LIVY-539] Fix incorrect default value in livy-client.conf.template\n\n## What changes were proposed in this pull request?\nJIRA LINK: https://issues.apache.org/jira/browse/LIVY-539\n\nThe template for `livy-client.conf` implies that the default value of `livy.rsc.launcher.port.range` is `10000~10110`, but this is in fact incorrect - the real default value is `10000~10010`, as you can [see here](https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/RSCConf.java#L54). This is really confusing, especially since:\n  * 10110 is wrong on the Spark side, the default configuration there won't open that many ports\n  * 10010 and 10110 look very similar to the human eye so it can take a long time to notice the discrepancy\n\nThis PR modifies the default conf template to reflect the real default value.\n\n## How was this patch tested?\n\nNo testing applicable, it's a configuration template fix.\n\nAuthor: Adrian Petrescu <apetresc@gmail.com>\n\nCloses #132 from apetresc/patch-1.","distinct":false,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/1d05d18cf037121722750504050195249616019f"},{"sha":"de4a794a320cfe90edaaedfb602de008080e499f","author":{"name":"Kevin Risden","email":"66e32a53aee277278658cf5d4b7ceaf23cd367ff@apache.org"},"message":"WIP doAs\n\nSigned-off-by: Kevin Risden <krisden@apache.org>","distinct":true,"url":"https://api.github.com/repos/risdenk/incubator-livy/commits/de4a794a320cfe90edaaedfb602de008080e499f"}]}
{"actor":{"display_login":"risdenk"}}