{"action":"created","comment":{"url":"https://api.github.com/repos/broadinstitute/cromwell/pulls/comments/76494754","id":76494754,"diff_hunk":"@@ -0,0 +1,93 @@\n+package cromwell.backend.impl.spark\n+\n+import java.nio.file.Path\n+\n+import com.typesafe.scalalogging.StrictLogging\n+import cromwell.core.{TailedWriter, UntailedWriter}\n+import cromwell.core.PathFactory.EnhancedPath\n+\n+import scala.sys.process._\n+import better.files._\n+import scala.language.postfixOps\n+import scala.util.{Failure, Success, Try}\n+\n+object SparkCommands {\n+  val SPARK_SUBMIT = \"spark-submit\"\n+  val SEPARATOR = \"--\"\n+  val EXECUTOR_CORES = \"total-executor-cores\"\n+  val EXECUTOR_MEMORY = \"executor-memory\"\n+  val APP_MAIN_CLASS = \"class\"\n+  val MASTER = \"master\"\n+  val DEPLOY_MODE = \"deploy-mode\"\n+  val SPARK_APP_WITH_ARGS = \"spark_app_with_args\"\n+  val MEMORY_UNIT = \"G\" //Accepts only GB\n+  val SHELL_CMD = \"sh\"\n+  val PIPELINE = \"&>\"\n+}\n+\n+class SparkCommands extends StrictLogging {\n+  import SparkCommands._\n+  import SparkClusterConstants._\n+  /**\n+    * Writes the script file containing the user's command from the WDL as well\n+    * as some extra shell code for monitoring jobs\n+    */\n+  def writeScript(instantiatedCommand: String, filePath: Path, containerRoot: Path) = {\n+    filePath.write(\n+      s\"\"\"#!/bin/sh\n+          |cd $containerRoot\n+          |$instantiatedCommand\n+          |echo $$? > rc\n+          |\"\"\".stripMargin)\n+  }\n+\n+  def sparkSubmitCommand(attributes: Map[String, Any]): String = {\n+    val stringBuilder = StringBuilder.newBuilder\n+    val commandPlaceHolder = \" %s%s %s \"\n+    val sparkHome = Try(sys.env(\"SPARK_HOME\")) match {\n+      case Success(s) => Some(s)\n+      case Failure(ex) => logger.warn(s\"Spark home does not exist picking up default command\")","path":"supportedBackends/spark/src/main/scala/cromwell/backend/impl/spark/SparkProcess.scala","position":49,"original_position":49,"commit_id":"590fcb6a9ea708cc5e00faefc4e5beb7e8b8339c","original_commit_id":"590fcb6a9ea708cc5e00faefc4e5beb7e8b8339c","user":{"login":"geoffjentry","id":961771,"avatar_url":"https://avatars.githubusercontent.com/u/961771?v=3","gravatar_id":"","url":"https://api.github.com/users/geoffjentry","html_url":"https://github.com/geoffjentry","followers_url":"https://api.github.com/users/geoffjentry/followers","following_url":"https://api.github.com/users/geoffjentry/following{/other_user}","gists_url":"https://api.github.com/users/geoffjentry/gists{/gist_id}","starred_url":"https://api.github.com/users/geoffjentry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geoffjentry/subscriptions","organizations_url":"https://api.github.com/users/geoffjentry/orgs","repos_url":"https://api.github.com/users/geoffjentry/repos","events_url":"https://api.github.com/users/geoffjentry/events{/privacy}","received_events_url":"https://api.github.com/users/geoffjentry/received_events","type":"User","site_admin":false},"body":"put logger on the next line","created_at":"2016-08-26T22:01:43Z","updated_at":"2016-08-26T22:01:43Z","html_url":"https://github.com/broadinstitute/cromwell/pull/1339#discussion_r76494754","pull_request_url":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339","_links":{"self":{"href":"https://api.github.com/repos/broadinstitute/cromwell/pulls/comments/76494754"},"html":{"href":"https://github.com/broadinstitute/cromwell/pull/1339#discussion_r76494754"},"pull_request":{"href":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339"}}},"pull_request":{"url":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339","id":82777944,"html_url":"https://github.com/broadinstitute/cromwell/pull/1339","diff_url":"https://github.com/broadinstitute/cromwell/pull/1339.diff","patch_url":"https://github.com/broadinstitute/cromwell/pull/1339.patch","issue_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1339","number":1339,"state":"open","locked":false,"title":"Cromwell 1087","user":{"login":"jainh","id":10890082,"avatar_url":"https://avatars.githubusercontent.com/u/10890082?v=3","gravatar_id":"","url":"https://api.github.com/users/jainh","html_url":"https://github.com/jainh","followers_url":"https://api.github.com/users/jainh/followers","following_url":"https://api.github.com/users/jainh/following{/other_user}","gists_url":"https://api.github.com/users/jainh/gists{/gist_id}","starred_url":"https://api.github.com/users/jainh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jainh/subscriptions","organizations_url":"https://api.github.com/users/jainh/orgs","repos_url":"https://api.github.com/users/jainh/repos","events_url":"https://api.github.com/users/jainh/events{/privacy}","received_events_url":"https://api.github.com/users/jainh/received_events","type":"User","site_admin":false},"body":"PR Documentation describing the SparkBackend added to Cromwell\r\n\r\nThis PR adds support for execution of spark jobs as task in a workflow using the existing wdl format, with restrictions on the environment like having a local file system, a shared file system or a network file system when running a spark job in the spark standalone cluster mode.\r\n\r\nThis implementation takes a wdl with the backend configuration specified as \"Spark\" and then generates the appropriate spark commands and monitoring process to ensure the job runs to completion. Meaning, details of the spark internals are completely abstracted from the user provided backends with different configurations containing different flavours of { master and deployMode } combinations are already set.\r\n\r\nInternally, we create a bash script containing a spark-submit (depending on the backend flavour selected at runtime) command using all the specified wdl runtime attributes which is then executed by Spark.  \r\n\r\nCurrent deploy modes supported for any spark job:\r\n  a - Client deploy mode using the spark standalone cluster manager\r\n  b - Cluster deploy mode using the spark standalone cluster manager\r\n  c - Client deploy mode using Yarn resource manager\r\n  d - Cluster deploy mode using Yarn resource manager\r\n  \r\nFuture PR Plans:\r\n  In this PR, the hadoop file system cannot be used as an input/output for the SBE because the Cromwell engine does not identify the protocol, and this results in the hdfs path being localized (soft-link, hard-link or copied).\r\n  This is not a problem until the SBE tries to evaluate the output after a successful execution, and because it cannot interpret the protocol, it tries to look for an hdfs output locally which results in an error. Note: This is only the case when the spark job writes the output to an hdfs location. Then cromwell cannot find the output file for evaluation.\r\n\r\n  In the near **Future**, we plan to provide an hdfs client similar to that of the gcs to add support for the hdfs, primarily because hdfs is spark's natural file system.\r\n  Note that this doesn't actually prevent spark from writing to the hdfs, in order words, the spark application can write or read from the hdfs if given hdfs locations as arguments.\r\n\r\nReason for restriction on environment:\r\n  In spark cluster mode, the assembly jar file containing the application has to exist in all the nodes of the cluster since the driver program can be started on any of the nodes in the cluster.\r\n  Known solution to this is to put the jar file in a shared file system like hdfs or a network file system, or a parallel distributed file system like lustre where all nodes in the cluster can access the file.\r\n  For this reason, we did all our testing using the lustre file system, though it works just fine on a local file system with replication.\r\n  \r\n  Also, note that as pointed out in the future PR plans section, the hadoop file system is not supported on this release.\r\n\r\nSupported File Systems:\r\n  Local File System\r\n  Network File System\r\n  Distributed file system\r\n\r\nPS: Please find attached read me for examples on How to use Spark Backend\r\n[readMe.md.zip](https://github.com/broadinstitute/cromwell/files/437890/readMe.md.zip)\r\n\r\nReviewers:  @geoffjentry  @francares \r\n\r\n","created_at":"2016-08-25T19:24:39Z","updated_at":"2016-08-26T22:01:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1eb4edc1ec255c3d8102bf68bdf47c9493b242ea","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339/commits","review_comments_url":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339/comments","review_comment_url":"https://api.github.com/repos/broadinstitute/cromwell/pulls/comments{/number}","comments_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1339/comments","statuses_url":"https://api.github.com/repos/broadinstitute/cromwell/statuses/590fcb6a9ea708cc5e00faefc4e5beb7e8b8339c","head":{"label":"broadinstitute:cromwell-1087","ref":"cromwell-1087","sha":"590fcb6a9ea708cc5e00faefc4e5beb7e8b8339c","user":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"repo":{"id":34136406,"name":"cromwell","full_name":"broadinstitute/cromwell","owner":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/broadinstitute/cromwell","description":"Workflow Execution Engine using WDL","fork":false,"url":"https://api.github.com/repos/broadinstitute/cromwell","forks_url":"https://api.github.com/repos/broadinstitute/cromwell/forks","keys_url":"https://api.github.com/repos/broadinstitute/cromwell/keys{/key_id}","collaborators_url":"https://api.github.com/repos/broadinstitute/cromwell/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/broadinstitute/cromwell/teams","hooks_url":"https://api.github.com/repos/broadinstitute/cromwell/hooks","issue_events_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/events{/number}","events_url":"https://api.github.com/repos/broadinstitute/cromwell/events","assignees_url":"https://api.github.com/repos/broadinstitute/cromwell/assignees{/user}","branches_url":"https://api.github.com/repos/broadinstitute/cromwell/branches{/branch}","tags_url":"https://api.github.com/repos/broadinstitute/cromwell/tags","blobs_url":"https://api.github.com/repos/broadinstitute/cromwell/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/broadinstitute/cromwell/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/broadinstitute/cromwell/git/refs{/sha}","trees_url":"https://api.github.com/repos/broadinstitute/cromwell/git/trees{/sha}","statuses_url":"https://api.github.com/repos/broadinstitute/cromwell/statuses/{sha}","languages_url":"https://api.github.com/repos/broadinstitute/cromwell/languages","stargazers_url":"https://api.github.com/repos/broadinstitute/cromwell/stargazers","contributors_url":"https://api.github.com/repos/broadinstitute/cromwell/contributors","subscribers_url":"https://api.github.com/repos/broadinstitute/cromwell/subscribers","subscription_url":"https://api.github.com/repos/broadinstitute/cromwell/subscription","commits_url":"https://api.github.com/repos/broadinstitute/cromwell/commits{/sha}","git_commits_url":"https://api.github.com/repos/broadinstitute/cromwell/git/commits{/sha}","comments_url":"https://api.github.com/repos/broadinstitute/cromwell/comments{/number}","issue_comment_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/comments{/number}","contents_url":"https://api.github.com/repos/broadinstitute/cromwell/contents/{+path}","compare_url":"https://api.github.com/repos/broadinstitute/cromwell/compare/{base}...{head}","merges_url":"https://api.github.com/repos/broadinstitute/cromwell/merges","archive_url":"https://api.github.com/repos/broadinstitute/cromwell/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/broadinstitute/cromwell/downloads","issues_url":"https://api.github.com/repos/broadinstitute/cromwell/issues{/number}","pulls_url":"https://api.github.com/repos/broadinstitute/cromwell/pulls{/number}","milestones_url":"https://api.github.com/repos/broadinstitute/cromwell/milestones{/number}","notifications_url":"https://api.github.com/repos/broadinstitute/cromwell/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/broadinstitute/cromwell/labels{/name}","releases_url":"https://api.github.com/repos/broadinstitute/cromwell/releases{/id}","deployments_url":"https://api.github.com/repos/broadinstitute/cromwell/deployments","created_at":"2015-04-17T19:39:36Z","updated_at":"2016-08-26T17:20:25Z","pushed_at":"2016-08-26T21:54:34Z","git_url":"git://github.com/broadinstitute/cromwell.git","ssh_url":"git@github.com:broadinstitute/cromwell.git","clone_url":"https://github.com/broadinstitute/cromwell.git","svn_url":"https://github.com/broadinstitute/cromwell","homepage":"","size":8761,"stargazers_count":81,"watchers_count":81,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":21,"mirror_url":null,"open_issues_count":174,"forks":21,"open_issues":174,"watchers":81,"default_branch":"develop"}},"base":{"label":"broadinstitute:develop","ref":"develop","sha":"b195e1f8297a256b002284fbd805bfc2e915521f","user":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"repo":{"id":34136406,"name":"cromwell","full_name":"broadinstitute/cromwell","owner":{"login":"broadinstitute","id":393552,"avatar_url":"https://avatars.githubusercontent.com/u/393552?v=3","gravatar_id":"","url":"https://api.github.com/users/broadinstitute","html_url":"https://github.com/broadinstitute","followers_url":"https://api.github.com/users/broadinstitute/followers","following_url":"https://api.github.com/users/broadinstitute/following{/other_user}","gists_url":"https://api.github.com/users/broadinstitute/gists{/gist_id}","starred_url":"https://api.github.com/users/broadinstitute/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broadinstitute/subscriptions","organizations_url":"https://api.github.com/users/broadinstitute/orgs","repos_url":"https://api.github.com/users/broadinstitute/repos","events_url":"https://api.github.com/users/broadinstitute/events{/privacy}","received_events_url":"https://api.github.com/users/broadinstitute/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/broadinstitute/cromwell","description":"Workflow Execution Engine using WDL","fork":false,"url":"https://api.github.com/repos/broadinstitute/cromwell","forks_url":"https://api.github.com/repos/broadinstitute/cromwell/forks","keys_url":"https://api.github.com/repos/broadinstitute/cromwell/keys{/key_id}","collaborators_url":"https://api.github.com/repos/broadinstitute/cromwell/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/broadinstitute/cromwell/teams","hooks_url":"https://api.github.com/repos/broadinstitute/cromwell/hooks","issue_events_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/events{/number}","events_url":"https://api.github.com/repos/broadinstitute/cromwell/events","assignees_url":"https://api.github.com/repos/broadinstitute/cromwell/assignees{/user}","branches_url":"https://api.github.com/repos/broadinstitute/cromwell/branches{/branch}","tags_url":"https://api.github.com/repos/broadinstitute/cromwell/tags","blobs_url":"https://api.github.com/repos/broadinstitute/cromwell/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/broadinstitute/cromwell/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/broadinstitute/cromwell/git/refs{/sha}","trees_url":"https://api.github.com/repos/broadinstitute/cromwell/git/trees{/sha}","statuses_url":"https://api.github.com/repos/broadinstitute/cromwell/statuses/{sha}","languages_url":"https://api.github.com/repos/broadinstitute/cromwell/languages","stargazers_url":"https://api.github.com/repos/broadinstitute/cromwell/stargazers","contributors_url":"https://api.github.com/repos/broadinstitute/cromwell/contributors","subscribers_url":"https://api.github.com/repos/broadinstitute/cromwell/subscribers","subscription_url":"https://api.github.com/repos/broadinstitute/cromwell/subscription","commits_url":"https://api.github.com/repos/broadinstitute/cromwell/commits{/sha}","git_commits_url":"https://api.github.com/repos/broadinstitute/cromwell/git/commits{/sha}","comments_url":"https://api.github.com/repos/broadinstitute/cromwell/comments{/number}","issue_comment_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/comments{/number}","contents_url":"https://api.github.com/repos/broadinstitute/cromwell/contents/{+path}","compare_url":"https://api.github.com/repos/broadinstitute/cromwell/compare/{base}...{head}","merges_url":"https://api.github.com/repos/broadinstitute/cromwell/merges","archive_url":"https://api.github.com/repos/broadinstitute/cromwell/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/broadinstitute/cromwell/downloads","issues_url":"https://api.github.com/repos/broadinstitute/cromwell/issues{/number}","pulls_url":"https://api.github.com/repos/broadinstitute/cromwell/pulls{/number}","milestones_url":"https://api.github.com/repos/broadinstitute/cromwell/milestones{/number}","notifications_url":"https://api.github.com/repos/broadinstitute/cromwell/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/broadinstitute/cromwell/labels{/name}","releases_url":"https://api.github.com/repos/broadinstitute/cromwell/releases{/id}","deployments_url":"https://api.github.com/repos/broadinstitute/cromwell/deployments","created_at":"2015-04-17T19:39:36Z","updated_at":"2016-08-26T17:20:25Z","pushed_at":"2016-08-26T21:54:34Z","git_url":"git://github.com/broadinstitute/cromwell.git","ssh_url":"git@github.com:broadinstitute/cromwell.git","clone_url":"https://github.com/broadinstitute/cromwell.git","svn_url":"https://github.com/broadinstitute/cromwell","homepage":"","size":8761,"stargazers_count":81,"watchers_count":81,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":21,"mirror_url":null,"open_issues_count":174,"forks":21,"open_issues":174,"watchers":81,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339"},"html":{"href":"https://github.com/broadinstitute/cromwell/pull/1339"},"issue":{"href":"https://api.github.com/repos/broadinstitute/cromwell/issues/1339"},"comments":{"href":"https://api.github.com/repos/broadinstitute/cromwell/issues/1339/comments"},"review_comments":{"href":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339/comments"},"review_comment":{"href":"https://api.github.com/repos/broadinstitute/cromwell/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/broadinstitute/cromwell/pulls/1339/commits"},"statuses":{"href":"https://api.github.com/repos/broadinstitute/cromwell/statuses/590fcb6a9ea708cc5e00faefc4e5beb7e8b8339c"}}}}
{"actor":{"display_login":"geoffjentry"}}