{"action":"created","issue":{"url":"https://api.github.com/repos/BD2KGenomics/toil/issues/797","repository_url":"https://api.github.com/repos/BD2KGenomics/toil","labels_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/797/labels{/name}","comments_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/797/comments","events_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/797/events","html_url":"https://github.com/BD2KGenomics/toil/pull/797","id":148701215,"number":797,"title":"Add YARNBatchSystem to Toil (resolves #796)","user":{"login":"fnothaft","id":3752466,"avatar_url":"https://avatars.githubusercontent.com/u/3752466?v=3","gravatar_id":"","url":"https://api.github.com/users/fnothaft","html_url":"https://github.com/fnothaft","followers_url":"https://api.github.com/users/fnothaft/followers","following_url":"https://api.github.com/users/fnothaft/following{/other_user}","gists_url":"https://api.github.com/users/fnothaft/gists{/gist_id}","starred_url":"https://api.github.com/users/fnothaft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fnothaft/subscriptions","organizations_url":"https://api.github.com/users/fnothaft/orgs","repos_url":"https://api.github.com/users/fnothaft/repos","events_url":"https://api.github.com/users/fnothaft/events{/privacy}","received_events_url":"https://api.github.com/users/fnothaft/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BD2KGenomics/toil/labels/in%20progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/BD2KGenomics/toil/labels/needs%20work","name":"needs work","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"hannes-ucsc","id":5143256,"avatar_url":"https://avatars.githubusercontent.com/u/5143256?v=3","gravatar_id":"","url":"https://api.github.com/users/hannes-ucsc","html_url":"https://github.com/hannes-ucsc","followers_url":"https://api.github.com/users/hannes-ucsc/followers","following_url":"https://api.github.com/users/hannes-ucsc/following{/other_user}","gists_url":"https://api.github.com/users/hannes-ucsc/gists{/gist_id}","starred_url":"https://api.github.com/users/hannes-ucsc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hannes-ucsc/subscriptions","organizations_url":"https://api.github.com/users/hannes-ucsc/orgs","repos_url":"https://api.github.com/users/hannes-ucsc/repos","events_url":"https://api.github.com/users/hannes-ucsc/events{/privacy}","received_events_url":"https://api.github.com/users/hannes-ucsc/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/BD2KGenomics/toil/milestones/8","html_url":"https://github.com/BD2KGenomics/toil/milestones/3.2.0","labels_url":"https://api.github.com/repos/BD2KGenomics/toil/milestones/8/labels","id":1438887,"number":8,"title":"3.2.0","description":"Better caching, auto-scaling, import/export of files on leader, promised requirements.","creator":{"login":"hannes-ucsc","id":5143256,"avatar_url":"https://avatars.githubusercontent.com/u/5143256?v=3","gravatar_id":"","url":"https://api.github.com/users/hannes-ucsc","html_url":"https://github.com/hannes-ucsc","followers_url":"https://api.github.com/users/hannes-ucsc/followers","following_url":"https://api.github.com/users/hannes-ucsc/following{/other_user}","gists_url":"https://api.github.com/users/hannes-ucsc/gists{/gist_id}","starred_url":"https://api.github.com/users/hannes-ucsc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hannes-ucsc/subscriptions","organizations_url":"https://api.github.com/users/hannes-ucsc/orgs","repos_url":"https://api.github.com/users/hannes-ucsc/repos","events_url":"https://api.github.com/users/hannes-ucsc/events{/privacy}","received_events_url":"https://api.github.com/users/hannes-ucsc/received_events","type":"User","site_admin":false},"open_issues":43,"closed_issues":30,"state":"open","created_at":"2015-12-02T04:12:13Z","updated_at":"2016-05-13T22:30:04Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2016-04-15T16:06:10Z","updated_at":"2016-05-17T14:36:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BD2KGenomics/toil/pulls/797","html_url":"https://github.com/BD2KGenomics/toil/pull/797","diff_url":"https://github.com/BD2KGenomics/toil/pull/797.diff","patch_url":"https://github.com/BD2KGenomics/toil/pull/797.patch"},"body":"Adds a YARNBatchSystem that enables Toil to run jobs on a cluster that is managed with the Apache Hadoop YARN resource manager. A job maps directly to a YARN application. Specifically, we run a batch command line under YARN using the YARN DistributedShell application. This way, we do not need to implement a YARN ApplicationManager, etc.\r\n\r\nThis is completely untested and doesn't have docs. I'll be adding tests in the next few days. @hannes-ucsc: IIRC, we'll need to add a YARN setup on Jenkins. I will ask you about this once I've got the tests done. I think it should be straightforward to do."},"comment":{"url":"https://api.github.com/repos/BD2KGenomics/toil/issues/comments/219737724","html_url":"https://github.com/BD2KGenomics/toil/pull/797#issuecomment-219737724","issue_url":"https://api.github.com/repos/BD2KGenomics/toil/issues/797","id":219737724,"user":{"login":"jpdna","id":7423296,"avatar_url":"https://avatars.githubusercontent.com/u/7423296?v=3","gravatar_id":"","url":"https://api.github.com/users/jpdna","html_url":"https://github.com/jpdna","followers_url":"https://api.github.com/users/jpdna/followers","following_url":"https://api.github.com/users/jpdna/following{/other_user}","gists_url":"https://api.github.com/users/jpdna/gists{/gist_id}","starred_url":"https://api.github.com/users/jpdna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpdna/subscriptions","organizations_url":"https://api.github.com/users/jpdna/orgs","repos_url":"https://api.github.com/users/jpdna/repos","events_url":"https://api.github.com/users/jpdna/events{/privacy}","received_events_url":"https://api.github.com/users/jpdna/received_events","type":"User","site_admin":false},"created_at":"2016-05-17T14:36:17Z","updated_at":"2016-05-17T14:36:17Z","body":"@fnothaft - in effort to test this and reproduce and resolve the remaining test failure you mentioned I have done the following:\r\n\r\n1) cloned the Yarn fork from your repo, and checked out the `yarn` branch on which  this PR is based, and setup virtualenv for toil then run `make test` as described at `http://toil.readthedocs.io/en/latest/installation.html#building-testing`\r\n\r\n2) installed Hadoop-2.6.4 in a local dir and set HADOOP_HOME to it.    Is there more config for Hadoop I should do?\r\n\r\n3) I tried starting Yarn daemons with `./sbin/start-yarn.sh` prior to running `make test` on toil, but get lots of erros about `java.net.BindException: Problem binding to [0.0.0.0:8031] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException`\r\n\r\nIf I don't start YARN first, the tests seem to run more happily, but in that case the end of the test output seems stuck at end producing forever:\r\n```\r\n6/05/17 10:29:52 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)\r\n16/05/17 10:29:53 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)\r\n16/05/17 10:29:54 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)\r\n...\r\n```\r\n\r\ncan you point me to the correct Yarn setup here for testing, which test to run, and what the current test failure  you are getting is so I know when I reproduce it properly?\r\n\r\n`"}}