{"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1665","repository_url":"https://api.github.com/repos/broadinstitute/cromwell","labels_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1665/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1665/comments","events_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1665/events","html_url":"https://github.com/broadinstitute/cromwell/issues/1665","id":188887253,"number":1665,"title":"The JES polling actor unexpectedly terminated while conducting polls. Making a new one...","user":{"login":"seandavi","id":92435,"avatar_url":"https://avatars.githubusercontent.com/u/92435?v=3","gravatar_id":"","url":"https://api.github.com/users/seandavi","html_url":"https://github.com/seandavi","followers_url":"https://api.github.com/users/seandavi/followers","following_url":"https://api.github.com/users/seandavi/following{/other_user}","gists_url":"https://api.github.com/users/seandavi/gists{/gist_id}","starred_url":"https://api.github.com/users/seandavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seandavi/subscriptions","organizations_url":"https://api.github.com/users/seandavi/orgs","repos_url":"https://api.github.com/users/seandavi/repos","events_url":"https://api.github.com/users/seandavi/events{/privacy}","received_events_url":"https://api.github.com/users/seandavi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"geoffjentry","id":961771,"avatar_url":"https://avatars.githubusercontent.com/u/961771?v=3","gravatar_id":"","url":"https://api.github.com/users/geoffjentry","html_url":"https://github.com/geoffjentry","followers_url":"https://api.github.com/users/geoffjentry/followers","following_url":"https://api.github.com/users/geoffjentry/following{/other_user}","gists_url":"https://api.github.com/users/geoffjentry/gists{/gist_id}","starred_url":"https://api.github.com/users/geoffjentry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geoffjentry/subscriptions","organizations_url":"https://api.github.com/users/geoffjentry/orgs","repos_url":"https://api.github.com/users/geoffjentry/repos","events_url":"https://api.github.com/users/geoffjentry/events{/privacy}","received_events_url":"https://api.github.com/users/geoffjentry/received_events","type":"User","site_admin":false},"assignees":[{"login":"geoffjentry","id":961771,"avatar_url":"https://avatars.githubusercontent.com/u/961771?v=3","gravatar_id":"","url":"https://api.github.com/users/geoffjentry","html_url":"https://github.com/geoffjentry","followers_url":"https://api.github.com/users/geoffjentry/followers","following_url":"https://api.github.com/users/geoffjentry/following{/other_user}","gists_url":"https://api.github.com/users/geoffjentry/gists{/gist_id}","starred_url":"https://api.github.com/users/geoffjentry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geoffjentry/subscriptions","organizations_url":"https://api.github.com/users/geoffjentry/orgs","repos_url":"https://api.github.com/users/geoffjentry/repos","events_url":"https://api.github.com/users/geoffjentry/events{/privacy}","received_events_url":"https://api.github.com/users/geoffjentry/received_events","type":"User","site_admin":false}],"milestone":null,"comments":10,"created_at":"2016-11-12T02:32:30Z","updated_at":"2016-11-14T22:10:49Z","closed_at":null,"body":"I was running Cromwell with a JES backend.  I got through about 300 out of 1000 workflows that were submitted as a batch; the rest are still listed as `Running`.  Oddly, the genomics API operations report that they are still running though there are no instances still up.  I am not sure if this is a Cromwell issue or a Google one, but here is what I am seeing from Cromwell.   Cromwell has been working well for smaller batches.  Any thoughts? \r\n\r\nThanks!\r\n\r\n```\r\n[ERROR] [11/12/2016 02:22:12.170] [cromwell-system-akka.actor.default-dispatcher-2686] [akka://cromwell-system/user/cromwell-service/$b/$a] The JES polling actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$8#-1275882726] unexpectedly terminated while conducting 11 polls. Making a new one...\r\n[INFO] [11/12/2016 02:22:12.170] [cromwell-system-akka.actor.default-dispatcher-2686] [akka://cromwell-system/user/cromwell-service/$b/$a] watching Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$9#-1852863496]\r\n[WARN] [11/12/2016 02:22:12.171] [cromwell-system-akka.dispatchers.backend-dispatcher-2573] [akka://cromwell-system/user/cromwell-service/WorkflowManagerActor/WorkflowActor-7f0d7c16-d434-4dd7-a7ba-c7e897701c9a/WorkflowExecutionActor-7f0d7c16-d434-4dd7-a7ba-c7e897701c9a/7f0d7c16-d434-4dd7-a7ba-c7e897701c9a-EngineJobExecutionActor-salmonRun.salmonQuant:NA:1/7f0d7c16-d434-4dd7-a7ba-c7e897701c9a-BackendJobExecutionActor-7f0d7c16:salmonRun.salmonQuant:-1:1/JesAsyncBackendJobExecutionActor] JesAsyncBackendJobExecutionActor [UUID(7f0d7c16)salmonRun.salmonQuant:NA:1]: Caught exception, retrying:\r\njava.lang.RuntimeException: Unexpected actor death!\r\n\tat cromwell.backend.impl.jes.statuspolling.JesPollingActorClient$$anonfun$pollingActorClientReceive$1.applyOrElse(JesPollingActorClient.scala:33)\r\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)\r\n\tat akka.actor.Actor$class.aroundReceive(Actor.scala:484)\r\n\tat cromwell.backend.impl.jes.JesAsyncBackendJobExecutionActor.aroundReceive(JesAsyncBackendJobExecutionActor.scala:82)\r\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)\r\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:495)\r\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:224)\r\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:234)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\n```"},"comment":{"url":"https://api.github.com/repos/broadinstitute/cromwell/issues/comments/260479729","html_url":"https://github.com/broadinstitute/cromwell/issues/1665#issuecomment-260479729","issue_url":"https://api.github.com/repos/broadinstitute/cromwell/issues/1665","id":260479729,"user":{"login":"seandavi","id":92435,"avatar_url":"https://avatars.githubusercontent.com/u/92435?v=3","gravatar_id":"","url":"https://api.github.com/users/seandavi","html_url":"https://github.com/seandavi","followers_url":"https://api.github.com/users/seandavi/followers","following_url":"https://api.github.com/users/seandavi/following{/other_user}","gists_url":"https://api.github.com/users/seandavi/gists{/gist_id}","starred_url":"https://api.github.com/users/seandavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seandavi/subscriptions","organizations_url":"https://api.github.com/users/seandavi/orgs","repos_url":"https://api.github.com/users/seandavi/repos","events_url":"https://api.github.com/users/seandavi/events{/privacy}","received_events_url":"https://api.github.com/users/seandavi/received_events","type":"User","site_admin":false},"created_at":"2016-11-14T22:10:49Z","updated_at":"2016-11-14T22:10:49Z","body":"Just a note that the second part of my issue was about operations stuck in the \"Running\" state.  It turns out that was a Google bug that has now been fixed.  "}}
{"actor":{"display_login":"seandavi"}}