{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/conductor/issues/776","repository_url":"https://api.github.com/repos/Netflix/conductor","labels_url":"https://api.github.com/repos/Netflix/conductor/issues/776/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/conductor/issues/776/comments","events_url":"https://api.github.com/repos/Netflix/conductor/issues/776/events","html_url":"https://github.com/Netflix/conductor/issues/776","id":362178925,"node_id":"MDU6SXNzdWUzNjIxNzg5MjU=","number":776,"title":"No Heartbeat / Failover?","user":{"login":"panthony","id":230462,"node_id":"MDQ6VXNlcjIzMDQ2Mg==","avatar_url":"https://avatars1.githubusercontent.com/u/230462?v=4","gravatar_id":"","url":"https://api.github.com/users/panthony","html_url":"https://github.com/panthony","followers_url":"https://api.github.com/users/panthony/followers","following_url":"https://api.github.com/users/panthony/following{/other_user}","gists_url":"https://api.github.com/users/panthony/gists{/gist_id}","starred_url":"https://api.github.com/users/panthony/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/panthony/subscriptions","organizations_url":"https://api.github.com/users/panthony/orgs","repos_url":"https://api.github.com/users/panthony/repos","events_url":"https://api.github.com/users/panthony/events{/privacy}","received_events_url":"https://api.github.com/users/panthony/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-20T13:06:39Z","updated_at":"2018-09-24T06:02:52Z","closed_at":null,"author_association":"NONE","body":"Hi there,\r\n\r\nSimilar to #162, I tried to see how conductor would behave if a worker just died, for this I created a simple worker that I killed right after starting the workflow.\r\n\r\nConductor would not see that the worker died and would still show the task as being \"in progress\", restarting the worker would not pick the task back either.\r\n\r\nThe closest thing that I found to an heart beat is `responseTimeoutSeconds` but I failed to make it works or understand how it works.\r\n\r\nI cannot use the \"timeoutSeconds\" feature because a same task could take few seconds to few weeks depending on its input.\r\n\r\nIs there a way for conductor to take back a task if the associated worker does not show any sign of life anymore ?\r\n\r\nAny pointer would be greatly appreciated.\r\n\r\nHere's the used config:\r\n\r\n**workflow**\r\n```json\r\n{\r\n  \"name\": \"test_heartbeat_workflow\",\r\n  \"version\": 1,\r\n  \"tasks\": [\r\n\r\n      {\r\n        \"name\": \"test_heartbeat\",\r\n        \"taskReferenceName\": \"test_heartbeat\",\r\n        \"type\": \"SIMPLE\"\r\n      }\r\n  ],\r\n  \"schemaVersion\": 2\r\n}\r\n```\r\n\r\n**taskDef**\r\n```\r\n[\r\n  {\r\n    \"name\": \"test_heartbeat\",\r\n    \"retryCount\": 50,\r\n    \"timeoutSeconds\": 0,\r\n    \"inputKeys\": [\r\n    ],\r\n    \"timeoutPolicy\": \"RETRY\",\r\n    \"retryLogic\": \"FIXED\",\r\n    \"retryDelaySeconds\": 5,\r\n    \"responseTimeoutSeconds\": 5\r\n  }\r\n]\r\n```\r\n\r\n**worker**\r\n```java\r\npackage com.oncrawl.analysis.conductor;\r\n\r\nimport com.netflix.conductor.client.worker.Worker;\r\nimport com.netflix.conductor.common.metadata.tasks.Task;\r\nimport com.netflix.conductor.common.metadata.tasks.TaskResult;\r\n\r\nimport java.util.concurrent.TimeUnit;\r\n\r\npublic class SampleWorker implements Worker {\r\n\r\n  @Override\r\n  public String getTaskDefName() {\r\n    return \"test_heartbeat\";\r\n  }\r\n\r\n  @Override\r\n  public TaskResult execute(Task task) {\r\n    System.out.println(\"executing \" + getTaskDefName());\r\n\r\n    try {\r\n      Thread.sleep(TimeUnit.MINUTES.toMillis(5L));\r\n    } catch (InterruptedException e) {\r\n      e.printStackTrace();\r\n    }\r\n\r\n    return new TaskResult(task);\r\n  }\r\n}\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/Netflix/conductor/issues/comments/423882879","html_url":"https://github.com/Netflix/conductor/issues/776#issuecomment-423882879","issue_url":"https://api.github.com/repos/Netflix/conductor/issues/776","id":423882879,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzg4Mjg3OQ==","user":{"login":"panthony","id":230462,"node_id":"MDQ6VXNlcjIzMDQ2Mg==","avatar_url":"https://avatars1.githubusercontent.com/u/230462?v=4","gravatar_id":"","url":"https://api.github.com/users/panthony","html_url":"https://github.com/panthony","followers_url":"https://api.github.com/users/panthony/followers","following_url":"https://api.github.com/users/panthony/following{/other_user}","gists_url":"https://api.github.com/users/panthony/gists{/gist_id}","starred_url":"https://api.github.com/users/panthony/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/panthony/subscriptions","organizations_url":"https://api.github.com/users/panthony/orgs","repos_url":"https://api.github.com/users/panthony/repos","events_url":"https://api.github.com/users/panthony/events{/privacy}","received_events_url":"https://api.github.com/users/panthony/received_events","type":"User","site_admin":false},"created_at":"2018-09-24T06:02:52Z","updated_at":"2018-09-24T06:02:52Z","author_association":"NONE","body":"@jvemugunta Thanks for your anwser, I'll take a look at your PR (once you reverted the format changes it makes it too hard to see what was truly updated) and if it does fix by issue I'll close it as duplicate ! "}}
{"actor":{"display_login":"panthony"}}