{"action":"reopened","issue":{"url":"https://api.github.com/repos/spotify/styx/issues/11","repository_url":"https://api.github.com/repos/spotify/styx","labels_url":"https://api.github.com/repos/spotify/styx/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/spotify/styx/issues/11/comments","events_url":"https://api.github.com/repos/spotify/styx/issues/11/events","html_url":"https://github.com/spotify/styx/issues/11","id":192719302,"number":11,"title":"Halting and re-triggering in short succession causes premature state termination","user":{"login":"rouzwawi","id":655452,"avatar_url":"https://avatars.githubusercontent.com/u/655452?v=3","gravatar_id":"","url":"https://api.github.com/users/rouzwawi","html_url":"https://github.com/rouzwawi","followers_url":"https://api.github.com/users/rouzwawi/followers","following_url":"https://api.github.com/users/rouzwawi/following{/other_user}","gists_url":"https://api.github.com/users/rouzwawi/gists{/gist_id}","starred_url":"https://api.github.com/users/rouzwawi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rouzwawi/subscriptions","organizations_url":"https://api.github.com/users/rouzwawi/orgs","repos_url":"https://api.github.com/users/rouzwawi/repos","events_url":"https://api.github.com/users/rouzwawi/events{/privacy}","received_events_url":"https://api.github.com/users/rouzwawi/received_events","type":"User","site_admin":false},"labels":[{"id":486382309,"url":"https://api.github.com/repos/spotify/styx/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-01T00:14:20Z","updated_at":"2016-12-02T17:02:59Z","closed_at":null,"body":"When an actively running Execution is halted (through the cli or api) Styx will `delete` the Pod in k8s. This process takes some time to complete as the container have to receive the `SIGTERM` signal and exit. It will get some time to exit gracefully (the so called grace period).\r\n\r\nIf the same Workflow Instance is triggered and a new Execution is started during this shutdown period, there will be two Pods in k8s which are associated with the same Workflow Instance (through a Pod annotation). When finally the first Pod exits (usually with status code 137, 128 + `SIGTERM(9)`), a `terminate(137)` event is generated for that Workflow Instance. This event is dispatched to the currently active state and we see a premature state transition.\r\n\r\nAt this point, the second execution Pod is orphaned.\r\n\r\nA sequence of events outlining this scenario:\r\n```\r\n2016-11-28T14:39:57       triggerExecution          Trigger id: ad-hoc-cli-1480343996990-18517\r\n2016-11-28T14:39:57       submit                    Execution description: ExecutionDescription{dockerImage=stagger:3, dockerArgs=[{}], secret=Optional.empty, commitSha=Optional.empty}\r\n2016-11-28T14:39:57       submitted                 Execution id: styx-run-0ab59615-979b-4717-80f3-deab984d1074\r\n2016-11-28T14:40:00       started\r\n2016-11-28T14:41:17       halt\r\n2016-11-28T14:41:22       triggerExecution          Trigger id: ad-hoc-cli-1480344082837-41704\r\n2016-11-28T14:41:23       submit                    Execution description: ExecutionDescription{dockerImage=stagger:3, dockerArgs=[{}], secret=Optional.empty, commitSha=Optional.empty}\r\n2016-11-28T14:41:23       submitted                 Execution id: styx-run-12994ebb-c7f7-4083-8844-952ecd63b3f8\r\n2016-11-28T14:41:27       started\r\n2016-11-28T14:41:48       terminate                 Exit code: 137\r\n2016-11-28T14:41:48       retryAfter                Delay (seconds): 180\r\n```\r\n\r\nThe `terminate` event at the end is actually coming from the first execution, `styx-run-0ab59615-979b-4717-80f3-deab984d1074`.\r\n"}}
{"actor":{"display_login":"rouzwawi"}}