{"action":"opened","issue":{"url":"https://api.github.com/repos/Yelp/paasta/issues/903","repository_url":"https://api.github.com/repos/Yelp/paasta","labels_url":"https://api.github.com/repos/Yelp/paasta/issues/903/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/paasta/issues/903/comments","events_url":"https://api.github.com/repos/Yelp/paasta/issues/903/events","html_url":"https://github.com/Yelp/paasta/issues/903","id":191154360,"number":903,"title":"2 minute delay before killing task causes downthenup to have overlapping versions","user":{"login":"EvanKrall","id":233961,"avatar_url":"https://avatars.githubusercontent.com/u/233961?v=3","gravatar_id":"","url":"https://api.github.com/users/EvanKrall","html_url":"https://github.com/EvanKrall","followers_url":"https://api.github.com/users/EvanKrall/followers","following_url":"https://api.github.com/users/EvanKrall/following{/other_user}","gists_url":"https://api.github.com/users/EvanKrall/gists{/gist_id}","starred_url":"https://api.github.com/users/EvanKrall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EvanKrall/subscriptions","organizations_url":"https://api.github.com/users/EvanKrall/orgs","repos_url":"https://api.github.com/users/EvanKrall/repos","events_url":"https://api.github.com/users/EvanKrall/events{/privacy}","received_events_url":"https://api.github.com/users/EvanKrall/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-22T23:36:09Z","updated_at":"2016-11-22T23:36:09Z","closed_at":null,"body":"A service (hipchat-irc-bridge) is configured with downthenup to avoid having duplicate copies running at the same time, but bounces seem to start the new version before the old one dies.\r\n\r\nAccording to `paasta logs`, paasta issued the kill at 15:23:56:\r\n\r\n```\r\n2016-11-22 15:23:56 [deploy] [norcal-devc] [main] - [debug] downthenup bounce killing drained task hipchat-irc-bridge.main.git7aad13ec.config5366305a.76aabad3-adcd-11e6-9ab4-16c8c3905dcb\r\n```\r\n\r\nHowever, according to `docker events --since=0 | grep hipchat` on the box where that old instance was running, the container wasn't killed until 15:25:57.\r\n\r\n```\r\n$ sudo docker events --since=0 | grep hipchat\r\n2016-11-22T15:25:57.190513683-08:00 container kill 3a44d6f73f06bb78ffa111c6c93af33ccaca65b697a3d5566bd97809f7e52b25 (image=docker-paasta.yelpcorp.com:443/services-hipchat-irc-bridge:paasta-7aad13ec4dbe02cbd22aec7b1f6fb0d59e17a492, name=mesos-c4b99c87-1a66-48f3-b31e-1686fdfea7c2-S86.a2a7f9b1-1673-4ea8-ba5c-3fdf0cace260)\r\n2016-11-22T15:25:57.276188294-08:00 container die 3a44d6f73f06bb78ffa111c6c93af33ccaca65b697a3d5566bd97809f7e52b25 (image=docker-paasta.yelpcorp.com:443/services-hipchat-irc-bridge:paasta-7aad13ec4dbe02cbd22aec7b1f6fb0d59e17a492, name=mesos-c4b99c87-1a66-48f3-b31e-1686fdfea7c2-S86.a2a7f9b1-1673-4ea8-ba5c-3fdf0cace260)\r\n2016-11-22T15:25:58.122977165-08:00 container stop 3a44d6f73f06bb78ffa111c6c93af33ccaca65b697a3d5566bd97809f7e52b25 (image=docker-paasta.yelpcorp.com:443/services-hipchat-irc-bridge:paasta-7aad13ec4dbe02cbd22aec7b1f6fb0d59e17a492, name=mesos-c4b99c87-1a66-48f3-b31e-1686fdfea7c2-S86.a2a7f9b1-1673-4ea8-ba5c-3fdf0cace260)\r\n```\r\n\r\nThis seems like there's a two-minute delay between the kill API call to Marathon and when marathon/mesos actually kills the container. Perhaps `taskKillGracePeriodSeconds` is defaulting to 120 somehow? The docs suggest that there is no default."}}
{"actor":{"display_login":"EvanKrall"}}