{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1435","repository_url":"https://api.github.com/repos/cloudfoundry/bosh","labels_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1435/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1435/comments","events_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1435/events","html_url":"https://github.com/cloudfoundry/bosh/issues/1435","id":176519619,"number":1435,"title":"Process in monit file failed to get killed","user":{"login":"Amit-PivotalLabs","id":3108318,"avatar_url":"https://avatars.githubusercontent.com/u/3108318?v=3","gravatar_id":"","url":"https://api.github.com/users/Amit-PivotalLabs","html_url":"https://github.com/Amit-PivotalLabs","followers_url":"https://api.github.com/users/Amit-PivotalLabs/followers","following_url":"https://api.github.com/users/Amit-PivotalLabs/following{/other_user}","gists_url":"https://api.github.com/users/Amit-PivotalLabs/gists{/gist_id}","starred_url":"https://api.github.com/users/Amit-PivotalLabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Amit-PivotalLabs/subscriptions","organizations_url":"https://api.github.com/users/Amit-PivotalLabs/orgs","repos_url":"https://api.github.com/users/Amit-PivotalLabs/repos","events_url":"https://api.github.com/users/Amit-PivotalLabs/events{/privacy}","received_events_url":"https://api.github.com/users/Amit-PivotalLabs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-13T00:36:16Z","updated_at":"2016-09-13T03:11:40Z","closed_at":null,"body":"Hi all,\r\n\r\nWe had an instance group with a release job whose monit file defined 1 process (consul) which bound to port 53.  We changed the release job to have two processes, the original process binding to 8600 and the new process (dnsmasq) binding to 53.  We deployed, and this worked as expected -- stopped consul which was binding to 53, and then started consul bound to 8600 and dnsmasq bound to 53.  We then reverted this change to the release, updated manifest to use the new release, and redeployed.  We expected that this would stop both consul and dnsmasq, and then start up consul binding to 53.  Instead, deploy failed because dnsmasq was still running bound to 53.\r\n\r\n1. Why didn’t dnsmasq get killed? Does anything look wrong with https://github.com/cloudfoundry-incubator/consul-release/blob/master/jobs/consul_agent/templates/dnsmasq_ctl.erb#L22-L25?\r\n2. To remediate this without having to SSH onto every vm and kill the dnsmasq process, I wanted to `bosh -d old-manifest.yml stop` to stop all processes, then `bosh -d new-manifest.yml deploy --recreate`.  Does this seem like the best way to do it?\r\n\r\nThanks\r\n\r\n/cc @cloudfoundry/cf-infrastructure @cloudfoundry/cf-release-integration @cloudfoundry/cf-capi "},"comment":{"url":"https://api.github.com/repos/cloudfoundry/bosh/issues/comments/246561097","html_url":"https://github.com/cloudfoundry/bosh/issues/1435#issuecomment-246561097","issue_url":"https://api.github.com/repos/cloudfoundry/bosh/issues/1435","id":246561097,"user":{"login":"cppforlife","id":185847,"avatar_url":"https://avatars.githubusercontent.com/u/185847?v=3","gravatar_id":"","url":"https://api.github.com/users/cppforlife","html_url":"https://github.com/cppforlife","followers_url":"https://api.github.com/users/cppforlife/followers","following_url":"https://api.github.com/users/cppforlife/following{/other_user}","gists_url":"https://api.github.com/users/cppforlife/gists{/gist_id}","starred_url":"https://api.github.com/users/cppforlife/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cppforlife/subscriptions","organizations_url":"https://api.github.com/users/cppforlife/orgs","repos_url":"https://api.github.com/users/cppforlife/repos","events_url":"https://api.github.com/users/cppforlife/events{/privacy}","received_events_url":"https://api.github.com/users/cppforlife/received_events","type":"User","site_admin":false},"created_at":"2016-09-13T03:11:39Z","updated_at":"2016-09-13T03:11:39Z","body":"does this happen consistently? can you throw in links to releases i can use to repro in my env?\n\nSent from my iPhone\n\n> On Sep 12, 2016, at 8:05 PM, Amit Gupta <notifications@github.com> wrote:\n> \n> @tylerschultz wrote:\n> \n> I believe this can be explained by a bug that has been present for a long time. When director tells monit to stop, it may take a while for monit to get around to actually running the ctl stop command. Monit may be busy doing other things, stopping other jobs, daydreaming... whatever it does. Unfortunately, the agent believes stopping is complete because it has issued the monit stop foo command. The agent reports back to director that the job is stopped. Director moves on and then issues an apply call. This call will result in the new job templates replacing the old templates (the templates that would stop dnsmasq are now replaced.) Monit now finnaly gets around to issuing the ctl stop command and the stop runs against the replaced templates!!!!\n> \n> Alas, there is a fix:\n> \n> Now I can't find the job property on a quick search, hopefully it has been released. Enabling the property will cause agent and director to wait for the stop command to complete, within a timeout, before moving on to the apply step. This seems like something that should be turned on by default, but there is concern that there are many jobs that don't stop well, and it'd be a breaking change for many releases.\n> \n> Looking at tracker, it appears the fix may not yet be released. Thinking about it more, it also occurs to me that it's odd that all of the instances suffered from this problem. This leads me to believe something else may be going on.\n> \n> [source]\n> \n> Thanks. Yeah, my suspicions are the same. We know about the race that can happen around stop, but this happened on many VMs in many different environments. So it's kinda mysterious. Let me know if you have any further thoughts on that first question.\n> \n> As for the second question, does that sound like a reasonable approach? Note that I believe I had to do the deploy with the --recreate since a simple bosh deploy left the VMs in a stopped state, and bosh start didn't start the VMs back up in the correct order (according to the manifest).\n> \n> —\n> You are receiving this because you are subscribed to this thread.\n> Reply to this email directly, view it on GitHub, or mute the thread.\n> \n"}}
{"actor":{"display_login":"cppforlife"}}