{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/2052","repository_url":"https://api.github.com/repos/hashicorp/nomad","labels_url":"https://api.github.com/repos/hashicorp/nomad/issues/2052/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/nomad/issues/2052/comments","events_url":"https://api.github.com/repos/hashicorp/nomad/issues/2052/events","html_url":"https://github.com/hashicorp/nomad/issues/2052","id":192985966,"node_id":"MDU6SXNzdWUxOTI5ODU5NjY=","number":2052,"title":"Need a way to cleanly shut down nodes","user":{"login":"blalor","id":109915,"node_id":"MDQ6VXNlcjEwOTkxNQ==","avatar_url":"https://avatars0.githubusercontent.com/u/109915?v=4","gravatar_id":"","url":"https://api.github.com/users/blalor","html_url":"https://github.com/blalor","followers_url":"https://api.github.com/users/blalor/followers","following_url":"https://api.github.com/users/blalor/following{/other_user}","gists_url":"https://api.github.com/users/blalor/gists{/gist_id}","starred_url":"https://api.github.com/users/blalor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blalor/subscriptions","organizations_url":"https://api.github.com/users/blalor/orgs","repos_url":"https://api.github.com/users/blalor/repos","events_url":"https://api.github.com/users/blalor/events{/privacy}","received_events_url":"https://api.github.com/users/blalor/received_events","type":"User","site_admin":false},"labels":[{"id":321148368,"node_id":"MDU6TGFiZWwzMjExNDgzNjg=","url":"https://api.github.com/repos/hashicorp/nomad/labels/client","name":"client","color":"bfdadc","default":false},{"id":218895785,"node_id":"MDU6TGFiZWwyMTg4OTU3ODU=","url":"https://api.github.com/repos/hashicorp/nomad/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":267728415,"node_id":"MDU6TGFiZWwyNjc3Mjg0MTU=","url":"https://api.github.com/repos/hashicorp/nomad/labels/thinking","name":"thinking","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-01T22:55:40Z","updated_at":"2018-09-12T16:29:14Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Nomad v0.5.0\r\n\r\nThere doesn't appear to be a way to cleanly shut down a client node in a way that allows allocations to be moved to other nodes and also accounts for the data in sticky ephemeral disks to be migrated.  I wrote a script to help my systemd service delay stopping the service until allocations have been moved, but there doesn't appear to be a way to monitor for the status of the migrated data.  If the data's not moved quickly, it could be lost when the node shuts down.\r\n\r\nSomething like `nomad shutdown` that blocks until the agent is completely idle would be ideal."},"comment":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/comments/420712799","html_url":"https://github.com/hashicorp/nomad/issues/2052#issuecomment-420712799","issue_url":"https://api.github.com/repos/hashicorp/nomad/issues/2052","id":420712799,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDcxMjc5OQ==","user":{"login":"dcparker88","id":1847494,"node_id":"MDQ6VXNlcjE4NDc0OTQ=","avatar_url":"https://avatars1.githubusercontent.com/u/1847494?v=4","gravatar_id":"","url":"https://api.github.com/users/dcparker88","html_url":"https://github.com/dcparker88","followers_url":"https://api.github.com/users/dcparker88/followers","following_url":"https://api.github.com/users/dcparker88/following{/other_user}","gists_url":"https://api.github.com/users/dcparker88/gists{/gist_id}","starred_url":"https://api.github.com/users/dcparker88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcparker88/subscriptions","organizations_url":"https://api.github.com/users/dcparker88/orgs","repos_url":"https://api.github.com/users/dcparker88/repos","events_url":"https://api.github.com/users/dcparker88/events{/privacy}","received_events_url":"https://api.github.com/users/dcparker88/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T16:29:14Z","updated_at":"2018-09-12T16:29:14Z","author_association":"NONE","body":"Not sure if this is relevant or related - but even when I have `leave_on_terminate` set in my config - it doesn't seem to fully leave. I've been doing some testing with the stuff above, and I can see in my logs the node is cleanly shutting down:\r\n\r\n```\r\nnomad: ==> Caught signal: terminated\r\nnomad: ==> Gracefully shutting down agent...\r\nnomad[2525]: agent: requesting shutdown\r\nnomad: 2018/09/12 16:11:52.306399 [INFO] agent: requesting shutdown\r\nnomad: 2018/09/12 16:11:52.306468 [INFO] client: shutting down\r\nnomad[2525]: client: shutting down\r\nnomad: 2018/09/12 16:11:52.320998 [INFO] agent: shutdown complete\r\nnomad[2525]: agent: shutdown complete\r\n```\r\n\r\nbut when I check `nomad node status` it still shows down:\r\n```\r\n$ nomad node status\r\nID        DC                 Name   Class   Drain  Eligibility  Status\r\n4995dacd  east     agent1     <none>  false  ineligible   down\r\n```\r\n\r\nis that expected behavior? I would expect once the node leaves the cluster it doesn't appear in the status anymore. "}}
{"actor":{"display_login":"dcparker88"}}