{"action":"closed","issue":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/4520","repository_url":"https://api.github.com/repos/hashicorp/nomad","labels_url":"https://api.github.com/repos/hashicorp/nomad/issues/4520/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/nomad/issues/4520/comments","events_url":"https://api.github.com/repos/hashicorp/nomad/issues/4520/events","html_url":"https://github.com/hashicorp/nomad/issues/4520","id":342396455,"node_id":"MDU6SXNzdWUzNDIzOTY0NTU=","number":4520,"title":"Nomad has running deployments for non existant jobs","user":{"login":"Magical-Chicken","id":9562766,"node_id":"MDQ6VXNlcjk1NjI3NjY=","avatar_url":"https://avatars2.githubusercontent.com/u/9562766?v=4","gravatar_id":"","url":"https://api.github.com/users/Magical-Chicken","html_url":"https://github.com/Magical-Chicken","followers_url":"https://api.github.com/users/Magical-Chicken/followers","following_url":"https://api.github.com/users/Magical-Chicken/following{/other_user}","gists_url":"https://api.github.com/users/Magical-Chicken/gists{/gist_id}","starred_url":"https://api.github.com/users/Magical-Chicken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Magical-Chicken/subscriptions","organizations_url":"https://api.github.com/users/Magical-Chicken/orgs","repos_url":"https://api.github.com/users/Magical-Chicken/repos","events_url":"https://api.github.com/users/Magical-Chicken/events{/privacy}","received_events_url":"https://api.github.com/users/Magical-Chicken/received_events","type":"User","site_admin":false},"labels":[{"id":656423475,"node_id":"MDU6TGFiZWw2NTY0MjM0NzU=","url":"https://api.github.com/repos/hashicorp/nomad/labels/deployments","name":"deployments","color":"5319e7","default":false},{"id":266756295,"node_id":"MDU6TGFiZWwyNjY3NTYyOTU=","url":"https://api.github.com/repos/hashicorp/nomad/labels/waiting-reply","name":"waiting-reply","color":"d4c5f9","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-07-18T16:10:05Z","updated_at":"2018-07-18T17:18:24Z","closed_at":"2018-07-18T17:18:24Z","author_association":"NONE","body":"Nomad has running deployments listed for jobs which were stopped several months ago and have long since been garbage collected. It should not be possible to have running deployments for non-existent jobs, they should have been canceled when the job was stopped and garbage collected.\r\n\r\nNomad Version: `Nomad v0.8.3 (c85483da3471f4bd3a7c3de112e95f551071769f)`\r\n\r\nThis causes error messages in the nomad server logs such as:\r\n```\r\nJul 17 22:13:15 ip-____________.us-west-2.compute.internal nomad[17598]: 2018/07/17 22:13:15.576347 [ERR] nomad.deployments_watcher: failed to track deployment \"80ea9bf5-5eb6-9555-e94a-af6920c2cfbc\": deployment \"80ea9bf5-5eb6-9555-e94a-af6920c2cfbc\" references unknown job \"zoolander-qa-_________________-job\"\r\n\r\nJul 17 22:13:15 ip-____________.us-west-2.compute.internal nomad[17598]: 2018/07/17 22:13:15.576433 [ERR] nomad.deployments_watcher: failed to track deployment \"952f1c85-fe80-8c73-9022-094d5143912e\": deployment \"952f1c85-fe80-8c73-9022-094d5143912e\" references unknown job \"zoolander-qa-_________________-job\"\r\n\r\nJul 17 22:13:15 ip-____________.us-west-2.compute.internal nomad[17598]: 2018/07/17 22:13:15.576485 [ERR] nomad.deployments_watcher: failed to track deployment \"a80eed84-068e-7b41-f858-911f80130fa9\": deployment \"a80eed84-068e-7b41-f858-911f80130fa9\" references unknown job \"zoolander-qa-_________________-job\"\r\n```\r\n\r\n```\r\n-[w-vm]- -[~]-\r\n-[10:39 AM]-nomad deployment status 952f1c85-fe80-8c73-9022-094d5143912e\r\nID          = 952f1c85\r\nJob ID      = zoolander-qa-_________________-job\r\nJob Version = 1\r\nStatus      = running\r\nDescription = Deployment is running\r\n\r\nDeployed\r\nTask Group  Desired  Placed  Healthy  Unhealthy\r\nservice     1        0       0        0\r\n-[w-vm]- -[~]-\r\n-[10:41 AM]-nomad status zoolander-qa-_________________-job\r\nUnable to resolve ID: \"zoolander-qa-_________________-job\"\r\n-[w-vm]- -[~]-\r\n-[10:41 AM]-curl ${NOMAD_ADDR}/v1/deployment/952f1c85-fe80-8c73-9022-094d5143912e | jq\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100   460  100   460    0     0   4509      0 --:--:-- --:--:-- --:--:--  4554\r\n{\r\n  \"CreateIndex\": 1031168,\r\n  \"ID\": \"952f1c85-fe80-8c73-9022-094d5143912e\",\r\n  \"JobCreateIndex\": 699544,\r\n  \"JobID\": \"zoolander-qa-_________________-job\",\r\n  \"JobModifyIndex\": 1029796,\r\n  \"JobVersion\": 1,\r\n  \"ModifyIndex\": 1031168,\r\n  \"Namespace\": \"default\",\r\n  \"Status\": \"running\",\r\n  \"StatusDescription\": \"Deployment is running\",\r\n  \"TaskGroups\": {\r\n    \"service\": {\r\n      \"AutoRevert\": false,\r\n      \"DesiredCanaries\": 0,\r\n      \"DesiredTotal\": 1,\r\n      \"HealthyAllocs\": 0,\r\n      \"PlacedAllocs\": 0,\r\n      \"PlacedCanaries\": null,\r\n      \"Promoted\": false,\r\n      \"UnhealthyAllocs\": 0\r\n    }\r\n  }\r\n}\r\n-[w-vm]- -[~]-\r\n-[10:42 AM]-curl ${NOMAD_ADDR}/v1/job/zoolander-qa-_________________-job\r\njob not found-[w-vm]- -[~]-\r\n-[10:42 AM]-\r\n-[w-vm]- -[~]-\r\n-[10:42 AM]-curl ${NOMAD_ADDR}/v1/job/zoolander-qa-_________________-job/deployments | jq\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100  6455    0  6455    0     0  50826      0 --:--:-- --:--:-- --:--:-- 51230\r\n[\r\n  {\r\n    \"CreateIndex\": 1030352,\r\n    \"ID\": \"0ec29870-d92d-2c81-e02a-ab398edc353c\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030352,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030298,\r\n    \"ID\": \"18e52c00-9609-2cf7-5375-19b5d06bfd18\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030298,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030321,\r\n    \"ID\": \"4cb195c7-8130-e5ef-cbb7-fe51d7b95a62\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030321,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030335,\r\n    \"ID\": \"4ec10131-5ba3-1ee7-81c3-38a684366dfa\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030335,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030348,\r\n    \"ID\": \"513f5a36-a590-30a8-453c-90f291f3027a\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030348,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030311,\r\n    \"ID\": \"5914c2db-7358-2c8a-334a-ea2114e62943\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030311,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030273,\r\n    \"ID\": \"5f53097b-7f48-02d4-197a-263aff49a2ff\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 1,\r\n    \"ModifyIndex\": 1030273,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030051,\r\n    \"ID\": \"7bb4e814-67cd-acc3-6601-441890a3a1d8\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030051,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030383,\r\n    \"ID\": \"80ea9bf5-5eb6-9555-e94a-af6920c2cfbc\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030383,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1031168,\r\n    \"ID\": \"952f1c85-fe80-8c73-9022-094d5143912e\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 1,\r\n    \"ModifyIndex\": 1031168,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030675,\r\n    \"ID\": \"a80eed84-068e-7b41-f858-911f80130fa9\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 1,\r\n    \"ModifyIndex\": 1030675,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1031087,\r\n    \"ID\": \"af867b89-1faf-9588-420e-684576ed0ebf\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 1,\r\n    \"ModifyIndex\": 1031087,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030308,\r\n    \"ID\": \"d4b0783e-84fe-0671-49fe-19b386323e25\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030308,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  },\r\n  {\r\n    \"CreateIndex\": 1030317,\r\n    \"ID\": \"f862ad54-e422-8e66-7af7-7e6aa8982843\",\r\n    \"JobCreateIndex\": 699544,\r\n    \"JobID\": \"zoolander-qa-_________________-job\",\r\n    \"JobModifyIndex\": 1029796,\r\n    \"JobVersion\": 2,\r\n    \"ModifyIndex\": 1030317,\r\n    \"Namespace\": \"default\",\r\n    \"Status\": \"running\",\r\n    \"StatusDescription\": \"Deployment is running\",\r\n    \"TaskGroups\": {\r\n      \"service\": {\r\n        \"AutoRevert\": false,\r\n        \"DesiredCanaries\": 0,\r\n        \"DesiredTotal\": 1,\r\n        \"HealthyAllocs\": 0,\r\n        \"PlacedAllocs\": 0,\r\n        \"PlacedCanaries\": null,\r\n        \"Promoted\": false,\r\n        \"UnhealthyAllocs\": 0\r\n      }\r\n    }\r\n  }\r\n]\r\n```\r\n\r\n"}}
{"actor":{"display_login":"dadgar"}}