{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1188","repository_url":"https://api.github.com/repos/hashicorp/consul","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1188/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1188/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1188/events","html_url":"https://github.com/hashicorp/consul/issues/1188","id":102227051,"node_id":"MDU6SXNzdWUxMDIyMjcwNTE=","number":1188,"title":"Unable to deregister a service","user":{"login":"drsnyder","id":21621,"node_id":"MDQ6VXNlcjIxNjIx","avatar_url":"https://avatars3.githubusercontent.com/u/21621?v=4","gravatar_id":"","url":"https://api.github.com/users/drsnyder","html_url":"https://github.com/drsnyder","followers_url":"https://api.github.com/users/drsnyder/followers","following_url":"https://api.github.com/users/drsnyder/following{/other_user}","gists_url":"https://api.github.com/users/drsnyder/gists{/gist_id}","starred_url":"https://api.github.com/users/drsnyder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drsnyder/subscriptions","organizations_url":"https://api.github.com/users/drsnyder/orgs","repos_url":"https://api.github.com/users/drsnyder/repos","events_url":"https://api.github.com/users/drsnyder/events{/privacy}","received_events_url":"https://api.github.com/users/drsnyder/received_events","type":"User","site_admin":false},"labels":[{"id":64219391,"node_id":"MDU6TGFiZWw2NDIxOTM5MQ==","url":"https://api.github.com/repos/hashicorp/consul/labels/bug","name":"bug","color":"fc2929","default":true},{"id":612601191,"node_id":"MDU6TGFiZWw2MTI2MDExOTE=","url":"https://api.github.com/repos/hashicorp/consul/labels/theme/operator-usability","name":"theme/operator-usability","color":"c5def5","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/hashicorp/consul/milestones/16","html_url":"https://github.com/hashicorp/consul/milestone/16","labels_url":"https://api.github.com/repos/hashicorp/consul/milestones/16/labels","id":2844406,"node_id":"MDk6TWlsZXN0b25lMjg0NDQwNg==","number":16,"title":"Unplanned","description":"Issues in here have been triaged by someone on the Consul team and may be addressed in a future release of Consul, but aren't yet slated for a particular version of Consul. Issues will move from here onto a milestone for a particular release.","creator":{"login":"slackpad","id":673509,"node_id":"MDQ6VXNlcjY3MzUwOQ==","avatar_url":"https://avatars0.githubusercontent.com/u/673509?v=4","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"open_issues":140,"closed_issues":18,"state":"open","created_at":"2017-10-17T19:53:25Z","updated_at":"2018-08-09T13:43:19Z","due_on":null,"closed_at":null},"comments":64,"created_at":"2015-08-20T20:14:01Z","updated_at":"2018-08-23T02:52:35Z","closed_at":"2018-02-27T18:21:54Z","author_association":"NONE","body":"I brought this to the attention of the mailing list [here](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/consul-tool/1wEHSx3Qp-Y). @slackpad asked me to go ahead and file a bug. Below summary of the issue from the discussion thread.\n\nWe have services that are being orphaned and we cannot deregister them. The orphans show up under one or more of the master nodes. In our configuration the master nodes are `dev-consul`, `dev-consul-s1`, and `dev-broker`.\n\nThe health check of the orphaned node looks something like the following:\n\n```\n{\n    \"Node\": \"dev-consul\",\n    \"CheckID\": \"service:discussion_8080\",\n    \"Name\": \"Service 'discussion' check\",\n    \"ServiceName\": \"discussion\",\n    \"Notes\": \"\",\n    \"Status\": \"critical\",\n    \"ServiceID\": \"discussion_8080\",\n    \"Output\": \"\"\n}\n```\n\nI attempted to deregister via:\n\n```\nuser@dev-consul $ curl -X PUT -d '{\"CheckID\": \"service:discussion_8080\", \"ServiceID\": \"discussion_8080\", \"Node\": \"dev-consul\", \"Datacenter\": \"dev\"}' http://localhost:8500/v1/catalog/deregister\n```\n\nThe node was removed but then reappears within 30-60s. As @slackpad's recommended, I tried deregistering with:\n\n```\nuser@dev-consul $ curl -v http://localhost:8500/v1/agent/service/deregister/discussion_8080\nuser@dev-consul $ curl -v -X PUT -d'{\"CheckID\": \"service:discussion_8080\", \"ServiceID\": \"discussion_8080\", \"Node\": \"dev-consul\", \"Datacenter\": \"dev\"}' http://localhost:8500/v1/catalog/deregister\n```\n\nBoth commands returned status `200 OK`. But that also failed. You can see the output in this [gist](https://gist.github.com/drsnyder/3016efb02efebc6afbe6) as well as the debug logs from consul.\n\nFrom the debug logs in consul we see:\n\n```\nAug 20 16:57:45 dev-broker consul[2221]: agent: Deregistered service 'discussion_8080'\nAug 20 16:57:45 dev-broker consul[2221]: agent: Check 'service:discussion_8080' in sync\nAug 20 16:57:45 dev-broker consul[2221]: agent: Deregistered check 'service:discussion_8080'\nAug 20 16:57:45 dev-broker consul[2221]: http: Request /v1/agent/service/deregister/discussion_8080 (19.73968ms)\nAug 20 16:57:46 dev-broker consul[2221]: http: Request /v1/agent/check/pass/service:discussion_8080, error: CheckID does not have associated TTL\nAug 20 16:57:46 dev-broker consul[2221]: http: Request /v1/agent/check/pass/service:discussion_8080 (246.298µs)\nAug 20 16:57:47 dev-broker consul[2221]: agent: Synced service 'discussion_8080' <--- SHADY!\n```\n\nThe annotation is from @slackpad. \n\nIt's also noteworthy that the orphans are always associated with one of the master nodes (e.g. `dev-consul`) and not the node (`dev-mesos`) that's running the service that was registered. I should also mention (it could be a coincidence) that the service (`discussion`) is also flapping though from what I can tell from the debug logs for consul on `dev-mesos` everything is fine.\n\nOur consul version:\n\n```\n$ consul version\nConsul v0.5.2\nConsul Protocol: 2 (Understands back to: 1)\n```\n\nThanks!\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/415265880","html_url":"https://github.com/hashicorp/consul/issues/1188#issuecomment-415265880","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1188","id":415265880,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTI2NTg4MA==","user":{"login":"mritd","id":13043245,"node_id":"MDQ6VXNlcjEzMDQzMjQ1","avatar_url":"https://avatars2.githubusercontent.com/u/13043245?v=4","gravatar_id":"","url":"https://api.github.com/users/mritd","html_url":"https://github.com/mritd","followers_url":"https://api.github.com/users/mritd/followers","following_url":"https://api.github.com/users/mritd/following{/other_user}","gists_url":"https://api.github.com/users/mritd/gists{/gist_id}","starred_url":"https://api.github.com/users/mritd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mritd/subscriptions","organizations_url":"https://api.github.com/users/mritd/orgs","repos_url":"https://api.github.com/users/mritd/repos","events_url":"https://api.github.com/users/mritd/events{/privacy}","received_events_url":"https://api.github.com/users/mritd/received_events","type":"User","site_admin":false},"created_at":"2018-08-23T02:52:35Z","updated_at":"2018-08-23T02:52:35Z","author_association":"NONE","body":"The reason I found this problem is: \r\n\r\n**Deregistered with a client of another agent**\r\n\r\nIn this case the deregister service will be resynchronized, so I created a small tool(https://github.com/Gozap/cclean) to clean up the unhealthy service."}}
{"actor":{"display_login":"mritd"}}