{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9658","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9658/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9658/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9658/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9658","id":180395579,"number":9658,"title":"Node freezes up after process restart","user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars.githubusercontent.com/u/7085343?v=3","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-09-30T20:29:39Z","updated_at":"2016-10-11T18:13:55Z","closed_at":null,"body":"This started from a problem @jseldess found while verifying https://github.com/cockroachdb/docs/pull/676. When you have cockroachdb running across 3 VMs in docker containers and tell one to quit, a fresh container comes up to take its place using the same storage. The problem is that SQL queries sent to it hang in the bowels of the storage and kv code.\r\n\r\nIn the past (https://github.com/cockroachdb/docs/pull/676#issuecomment-250778806), I've seen the main query-processing thread and a few others get stuck in `storage.(Replica).redirectOnOrAcquireLease`, but this afternoon's repro has a number of threads (including a query-processing thread) stuck in `kv.(*DistSender).sendToReplicas`.\r\n\r\nI grabbed a bunch of debug info, but was interestingly unable to get info from the `/_status/raft` page because requests to that endpoint also hang.\r\n\r\nReproducing is really easy if you follow the instructions in https://github.com/cockroachdb/docs/pull/676. I haven't yet tried reproducing it on raw VMs rather than using containers.\r\n\r\n@tschottdorf @tamird \r\n\r\n[stopper.txt](https://github.com/cockroachdb/docs/files/503757/stopper.txt)\r\n[goroutines.txt](https://github.com/cockroachdb/docs/files/503758/goroutines.txt)\r\n[gossip.txt](https://github.com/cockroachdb/docs/files/503756/gossip.txt)\r\n[raft-status.txt](https://github.com/cockroachdb/docs/files/503755/raft-status.txt)\r\n[ranges.txt](https://github.com/cockroachdb/docs/files/503754/ranges.txt)\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/252998900","html_url":"https://github.com/cockroachdb/cockroach/issues/9658#issuecomment-252998900","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9658","id":252998900,"user":{"login":"jseldess","id":15893490,"avatar_url":"https://avatars.githubusercontent.com/u/15893490?v=3","gravatar_id":"","url":"https://api.github.com/users/jseldess","html_url":"https://github.com/jseldess","followers_url":"https://api.github.com/users/jseldess/followers","following_url":"https://api.github.com/users/jseldess/following{/other_user}","gists_url":"https://api.github.com/users/jseldess/gists{/gist_id}","starred_url":"https://api.github.com/users/jseldess/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jseldess/subscriptions","organizations_url":"https://api.github.com/users/jseldess/orgs","repos_url":"https://api.github.com/users/jseldess/repos","events_url":"https://api.github.com/users/jseldess/events{/privacy}","received_events_url":"https://api.github.com/users/jseldess/received_events","type":"User","site_admin":false},"created_at":"2016-10-11T18:13:54Z","updated_at":"2016-10-11T18:13:54Z","body":"I just tried passing `--advertise-host` and had the same problem (sql queries hung). Is there anything wrong with this service creation command? \r\n\r\n```\r\nsudo docker service create --replicas 1 --name cockroachdb-2 --network cockroachdb \\\r\n--mount type=volume,source=cockroachdb-2,target=/cockroach/cockroach-data,volume-driver=local \\\r\n--stop-grace-period 60s \\\r\ncockroachdb/cockroach:beta-20161006 start --advertise-host=cockroachdb-2 --logtostderr --insecure --join=cockroachdb-0:26257\r\n```"}}
{"actor":{"display_login":"jseldess"}}