{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/4220","labels_url":"https://api.github.com/repos/coreos/etcd/issues/4220/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/4220/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/4220/events","html_url":"https://github.com/coreos/etcd/issues/4220","id":126866265,"number":4220,"title":"Unable to start after full disk","user":{"login":"Dynom","id":39014,"avatar_url":"https://avatars.githubusercontent.com/u/39014?v=3","gravatar_id":"","url":"https://api.github.com/users/Dynom","html_url":"https://github.com/Dynom","followers_url":"https://api.github.com/users/Dynom/followers","following_url":"https://api.github.com/users/Dynom/following{/other_user}","gists_url":"https://api.github.com/users/Dynom/gists{/gist_id}","starred_url":"https://api.github.com/users/Dynom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dynom/subscriptions","organizations_url":"https://api.github.com/users/Dynom/orgs","repos_url":"https://api.github.com/users/Dynom/repos","events_url":"https://api.github.com/users/Dynom/events{/privacy}","received_events_url":"https://api.github.com/users/Dynom/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-01-15T12:48:03Z","updated_at":"2016-01-15T16:01:36Z","closed_at":null,"body":"It's related to #3591, but that ticket didn't help me, at all. Etcd2 is refusing to start and it's seriously missing information to figure out what to do next. It's highly frustrating.\r\n\r\nI've a cluster of 5 machines. Due to a bug in some software the disk on one node got full. Now Etcd fails to start, claiming a file can't be found, no idea what file it tries to locate.\r\n\r\n* In `snap/` there were several files ending on `.snap.broken`. All were 0 bytes large. I renamed them to .snap, but after a start of Etcd2, they got renamed back to `.snap.broken`. I removed them.\r\n* In `member/` there were several files ending on `.wal` and one ending on `.wal.broken`. I stopped Etcd2, renamed the file to `.wal` and res-started Etcd2. The result is in the logs:\r\n\r\nThe log:\r\n```\r\n$ etcd2 --debug\r\n2016-01-15 12:36:05.944772 I | flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.31.34.46:2379\r\n2016-01-15 12:36:05.944840 I | flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\n2016-01-15 12:36:05.944861 I | flags: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/ff...xx\r\n2016-01-15 12:36:05.944885 I | flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1200\r\n2016-01-15 12:36:05.944913 I | flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.34.46:2380\r\n2016-01-15 12:36:05.944942 I | flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379\r\n2016-01-15 12:36:05.944963 I | flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.31.34.46:2380\r\n2016-01-15 12:36:05.944990 I | flags: recognized and used environment variable ETCD_NAME=39d22783e74d48528fdfe346314733a2\r\n2016-01-15 12:36:05.945028 I | flags: recognized and used environment variable ETCD_PROXY=on\r\n2016-01-15 12:36:05.945115 I | etcdmain: etcd Version: 2.2.0\r\n2016-01-15 12:36:05.945135 I | etcdmain: Git SHA: e4561dd\r\n2016-01-15 12:36:05.945148 I | etcdmain: Go Version: go1.4.2\r\n2016-01-15 12:36:05.945161 I | etcdmain: Go OS/Arch: linux/amd64\r\n2016-01-15 12:36:05.945175 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 2\r\n2016-01-15 12:36:05.945238 N | etcdmain: the server is already initialized as member before, starting as etcd member...\r\n2016-01-15 12:36:05.945310 I | etcdmain: listening for peers on http://172.31.34.46:2380\r\n2016-01-15 12:36:05.945364 I | etcdmain: listening for client requests on http://0.0.0.0:2379\r\n2016-01-15 12:36:05.945606 W | etcdserver: discovery token ignored since a cluster has already been initialized. Valid log found at \"/var/lib/etcd2/member/wal\"\r\n2016-01-15 12:36:05.945653 I | etcdserver: name = 39d22783e74d48528fdfe346314733a2\r\n2016-01-15 12:36:05.945687 I | etcdserver: data dir = /var/lib/etcd2\r\n2016-01-15 12:36:05.945702 I | etcdserver: member dir = /var/lib/etcd2/member\r\n2016-01-15 12:36:05.945716 I | etcdserver: heartbeat = 100ms\r\n2016-01-15 12:36:05.945729 I | etcdserver: election = 1200ms\r\n2016-01-15 12:36:05.945741 I | etcdserver: snapshot count = 10000\r\n2016-01-15 12:36:05.945753 I | etcdserver: discovery URL= https://discovery.etcd.io/ff...xx\r\n2016-01-15 12:36:05.945770 I | etcdserver: advertise client URLs = http://172.31.34.46:2379\r\n2016-01-15 12:36:05.945881 C | etcdserver: open wal error: wal: file not found\r\n```\r\n\r\nNo idea what file Etcd2 tries to load, all files I have are in the directory:\r\n```\r\n# ls -halF /var/lib/etcd2/member/wal\r\ntotal 348M\r\ndrwx------ 2 etcd etcd 4.0K Jan 15 12:36 ./\r\ndrwx------ 4 etcd etcd 4.0K Jan 15 12:36 ../\r\n-rw-r--r-- 1 etcd etcd  42M Oct 29 13:11 000000000000006f-0000000001730472.wal\r\n-rw------- 1 etcd etcd  62M Jan 14 23:02 0000000000000291-00000000087c630f.wal\r\n-rw------- 1 etcd etcd  62M Jan 15 02:26 0000000000000292-00000000087fb746.wal\r\n-rw------- 1 etcd etcd  62M Jan 15 05:50 0000000000000293-0000000008830b7e.wal\r\n-rw------- 1 etcd etcd  62M Jan 15 09:14 0000000000000294-0000000008865f60.wal\r\n-rw------- 1 etcd etcd  33M Jan 15 11:06 0000000000000295-000000000889b34e.wal\r\n```\r\n\r\nOddly, it's the one from Oct 29 that was the broken file:\r\n```\r\n-rw-r--r-- 1 etcd etcd  42M Oct 29 13:11 000000000000006f-0000000001730472.wal.broken\r\n```\r\n\r\nMy highest priority is to get my cluster back in a running state. So my question is: **What to do next?** Preferably I re-introduce this node back to the cluster, otherwise I have to copy lots of data to a new node. In theory it can be done, however.\r\n\r\nI'm currently inclined to start the steps on: https://coreos.com/etcd/docs/latest/runtime-configuration.html#replace-a-failed-machine but I feel that should be too heavy for my situation."},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/172000523","html_url":"https://github.com/coreos/etcd/issues/4220#issuecomment-172000523","issue_url":"https://api.github.com/repos/coreos/etcd/issues/4220","id":172000523,"user":{"login":"Dynom","id":39014,"avatar_url":"https://avatars.githubusercontent.com/u/39014?v=3","gravatar_id":"","url":"https://api.github.com/users/Dynom","html_url":"https://github.com/Dynom","followers_url":"https://api.github.com/users/Dynom/followers","following_url":"https://api.github.com/users/Dynom/following{/other_user}","gists_url":"https://api.github.com/users/Dynom/gists{/gist_id}","starred_url":"https://api.github.com/users/Dynom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dynom/subscriptions","organizations_url":"https://api.github.com/users/Dynom/orgs","repos_url":"https://api.github.com/users/Dynom/repos","events_url":"https://api.github.com/users/Dynom/events{/privacy}","received_events_url":"https://api.github.com/users/Dynom/received_events","type":"User","site_admin":false},"created_at":"2016-01-15T16:01:36Z","updated_at":"2016-01-15T16:01:36Z","body":"I've already worked around the problem by migrating to a new node, so I\r\n* Created a new instance\r\n* Copied the data\r\n* Added it as a member to the cluster\r\n* Deleted the broken member from the cluster\r\n\r\n**I've a question remaining:**\r\nOnce I empty the etcd2 directory, can safely re-add it as a new member on the same IP address? It was a member according to the original discovery URL (on discovery.etcd.io).\r\n"}}