{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8473","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8473/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8473/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8473/events","html_url":"https://github.com/cockroachdb/cockroach/issues/8473","id":170702787,"number":8473,"title":"storage: out of disk space","user":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"assignees":[{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/5","html_url":"https://github.com/cockroachdb/cockroach/milestones/Q3","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/5/labels","id":1875804,"number":5,"title":"Q3","description":"","creator":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"open_issues":122,"closed_issues":247,"state":"open","created_at":"2016-07-11T13:25:51Z","updated_at":"2016-09-29T15:48:43Z","due_on":"2016-10-01T04:00:00Z","closed_at":null},"comments":6,"created_at":"2016-08-11T17:39:27Z","updated_at":"2016-09-29T19:11:35Z","closed_at":null,"body":"A node should fail gracefully when out of disk space."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/250562778","html_url":"https://github.com/cockroachdb/cockroach/issues/8473#issuecomment-250562778","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8473","id":250562778,"user":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T19:11:35Z","updated_at":"2016-09-29T19:11:35Z","body":"> Runaway process (outside of CockroachDB's control) starts sucking up disk space. For example, logging excessively. This can happen rapidly, maybe even faster than we can rebalance ranges away.\r\n\r\nThere's only so much we can do here.  By panicking we allow the admin time to clean up the disk or moving the data to a larger drive before starting the node back up.\r\n\r\n> Poor rebalancing heuristics move too many ranges to a node. This should be considered a bug which we should fix.\r\n\r\nAgreed.  We already have safeguards in place for this, but It would be nice to have some of the other options, including the high/low watermark levels for rebalancing, but as of right now, I don't think they would really help.  Moving to rebalancing based on size instead of range count would go a long way to fixing our issues. \r\n\r\n> Natural range growth. A node with 10,000 32MB ranges (312.5 GB) can see each of them double in size (625 GB) causing it to run out of disk space. Rebalancing can help here as long as there is capacity in the cluster. If the cluster is out of space, panicing seems problematic. How do I read data from a full cluster?\r\n\r\nI see reading data from a cluster that is over capacity as a non-issue at this point.  We might be able to add a mechanism to do so from the command line or some other solution, but for now, my advice would be to copy the data to machines with more disk space and restart the cluster.  This could easily be a ver 2 or 3 feature, but shouldn't be a goal for 1.0.\r\n\r\n> Is there any evidence that we corrupt data when we run out of disk space? I thought the problem was that a node crashed when that happened. RocksDB certainly shouldn't corrupt data if it can't write.\r\n\r\nI went looking and but I can't find clear evidence. But I did find some things that depend on rocksdb that found after disk full errors their checksums were not correct (after recovery).  RocksDB tends to just suggest freeing up space (https://github.com/facebook/rocksdb/issues/919)  But why even let it get that far?  If we drain the node when close to full, it would ensure we do our best to not cross that line.  Obviously, there's no guarantee that we would be able to stop rocksDB from panicking as we only have so much control over disk space.\r\n"}}
{"actor":{"display_login":"BramGruneir"}}