{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9545","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9545/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9545/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9545/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9545","id":179295890,"number":9545,"title":"stability: drastically reducing range_max_bytes on a running cluster causes stall","user":{"login":"tamird","id":1535036,"avatar_url":"https://avatars.githubusercontent.com/u/1535036?v=3","gravatar_id":"","url":"https://api.github.com/users/tamird","html_url":"https://github.com/tamird","followers_url":"https://api.github.com/users/tamird/followers","following_url":"https://api.github.com/users/tamird/following{/other_user}","gists_url":"https://api.github.com/users/tamird/gists{/gist_id}","starred_url":"https://api.github.com/users/tamird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tamird/subscriptions","organizations_url":"https://api.github.com/users/tamird/orgs","repos_url":"https://api.github.com/users/tamird/repos","events_url":"https://api.github.com/users/tamird/events{/privacy}","received_events_url":"https://api.github.com/users/tamird/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"jordanlewis","id":43821,"avatar_url":"https://avatars.githubusercontent.com/u/43821?v=3","gravatar_id":"","url":"https://api.github.com/users/jordanlewis","html_url":"https://github.com/jordanlewis","followers_url":"https://api.github.com/users/jordanlewis/followers","following_url":"https://api.github.com/users/jordanlewis/following{/other_user}","gists_url":"https://api.github.com/users/jordanlewis/gists{/gist_id}","starred_url":"https://api.github.com/users/jordanlewis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordanlewis/subscriptions","organizations_url":"https://api.github.com/users/jordanlewis/orgs","repos_url":"https://api.github.com/users/jordanlewis/repos","events_url":"https://api.github.com/users/jordanlewis/events{/privacy}","received_events_url":"https://api.github.com/users/jordanlewis/received_events","type":"User","site_admin":false},"assignees":[{"login":"jordanlewis","id":43821,"avatar_url":"https://avatars.githubusercontent.com/u/43821?v=3","gravatar_id":"","url":"https://api.github.com/users/jordanlewis","html_url":"https://github.com/jordanlewis","followers_url":"https://api.github.com/users/jordanlewis/followers","following_url":"https://api.github.com/users/jordanlewis/following{/other_user}","gists_url":"https://api.github.com/users/jordanlewis/gists{/gist_id}","starred_url":"https://api.github.com/users/jordanlewis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordanlewis/subscriptions","organizations_url":"https://api.github.com/users/jordanlewis/orgs","repos_url":"https://api.github.com/users/jordanlewis/repos","events_url":"https://api.github.com/users/jordanlewis/events{/privacy}","received_events_url":"https://api.github.com/users/jordanlewis/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2016-09-26T17:58:47Z","updated_at":"2016-09-28T12:52:05Z","closed_at":null,"body":"1. `roachdemo -n 3`\r\n1. `block_writer`\r\n1. get a coffee\r\n1. `echo \"range_max_bytes: 1048577\" | cockroach zone set datablocks --file=-`\r\n1. watch block_writer, it will stall\r\n1. `tail cockroach-data/1/logs/cockroach.INFO`:\r\n\r\n\t```\r\n\tI160926 13:54:56.948066 69 storage/replica_raftstorage.go:290  [n1,s1,r401:/Table/51/1/{177428…-230647…}]: not generating snapshot because replica is too large: 21944985 > 2 * 1048577\r\n\tI160926 13:54:56.948152 71 storage/replica_raftstorage.go:290  [n1,s1,r401:/Table/51/1/{177428…-230647…}]: not generating snapshot because replica is too large: 21944985 > 2 * 1048577\r\n\tI160926 13:54:57.546802 74 storage/replica_raftstorage.go:290  [n1,s1,r410:/Table/51/1/{756457…-807945…}]: not generating snapshot because replica is too large: 21395821 > 2 * 1048577\r\n\tI160926 13:54:57.546940 73 storage/replica_raftstorage.go:290  [n1,s1,r409:/{Table/51/1/8…-Max}]: not generating snapshot because replica is too large: 21410643 > 2 * 1048577\r\n\tI160926 13:54:57.547118 74 storage/replica_raftstorage.go:290  [n1,s1,r412:/Table/51/1/{653153…-114755…}]: not generating snapshot because replica is too large: 20730984 > 2 * 1048577\r\n\tI160926 13:54:57.793041 75 storage/replica_raftstorage.go:290  [n1,s1,r401:/Table/51/1/{177428…-230647…}]: not generating snapshot because replica is too large: 21944985 > 2 * 1048577\r\n\tI160926 13:54:57.793060 75 storage/replica_raftstorage.go:290  [n1,s1,r401:/Table/51/1/{177428…-230647…}]: not generating snapshot because replica is too large: 21944985 > 2 * 1048577\r\n\tI160926 13:54:57.949481 73 storage/replica_raftstorage.go:290  [n1,s1,r401:/Table/51/1/{177428…-230647…}]: not generating snapshot because replica is too large: 21944985 > 2 * 1048577\r\n\tI160926 13:54:57.949497 68 storage/replica_raftstorage.go:290  [n1,s1,r413:/Table/51/1/{297339…-347337…}]: not generating snapshot because replica is too large: 20984435 > 2 * 1048577\r\n\tI160926 13:54:57.949515 73 storage/replica_raftstorage.go:290  [n1,s1,r401:/Table/51/1/{177428…-230647…}]: not generating snapshot because replica is too large: 21944985 > 2 * 1048577\r\n\t```\r\n1. Profit\r\n\r\nSeems like we're declining raft-requested snapshots, which is very bad indeed. The following diff seems to fix the problem (but needs tests):\r\n```diff\r\ndiff --git a/storage/replica_raftstorage.go b/storage/replica_raftstorage.go\r\nindex c4e9a20..71b9044 100644\r\n--- a/storage/replica_raftstorage.go\r\n+++ b/storage/replica_raftstorage.go\r\n@@ -269,7 +269,7 @@ func (r *Replica) GetFirstIndex() (uint64, error) {\r\n // Snapshot implements the raft.Storage interface.\r\n // Snapshot requires that the replica lock is held.\r\n func (r *Replica) Snapshot() (raftpb.Snapshot, error) {\r\n-\tsnap, err := r.SnapshotWithContext(context.Background())\r\n+\tsnap, err := r.SnapshotWithContext(context.Background(), false)\r\n \tif err != nil {\r\n \t\treturn raftpb.Snapshot{}, err\r\n \t}\r\n@@ -279,10 +279,10 @@ func (r *Replica) Snapshot() (raftpb.Snapshot, error) {\r\n // SnapshotWithContext is main implementation for Snapshot() but it takes a\r\n // context to allow tracing. If this method returns without error, callers\r\n // must eventually call CloseOutSnap to ready this replica for more snapshots.\r\n-func (r *Replica) SnapshotWithContext(ctx context.Context) (*OutgoingSnapshot, error) {\r\n+func (r *Replica) SnapshotWithContext(ctx context.Context, checkSize bool) (*OutgoingSnapshot, error) {\r\n \trangeID := r.RangeID\r\n \r\n-\tif r.exceedsDoubleSplitSizeLocked() {\r\n+\tif checkSize && r.exceedsDoubleSplitSizeLocked() {\r\n \t\tmaxBytes := r.mu.maxBytes\r\n \t\tsize := r.mu.state.Stats.Total()\r\n \t\tlog.Infof(ctx,\r\n@@ -341,7 +341,7 @@ func (r *Replica) GetSnapshot(ctx context.Context) (*OutgoingSnapshot, error) {\r\n \t\t<-doneChan\r\n \r\n \t\tr.mu.Lock()\r\n-\t\tsnap, err := r.SnapshotWithContext(ctx)\r\n+\t\tsnap, err := r.SnapshotWithContext(ctx, true)\r\n \t\tif err == nil {\r\n \t\t\tr.mu.outSnap.claimed = true\r\n \t\t}\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/250157490","html_url":"https://github.com/cockroachdb/cockroach/issues/9545#issuecomment-250157490","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9545","id":250157490,"user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2016-09-28T12:52:05Z","updated_at":"2016-09-28T12:52:05Z","body":"Ah, got it. This looks good then."}}
{"actor":{"display_login":"petermattis"}}