{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/385","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/385/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/385/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/385/events","html_url":"https://github.com/scylladb/scylla/issues/385","id":107481454,"number":385,"title":"OOM during cassandra-stress write test","user":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":72,"closed_issues":222,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2016-03-01T14:48:35Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2015-09-21T10:09:28Z","updated_at":"2016-03-01T14:53:20Z","closed_at":null,"body":"I'm running a 2-cpu Scylla nodes on the same machine, with 1GB of memory each, and adding 11 million partitions to both nodes (RF=2) with a cassandra-stress write run:\r\n\r\n```\r\n/home/nyh/cassandra/cassandra.git/tools/bin/cassandra-stress write n=11000000 -schema \"replication(strategy=org.apache.cassandra.locator.SimpleStrategy,factor=2)\" \r\n```\r\n\r\nThis eventually (after about 8 million partitions) runs out-of-memory:\r\n\r\n```\r\nERROR [shard 0] database - failed to write sstable: std::bad_alloc\r\nERROR [shard 0] compaction_manager - compaction failed: std::bad_alloc\r\nERROR [shard 0] database - failed to write sstable: std::bad_alloc\r\nscylla: sstables/sstables.cc:1129: void sstables::prepare_summary(sstables::summary&, uint64_t): Assertion `expected_partition_count >= 1' failed.\r\n```\r\n\r\nI can reproduce this problem, and it always fails during the compaction phase. \r\nLooking at the sstable directory at the time at the crash, we have generation 127 currently written by a compaction (so it still has a tmp TOC),  generations 128 through 133 have all been successfully flushed to disk (but not yet compacted), generation 134 is partially flushed, and generation 135 has empty files - my guess (?) is that this is either an attempt to start a second compaction in parallel, or an attempt to start flushing another memtable while the previous one is still being flushed.\r\n\r\nWhen I try to do the same thing with only one node, it does *not* run out of memory, which is surprising because one would expect that in the two-node test, each node does the same as the single node, so the single node would run out of memory in the same way - but it doesn't.\r\n\r\nSo my suspicion (not yet completely founded) is that I have a problem with backlog of unflushed memtables which fill the memory: The two nodes run on the same machine (my laptop) so share the same disk. With a single node, I see in iostat a peak write of about 25 MB/sec, so it is possible (?) my laptop's SSDs cannot handle the full write speed of two nodes (the maximum I'm seeing is 35 MB/sec), and while two nodes are busy doing a long compaction at full speed, they cannot flush memtables quickly enough and those start to pile up and deplete the available memory. Again, this is just a guess."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/190754733","html_url":"https://github.com/scylladb/scylla/issues/385#issuecomment-190754733","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/385","id":190754733,"user":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"created_at":"2016-03-01T14:53:20Z","updated_at":"2016-03-01T14:53:20Z","body":"@slivne, #642 is unrelated, because in that issue the problem was that we didn't leave enough memory *outside* scylla, causing the kernel to kill random programs including scylla. In this issue my problem was that scylla ran out of its own memory.\r\n\r\nI'm trying to reproduce this problem with the latest scylla, and will update whether I can still reproduce it, or it appears to have been fixed."}}