{"action":"closed","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/993","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/993/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/993/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/993/events","html_url":"https://github.com/scylladb/scylla/issues/993","id":138733531,"number":993,"title":"memory misacounting can lead to database hanging","user":{"login":"glommer","id":331197,"avatar_url":"https://avatars.githubusercontent.com/u/331197?v=3","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/showstopper","name":"showstopper","color":"e11d21"}],"state":"closed","locked":false,"assignee":{"login":"pdziepak","id":6995,"avatar_url":"https://avatars.githubusercontent.com/u/6995?v=3","gravatar_id":"","url":"https://api.github.com/users/pdziepak","html_url":"https://github.com/pdziepak","followers_url":"https://api.github.com/users/pdziepak/followers","following_url":"https://api.github.com/users/pdziepak/following{/other_user}","gists_url":"https://api.github.com/users/pdziepak/gists{/gist_id}","starred_url":"https://api.github.com/users/pdziepak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdziepak/subscriptions","organizations_url":"https://api.github.com/users/pdziepak/orgs","repos_url":"https://api.github.com/users/pdziepak/repos","events_url":"https://api.github.com/users/pdziepak/events{/privacy}","received_events_url":"https://api.github.com/users/pdziepak/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":80,"closed_issues":236,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2016-03-07T09:15:39Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-03-05T23:30:00Z","updated_at":"2016-03-07T09:15:39Z","closed_at":"2016-03-07T09:15:39Z","body":"When running scylla without the row cache, the database can hang.\r\n\r\nMy command line is as follows:\r\n\r\n```\r\nCPU=$1\r\nIP=\"127.0.0.$(($CPU+1))\"\r\nSEED=\"127.0.0.1\"\r\n$GDB ./build/release/scylla --enable-cache 0 --collectd 0 --smp 1 --cpuset $CPU -m 512M --api-address $IP --rpc-address $IP --listen-address $IP --seed-provider-parameters seeds=$SEED --datadir /var/lib/scylla/data/$CPU --commitlog-directory /var/lib/scylla/commitlog/$CPU --max-io-requests 10 | tee /tmp/log.$IP.txt\r\n```\r\nThe following load is then applied:\r\n\r\n```\r\ncassandra-stress write cl=ONE n=1000000 -rate threads=10  -node 127.0.0.1 -pop seq=1..1000000 -col 'size=FIXED(64) n=FIXED(5)'; \r\n```\r\n\r\nThe database hangs on database::throttle(), and when it does so, I can verify that `_dirty_memory_region_group.memory_used()` is negative (therefore converted to a large number).\r\n\r\nBecause this is a hard condition to verify, I have sprinkled some assertions in the logalloc to try to catch the moment where a region counter goes negative. It always seem to happen at the memtable destructor.\r\n\r\n### Conditions affecting this issue:\r\n\r\n- When running the cache, this issue doesn't show. The first step update_cache takes is to merge the memtable region's with its own. That step is the only step necessary to hide the problem - indicating that it is indeed an accounting bug. It is probably just hiding the issue, though. The cache region being bigger, it can hold a negative new region without going negative itself.\r\n\r\n- Giving the shard more memory makes the issue harder to reproduce. Giving the shard less memory makes the issue easier to reproduce. That made me suspect region compaction, but I have seen no region compactions taking place. Most likely, less memory just makes memtables getting flushed more often.\r\n\r\n\r\n\r\n\r\n"}}