{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3155","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3155/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3155/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3155/events","html_url":"https://github.com/scylladb/scylla/issues/3155","id":291757804,"number":3155,"title":"Failure to start (after stopping) a scylla node in a cluster with 1000 keyspaces & tables - commitlog_replayer - bad_alloc","user":{"login":"roydahan","id":20959584,"avatar_url":"https://avatars3.githubusercontent.com/u/20959584?v=4","gravatar_id":"","url":"https://api.github.com/users/roydahan","html_url":"https://github.com/roydahan","followers_url":"https://api.github.com/users/roydahan/followers","following_url":"https://api.github.com/users/roydahan/following{/other_user}","gists_url":"https://api.github.com/users/roydahan/gists{/gist_id}","starred_url":"https://api.github.com/users/roydahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roydahan/subscriptions","organizations_url":"https://api.github.com/users/roydahan/orgs","repos_url":"https://api.github.com/users/roydahan/repos","events_url":"https://api.github.com/users/roydahan/events{/privacy}","received_events_url":"https://api.github.com/users/roydahan/received_events","type":"User","site_admin":false},"labels":[{"id":160570956,"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420","default":true}],"state":"open","locked":false,"assignee":{"login":"elcallio","id":7113590,"avatar_url":"https://avatars1.githubusercontent.com/u/7113590?v=4","gravatar_id":"","url":"https://api.github.com/users/elcallio","html_url":"https://github.com/elcallio","followers_url":"https://api.github.com/users/elcallio/followers","following_url":"https://api.github.com/users/elcallio/following{/other_user}","gists_url":"https://api.github.com/users/elcallio/gists{/gist_id}","starred_url":"https://api.github.com/users/elcallio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elcallio/subscriptions","organizations_url":"https://api.github.com/users/elcallio/orgs","repos_url":"https://api.github.com/users/elcallio/repos","events_url":"https://api.github.com/users/elcallio/events{/privacy}","received_events_url":"https://api.github.com/users/elcallio/received_events","type":"User","site_admin":false},"assignees":[{"login":"elcallio","id":7113590,"avatar_url":"https://avatars1.githubusercontent.com/u/7113590?v=4","gravatar_id":"","url":"https://api.github.com/users/elcallio","html_url":"https://github.com/elcallio","followers_url":"https://api.github.com/users/elcallio/followers","following_url":"https://api.github.com/users/elcallio/following{/other_user}","gists_url":"https://api.github.com/users/elcallio/gists{/gist_id}","starred_url":"https://api.github.com/users/elcallio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elcallio/subscriptions","organizations_url":"https://api.github.com/users/elcallio/orgs","repos_url":"https://api.github.com/users/elcallio/repos","events_url":"https://api.github.com/users/elcallio/events{/privacy}","received_events_url":"https://api.github.com/users/elcallio/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/24","html_url":"https://github.com/scylladb/scylla/milestone/24","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/24/labels","id":2618466,"number":24,"title":"2.2","description":"","creator":{"login":"slivne","id":3465480,"avatar_url":"https://avatars0.githubusercontent.com/u/3465480?v=4","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"open_issues":96,"closed_issues":32,"state":"open","created_at":"2017-07-03T12:58:51Z","updated_at":"2018-02-12T19:11:20Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2018-01-25T23:58:18Z","updated_at":"2018-02-13T13:31:58Z","closed_at":null,"author_association":"NONE","body":"*Installation details*\r\nScylla version (or git commit hash): 2.0.2\r\nCluster size: 4\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): CentOS\r\nInstance-type: n1-highmem-16 (16 vCPUs, 104 GB memory)\r\n\r\nAfter populating 4TB into a cluster with 1000 keyspaces and each keyspace has one table, a restart to one of scylla nodes resulted in failure and not refuse to start.\r\n\r\n\r\nThe following errors and warnings appears in the logs:\r\n```\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:21:51 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\n\r\nJan 25 13:21:51 err    |  [shard 11] database - failed to move memtable for /var/lib/scylla/data/keyspace198/standard1-c3619bf0012211e8b1eb000000000003/keyspace198-standard1-ka-2107-Data.db to\r\ncache: std::bad_alloc (std::bad_alloc)\r\n\r\nJan 25 13:22:21 warning|  [shard 6] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 8] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 9] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 2] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 1] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 11] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 13] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 14] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:22:21 warning|  [shard 6] commitlog_replayer - error replaying: std::bad_alloc (std::bad_alloc)\r\n\r\n\r\nJan 25 13:29:30 warning|  [shard 0] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 0] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 0] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 13] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 1] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 15] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 13] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 5] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 9] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 15] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 9] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 1] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 4] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 9] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 13] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\nJan 25 13:29:30 warning|  [shard 15] sstable - Exception when deleting sstable file: std::bad_alloc (std::bad_alloc)\r\n\r\nJan 25 13:29:30 warning|  [shard 10] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x961c74, 0x4ecda9, 0x5408ca, 0x542170, 0x59\r\nadfd, 0x7efd598e8e24, 0x7efd5961634c\r\n2018-01-25 13:29:30,933 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:29:30 warning|  [shard 10] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x961c74, 0x4ecda9, 0x5408ca, 0x542170, 0x59\r\nadfd, 0x7efd598e8e24, 0x7efd5961634c\r\n2018-01-25 13:29:30,933 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:29:30 warning|  [shard 10] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x961c74, 0x4ecda9, 0x5408ca, 0x542170, 0x59\r\nadfd, 0x7efd598e8e24, 0x7efd5961634c\r\n2018-01-25 13:29:30,933 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:29:30 warning|  [shard 10] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x961c74, 0x4ecda9, 0x5408ca, 0x542170, 0x59\r\nadfd, 0x7efd598e8e24, 0x7efd5961634c\r\n2018-01-25 13:29:30,933 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:29:30 warning|  [shard 10] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x961c74, 0x4ecda9, 0x5408ca, 0x542170, 0x59\r\nadfd, 0x7efd598e8e24, 0x7efd5961634c\r\n\r\n\r\nJan 25 13:34:37 warning|  [shard 0] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x4f2524, 0x4ecda9, 0x5408ca, 0x5c5834, 0x475\r\n1ff, 0x7efd5953fc04, 0x4e8e2e\r\n2018-01-25 13:34:37,126 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:34:37 warning|  [shard 0] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x4f2524, 0x4ecda9, 0x5408ca, 0x5c5834, 0x475\r\n1ff, 0x7efd5953fc04, 0x4e8e2e\r\n2018-01-25 13:34:37,126 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:34:37 warning|  [shard 0] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x4f2524, 0x4ecda9, 0x5408ca, 0x5c5834, 0x475\r\n1ff, 0x7efd5953fc04, 0x4e8e2e\r\n2018-01-25 13:34:37,126 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:34:37 warning|  [shard 0] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x4f2524, 0x4ecda9, 0x5408ca, 0x5c5834, 0x475\r\n1ff, 0x7efd5953fc04, 0x4e8e2e\r\n2018-01-25 13:34:37,127 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:34:37 warning|  [shard 0] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x4f2524, 0x4ecda9, 0x5408ca, 0x5c5834, 0x475\r\n1ff, 0x7efd5953fc04, 0x4e8e2e\r\n2018-01-25 13:34:37,127 remote           L0197 DEBUG| [10.142.0.12] [stdout] Jan 25 13:34:37 warning|  [shard 0] seastar - Exceptional future ignored: std::bad_alloc (std::bad_alloc), backtrace: 0x4f0173, 0x4f01d1, 0x55c22a, 0x4f2524, 0x4ecda9, 0x5408ca, 0x5c5834, 0x475\r\n1ff, 0x7efd5953fc04, 0x4e8e2e\r\n```"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/365267260","html_url":"https://github.com/scylladb/scylla/issues/3155#issuecomment-365267260","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3155","id":365267260,"user":{"login":"elcallio","id":7113590,"avatar_url":"https://avatars1.githubusercontent.com/u/7113590?v=4","gravatar_id":"","url":"https://api.github.com/users/elcallio","html_url":"https://github.com/elcallio","followers_url":"https://api.github.com/users/elcallio/followers","following_url":"https://api.github.com/users/elcallio/following{/other_user}","gists_url":"https://api.github.com/users/elcallio/gists{/gist_id}","starred_url":"https://api.github.com/users/elcallio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elcallio/subscriptions","organizations_url":"https://api.github.com/users/elcallio/orgs","repos_url":"https://api.github.com/users/elcallio/repos","events_url":"https://api.github.com/users/elcallio/events{/privacy}","received_events_url":"https://api.github.com/users/elcallio/received_events","type":"User","site_admin":false},"created_at":"2018-02-13T13:31:58Z","updated_at":"2018-02-13T13:31:58Z","author_association":"CONTRIBUTOR","body":"If the problem is huge blobs in the mutations (how large is each row in the data set?), it might be we need to move the reader to use a segmented approach to avoid loading large data rows and duplicating the data several times (each mutation is read into a single buffer). \r\n\r\nBut otoh, there should have been issues creating the data set if we have problems with the mutation sizes..."}}
{"actor":{"display_login":"elcallio"}}