{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1627","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1627/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1627/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1627/events","html_url":"https://github.com/scylladb/scylla/issues/1627","id":174188144,"number":1627,"title":"[1.3 AMI] bad_alloc error messages on Nodes","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"assignees":[{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/15","html_url":"https://github.com/scylladb/scylla/milestones/1.4","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/15/labels","id":1794297,"number":15,"title":"1.4","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":62,"closed_issues":24,"state":"open","created_at":"2016-05-29T08:33:08Z","updated_at":"2016-09-20T18:21:12Z","due_on":"2016-09-30T00:00:00Z","closed_at":null},"comments":7,"created_at":"2016-08-31T05:33:35Z","updated_at":"2016-09-20T18:22:42Z","closed_at":null,"body":"*Installation details*\r\n\r\nNode size/AMI: Cluster lmr-scylla-db-cluster-4ae0d54d (AMI: ami-dd46ffca Type: c3.xlarge) - 1.3 AMI, 6 nodes.\r\n\r\n*Issue description*\r\n\r\nAfter a short while running the following cassandra-stress command:\r\n\r\n    2016-08-31 02:12:26,165 remote           L0832 DEBUG| Remote [centos@54.159.156.182]: Running 'cassandra-stress write cl=QUORUM duration=120m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=1000 -pop seq=1..50000000 -node 172.30.0.219'\r\n\r\nWe start seeing `std::bad_alloc` error messages:\r\n\r\n    172.30.0.222#2: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,827 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 2] storage_proxy - Failed to apply mutation from 172.30.0.218#0: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,827 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 1] storage_proxy - Failed to apply mutation from 172.30.0.219#1: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,828 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 2] storage_proxy - Failed to apply mutation from 172.30.0.218#0: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,828 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 1] storage_proxy - Failed to apply mutation from 172.30.0.219#1: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,828 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 2] storage_proxy - Failed to apply mutation from 172.30.0.218#0: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,828 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 1] storage_proxy - Failed to apply mutation from 172.30.0.221#2: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:04,828 remote           L0194 DEBUG| [54.165.23.117] [stdout] Aug 31 05:18:04 ip-172-30-0-223 scylla[2297]:  [shard 2] storage_proxy - Failed to apply mutation from 172.30.0.219#2: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:27,759 remote           L0194 DEBUG| [52.87.177.83] [stdout] Aug 31 05:18:27 ip-172-30-0-220 scylla[2285]:  [shard 2] storage_proxy - Failed to apply mutation from 172.30.0.219#3: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:27,760 remote           L0194 DEBUG| [52.87.177.83] [stdout] Aug 31 05:18:27 ip-172-30-0-220 scylla[2285]:  [shard 2] storage_proxy - Failed to apply mutation from 172.30.0.223#2: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:27,760 remote           L0194 DEBUG| [52.87.177.83] [stdout] Aug 31 05:18:27 ip-172-30-0-220 scylla[2285]:  [shard 2] storage_proxy - exception during mutation write to 172.30.0.220: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:27,761 remote           L0194 DEBUG| [52.87.177.83] [stdout] Aug 31 05:18:27 ip-172-30-0-220 scylla[2285]:  [shard 0] storage_proxy - exception during mutation write to 172.30.0.220: std::bad_alloc (std::bad_alloc)\r\n    2016-08-31 02:18:27,761 remote           L0194 DEBUG| [52.87.177.83] [stdout] Aug 31 05:18:27 ip-172-30-0-220 scylla[2285]:  [shard 0] storage_proxy - exception during mutation write to 172.30.0.220: std::bad_alloc (std::bad_alloc)\r\n\r\nThe cluster manages to keep throughput and load stable,  and the nodes keep being filled, so apparently the error is not critical. However, it stands to reason that, when users see this error they will certainly think something is wrong.\r\n\r\nIf this is a harmless system condition, then I'd suggest that user logging must be improved and the messages should be suppressed."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/248387976","html_url":"https://github.com/scylladb/scylla/issues/1627#issuecomment-248387976","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1627","id":248387976,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2016-09-20T18:22:42Z","updated_at":"2016-09-20T18:22:42Z","body":"@tgrabiec I think this is more urgent then the #1640  "}}
{"actor":{"display_login":"slivne"}}