{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/926","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/926/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/926/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/926/events","html_url":"https://github.com/scylladb/scylla/issues/926","id":135110366,"number":926,"title":"[AMI 0.18] Longevity test failure: Cassandra-stress doesn't survive stop/start monkey","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":79,"closed_issues":205,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2016-02-24T16:59:38Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2016-02-20T18:57:58Z","updated_at":"2016-02-24T20:46:48Z","closed_at":null,"body":"Running the longevity test for 24 hour with our stop/start monkey revealed failures in cassandra-stress to survive until the end. The stresser stays up for about 6.5 hours:\r\n\r\n    22:47:51 remote           L0773 DEBUG| Remote [centos@54.183.232.21]: Running 'cassandra-stress write cl=QUORUM duration=1440m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=4 -node 172.31.27.192'\r\n    ...\r\n    05:32:10 remote           L0182 DEBUG| [54.183.232.21] [stdout] java.io.IOException: Operation x10 on key(s) [50364f364d34314f4e30]: Error executing: (UnavailableException): Not enough replicas available for query at consistency QUORUM (2 required but only 1 alive)\r\n\r\nOther than an error message already reported on bug #900 (samples follow):\r\n\r\n    22:31:29 remote           L0182 DEBUG| [54.183.212.72] [stdout] Feb 19 22:31:29 ip-172-31-27-193.us-west-1.compute.internal scylla[894]: WARN  [shard 0] failure_detector - Not marking nodes down due to local pause of 128281 > 5000 (milliseconds)\r\n    22:31:35 remote           L0182 DEBUG| [52.53.231.216] [stdout] Feb 19 22:31:35 ip-172-31-27-194.us-west-1.compute.internal scylla[924]: WARN  [shard 0] failure_detector - Not marking nodes down due to local pause of 128325 > 5000 (milliseconds)\r\n    22:31:40 remote           L0182 DEBUG| [54.193.93.33] [stdout] Feb 19 22:31:41 ip-172-31-27-196.us-west-1.compute.internal scylla[903]: WARN  [shard 0] failure_detector - Not marking nodes down due to local pause of 128232 > 5000 (milliseconds)\r\n    22:31:47 remote           L0182 DEBUG| [54.183.181.107] [stdout] Feb 19 22:31:47 ip-172-31-27-197.us-west-1.compute.internal scylla[1180]: WARN  [shard 0] failure_detector - Not marking nodes down due to local pause of 128291 > 5000 (milliseconds)\r\n\r\nAnd the one already reported on bug #902 (samples follow):\r\n\r\n    22:31:35 remote           L0182 DEBUG| [52.53.231.216] [stdout] Feb 19 22:31:36 ip-172-31-27-194.us-west-1.compute.internal scylla-jmx[1971]: Exception in thread \"Dropped messages\" java.lang.NullPointerException\r\n\r\nThere were no other suspicious error messages to report.\r\n\r\nWhat the test does\r\n------------------\r\n\r\nThe workflow of a longevity test is:\r\n\r\n1. Create a scylla cluster (referred as DB cluster from now on) with number of nodes (default size is 6) and node size (default is t2.micro, the bug report is using c3.large) defined by the user on a config file\r\n2. Create a number of loader nodes, that will stress the DB cluster, typically by running cassandra stress on it\r\n3. The cluster object hosts a Nemesis (that terminology was borrowed from Jepsen), a thread that randomly picks a non-seed node and disrupt it in some way.\r\n4. Nemesis will sleep for a specified time interval, then execute the disruption function, then sleep again, rinse and repeat.\r\n5. Cassandra-stress starts. The default command line looks like:\r\n\r\n    `cassandra-stress write cl=QUORUM duration=1440m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=4 -node 172.31.27.192`\r\n\r\n6. At all times, we have at least initial cluster size - 1 (6 -1 = 5) nodes active, which should be enough to keep c-s going. The disruption functions operate on a single node, and operations that actually remove a node permanently from the cluster will add a new node to compensate the one that was removed.\r\n7. Given all that, we do expect that c-s survives until the specified time period (the report is using 24 hours). If it doesn't, chances are that we have a bug.\r\n\r\nWhat the nemesis does\r\n---------------------\r\n\r\n1. Stop the AWS instance of the target node.\r\n2. Start the AWS instance of the target node.\r\n3. Wait until the DB services are up (port 9042 occuppied)\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/188447851","html_url":"https://github.com/scylladb/scylla/issues/926#issuecomment-188447851","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/926","id":188447851,"user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"created_at":"2016-02-24T20:46:48Z","updated_at":"2016-02-24T20:46:48Z","body":"This test was repeated with a shorter nemesis interval, and the result was the same, only the stresser survived for less time (now ~2.5 hours):\r\n\r\n    07:28:21 remote           L0773 DEBUG| Remote [centos@54.67.8.166]: Running 'cassandra-stress write cl=QUORUM duration=1440m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3\r\n     native -rate threads=4 -node 172.31.25.176'\r\n\r\n    ...\r\n\r\n    09:40:25 remote           L0182 DEBUG| [54.67.8.166] [stdout] java.io.IOException: Operation x10 on key(s) [32363238324b314b3130]: Error executing: (UnavailableException): Not enough replicas available for query at consistency QUORUM (2 required but only 1 alive)\r\n"}}