{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/962","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/962/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/962/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/962/events","html_url":"https://github.com/scylladb/scylla/issues/962","id":137337753,"number":962,"title":"[AMI 0.18] Longevity test failure: (DrainerMonkey) - cassandra-stress does not survive","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"asias","id":320076,"avatar_url":"https://avatars.githubusercontent.com/u/320076?v=3","gravatar_id":"","url":"https://api.github.com/users/asias","html_url":"https://github.com/asias","followers_url":"https://api.github.com/users/asias/followers","following_url":"https://api.github.com/users/asias/following{/other_user}","gists_url":"https://api.github.com/users/asias/gists{/gist_id}","starred_url":"https://api.github.com/users/asias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asias/subscriptions","organizations_url":"https://api.github.com/users/asias/orgs","repos_url":"https://api.github.com/users/asias/repos","events_url":"https://api.github.com/users/asias/events{/privacy}","received_events_url":"https://api.github.com/users/asias/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/12","html_url":"https://github.com/scylladb/scylla/milestones/1.1","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/12/labels","id":1661891,"number":12,"title":"1.1","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":51,"closed_issues":2,"state":"open","created_at":"2016-03-23T13:18:57Z","updated_at":"2016-03-24T10:34:23Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2016-02-29T18:52:31Z","updated_at":"2016-03-25T21:27:25Z","closed_at":null,"body":"The execution of the drainer monkey on the most recent AMI, now with 5 minutes of nemesis interval, causes cassandra-stress to fail in about 15 minutes:\r\n\r\n    03:19:45 remote           L0773 DEBUG| Remote [centos@54.193.83.106]: Running 'cassandra-stress write cl=QUORUM duration=1440m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=4 -node 172.31.30.65'\r\n    ...\r\n    03:34:44 remote           L0182 DEBUG| [54.193.83.106] [stdout] java.io.IOException: Operation x10 on key(s) [36324f4e33304b313231]: Error executing: (UnavailableException): Not enough replicas available for query at consistency QUORUM (2 required but only 1 alive)\r\n\r\nUnlike the previous issue found with this nemesis, #925, the nodes don't seem to be running out of space.\r\n\r\nWhat the test does\r\n------------------\r\n\r\nThe workflow of a longevity test is:\r\n\r\n1. Create a scylla cluster (referred as DB cluster from now on) with number of nodes (default size is 6) and node size (default is t2.micro, the bug report is using c3.large) defined by the user on a config file\r\n2. Create a number of loader nodes, that will stress the DB cluster, typically by running cassandra stress on it\r\n3. The cluster object hosts a Nemesis (that terminology was borrowed from Jepsen), a thread that randomly picks a non-seed node and disrupt it in some way.\r\n4. Nemesis will sleep for a specified time interval, then execute the disruption function, then sleep again, rinse and repeat.\r\n5. Cassandra-stress starts. The default command line looks like:\r\n\r\n    `cassandra-stress write cl=QUORUM duration=1440m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=4 -node 172.31.27.192`\r\n\r\n6. At all times, we have at least initial cluster size - 1 (6 -1 = 5) nodes active, which should be enough to keep c-s going. The disruption functions operate on a single node, and operations that actually remove a node permanently from the cluster will add a new node to compensate the one that was removed.\r\n7. Given all that, we do expect that c-s survives until the specified time period (the report is using 24 hours). If it doesn't, chances are that we have a bug.\r\n\r\nWhat the nemesis does\r\n---------------------\r\n\r\n1. Execute the following command in the target node:\r\n\r\n    `nodetool -h localhost drain`\r\n\r\n2. Stop the AWS instance of the target node.\r\n3. Start the AWS instance of the target node.\r\n4. Wait until the DB services are up (port 9042 occuppied)\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/201516508","html_url":"https://github.com/scylladb/scylla/issues/962#issuecomment-201516508","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/962","id":201516508,"user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"created_at":"2016-03-25T21:27:25Z","updated_at":"2016-03-25T21:27:25Z","body":"I still observe this using the latest 1.0-rc1 AMI\r\n\r\n    18:29:36 remote           L0773 DEBUG| Remote [centos@54.191.177.102]: Running 'cassandra-stress write cl=QUORUM duration=1440m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=1000 -node 172.31.42.50'\r\n    ...\r\n    23:44:07 remote           L0182 DEBUG| [54.191.177.102] [stdout] java.io.IOException: Operation x10 on key(s) [304b384f32364d4c4f30]: Error executing: (WriteTimeoutException): Cassandra timeout during write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write)\r\n"}}