{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1783","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1783/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1783/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1783/events","html_url":"https://github.com/scylladb/scylla/issues/1783","id":185440139,"number":1783,"title":"1.4 RC2] Scylla Cluster does not survive CorruptThenRebuild nemesis testing - larger data set","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[{"id":302689228,"url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/15","html_url":"https://github.com/scylladb/scylla/milestones/1.4","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/15/labels","id":1794297,"number":15,"title":"1.4","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":53,"state":"open","created_at":"2016-05-29T08:33:08Z","updated_at":"2016-10-31T11:27:47Z","due_on":"2016-09-30T00:00:00Z","closed_at":null},"comments":10,"created_at":"2016-10-26T16:10:46Z","updated_at":"2016-11-11T03:36:51Z","closed_at":null,"body":"_Installation details_\nScylla version (or git commit hash): 1.4 RC2 AMI\nCluster config: 6 nodes, i2.2xlarge\n\nRunning the CorruptThenRebuild nemesis from scylla-cluster-tests, we verified that the Cluster doesn't survive a cycle of nemesis addition. Moreover, we verify numerous occurrences of the issue #1776, but now with slightly different error messages.\n\nThe load lasted for about 2 hours, when cassandra-stress failed with:\n\n```\n2016-10-25 16:30:32,978 remote           L0194 DEBUG| [54.186.190.126] [stdout] java.io.IOException: Operation x10 on key(s) [4d3931334f4e394d3330]: Error executing: (WriteTimeoutException): Cassandra timeout during write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write)\n```\n\nWe verified several occurrences of #1776, now with different error messages:\n\n```\n2016-10-26 06:05:45,961 nemesis          L0108 ERROR| sdcm.nemesis.CorruptThenRebuildMonkey: nodetool command 'nodetool -h localhost rebuild' failed on node Node scylla-1-4-rc2-ld-corrupt-then-rebuild-scylla-db-node-bc66384f-3 [54.191.91.61 | 172.31.44.157] (seed: False): Command: /bin/ssh -a -x  -o ControlPath=/var/tmp/ssh-masterxAyeDW/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/var/tmp/tmp26mTlJ -o BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l centos -p 22 -i /var/tmp/scylla-1-4-rc2-ld-corrupt-then-rebuild-longevity-test-456aabcf.pem 54.191.91.61 \"nodetool -h localhost rebuild\"\nExit status: 2\nDuration: 9.46505784988\nStdout:\n\nStderr:\nerror: null\n-- StackTrace --\njava.io.EOFException\n        at java.io.DataInputStream.readByte(DataInputStream.java:267)\n        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:215)\n        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)\n        at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)\n        at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)\n        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)\n        at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)\n        at com.sun.proxy.$Proxy6.rebuild(Unknown Source)\n        at org.apache.cassandra.tools.NodeProbe.rebuild(NodeProbe.java:1038)\n        at org.apache.cassandra.tools.NodeTool$Rebuild.execute(NodeTool.java:1779)\n        at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:288)\n        at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)\n```\n\nThe cassandra-stress ran on a 1 loader, c4.8xlarge, and the reference command is:\n\n```\ncassandra-stress write cl=QUORUM duration=10080m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate threads=200 -col 'size=FIXED(1000) n=FIXED(1)' -pop seq=1..250000000\n```\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/259873411","html_url":"https://github.com/scylladb/scylla/issues/1783#issuecomment-259873411","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1783","id":259873411,"user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"created_at":"2016-11-11T03:36:51Z","updated_at":"2016-11-11T03:36:51Z","body":"About 28 hours after the test started, the c-s load died. So this is still an issue, although there are no more coredumps to be seen."}}
{"actor":{"display_login":"lmr"}}