{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1004","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1004/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1004/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1004/events","html_url":"https://github.com/scylladb/scylla/issues/1004","id":139391015,"number":1004,"title":"backup_restore: test snapshot_test.TestSnapshot.test_basic_snapshot_and_restore_with_refresh failes with smp=2","user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/showstopper","name":"showstopper","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"vladzcloudius","id":6103981,"avatar_url":"https://avatars.githubusercontent.com/u/6103981?v=3","gravatar_id":"","url":"https://api.github.com/users/vladzcloudius","html_url":"https://github.com/vladzcloudius","followers_url":"https://api.github.com/users/vladzcloudius/followers","following_url":"https://api.github.com/users/vladzcloudius/following{/other_user}","gists_url":"https://api.github.com/users/vladzcloudius/gists{/gist_id}","starred_url":"https://api.github.com/users/vladzcloudius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vladzcloudius/subscriptions","organizations_url":"https://api.github.com/users/vladzcloudius/orgs","repos_url":"https://api.github.com/users/vladzcloudius/repos","events_url":"https://api.github.com/users/vladzcloudius/events{/privacy}","received_events_url":"https://api.github.com/users/vladzcloudius/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":74,"closed_issues":252,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2016-03-10T06:30:27Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2016-03-08T20:56:41Z","updated_at":"2016-03-10T13:36:01Z","closed_at":null,"body":"running the test with scylla using smp=1 passes\r\n\r\nrunning the test with scylla using smp=2 fails\r\n\r\nusing latest ccm and latest dtest\r\n```\r\nexport SCYLLA_EXT_OPTS=\"--smp 2\"\r\nshlomi@localhost~/urchin-dtest (master)$ export CASSANDRA_DIR=~/scylla/build/release; pkill -9 scylla; pkill -9 java; rm -Rf core.*; KEEP_TEST_DIR=true REUSE_CLUSTER=false DEBUG=true PRINT_DEBUG=true nosetests -v -s snapshot_test.py:TestSnapshot.test_basic_snapshot_and_restore_with_refresh\r\ntest_basic_snapshot_and_restore_with_refresh (snapshot_test.TestSnapshot) ... cluster ccm directory: /home/shlomi/.dtest/dtest-oHlIO1\r\nMaking snapshot....\r\nRunning snapshot cmd: snapshot ks -cf cf -t basic\r\nsnapshot_dir is : /home/shlomi/.dtest/dtest-oHlIO1/test/node1/data/ks/cf-e95c77b0e56f11e5b349000000000001/snapshots/basic\r\nsnapshot copy is : /tmp/tmp8_t4QE\r\nRestoring snapshot....\r\nCopying from /tmp/tmp8_t4QE/ks/cf to /home/shlomi/.dtest/dtest-oHlIO1/test/node1/data/ks/cf-ec43e030e56f11e5b349000000000001/\r\nERROR\r\n\r\n======================================================================\r\nERROR: test_basic_snapshot_and_restore_with_refresh (snapshot_test.TestSnapshot)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File \"/home/shlomi/urchin-dtest/snapshot_test.py\", line 96, in test_basic_snapshot_and_restore_with_refresh\r\n    self.basic_snapshot_and_restore(use_sstableloader=False)\r\n  File \"/home/shlomi/urchin-dtest/snapshot_test.py\", line 143, in basic_snapshot_and_restore\r\n    node1.nodetool('refresh ks cf')\r\n  File \"/home/shlomi/urchin-ccm/ccmlib/node.py\", line 693, in nodetool\r\n    raise NodetoolError(\" \".join(args), exit_status, stdout, stderr)\r\nNodetoolError: Nodetool command '/home/shlomi/scylla/resources/cassandra/bin/nodetool -h localhost -p 7100 refresh ks cf' failed; exit status: 2; stderr: error: /home/shlomi/.dtest/dtest-oHlIO1/test/node1/data/ks/cf-ec43e030e56f11e5b349000000000001/ks-cf-ka-1-Filter.db: file not found\r\n-- StackTrace --\r\njava.lang.RuntimeException: /home/shlomi/.dtest/dtest-oHlIO1/test/node1/data/ks/cf-ec43e030e56f11e5b349000000000001/ks-cf-ka-1-Filter.db: file not found\r\n\tat com.scylladb.jmx.api.APIClient.getException(APIClient.java:115)\r\n\tat com.scylladb.jmx.api.APIClient.post(APIClient.java:102)\r\n\tat org.apache.cassandra.service.StorageService.loadNewSSTables(StorageService.java:1123)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)\r\n\tat com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)\r\n\tat com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)\r\n\tat com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)\r\n\tat com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)\r\n\tat com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)\r\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)\r\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)\r\n\tat javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)\r\n\tat javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)\r\n\tat javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)\r\n\tat javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)\r\n\tat javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)\r\n\tat sun.rmi.transport.Transport$1.run(Transport.java:200)\r\n\tat sun.rmi.transport.Transport$1.run(Transport.java:197)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat sun.rmi.transport.Transport.serviceCall(Transport.java:196)\r\n\tat sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)\r\n\tat sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)\r\n\tat sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\n\r\n-------------------- >> begin captured logging << --------------------\r\ndtest: DEBUG: cluster ccm directory: /home/shlomi/.dtest/dtest-oHlIO1\r\ndtest: DEBUG: Making snapshot....\r\ndtest: DEBUG: Running snapshot cmd: snapshot ks -cf cf -t basic\r\ndtest: DEBUG: snapshot_dir is : /home/shlomi/.dtest/dtest-oHlIO1/test/node1/data/ks/cf-e95c77b0e56f11e5b349000000000001/snapshots/basic\r\ndtest: DEBUG: snapshot copy is : /tmp/tmp8_t4QE\r\ndtest: DEBUG: Restoring snapshot....\r\ndtest: DEBUG: Copying from /tmp/tmp8_t4QE/ks/cf to /home/shlomi/.dtest/dtest-oHlIO1/test/node1/data/ks/cf-ec43e030e56f11e5b349000000000001/\r\n--------------------- >> end captured logging << ---------------------\r\n\r\n----------------------------------------------------------------------\r\nRan 1 test in 16.159s\r\n\r\nFAILED (errors=1)\r\n```\r\n\r\nchecking the snapshot backup directory\r\n\r\n```\r\nls /tmp/tmp8_t4QE/ks/cf/\r\nks-cf-ka-2-CompressionInfo.db  ks-cf-ka-2-Filter.db      ks-cf-ka-2-Summary.db          ks-cf-ka-3-Data.db      ks-cf-ka-3-Index.db       ks-cf-ka-3-TOC.txt\r\nks-cf-ka-2-Data.db             ks-cf-ka-2-Index.db       ks-cf-ka-2-TOC.txt             ks-cf-ka-3-Digest.sha1  ks-cf-ka-3-Statistics.db  manifest.json\r\nks-cf-ka-2-Digest.sha1         ks-cf-ka-2-Statistics.db  ks-cf-ka-3-CompressionInfo.db  ks-cf-ka-3-Filter.db    ks-cf-ka-3-Summary.db\r\n```\r\n\r\nthere is not ks-cf-ka-1-*\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/194840488","html_url":"https://github.com/scylladb/scylla/issues/1004#issuecomment-194840488","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1004","id":194840488,"user":{"login":"glommer","id":331197,"avatar_url":"https://avatars.githubusercontent.com/u/331197?v=3","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"created_at":"2016-03-10T13:36:01Z","updated_at":"2016-03-10T13:36:01Z","body":"It's okay not to have an SSTable with the generation number 1.\r\nAs a matter of fact we're not expecting to see one, since the code that creates generations do _gen * smp_count + shard_id\r\n\r\n_gen starts with 1, so for SMP 2, your first two SSTables will be 1 * 2 + 0 = 2, and 1 * 2 + 1 = 3.\r\n\r\nIf gen started with 0, we'd have an SSTable with generation number 0 (0 * 0 + 0). That's not the end of the world, but we are not sure Cassandra could handle it, since they always use positive numbers. So we start _gen from 1.\r\n"}}