{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3262","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3262/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3262/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3262/events","html_url":"https://github.com/scylladb/scylla/issues/3262","id":302982521,"number":3262,"title":"Database fails to start due to sstable confusion","user":{"login":"jdness","id":1036099,"avatar_url":"https://avatars0.githubusercontent.com/u/1036099?v=4","gravatar_id":"","url":"https://api.github.com/users/jdness","html_url":"https://github.com/jdness","followers_url":"https://api.github.com/users/jdness/followers","following_url":"https://api.github.com/users/jdness/following{/other_user}","gists_url":"https://api.github.com/users/jdness/gists{/gist_id}","starred_url":"https://api.github.com/users/jdness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdness/subscriptions","organizations_url":"https://api.github.com/users/jdness/orgs","repos_url":"https://api.github.com/users/jdness/repos","events_url":"https://api.github.com/users/jdness/events{/privacy}","received_events_url":"https://api.github.com/users/jdness/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-03-07T06:45:13Z","updated_at":"2018-03-07T15:11:07Z","closed_at":null,"author_association":"NONE","body":"*Installation details*\r\nScylla version 2.1.rc3-0.20180207.b98d5b30d\r\nCluster size: 1\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): AWS i3.xlarge\r\n\r\nWe had some issues with materialized views a few weeks ago and I ended up needing to move aside the views and indexes directories to save the cluster.  Scylla recreate those directories and we were able to subsequently recreated the views and indexes from our saved schema and repopulated the data by updating the base tables.  However, we later started getting periodic error messages in the journalctl log about missing sstables:\r\n\r\n```\r\nMar 07 06:29:38 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-64-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-64-Data.db is empty)\r\nMar 07 06:29:48 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-68-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-68-Data.db is empty)\r\nMar 07 06:29:58 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-72-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-72-Data.db is empty)\r\nMar 07 06:30:08 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-76-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-76-Data.db is empty)\r\nMar 07 06:30:18 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-80-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-80-Data.db is empty)\r\nMar 07 06:30:28 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-84-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-84-Data.db is empty)\r\nMar 07 06:30:38 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-88-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-88-Data.db is empty)\r\nMar 07 06:30:48 scylladb scylla[24302]:  [shard 0] database - failed to write sstable /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-92-Data.db: std::runtime_error (first key of summary of /var/lib/scylla/data/veraminetest/tracked_urls_usersid_idx_index-0324aac01dde11e89244000000000002/veraminetest-tracked_urls_usersid_idx_index-ka-92-Data.db is empty)\r\n```\r\n\r\nIn effort to resolve this issue, I tried nodetool drain.  The drain operation never finished, instead getting stuck in a loop failing to write to those sstables.  I also tried systemctl restart scylla-server and that also never completes.  It begins to replay the pending Commitlog files and eventually gets into a state where it is attempting to access missing sstables every 10 seconds and finally restarts the loop all over again.\r\n\r\nI don't need the data in these tables, indexes, or views.  Is there a way I can just wipe out references to the \"stuck\" indexes?  Or, any other suggestions to best resolve this state?  I attached the most recent couple days of journalctl output, in case that helps.  Thanks!\r\n\r\n[journal.txt](https://github.com/scylladb/scylla/files/1788071/journal.txt)\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/371169405","html_url":"https://github.com/scylladb/scylla/issues/3262#issuecomment-371169405","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3262","id":371169405,"user":{"login":"avikivity","id":1017210,"avatar_url":"https://avatars1.githubusercontent.com/u/1017210?v=4","gravatar_id":"","url":"https://api.github.com/users/avikivity","html_url":"https://github.com/avikivity","followers_url":"https://api.github.com/users/avikivity/followers","following_url":"https://api.github.com/users/avikivity/following{/other_user}","gists_url":"https://api.github.com/users/avikivity/gists{/gist_id}","starred_url":"https://api.github.com/users/avikivity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avikivity/subscriptions","organizations_url":"https://api.github.com/users/avikivity/orgs","repos_url":"https://api.github.com/users/avikivity/repos","events_url":"https://api.github.com/users/avikivity/events{/privacy}","received_events_url":"https://api.github.com/users/avikivity/received_events","type":"User","site_admin":false},"created_at":"2018-03-07T15:11:07Z","updated_at":"2018-03-07T15:11:07Z","author_association":"CONTRIBUTOR","body":"> I'm not an expert on sstable internals, but that error message suggests that the sstable is perhaps corrupt. I think you can just remove the sstable (unless there's valuable data there), but let's check with @raphaelsc first.\r\n\r\nThat error happened when writing the sstable. Looks like the sstable writer thinks the primary key is empty."}}
{"actor":{"display_login":"avikivity"}}