{"action":"created","issue":{"url":"https://api.github.com/repos/apache/accumulo/issues/535","repository_url":"https://api.github.com/repos/apache/accumulo","labels_url":"https://api.github.com/repos/apache/accumulo/issues/535/labels{/name}","comments_url":"https://api.github.com/repos/apache/accumulo/issues/535/comments","events_url":"https://api.github.com/repos/apache/accumulo/issues/535/events","html_url":"https://github.com/apache/accumulo/issues/535","id":334181248,"node_id":"MDU6SXNzdWUzMzQxODEyNDg=","number":535,"title":"WAL recovery enhancements and tooling","user":{"login":"adamjshook","id":1372828,"node_id":"MDQ6VXNlcjEzNzI4Mjg=","avatar_url":"https://avatars1.githubusercontent.com/u/1372828?v=4","gravatar_id":"","url":"https://api.github.com/users/adamjshook","html_url":"https://github.com/adamjshook","followers_url":"https://api.github.com/users/adamjshook/followers","following_url":"https://api.github.com/users/adamjshook/following{/other_user}","gists_url":"https://api.github.com/users/adamjshook/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjshook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjshook/subscriptions","organizations_url":"https://api.github.com/users/adamjshook/orgs","repos_url":"https://api.github.com/users/adamjshook/repos","events_url":"https://api.github.com/users/adamjshook/events{/privacy}","received_events_url":"https://api.github.com/users/adamjshook/received_events","type":"User","site_admin":false},"labels":[{"id":869766068,"node_id":"MDU6TGFiZWw4Njk3NjYwNjg=","url":"https://api.github.com/repos/apache/accumulo/labels/bug","name":"bug","color":"e99695","default":true},{"id":942637931,"node_id":"MDU6TGFiZWw5NDI2Mzc5MzE=","url":"https://api.github.com/repos/apache/accumulo/labels/v1.9.2","name":"v1.9.2","color":"1d76db","default":false}],"state":"open","locked":false,"assignee":{"login":"adamjshook","id":1372828,"node_id":"MDQ6VXNlcjEzNzI4Mjg=","avatar_url":"https://avatars1.githubusercontent.com/u/1372828?v=4","gravatar_id":"","url":"https://api.github.com/users/adamjshook","html_url":"https://github.com/adamjshook","followers_url":"https://api.github.com/users/adamjshook/followers","following_url":"https://api.github.com/users/adamjshook/following{/other_user}","gists_url":"https://api.github.com/users/adamjshook/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjshook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjshook/subscriptions","organizations_url":"https://api.github.com/users/adamjshook/orgs","repos_url":"https://api.github.com/users/adamjshook/repos","events_url":"https://api.github.com/users/adamjshook/events{/privacy}","received_events_url":"https://api.github.com/users/adamjshook/received_events","type":"User","site_admin":false},"assignees":[{"login":"adamjshook","id":1372828,"node_id":"MDQ6VXNlcjEzNzI4Mjg=","avatar_url":"https://avatars1.githubusercontent.com/u/1372828?v=4","gravatar_id":"","url":"https://api.github.com/users/adamjshook","html_url":"https://github.com/adamjshook","followers_url":"https://api.github.com/users/adamjshook/followers","following_url":"https://api.github.com/users/adamjshook/following{/other_user}","gists_url":"https://api.github.com/users/adamjshook/gists{/gist_id}","starred_url":"https://api.github.com/users/adamjshook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamjshook/subscriptions","organizations_url":"https://api.github.com/users/adamjshook/orgs","repos_url":"https://api.github.com/users/adamjshook/repos","events_url":"https://api.github.com/users/adamjshook/events{/privacy}","received_events_url":"https://api.github.com/users/adamjshook/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-06-20T17:34:24Z","updated_at":"2018-06-20T19:38:09Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The changes in #458 introduced a number of state checks that we are experiencing.  As far as I know there isn't any tooling around troubleshooting corrupt WAL files that are unrecoverable by Accumulo (please correct me if I am wrong!) and replacing the WAL with an empty one isn't a good option.\r\n\r\nI'd like to know a little more about the various `IllegalStateExceptions` themselves and what kind of WAL surgery could be done.  Either creating a new tool to manually assist with recovery of the mutations or enhancements to the process itself so Accumulo can make the choices for us."},"comment":{"url":"https://api.github.com/repos/apache/accumulo/issues/comments/398871117","html_url":"https://github.com/apache/accumulo/issues/535#issuecomment-398871117","issue_url":"https://api.github.com/repos/apache/accumulo/issues/535","id":398871117,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODg3MTExNw==","user":{"login":"ctubbsii","id":1280725,"node_id":"MDQ6VXNlcjEyODA3MjU=","avatar_url":"https://avatars3.githubusercontent.com/u/1280725?v=4","gravatar_id":"","url":"https://api.github.com/users/ctubbsii","html_url":"https://github.com/ctubbsii","followers_url":"https://api.github.com/users/ctubbsii/followers","following_url":"https://api.github.com/users/ctubbsii/following{/other_user}","gists_url":"https://api.github.com/users/ctubbsii/gists{/gist_id}","starred_url":"https://api.github.com/users/ctubbsii/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ctubbsii/subscriptions","organizations_url":"https://api.github.com/users/ctubbsii/orgs","repos_url":"https://api.github.com/users/ctubbsii/repos","events_url":"https://api.github.com/users/ctubbsii/events{/privacy}","received_events_url":"https://api.github.com/users/ctubbsii/received_events","type":"User","site_admin":false},"created_at":"2018-06-20T19:38:09Z","updated_at":"2018-06-20T19:38:09Z","author_association":"MEMBER","body":"I spoke to @keith-turner at length about this, and we (mostly Keith) came to the conclusion that these two errors *might* occur if you stop writing to one tablet, but continue writing mutations to another, and the logs roll over. In both cases, these exceptions could be thrown as a false positive, when there is no data to recover for that tablet, because the WAL containing the `COMPACTION_START` event could have been garbage collected.\r\n\r\nWorse, this scenario may not be tested for, because our continuous ingest tests don't ever stop writing to a tablet.\r\n\r\nThe workaround would be to inspect the WALs and verify that there is no data for the tablet which produced the exception during recovery, and remove the entries in the affected tablet, and to repeat for each affected tablet. This is not ideal, but if somebody can verify that this is what is happening (it's still just speculation right now), we could proceed with a fix for 1.9.2. The good news is that there shouldn't be any data loss, if this is what is happening. It's just an error when there's no data necessary to recover.\r\n\r\nSome possible fixes we discussed, if the issue can be verified:\r\n\r\n1. Check that there are no data events in the WALs for that tablet, before throwing the exception.\r\n2. Don't mark a WAL inactive prematurely, even if it has only a `COMPACTION_START` event with no data.\r\n\r\nMore investigation is needed to verify the problem, and possible fixes, though."}}
{"actor":{"display_login":"ctubbsii"}}