{"action":"created","issue":{"url":"https://api.github.com/repos/chrislusf/seaweedfs/issues/321","repository_url":"https://api.github.com/repos/chrislusf/seaweedfs","labels_url":"https://api.github.com/repos/chrislusf/seaweedfs/issues/321/labels{/name}","comments_url":"https://api.github.com/repos/chrislusf/seaweedfs/issues/321/comments","events_url":"https://api.github.com/repos/chrislusf/seaweedfs/issues/321/events","html_url":"https://github.com/chrislusf/seaweedfs/issues/321","id":160594442,"number":321,"title":"scan volume file with corrupted needle in it ","user":{"login":"hxiaodon","id":10990210,"avatar_url":"https://avatars.githubusercontent.com/u/10990210?v=3","gravatar_id":"","url":"https://api.github.com/users/hxiaodon","html_url":"https://github.com/hxiaodon","followers_url":"https://api.github.com/users/hxiaodon/followers","following_url":"https://api.github.com/users/hxiaodon/following{/other_user}","gists_url":"https://api.github.com/users/hxiaodon/gists{/gist_id}","starred_url":"https://api.github.com/users/hxiaodon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hxiaodon/subscriptions","organizations_url":"https://api.github.com/users/hxiaodon/orgs","repos_url":"https://api.github.com/users/hxiaodon/repos","events_url":"https://api.github.com/users/hxiaodon/events{/privacy}","received_events_url":"https://api.github.com/users/hxiaodon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-16T07:24:14Z","updated_at":"2016-06-16T09:52:23Z","closed_at":null,"body":"Chris,\r\n\r\nSuppose we have the following prerequisite:\r\n1.  Volume replication policy is 000\r\n2.  User upload files with high concurrency and throughput\r\n\r\nIf  the physical machine is suddenly power down, the last needle which is being appended to a volume file possibly could lead to writing some fragmented data(dirty data) into disk(Disk write caching), I could think of 2 cases:\r\n1 .Needle header is fragmented\r\n2. Needle body is  fragmented\r\n\r\nWhen this server is recovered,  we would try to fix the index file because the index file may be corrupted also. The fix could work because the corrupted file is the last one\r\n\r\n+-------------+\r\n|SuperBlock   |\r\n+-------------+\r\n|Needle1      |\r\n+-------------+\r\n|Needle2  corrupted    |\r\n\r\nAfter the index is recreated and the volume server is online, the write operation will continue to append needle(Per seaweedfs's design, offset alignment will be done from the end of the file  once there is a write operation) ，the currrent data structure is:\r\n\r\n+-------------+\r\n|SuperBlock   |\r\n+-------------+\r\n|Needle1      |\r\n+-------------+\r\n|Needle2  corrupted    |\r\n+-------------+\r\n|Needle3      |\r\n+-------------+\r\n|Needle ...   |\r\n+-------------+\r\n\r\nThis could work as long as index file is OK, we just loss one file.  But if the index file is corrupted again,  weed fix could not find the Needle3's offset by adding current offset with Needle2's data size and padding size,all other needles after 3 will not be added into the index file\r\n\r\n![image](https://cloud.githubusercontent.com/assets/10990210/16108347/2c3370c6-33d5-11e6-8db8-683ca6139cfd.png)\r\n\r\nChris,please help to review above draft code, I use iteratively scanning to find the next valid needle's position(maybe not a good idea if the corrupted needle data writing into disk is very large)\r\n\r\nI pop up this extreme corner case because I want to provide a more stronger disaster recovery solution when no backup data in other machine    \r\n\r\nHope for your suggestion, thanks!\r\n\r\n \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/chrislusf/seaweedfs/issues/comments/226440105","html_url":"https://github.com/chrislusf/seaweedfs/issues/321#issuecomment-226440105","issue_url":"https://api.github.com/repos/chrislusf/seaweedfs/issues/321","id":226440105,"user":{"login":"hxiaodon","id":10990210,"avatar_url":"https://avatars.githubusercontent.com/u/10990210?v=3","gravatar_id":"","url":"https://api.github.com/users/hxiaodon","html_url":"https://github.com/hxiaodon","followers_url":"https://api.github.com/users/hxiaodon/followers","following_url":"https://api.github.com/users/hxiaodon/following{/other_user}","gists_url":"https://api.github.com/users/hxiaodon/gists{/gist_id}","starred_url":"https://api.github.com/users/hxiaodon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hxiaodon/subscriptions","organizations_url":"https://api.github.com/users/hxiaodon/orgs","repos_url":"https://api.github.com/users/hxiaodon/repos","events_url":"https://api.github.com/users/hxiaodon/events{/privacy}","received_events_url":"https://api.github.com/users/hxiaodon/received_events","type":"User","site_admin":false},"created_at":"2016-06-16T09:52:23Z","updated_at":"2016-06-16T09:52:23Z","body":"I tried to verify the data integrity at my local, with ScanVolumeFile's 6th parameter be true(readNeedleBody) and modify below code:\r\n![image](https://cloud.githubusercontent.com/assets/10990210/16112565/6c948cee-33ea-11e6-9d81-41dd661633d2.png)\r\nAlthough it took a little while to find the next valid needle(without barrier's needle's help, crc checksum is my lighthouse), it works. And the following needles' data is not corrupted, the scanning work triggered by abnormal branch will not run any more\r\n"}}
{"actor":{"display_login":"hxiaodon"}}