{"action":"created","issue":{"url":"https://api.github.com/repos/mcgid/morenines/issues/3","repository_url":"https://api.github.com/repos/mcgid/morenines","labels_url":"https://api.github.com/repos/mcgid/morenines/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/mcgid/morenines/issues/3/comments","events_url":"https://api.github.com/repos/mcgid/morenines/issues/3/events","html_url":"https://github.com/mcgid/morenines/issues/3","id":148993556,"number":3,"title":"Write entirely new index files instead of overwriting","user":{"login":"mcgid","id":6307566,"avatar_url":"https://avatars.githubusercontent.com/u/6307566?v=3","gravatar_id":"","url":"https://api.github.com/users/mcgid","html_url":"https://github.com/mcgid","followers_url":"https://api.github.com/users/mcgid/followers","following_url":"https://api.github.com/users/mcgid/following{/other_user}","gists_url":"https://api.github.com/users/mcgid/gists{/gist_id}","starred_url":"https://api.github.com/users/mcgid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcgid/subscriptions","organizations_url":"https://api.github.com/users/mcgid/orgs","repos_url":"https://api.github.com/users/mcgid/repos","events_url":"https://api.github.com/users/mcgid/events{/privacy}","received_events_url":"https://api.github.com/users/mcgid/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mcgid/morenines/labels/bug","name":"bug","color":"ee0701"},{"url":"https://api.github.com/repos/mcgid/morenines/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-17T20:31:40Z","updated_at":"2016-04-17T20:53:05Z","closed_at":null,"body":"Currently, the `update` command reads the existing index file, and after modifications to the content, overwrites it. If the power goes out, the process is killed, etc., the index could be left in an inconsistent state. That is unacceptable; corruption should be practically impossible.\r\n\r\nIf we write the index as a new file, there is also no reason to remove previous versions of the index provided we don't use them for anything. Many thousands of old versions would have to accumulate before it was of any concern. (At that point, we could just add some sort of garbage collection command that would tarball the old files, delete files older than a certain point, a combination of those, etc.)\r\n\r\nSolving this will require modifying the behaviour of `update` such that it is fail-safe: an index should not be semantically identified as the current, newest index until it has, as far as we are able to know[1], been completely written to disk.\r\n\r\n---\r\n\r\n[1]: Obviously, with modern filesystems, actually finding out if a file has been written is nontrivial. Going beyond the simple implication of \"`file.write()` has returned\" is not presently worth it."},"comment":{"url":"https://api.github.com/repos/mcgid/morenines/issues/comments/211114544","html_url":"https://github.com/mcgid/morenines/issues/3#issuecomment-211114544","issue_url":"https://api.github.com/repos/mcgid/morenines/issues/3","id":211114544,"user":{"login":"mcgid","id":6307566,"avatar_url":"https://avatars.githubusercontent.com/u/6307566?v=3","gravatar_id":"","url":"https://api.github.com/users/mcgid","html_url":"https://github.com/mcgid","followers_url":"https://api.github.com/users/mcgid/followers","following_url":"https://api.github.com/users/mcgid/following{/other_user}","gists_url":"https://api.github.com/users/mcgid/gists{/gist_id}","starred_url":"https://api.github.com/users/mcgid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcgid/subscriptions","organizations_url":"https://api.github.com/users/mcgid/orgs","repos_url":"https://api.github.com/users/mcgid/repos","events_url":"https://api.github.com/users/mcgid/events{/privacy}","received_events_url":"https://api.github.com/users/mcgid/received_events","type":"User","site_admin":false},"created_at":"2016-04-17T20:53:05Z","updated_at":"2016-04-17T20:53:05Z","body":"There are two issues to deal with in picking a solution: how to name each index file when writing, and how to identify which index file is the correct/newest one when reading.\r\n\r\n##### Naming Index Files\r\n\r\nSome immediately obvious approaches:\r\n\r\n1. Include a timestamp.\r\n2. Include a monotonic counter.\r\n3. Hash the content of the index file and write it into a content-addressable-storage dir+name path (à la Git).\r\n4. Include a random UUID.\r\n\r\n##### Finding the Newest Index File\r\n\r\n1. If the index files have timestamped filenames, lexicographically sort the names and pick the one with the newest timestamp.\r\n2. If the index files have a monotonic counter in their filenames, parse all of the names into integers and pick the one with the largest counter value. \r\n3. Regardless of file names, keep a separate `HEAD` file, that contains the name of the newest index file.\r\n\r\nEach of these approaches has problems.\r\n\r\n- **Timestamps**: Daylight Savings Time breaks this. Time zone changes could break this if we're not careful. In general, dates are a terrible means of identifying things.\r\n- **Counter**: This requires converting between strings and integers, which doesn't have the best track record. It's a one-liner in Python, however, and \"should\" work fine.\r\n- **Separate `HEAD` file**: It seems questionable to solve the problem of overwriting a file by overwriting a different file; the same problems of atomicity and durability would apply, regardless of the fact that this would be a much smaller file.\r\n\r\n    Git does this, but cargo-culting isn't a good reason. Additionally, it should be possible to read all of Git's object files and re-create the entire graph from scratch, using (among other things) the `parent` headers. As they currently exist, this would not be possible with the morenines index files.\r\n\r\n"}}