{"action":"created","issue":{"url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/issues/1763","repository_url":"https://api.github.com/repos/cyrusimap/cyrus-imapd","labels_url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/issues/1763/labels{/name}","comments_url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/issues/1763/comments","events_url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/issues/1763/events","html_url":"https://github.com/cyrusimap/cyrus-imapd/issues/1763","id":197772777,"number":1763,"title":"Backups for SMB (lock entire server for a moment while taking a snapshot)","user":{"login":"brong","id":17645,"avatar_url":"https://avatars.githubusercontent.com/u/17645?v=3","gravatar_id":"","url":"https://api.github.com/users/brong","html_url":"https://github.com/brong","followers_url":"https://api.github.com/users/brong/followers","following_url":"https://api.github.com/users/brong/following{/other_user}","gists_url":"https://api.github.com/users/brong/gists{/gist_id}","starred_url":"https://api.github.com/users/brong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brong/subscriptions","organizations_url":"https://api.github.com/users/brong/orgs","repos_url":"https://api.github.com/users/brong/repos","events_url":"https://api.github.com/users/brong/events{/privacy}","received_events_url":"https://api.github.com/users/brong/received_events","type":"User","site_admin":false},"labels":[{"id":428554134,"url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/labels/3.0","name":"3.0","color":"fbca04","default":false},{"id":378124159,"url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-27T23:44:08Z","updated_at":"2016-12-27T23:45:47Z","closed_at":null,"body":"Via email from Anatoli <me@anatoli.ws>\r\n\r\n1. Small (SMB) deployments with a single server and somehow limited physical resources (e.g. disk space).\r\n\r\nHere as an example comes the excellent backup mechanism Ellie implemented that suits well the needs of medium to large deployments, but IMO that's not the best approach for small deployments, as it requires a separate server or, if ran at the same server just for the safe data-to-disk synchronization, twice the disk space.\r\n\r\nA better approach for small deployments, as I see it (and I believe it's highly demanded by the community), would be to have an executable that would instruct Cyrus daemon to synchronize to disk all the internal structures and lock (stop writing to disk) for a defined period. The lock could be implemented by hanging on network write requests or by writing them to temporary files, or by accumulating the changes in memory (the latter approach has a potential for data loss).\r\n\r\nOnce the flush is performed and the lock is applied, a (custom) backup script could create a snapshot of the partition that would hold the Cyrus data in a safe-to-backup state. Immediately after creating the snapshot, the lock would be released and the daemon would continue its normal operation. Then the backup script would be able to safely backup the data, e.g. create an incremental backup and upload it to some external storage, then destroy the snapshot.\r\n\r\nUsage example: cyrus_sync2disk --lock=5 -> returns 0 when the data is synced and a lock for 5 seconds is obtained. cyrus_sync2disk --unlock -> returns 0 if the lock has been released and 1 if there was no active lock (e.g a previous lock has expired), so the backup script knows if it performed the required operations with the lock still in place or if it should perform the lock-snapshot-unlock operation again. The short timeout is to protect the daemon from an infinite lock if a backup script fails to unlock it."},"comment":{"url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/issues/comments/269399088","html_url":"https://github.com/cyrusimap/cyrus-imapd/issues/1763#issuecomment-269399088","issue_url":"https://api.github.com/repos/cyrusimap/cyrus-imapd/issues/1763","id":269399088,"user":{"login":"brong","id":17645,"avatar_url":"https://avatars.githubusercontent.com/u/17645?v=3","gravatar_id":"","url":"https://api.github.com/users/brong","html_url":"https://github.com/brong","followers_url":"https://api.github.com/users/brong/followers","following_url":"https://api.github.com/users/brong/following{/other_user}","gists_url":"https://api.github.com/users/brong/gists{/gist_id}","starred_url":"https://api.github.com/users/brong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brong/subscriptions","organizations_url":"https://api.github.com/users/brong/orgs","repos_url":"https://api.github.com/users/brong/repos","events_url":"https://api.github.com/users/brong/events{/privacy}","received_events_url":"https://api.github.com/users/brong/received_events","type":"User","site_admin":false},"created_at":"2016-12-27T23:45:47Z","updated_at":"2016-12-27T23:45:47Z","body":"I understand what you want here, and I mostly like the idea.\r\n\r\nThe one thing that gives me pause is that it requires a single lock against ALL cyrus processes.  Right now, there's no global lock that processes take while making changes, and we'd need to add one.  I would want to make it be something that needs to be turned on in config so that sites which DON'T need it don't have to pay the extra locking cost.\r\n\r\nBut the design is definitely viable.  I want to do some other things with locking as well, like a single global lock for moves between users, renames, etc - so that we don't have lock ordering issues with those things."}}
{"actor":{"display_login":"brong"}}