{"action":"opened","issue":{"url":"https://api.github.com/repos/borgbackup/borg/issues/635","repository_url":"https://api.github.com/repos/borgbackup/borg","labels_url":"https://api.github.com/repos/borgbackup/borg/issues/635/labels{/name}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/635/comments","events_url":"https://api.github.com/repos/borgbackup/borg/issues/635/events","html_url":"https://github.com/borgbackup/borg/issues/635","id":131801470,"number":635,"title":"Backup to multiple remote locations","user":{"login":"oysols","id":10567774,"avatar_url":"https://avatars.githubusercontent.com/u/10567774?v=3","gravatar_id":"","url":"https://api.github.com/users/oysols","html_url":"https://github.com/oysols","followers_url":"https://api.github.com/users/oysols/followers","following_url":"https://api.github.com/users/oysols/following{/other_user}","gists_url":"https://api.github.com/users/oysols/gists{/gist_id}","starred_url":"https://api.github.com/users/oysols/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oysols/subscriptions","organizations_url":"https://api.github.com/users/oysols/orgs","repos_url":"https://api.github.com/users/oysols/repos","events_url":"https://api.github.com/users/oysols/events{/privacy}","received_events_url":"https://api.github.com/users/oysols/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-06T01:07:46Z","updated_at":"2016-02-06T01:07:46Z","closed_at":null,"body":"Currently I don't see how the backup can be synced to multiple remote locations without running the backup multiple times. It seems reasonable to first do a local backup, then sync the backup files to the remote locations.\r\n\r\nRsync can be used to sync the backup files, but due to how borg compacts the segment files this will result in a lot of duplicated data transfer if there are few or no changes to the backup set.\r\n\r\nEach time borg creates a new backup, a new segment file is created. If there are no or few changes this file is very small. However, borg combines this segment file with the existing segment file to minimize the total amout of files. Using rsync to update the changes results in copying this combined file, and results in an average of (maximum segment file size(5MB) / 2 =) 2.5 MB extra data transfered for every single backup.\r\n\r\nI have tried a simple hack by simply not compacting the segment files, and this works quite well. (very limited testing) There might however be good reasons for compacting the segments, such as being\r\nable to delete old backups.\r\n\r\nMy proposal is to only compact segments when there is enough data to fill a new segment file. This will drastically decrease the amount of duplicated data transfer.\r\n\r\nOf course, any other solution for syncing the backup files will be appreciated.\r\n\r\nIt might be interesting to look at the paper describing Duplicati 2.0 block based storage model. Specifically the assumption of a \"dumb\" storage backend.\r\nhttps://duplicati.googlecode.com/files/Block-basedstorageformat.pdf"}}