{"action":"created","issue":{"url":"https://api.github.com/repos/GFDRR/thinkhazard/issues/530","repository_url":"https://api.github.com/repos/GFDRR/thinkhazard","labels_url":"https://api.github.com/repos/GFDRR/thinkhazard/issues/530/labels{/name}","comments_url":"https://api.github.com/repos/GFDRR/thinkhazard/issues/530/comments","events_url":"https://api.github.com/repos/GFDRR/thinkhazard/issues/530/events","html_url":"https://github.com/GFDRR/thinkhazard/issues/530","id":142666453,"number":530,"title":"Archive database","user":{"login":"pgiraud","id":319774,"avatar_url":"https://avatars.githubusercontent.com/u/319774?v=3","gravatar_id":"","url":"https://api.github.com/users/pgiraud","html_url":"https://github.com/pgiraud","followers_url":"https://api.github.com/users/pgiraud/followers","following_url":"https://api.github.com/users/pgiraud/following{/other_user}","gists_url":"https://api.github.com/users/pgiraud/gists{/gist_id}","starred_url":"https://api.github.com/users/pgiraud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pgiraud/subscriptions","organizations_url":"https://api.github.com/users/pgiraud/orgs","repos_url":"https://api.github.com/users/pgiraud/repos","events_url":"https://api.github.com/users/pgiraud/events{/privacy}","received_events_url":"https://api.github.com/users/pgiraud/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"arnaud-morvan","id":6095494,"avatar_url":"https://avatars.githubusercontent.com/u/6095494?v=3","gravatar_id":"","url":"https://api.github.com/users/arnaud-morvan","html_url":"https://github.com/arnaud-morvan","followers_url":"https://api.github.com/users/arnaud-morvan/followers","following_url":"https://api.github.com/users/arnaud-morvan/following{/other_user}","gists_url":"https://api.github.com/users/arnaud-morvan/gists{/gist_id}","starred_url":"https://api.github.com/users/arnaud-morvan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arnaud-morvan/subscriptions","organizations_url":"https://api.github.com/users/arnaud-morvan/orgs","repos_url":"https://api.github.com/users/arnaud-morvan/repos","events_url":"https://api.github.com/users/arnaud-morvan/events{/privacy}","received_events_url":"https://api.github.com/users/arnaud-morvan/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-03-22T14:27:25Z","updated_at":"2016-03-25T14:08:34Z","closed_at":null,"body":"Each time the data is published, the public database should be archived.\r\nWe need to define how the archive should be done.\r\nCreating a dump each time the data is published doesn't seem to good solution since 95% of the data is similar from one version to an other. Instead, we should investigate the usage of `barman`."},"comment":{"url":"https://api.github.com/repos/GFDRR/thinkhazard/issues/comments/201296521","html_url":"https://github.com/GFDRR/thinkhazard/issues/530#issuecomment-201296521","issue_url":"https://api.github.com/repos/GFDRR/thinkhazard/issues/530","id":201296521,"user":{"login":"arnaud-morvan","id":6095494,"avatar_url":"https://avatars.githubusercontent.com/u/6095494?v=3","gravatar_id":"","url":"https://api.github.com/users/arnaud-morvan","html_url":"https://github.com/arnaud-morvan","followers_url":"https://api.github.com/users/arnaud-morvan/followers","following_url":"https://api.github.com/users/arnaud-morvan/following{/other_user}","gists_url":"https://api.github.com/users/arnaud-morvan/gists{/gist_id}","starred_url":"https://api.github.com/users/arnaud-morvan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arnaud-morvan/subscriptions","organizations_url":"https://api.github.com/users/arnaud-morvan/orgs","repos_url":"https://api.github.com/users/arnaud-morvan/repos","events_url":"https://api.github.com/users/arnaud-morvan/events{/privacy}","received_events_url":"https://api.github.com/users/arnaud-morvan/received_events","type":"User","site_admin":false},"created_at":"2016-03-25T14:08:34Z","updated_at":"2016-03-25T14:08:34Z","body":"After some reading on this, I learn that : \r\n\r\nPostgreSQL have a continuous archiving system : http://www.postgresql.org/docs/9.4/static/continuous-archiving.html\r\nThis permit to have a base backup (can be not consistent) and a transation log (called WAL). With this it is possible to recover any state of the database at any time. This is not exactly what we need, we only want to keep state of the database at publish time, not the full transaction log.\r\n\r\nBarman propose an incremental solution, base on rsync and filesystem hard links to share files that do not change between consecutive backups. This seems to respond precesely to our needs. http://docs.pgbarman.org/release/1.6.0/#incremental-backup\r\n\r\nSo it should be possible, in publish script, to backup the processing database locally (main purpose of barman is to backup remotely), in a incremental cluster. After that, publishing will recover that backup on public consultation database."}}