{"action":"created","issue":{"url":"https://api.github.com/repos/vatesfr/xen-orchestra/issues/3395","repository_url":"https://api.github.com/repos/vatesfr/xen-orchestra","labels_url":"https://api.github.com/repos/vatesfr/xen-orchestra/issues/3395/labels{/name}","comments_url":"https://api.github.com/repos/vatesfr/xen-orchestra/issues/3395/comments","events_url":"https://api.github.com/repos/vatesfr/xen-orchestra/issues/3395/events","html_url":"https://github.com/vatesfr/xen-orchestra/issues/3395","id":358058511,"node_id":"MDU6SXNzdWUzNTgwNTg1MTE=","number":3395,"title":"More consistent behaviour for delta backup snapshots","user":{"login":"Samuel-BF","id":36996277,"node_id":"MDQ6VXNlcjM2OTk2Mjc3","avatar_url":"https://avatars0.githubusercontent.com/u/36996277?v=4","gravatar_id":"","url":"https://api.github.com/users/Samuel-BF","html_url":"https://github.com/Samuel-BF","followers_url":"https://api.github.com/users/Samuel-BF/followers","following_url":"https://api.github.com/users/Samuel-BF/following{/other_user}","gists_url":"https://api.github.com/users/Samuel-BF/gists{/gist_id}","starred_url":"https://api.github.com/users/Samuel-BF/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Samuel-BF/subscriptions","organizations_url":"https://api.github.com/users/Samuel-BF/orgs","repos_url":"https://api.github.com/users/Samuel-BF/repos","events_url":"https://api.github.com/users/Samuel-BF/events{/privacy}","received_events_url":"https://api.github.com/users/Samuel-BF/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-07T13:02:53Z","updated_at":"2018-09-07T13:34:37Z","closed_at":null,"author_association":"NONE","body":"<!--\r\nWelcome to the issue section of Xen Orchestra!\r\n\r\nHere you can:\r\n- report an issue\r\n- propose an enhancement\r\n- ask a question\r\n\r\nPlease, respect this template as much as possible, it helps us sort\r\nthe issues :)\r\n-->\r\n### Summary\r\n\r\n- from a user point of view, we want to define a retention schema using \"backup schedules\" but it fails as we can't predict which schedule will apply when they are conflicting.\r\n- for delta backups (and for other jobs such as CR/DR i guess), there is one snapshot made per retention period, which is pointless (you only need one snapshot per backup job).\r\n\r\n\r\n### Context\r\n\r\n- **XO origin**: the sources\r\n- **Versions**:\r\n   - Node: v8.11.4\r\n   - xo-web: 5.26\r\n   - xo-server: 5.26\r\n\r\n### Expected behavior\r\n\r\nLet's suppose you want to keep 1 backup/month for 6 month, 1 backup/week for 4 weeks and 1 backup/day for 8 days. For efficiency reasons, you use delta backups. With backup-NG, you create a backup job with 3 schedules :\r\n\r\n- monthly : retention = 6 / cron = 0 0 1 * *\r\n- weekly : retention = 4 / cron = 0 0 * * 1\r\n- daily : retention = 8 / cron = 0 0 * * *\r\n\r\nWith this scheme, you expect that a delta backup occurs daily and that one or several merges occurs depending on which day of week or month you are. The schedules are defining only the merges. So **I expect to have only one snapshot on the hypervisor.** \r\n\r\nAnd after a while, I should have backups for : 04-01,  05-01, 06-01, 07-01, 08-01 (monthly backups), 09-01 (monthly and daily backups), 08-13, 08-20, 08-27 (weekly backups), 09-03 (weekly & daily backup), 08-31, 09-02, 09-04, 09-05, 09-06, 09-07 (daily backups). **I will have at least 16 backups and at most 17 backups** (I can't have 18 backups, as there will always be one backup which is both weekly and daily).\r\n\r\nAnd, more important, **I will have a backup per week and a backup per month**. \r\n\r\n\r\n### Current behavior\r\n\r\nAs schedules have the same time (0:0), the schedule running monday D is either the weekly or the daily one. And the schedule running the 1st of month is either the monthly either the daily one (this doesn't seems predictable). This has several consequences : \r\n\r\n- there is 3 snapshots kept on the hypervisor (1/schedule), which is useless (and I can remove the all-but-the-youngest snapshots without damage for future backups). This represents wasted storage.\r\n- If a daily snapshot occurs rather than the monthly one, it will not be kept more than 8 days. I may have holes in my backup timetables.\r\n\r\n\r\nSee for example what I got for a similar scheme with more frequent backups (7 x 0,10,20,30,40,50 * * * *, 4 x 10,40 * * * *, 20 x 10 * * * * =  7 x 1 backup per 10 min, 4 x 1 backup at *:{10,40} and 20 x 1 backup at *:10) : \r\n```\r\n$ jq '.scheduleId + \" \" + .vmSnapshot.snapshot_time' *.json -r\r\n73be1820-c879-49b5-a716-8e9569db5325 20180905T16:10:15Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180905T19:12:48Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180905T22:10:21Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T00:10:07Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T01:10:19Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T03:10:06Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T05:10:05Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T08:10:06Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T09:10:06Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T12:10:05Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T14:10:07Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T16:10:32Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T19:14:43Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180906T22:10:14Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180907T00:10:11Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180907T02:10:07Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180907T05:10:05Z\r\n12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T06:10:06Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180907T07:10:07Z\r\n12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T07:40:07Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180907T08:10:11Z\r\n12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T08:40:07Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T08:50:07Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:10:17Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:20:12Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:30:06Z\r\n12737e38-9763-48f0-be04-3b9ba80e05d3 20180907T09:40:06Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T09:50:08Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T10:00:08Z\r\n73be1820-c879-49b5-a716-8e9569db5325 20180907T10:10:12Z\r\n76bdde01-9dda-4085-97be-abb3791fa6bc 20180907T10:20:24Z\r\n```\r\n\r\nThere are 31 backups (20+4+7) and holes of several hours between \"hourly\" backups.\r\n\r\n### Workarounds\r\n\r\n1. To prevent holes, schedules should not occur at the same time (but this is a waste of resources, since you have to uselessly snapshot and backup twice on mondays)\r\n2. Manually remove extra snapshots when needed (there is no 'snapshot.remove' task in jobs)"},"comment":{"url":"https://api.github.com/repos/vatesfr/xen-orchestra/issues/comments/419440871","html_url":"https://github.com/vatesfr/xen-orchestra/issues/3395#issuecomment-419440871","issue_url":"https://api.github.com/repos/vatesfr/xen-orchestra/issues/3395","id":419440871,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTQ0MDg3MQ==","user":{"login":"Jarli01","id":14837699,"node_id":"MDQ6VXNlcjE0ODM3Njk5","avatar_url":"https://avatars2.githubusercontent.com/u/14837699?v=4","gravatar_id":"","url":"https://api.github.com/users/Jarli01","html_url":"https://github.com/Jarli01","followers_url":"https://api.github.com/users/Jarli01/followers","following_url":"https://api.github.com/users/Jarli01/following{/other_user}","gists_url":"https://api.github.com/users/Jarli01/gists{/gist_id}","starred_url":"https://api.github.com/users/Jarli01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jarli01/subscriptions","organizations_url":"https://api.github.com/users/Jarli01/orgs","repos_url":"https://api.github.com/users/Jarli01/repos","events_url":"https://api.github.com/users/Jarli01/events{/privacy}","received_events_url":"https://api.github.com/users/Jarli01/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T13:34:37Z","updated_at":"2018-09-07T13:34:37Z","author_association":"NONE","body":"I believe the biggest reason this won't work is that each snapshot has a unique UUID and multiple snapshots makeup the chain. Removing a link breaks the chain, thus you are required to have multiple chains with snapshots when you have multiple retention periods in cases like this. \r\n\r\n"}}
{"actor":{"display_login":"Jarli01"}}