{"action":"created","issue":{"url":"https://api.github.com/repos/nodejs/build/issues/308","labels_url":"https://api.github.com/repos/nodejs/build/issues/308/labels{/name}","comments_url":"https://api.github.com/repos/nodejs/build/issues/308/comments","events_url":"https://api.github.com/repos/nodejs/build/issues/308/events","html_url":"https://github.com/nodejs/build/issues/308","id":127372629,"number":308,"title":"Backup - the complete edition","user":{"login":"jbergstroem","id":176984,"avatar_url":"https://avatars.githubusercontent.com/u/176984?v=3","gravatar_id":"","url":"https://api.github.com/users/jbergstroem","html_url":"https://github.com/jbergstroem","followers_url":"https://api.github.com/users/jbergstroem/followers","following_url":"https://api.github.com/users/jbergstroem/following{/other_user}","gists_url":"https://api.github.com/users/jbergstroem/gists{/gist_id}","starred_url":"https://api.github.com/users/jbergstroem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbergstroem/subscriptions","organizations_url":"https://api.github.com/users/jbergstroem/orgs","repos_url":"https://api.github.com/users/jbergstroem/repos","events_url":"https://api.github.com/users/jbergstroem/events{/privacy}","received_events_url":"https://api.github.com/users/jbergstroem/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-01-19T05:56:03Z","updated_at":"2016-01-26T06:12:18Z","closed_at":null,"body":"Lets tackle this thing once and for all. I think there's three things we should accomplish with this issue; one being _how_ to backup, second _where_ and third _what_.\r\n\r\n### How \r\n\r\nMy preferred tool for managing backups is [rsnapshot](http://rsnapshot.org). It makes rotating daily/weekly/monthly a breeze and manages directory structures by hard symlinking stuff (meaning it doesn't store the same file twice). Not sure about options here, feel free to add your own. If we're backing up databases we can just pipe mysqldump/pg_dump to xz and store it in a folder.\r\n\r\nIf we were to use rsync, we'd create a specific backup ssh keypair and either create a user for backing up (forces us to run through permission) or use root (boo). \r\n\r\n\r\n#### Where\r\n\r\nLuckfully the sponsors we have available to us have lots of storage. I'd like to avoid object storage since it makes using rsync harder. This means the best case scenario would be using a provider that allows us to grow a disk (no need to waste costs on 1T when we'd consume 50G). The option closest to mind would be smartos on joyent which through zones/zfs allows us to transparently (read: admin click) grow the disk. We should start with ~300G and try to expand around the 75% rule. A cron job that reminds us about disk consumption would likely be helpful.\r\n\r\n\r\n#### What\r\n\r\nAssuming I've forgotten most stuff, here's what we want to store:\r\n\r\n - nodejs/iojs www\r\n   - copy of `/home/www` - so we quickly can redeploy as well as archiving binaries\r\n   - `/var/log/nginx/*` - this is already logrotated\r\n - jenkins host\r\n   - `/var/lib/jenkins` - this is a tricky one. I'd like to make exceptions for all build history -- this accumulates around 170G at the moment.\r\n  - `/var/log/nginx/jenkins-access.log` access logs for jenkins\r\n - benchmarking host\r\n   - mysqldump of raw data\r\n\r\nPlease add more here -- I'd personally like to avoid backing up stuff we can regenerate; but if it saves us time it might be worth it.\r\n\r\n##### Open questions\r\n\r\n - (jenkins host) where do we store /etc/defaults/jenkins? suggesting part of ansible scripts\r\n - (www host) how do we treat the cold storage in /home/nodejs/cloudfuse?\r\n\r\n\r\n\r\nI can attempt to PR an ansible playbook that would set up this host at joyent (and edit hosts to copy ssh key/modify permission as needed) after we've agreed on what path we should choose."},"comment":{"url":"https://api.github.com/repos/nodejs/build/issues/comments/174850500","html_url":"https://github.com/nodejs/build/issues/308#issuecomment-174850500","issue_url":"https://api.github.com/repos/nodejs/build/issues/308","id":174850500,"user":{"login":"rvagg","id":495647,"avatar_url":"https://avatars.githubusercontent.com/u/495647?v=3","gravatar_id":"","url":"https://api.github.com/users/rvagg","html_url":"https://github.com/rvagg","followers_url":"https://api.github.com/users/rvagg/followers","following_url":"https://api.github.com/users/rvagg/following{/other_user}","gists_url":"https://api.github.com/users/rvagg/gists{/gist_id}","starred_url":"https://api.github.com/users/rvagg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvagg/subscriptions","organizations_url":"https://api.github.com/users/rvagg/orgs","repos_url":"https://api.github.com/users/rvagg/repos","events_url":"https://api.github.com/users/rvagg/events{/privacy}","received_events_url":"https://api.github.com/users/rvagg/received_events","type":"User","site_admin":false},"created_at":"2016-01-26T06:12:18Z","updated_at":"2016-01-26T06:12:18Z","body":"Additional directories on nodejs.org we need to save:\r\n\r\n* /home/dist/nodejs/ (binaries, vital, most important thing we can save!)\r\n* /home/dist/iojs/ (binaries)\r\n* /home/dist/metrics/ (questionable, they are generated from the log files but can take a day or more to generate, nearly 10G atm)\r\n* /home/nodejs/old/ (historical stuff from Joyent server, worth keeping, won't ever change though)\r\n* /home/libuv/www/dist/\r\n\r\nI'm also a fan of just bundling up and storing /etc/ since it's so small and often contains stuff that you wish you kept but didn't realise wasn't stored anywhere else."}}