backup (Blog) en New Hub feature: Auto-Restore TKLBAM backup to a new cloud server <p> Since we <a href="">announced</a> the release of <a href="">TurnKey Hub v1.0</a> two weeks ago, we followed up with the <a href="">two top issues users reported</a>, and continued to receive awesome feedback - you guys rock, keep it coming!</p> <p> A common question we received was related to restoring a backup to a new server. This wasn&#39;t a new question either, last year <a href="">Phil Bower said</a>: <em>&quot;I only want to run my instances while I&#39;m using them so I&#39;m interested in figuring out the fastest way to create new instances and restore backups to them.&quot;</em></p> <p> In light of this, we just released a new Hub feature that streamlines the restore process depending on your use case. It also makes testing your backups even easier!</p> <h2> Restore this backup to a new cloud server</h2> <p> Alice is a consultant developing a new site for her client on Turnkey Wordpress in a local virtual machine. She is regularly backing up her work with <a href="">TKLBAM</a> and has reached version 1.0 status - time for production.</p> <p> Alice logs into her Hub account, goes to the backup dashboard and clicks <em>&quot;Restore this backup to a new cloud server&quot;</em>. The Hub knows which appliance is related to her backup and launches a Wordpress instance. TKLBAM is automatically initialized and her backup is restored, essentially migrating her local VM to the cloud.</p> <p> <img alt="Restore on Launch 1" src="" style="width: 600px; height: 213px; " /></p> <p> Alice jumps up and down with excitement. She immediately calls up her client to give them the great news. Development to production in 3 minutes flat!</p> <h2> Launch a new server like this one</h2> <p> Bob (aka Evil Bob) has a highly customized TurnKey LAMP instance running in the cloud. He has added user accounts, installed packages, saved data on the filesystem as well as MySQL. Additionally, he configured an EBS volume and Elastic IP when he launched the instance. He even customized the firewall rules.</p> <p> Bob only needs the instance to be up about for a few days every week while crunching data for his evil plan, but launching a new instance, attaching the EBS, associating the EIP, tweaking the firewall rules and manually restoring his TKLBAM backup is a little tedious.</p> <p> Tedious? Not any more! Once a week Bob logs into his Hub account, goes to the server dashboard and clicks &quot;Launch a new server like this one&quot;. &nbsp;He clicks the checkbox to automatically restore his backup, and he&#39;s off! The Hub launches a new server, automatically attaches his EBS volume (<a href="">EBSMount</a> mounts it), associates his Elastic IP, configures his custom firewall rules and restores his backup.&nbsp;</p> <p> <img alt="Restore on launch 2" src="" style="width: 600px; height: 198px; " /></p> <p> The lights dimm, Evil Bob&#39;s face lights up, raises his chin and lets out an evil laugh - MUHAHA!!</p> <p> <strong>Note:</strong> The new restore features are not available for passphrase protected backups or legacy builds.</p> aws backup cloud ec2 hub restore tklbam Thu, 14 Apr 2011 16:19:00 +0000 Alon Swartz 2212 at Announcing TurnKey Hub v1.0 - now officially out of private beta <p style="text-align: center; padding-top: 10px;"> <a href=""><img alt="Hub Front" src="" style="width: 640px; height: 461px; " /></a></p> <p> When we first announced the <a href="">TurnKey Hub private beta</a> about 9 months ago, we had limited capacity (invitation only) and a modest feature set. Since then we tested, bugfixed, removed bottlenecks and added features, constantly improving the Hub with the help and feedback from our excellent beta users. Thank you so much!</p> <p> With the release of <a href="">TurnKey 11</a> which was tightly integrated with TKLBAM and the Hub, the amount of Hub invitation requests exploded. We were prepared for this and managed to scale the Hub smoothly without any serious issues.</p> <p> With several months of testing, feedback and bugfixes under our belt we are now confident enough to officially announce, a bit earlier than planned, that the Hub is out of private beta. As of today, the <a href="">Hub</a> is open to all, and new users will no longer be required to request an invitation.</p> <p> Existing users can rest easy though. We will continue to carefully monitor the Hub&#39;s performance. There should be no interruptions to the service. Worse case scenario, if we start hitting unforseen capacity issues we will temprarily reintroduce the limit on new signups.</p> <h2> Review of notable changes since the initial release</h2> <p> <strong>TurnKey Backup and Migration</strong></p> <ul> <li> A few months into the private beta we announced support for <a href=" ">TurnKey Backup and Migration</a> (AKA TKLBAM), which amongst other uses makes previously difficult tasks such as <a href=" ">testing your backups</a> much easier.</li> <li> In response to demand, we&#39;ve added support for configurable backup retention. Users can specify how many full backups they would like to keep for any given server backup (set to unlimited by default).</li> </ul> <p> <strong>TurnKey Cloud Servers</strong></p> <ul> <li> Support for TurnKey Linux 11 images (legacy images still available to ease migration).</li> <li> Basic pre-launch configuration: No more having to fiddle with the <a href="">default passwords</a> after an instance launches. The Hub supports <a href="">pre-seeding appliance configuration before launch</a>. This makes up for not having console access that would usually be required for first boot configuration.</li> <li> TKLBAM pre-initialization: No more having to cut and paste your Hub APIKEY to initialize TKLBAM. The Hub pre-initializes TKLBAM automatically when the instance is first launched.</li> <li> Upgradeable Kernels: We&#39;ve figured out how to make it easy to update the kernel via pv-grub.</li> <li> Preset launch region automatically chosen by geo-location of user.</li> </ul> <p> <strong>General stuff</strong></p> <ul> <li> Performance optimizations, improved stability and error handling.</li> <li> Refined the look and feel with an update to the theme.</li> <li> We now try harder to explain how the Hub works and what it&#39;s good for before and after you sign up. For example we&#39;ve added nice visual tours of the <a href="">Backup and migration</a> and <a href="">Cloud servers</a> features.</li> <li> We&#39;ve added a <a href=" ">pricing page</a> answering frequently asked questions. Yes, the Hub is still free. You pay Amazon directly for the cloud resources you use.</li> <li> Improved start page to get you going once you sign up. Once you setup your account, this transforms into a dashboard that provides a high level overview and quick access links.</li> <li> New and improved notifications (<a href="">growl style</a>).</li> <li> Removed invitation requirement and added support for OpenID signup and authentication.</li> <li> Added functionality to change account email.</li> <li> Full internationalization support (<a href="">UTF-8</a>).</li> <li> APT archive geo-location API service for choosing the <a href="">closest package archive</a>.</li> <li> Link to <a href="">Privacy policy</a>.</li> </ul> <p> As usual, feedback is appreciated. If you don&#39;t have a TurnKey Hub account yet, go get one now or try out the demo. If you already have a TurnKey Hub account, go check out the new stuff.</p> <p> The TurnKey Hub lives at: <strong><a href=""></a></strong></p> aws backup cloud ec2 hub news tklbam Wed, 30 Mar 2011 12:42:48 +0000 Alon Swartz 2166 at Passphrase dictionary attack countermeasures in tklbam's keying mechanism <h2> Background: how a backup key works</h2> <p> In TKLBAM the <em>backup key</em> is a <em>secret</em> encrypted with a passphrase which is uploaded to the <a href="" rel="nofollow">Hub</a>.&nbsp; Decrypting the <em>backup key</em> yields the <em>secret</em> which is passed on to duplicity (and eventually to GnuPG) to be used as the symmetric key with which backup volumes are encrypted on backup and decrypted on restore.</p> <p> When you create a new backup, or change the passphrase on an existing backup, a new <em>backup key</em> is uploaded to the Hub where it is stored in the key field for that backup record.</p> <p> When you restore, tklbam downloads the <em>backup key</em> from the Hub and decrypts it locally on the computer performing the restore. Note that the Hub only allows you to download the <em>backup key</em> for backup records to which you have access (e.g., you are the owner).</p> <h2> Only you can decrypt your passphrase protected backups</h2> <p> All of this matters because it means that as long as you use a passphrase to protect the key, <strong>even the Hub can&#39;t decrypt your backups, only you can</strong> - provided you remember the passphrase (or failing that, at least have the escrow key stored in a safe place).</p> <p> In other words, the decryption of the <em>backup key</em> happens locally and at no point does the passphrase reach the Hub, so we can&#39;t decrypt your backup even if you asked us to. Neither can an attacker that has theoretically compromised the Hub, or a government agency that comes kicking down our door with a court warrant.</p> <h2> The problem with cryptographic passphrases</h2> <p> But wait. If an attacker has local access to the key, his ability to run dictionary attacks to find the key&#39;s passphrase is limited only by the computational resources he can throw at it.</p> <p> Remember there&#39;s a critical difference between a passphrase used for authentication purposes (e.g., to an online service) and a passphrase used for cryptographic purposes.</p> <p> By contrast, a passphrase used for authenticating to an online service doesn&#39;t need to be as strong as a passphrase that is used cryptographically because with an online service, even if no explicit countermeasures are used (e.g., IP blacklisting on too many failed attempts) there is still a network between the attacker and the service. The available bandwidth places a debilitating upper limit on how many passphrases can be tried per second. Also, in practice there are usually&nbsp; bottlenecks in other places which would slow down an online dictionary attack even further.</p> <p> But a passphrase used for cryptographic purposes assumes the attacker has access to the ciphertext, and that&#39;s a whole different ball game.</p> <p> To better understand what we&#39;re up against, here&#39;s the formula for calculating the size of the passphrase search space:</p> <pre class="literal-block"> log(howmany_different_possible_values ** howmany_values) / log(2) </pre> <p> For example, consider a typical 6 letter password.</p> <p> 6 ASCII printable letters = maximum 42-bits of search space.</p> <p> That&#39;s a maximum of 4 trillion possible combinations. Which sounds like a lot. But it really isn&#39;t, since:</p> <ol class="arabic"> <li> <p class="first"> You can probably squeeze out about 1 million local passphrase tests per second from a modern multi-core workstation, assuming a typical passphrase encryption method is used.</p> </li> <li> <p class="first"> This is one of those problems that are trivial to parallelize.</p> <p> If you rent just 100 computers (e.g., in the cloud) you could exhaustively search through 42-bits in about 5 days.</p> <p> And remember, today the bad guys often have millions of computers at their disposal via botnets.</p> </li> <li> <p class="first"> People are very bad at choosing truly random passwords. A clever attacker will try the low hanging fruit first, so they&#39;re likely to find out your passphrase much sooner than by brute forcing blindly through the full search space.</p> <p> For example, say you know a 6 letter password is much too short for an encryption key and instead you&#39;re using a longer random combination of 10,000 common English words:</p> <ul class="simple"> <li> 2 words = 18-bits worth of search space.</li> <li> 3 words = 27-bits worth of search space.</li> <li> 4 words = 36-bits worth of search space.</li> </ul> <p> English words aren&#39;t very random so your &quot;paranoid&quot; 3 word, 17 letter passphrase may actually be easier to crack than a truly random combination of just 4 ASCII printable characters (28-bits).</p> <p> For comparison, let&#39;s see what happens if you use 6 random individual characters.</p> <p> If you just use random lowercase characters the search space is reduced to 27-bits which is 32,768 times easier to search through than the full 42-bit search space of 6-letter ASCII printable passwords.</p> <p> If you just use random lowercase characters and numbers, the search space is 30-bits which is 4,096 times easier to search through.</p> <p> If you just use random lowercase and uppercase characters and numbers, the search space is 35-bits which is 128 times easier to search through.</p> </li> </ol> <p> The good news is that each bit of search space doubles the expense for the attacker.</p> <p> The bad news is that it takes a truly random combination of 11 uppercase, lowercase characters and numbers just to reach 64-bits worth of search space, and a 10M strong botnet could crack even that in an average of 10 days.</p> <p> Bottom line: even your supposedly ultra-paranoid passphrase (e.g., r0m4n14nv4mp1r344rdv4rkn3st) of 4 random words from a dictionary of 150K words (in l33t speak) only has about 50-bits worth of entropy, despite being 27 characters long. A 10,000 botnet could crack that in about a day.</p> <h2> Countermeasures: increase computational cost</h2> <p> Though it&#39;s impossible to prevent these attacks entirely I&#39;ve implemented a couple of countermeasures in the way TKLBAM generates passphrase protected keys:</p> <p> 1) <strong>The first trick</strong>: increase how computationally expensive it is to calculate the cipher key from the passphrase:</p> <pre class="literal-block"> def _repeat(f, input, count): for x in xrange(count): input = f(input) return input def _cipher_key(passphrase, repeats): cipher_key = _repeat(lambda k: hashlib.sha256(k).digest(), passphrase, repeats) </pre> <p> The principle is that calculating a hash costs CPU time so by feeding the hash into itself enough times we can linearly increase how expensive it is to map the passphrase-space to the key-space.</p> <p> For example, repeating the hash routine 100,000 times takes about a quarter second on one of the cores of my computer. If I use all 4 cores this limits me to generating 16 cipher keys per second. Down from 1.6 million cipher keys per second. So that&#39;s one way to dramatically reduce the practical feasibility of a dictionary or exhaustive brute force attack.</p> <p> Note that an attacker can&#39;t circumvent this calculation by searching through the key-space directly because even after we increase the cost of generating the passphrase space a 100,000 times over, the cost of trying to bruteforce the 256-bit key-space directly is still countless trillions of times greater.</p> <p> The weakness of this technique is that an attacker would have to pay the cost of mapping the passphrase-space (e.g., a dictionary) to the key-space only once when trying to crack multiple keys.</p> <p> 2) <strong>The second trick</strong>: increase how computationally expensive it is to decrypt the key packet by increasing the number of times we pass it through encryption:</p> <pre class="literal-block"> def _cipher(cipher_key): return, AES.MODE_CBC) ciphertext = _repeat(lambda v: _cipher(cipher_key).encrypt(v), _pad(plaintext), cipher_repeats) </pre> <p> This part of the computational expense is key-specific so trading off memory to pre-calculate the mapped key-space won&#39;t help you with this step.</p> <h2> Implementation notes</h2> <h3> Embedding repeat parameters in the key packet</h3> <p> The current implementation hardwires 100,000 repeats of the hash, and another 100,000 repeats of the cipher.</p> <p> This makes searching through the passphrase-space about 200,000 times more expensive. On my workstation it takes 0.5 seconds to encrypt or decrypt a key (per-core).</p> <p> I&#39;m not sure these are the ideal parameters but they are in the ball park of how much you can increase the computational expense before usability suffers.</p> <p> That&#39;s not to say you couldn&#39;t go higher, but there&#39;s a practical upper boundary to that too. If you&#39;re willing to wait about a minute for key generation/decryption you could increase the computational expense about 100 times over and that would give you 100 times better protection or allow you to use a password that is 100 times weaker with the same protection.</p> <p> Just in case, to allow the number of repeats to change or be user configurable in the future the key formatter routine embeds the repeat parameters into the unencrypted portion of the key packet. This allows the key parsing routine to extract these parameters from the key itself so it can just as easily parse a 0.5 second key (I.e., the current default) as a 5 second, or 50 second key.</p> <h3> Embedding a version id</h3> <p> Just to make sure the key format is future proof I&#39;m also embedding a version id into it.</p> <p> Embedding a version costs almost nothing (an extra byte) and makes it easier to support incompatible changes to the key format should the need arise (e.g., changing of cipher/hash, changing the format, etc.).</p> <p> Worst case scenario, we increment the version and implement a new incompatible key format. Old clients won&#39;t be able to understand the new key format but will at least fail reliably, and new clients will be able to support both new and old key formats.</p> backup passphrase security tklbam Mon, 13 Sep 2010 09:14:11 +0000 Liraz Siri 1397 at TKLBAM: a new kind of smart backup/restore system that just works <h2> <a name="drumroll"></a>Drum roll please...</h2> <p> Today, I&#39;m proud to officially unveil TKLBAM (AKA TurnKey Linux Backup and Migration): the easiest, most powerful system-level backup anyone has ever seen. Skeptical? I would be too. But if you read all the way through you&#39;ll see I&#39;m not exaggerating and I have <a href="#screencast">the screencast</a> to prove it. Aha!</p> <p> <img alt="" src="" style="border: 1px solid rgb(170, 170, 170); margin-left: 9px; width: 650px; height: 317px;" /></p> <p> This was the missing piece of the puzzle that has been holding up the Ubuntu Lucid based release batch. You&#39;ll soon understand why and hopefully agree it was worth the wait.</p> <h2> <a name="design"></a>We set out to design the ideal backup system</h2> <p> Imagine the ideal backup system. That&#39;s what we did.</p> <h3> Pain free</h3> <p> A fully automated backup and restore system with no pain. That you wouldn&#39;t need to configure. That just magically knows what to backup and, just as importantly, what NOT to backup, to create super efficient, encrypted backups of changes to files, databases, package management state, even users and groups.</p> <h3> Migrate anywhere</h3> <p> An automated backup/restore system so powerful it would double as <em>a migration mechanism</em> to move or copy fully working systems anywhere in minutes instead of hours or days of error prone, frustrating manual labor.</p> <p> It would be so easy you would, shockingly enough, actually <a href="">test your backups</a>. No more excuses. As frequently as you know you should be, avoiding unpleasant surprises at the worst possible timing.</p> <p> One turn-key tool, simple and generic enough that you could just as easily use it to migrate a system:</p> <ul class="simple"> <li> from Ubuntu Hardy to Ubuntu Lucid (get it now?)</li> <li> from a local deployment, to a cloud server</li> <li> from a cloud server to any VPS</li> <li> from a virtual machine to bare metal</li> <li> from Ubuntu to Debian</li> <li> from 32-bit to 64-bit</li> </ul> <h3> System smart</h3> <p> Of course, you can&#39;t do that with a conventional backup. It&#39;s too dumb. You need a vertically integrated backup that has system level awareness. That knows, for example, which configuration files you changed and which you didn&#39;t touch since installation. That can leverage the package management system to get appropriate versions of system binaries from package repositories instead of wasting backup space.</p> <p> This backup tool would be smart enough to protect you from all the small paper-cuts that conspire to make restoring an ad-hoc backup such a nightmare. It would transparently handle technical stuff you&#39;d rather not think about like fixing ownership and permission issues in the restored filesystem after merging users and groups from the backed up system.</p> <h3> Ninja secure, dummy proof</h3> <p> It would be a tool you could trust to always encrypt your data. But it would still allow you to choose how much convenience you&#39;re willing to trade off for security.</p> <p> If data stealing ninjas keep you up at night, you could enable strong cryptographic passphrase protection for your encryption key that includes <a href="">special countermeasures against dictionary attacks</a>. But since your backup&#39;s worst enemy is probably staring you in the mirror, it would need to allow you to <a href="">create an escrow key</a> to store in a safe place in case you ever forget your super-duper passphrase.</p> <p> On the other hand, nobody wants excessive security measures forced down their throats when they don&#39;t need them and in that case, the ideal tool would be designed to optimize for convenience. Your data would still be encrypted, but the key management stuff would happen transparently.</p> <h3> Ultra data durability</h3> <p> By default, your AES encrypted backup volumes would be uploaded to inexpensive, ultra-durable cloud storage designed to provide <a href="">%99.999999999</a> durability. To put 11 nines of reliability in perspective, if you stored 10,000 backup volumes you could expect to lose a single volume once every 10 million years.</p> <p> For maximum network performance, you would be routed automatically to the cloud storage datacenter <a href="">closest to you</a>.</p> <h3> Open source goodness</h3> <p> Naturally, the ideal backup system would be open source. You don&#39;t have to care about free software ideology to appreciate the advantages. As far as I&#39;m concerned any code running on my servers doing something as critical as encrypted backups should be available for peer review and modification. No proprietary secret sauce. No pacts with a cloudy devil that expects you to give away your freedom, nay worse, your data, in exchange for a little bit of vendor-lock-in-flavored convenience.</p> <h3> Tall order huh?</h3> <p> All of this and more is what we set out to accomplish with TKLBAM. But this is not our wild eyed vision for a future backup system. We took our ideal and we made it work. In fact, we&#39;ve been experimenting with increasingly sophisticated prototypes for a few months now, privately eating our own dog food, working out the kinks. This stuff is complex so there may be a few rough spots left, but the foundation should be stable by now.</p> <h2> Seeing is believing: a simple usage example</h2> <p> We have two installations of TurnKey Drupal6:</p> <ol class="arabic simple"> <li> Alpha, a virtual machine on my local laptop. I&#39;ve been using it to develop the TurnKey Linux web site.</li> <li> Beta, an EC2 instance I just launched from the TurnKey Hub.</li> </ol> <p> In the <a href="">new TurnKey Linux 11.0</a> appliances, TKLBAM comes pre-installed. With older versions you&#39;ll need to install it first:</p> <pre> apt-get update apt-get install tklbam webmin-tklbam </pre> <p> You&#39;ll also need to link TKLBAM to your TurnKey Hub account by providing the API-KEY. You can do that via the <a href="">new Webmin module</a>, or on the command line:</p> <pre class="literal-block"> tklbam-init QPINK3GD7HHT3A </pre> <p> I now log into Alpha&#39;s command line as root (e.g., via the console, SSH or web shell) and do the following:</p> <pre class="literal-block"> tklbam-backup </pre> <p> It&#39;s <em>that</em> simple. Unless you want to change defaults, no arguments or additional configuration required.</p> <p> When the backup is done a new backup record will show up in my Hub account:</p> <p> <img alt="" src="" style="border: 1px solid rgb(170, 170, 170); margin-left: 9px; width: 650px; height: 317px;" /></p> <p> To restore I log into Beta and do this:</p> <pre class="literal-block"> tklbam-restore 1 </pre> <p> That&#39;s it! To see it in action watch the video below or better yet log into your <a href="">TurnKey Hub</a> account and try it for yourself.</p> <h2> <a name="screencast"></a>Quick screencast (2 minutes)</h2> <p> <iframe width="670" height="530" src="" frameborder="0" allowfullscreen></iframe> </p> <p> Best viewed full-screen. Having problems with playback? Try the <a href="">YouTube</a> version.<br /> <br /> The screencast shows TKLBAM command line usage, but users who dislike the command line can now do everything from the comfort of their web browser, thanks to the <a href="">new Webmin module</a>.</p> <h2> Getting started</h2> <p> TKLBAM&#39;s front-end interface is provided by the <a href="">TurnKey Hub</a>, an Amazon-powered cloud backup and server deployment web service currently in private beta.</p> <p> If you don&#39;t have a Hub account already, <a href="">request an invitation</a>. We&#39;ll do our best to grant them as fast as we can scale capacity on a first come, first served basis. <strong>Update: </strong>currently we&#39;re doing ok in terms of capacity so we&#39;re granting invitation requests within the hour.</p> <p> To get started log into your Hub account and follow the basic usage instructions. For more detail, <a href="">see the documentation</a>.</p> <p> Feel free to ask any questions in the comments below. But you&#39;ll probably want to check with <a href="">the FAQ</a> first to see if they&#39;ve already been answered.</p> <h2> Upcoming features</h2> <ul class="simple"> <li> <strong>PostgreSQL support</strong>: PostgreSQL support is in development but currently only MySQL is supported. That means TKLBAM doesn&#39;t yet work on the three PostgreSQL based TurnKey appliances (PostgreSQL, LAPP, and OpenBravo).</li> <li> <strong>Built-in integration</strong>: TKLBAM will be included by default in all future versions of TurnKey appliances. In the future when you launch a cloud server from the Hub it will be ready for action immediately. No installation or initialization necessary.</li> <li> <strong>Webmin integration</strong>: we realize not everyone is comfortable with the command line, so we&#39;re going to look into developing a custom webmin module for TKLBAM. <strong>Update</strong>: we&#39;ve added the <a href="">new TKLBAM webmin module</a> to the 11.0 RC images based on Lucid. In older images, the webmin-tklbam package can also be installed via the package manager.</li> </ul> <h2> Special salute to the TurnKey community</h2> <p> First, many thanks to the brave souls who tested TKLBAM and provided feedback even before we officially announced it. Remember, with enough eyeballs all bugs are shallow, so if you come across anything else, don&#39;t rely on someone else to report it. Speak up!</p> <p> Also, as usual during a development cycle we haven&#39;t been able to spend as much time on the community forums as we&#39;d like. Many thanks to everyone who helped keep the community alive and kicking in our relative absence.</p> <p> Remember, if the TurnKey community has helped you, try to pay it forward when you can by helping others.</p> <p> Finally, I&#39;d like to give extra special thanks to three key individuals that have gone above and beyond in their contributions to the community.</p> <p> By alphabetical order:</p> <ul class="simple"> <li> <a href="">Adrian Moya</a>: for developing appliances that rival some of our best work.</li> <li> <a href="">Basil Kurian</a>: for storming through appliance development at a rate I can barely keep up with.</li> <li> <a href="">JedMeister</a>: for continuing to lead as our most helpful and tireless community member for nearly a year and a half now. This guy is a frigging one man support army.</li> </ul> <p> Also special thanks to Bob Marley, the legend who&#39;s been inspiring us as of late to keep jamming till the sun was shining. :)</p> <h2> Final thoughts</h2> <p> TKLBAM is a major milestone for TurnKey. We&#39;re very excited to finally unveil it to the world. It&#39;s actually been a not-so-secret part of our vision from the start. A chance to show how TurnKey can innovate beyond just bundling off the shelf components.</p> <p> With TKLBAM out of the way we can now focus on pushing out the next release batch of Lucid based appliances. Thanks to the amazing work done by our star TKLPatch developers, we&#39;ll be able to significantly expand our library so by the next release we&#39;ll be showcasing even more of the world&#39;s best open source software. Stir It Up!</p> amazon backup cloud hub news s3 tklbam ubuntu Wed, 08 Sep 2010 13:32:47 +0000 Liraz Siri 1542 at Backups are hard, making sure you got it right - harder <p> According to Murphy&#39;s Law, everything that can go wrong, eventually will&nbsp;go wrong.</p> <p> This is true for backups on multiple levels. A backup is often our last&nbsp;line of defense when things go wrong, but so many things can go wrong&nbsp;with the backup itself that we usually don&#39;t find out about it until, well, horror of horrors, the backup fails.</p> <p> On the surface, backups can fail for zillions of reasons.</p> <p> If you&#39;re ahead of the game you can probably think of at least a&nbsp;dozen reasons why your backups will fail exactly when you need them the&nbsp;most. If you can&#39;t, you&#39;ve most likely been lulled into a false sense of&nbsp;security. You don&#39;t even know what you don&#39;t know.</p> <p> But these are just symptoms of a deeper underlying problem. Yes, backups&nbsp;are hard to get right, but the real stinger is that they&#39;re even harder&nbsp;to test. This is because restoring a system from backup into production is&nbsp;usually a labor intensive, error prone, time consuming exercise in pain.</p> <p> In the real world, it wouldn&#39;t matter if things go wrong, if (and that&#39;s a big if) we could easily&nbsp;simulate the worst case scenario on demand: restore our systems from&nbsp;backup and verify that everything works - or verify that they don&#39;t, fix it, rinse-repeat.</p> <p> Bottom line? Few test their backups, and nobody tests them frequently&nbsp;enough.</p> <p> Sure, if you have the resources (not many do), you can bruteforce your way around the problem.&nbsp;For example, with the&nbsp;right setup you could do frequent bit-for-bit identical snapshots of the&nbsp;underlying storage media and send them safely off site via a very&nbsp;high-bandwidth network connection (and vice versa). But such bruteforce&nbsp;backup strategies are hugely inefficient and often impractical due to&nbsp;costs.</p> <p> In the words of Mat Kearney - what is a boy to do?</p> backup tklbam Tue, 31 Aug 2010 16:46:57 +0000 Alon Swartz 1515 at Exploring S3 based filesystems S3FS and S3Backer <p>In the last couple of days I've been researching <a href="">Amazon S3</a> based filesystems, to figure out if maybe we could integrate that into an easy to use backup solution for TurnKey Linux appliances.</p> <p>Note that S3 could only be a part of the solution. It wouldn't be a good idea to rely exclusively on S3 based automatic backups because of the problematic security architecture it creates. If an attacker compromises your server, he can easily compromise and subvert or destroy any S3 based automatic backups. That's bad news.</p> <h2>S3 performance, limitations and costs</h2> <h3>S3 performance</h3> <ul> <li> <p class="first">S3 itself is faster than I realized. I've been fully saturating our server's network connection and uploading/downloading objects to S3 at 10MBytes/s.</p> </li> <li> <p class="first">Each S3 transaction comes with fixed overhead of 200ms for writes and about 350ms for reads.</p> <p>This means you can only access about 3 objects a second sequentially, which will of course massively impact your data throughput. (e.g., if you read many 1 bytes objects sequentially you'll get 3 bytes a second)</p> </li> </ul> <h3>S3 performance variability</h3> <p>S3 is usually very fast, but it's based on a complex distributed storage network behind the scenes that is known to vary in its behavior and performance characteristics.</p> <p>Use it long enough and you will come across requests that take 10 seconds to complete instead of 300ms. Point is, you can't rely on the average behavior ALWAYS happening.</p> <h3>S3 limitations</h3> <ul class="simple"> <li>Objects can contain a maximum of 5GB.</li> <li>You can't update part of an object. If you want to update 1 byte in a 1GB object you'll have to reupload the entire GB.</li> </ul> <h3>S3 costs</h3> <p>Three components:</p> <ol class="arabic"> <li> <div>Storage: $0.15 GB/month</div> </li> <li> <div>Data transfer: $0.1 GB in, $0.17 GB out</div> </li> <li> <div>Requests: $0.01 per 1000 PUT requests, $0.01 per 10,000 GET and other requests</div> </li> </ol> <p>A word of caution, some people using S3 based filesystems have made the mistake of focusing on just the storage costs and forgotten about other expenses, especially requests, which look so inexpensive.</p> <p>You need to watch out for that because using one filesystem under the default configuration (4KB blocks), storing 50GB of data cost $130 just in PUT request fees, more than 17X the storage fees!</p> <h2>Filesystems</h2> <h3>s3fs</h3> <h4>Which s3fs?</h4> <p>It's a bit confusing but there are two working projects competing for the name &quot;s3fs&quot;, both based on <a href="">FUSE</a>.</p> <p>One is implemented in C++, last release Aug 2008:</p> <p><a href="" title=""></a></p> <p>Another implemented in Python, last release May 2008:</p> <p><a href="" title=""></a></p> <p>I've only tried the C++ project, which is better known and more widely used (e.g., the Python project comes with warnings regarding data loss) so when I say s3fs I mean the C++ project on Google Code.</p> <h4>Description</h4> <p><a href="">s3fs</a> is a direct mapping of S3 to a filesystem paradigm. Files are mapped to objects. Filesystem metadata (e.g., ownership and file modes) are stored inside the object's meta data. Filenames are keys, with &quot;/&quot; as the delimiter to make listing more efficient, etc.</p> <p>That's significant because it means there is nothing terribly magical about a bucket being read/written to by s3fs, and in fact you can mount any bucket with s3fs to explore it as a filesystem.</p> <p>s3fs's main advantage is its simplicity. There are however a few gotchas:</p> <ul> <li> <p class="first">If you're using s3fs to access a bucket it didn't create and have objects in it that have directory-like components in their names (e.g., mypath/myfile), you'll need to create a dummy directory in order to see them (e.g., mkdir mypath).</p> </li> <li> <p class="first">The project seems to be &quot;regretware&quot;. The last open source release was in August 2008. Since then the author seems to have continued all development of new features (e.g., encryption, compression, multi-user access) as a commercial license (subcloud), and with that inherent conflict of interest the future of the GPLed licensed open source version is uncertain.</p> <p>In fact a few of the unresolved bugs (e.g., deep directory renames) in the open source version have been long fixed in the proprietary version.</p> </li> <li> <p class="first">No embedded documentation. Probably another side-effect of the proprietary version, though the available options are documented no the web site.</p> </li> <li> <p class="first">Inherits S3's limitations: no file can be over 5GB, and you can't partially update a file so changing a single byte will re-upload the entire file.</p> </li> <li> <p class="first">Inherits S3's performance characteristics: operation on many small files are very efficient (each is a separate S3 object after all)</p> </li> <li> <p class="first">Though S3 supports partial/chunked downloads, s3fs doesn't take advantage of this so if you want to read just one byte of a 1GB file, you'll have to download the entire GB.</p> <p>OTOH, s3fs supports a disk cache, which can be used to mitigate this limitation.</p> </li> <li> <p class="first">Watch out, the ACL for objects/files you update/write to will be reset to s3fs's global ACL (e.g., by default &quot;private&quot;). So if you rely on a richer ACL configuration for objects in your bucket you'll want to access your S3FS bucket in read-only mode.</p> </li> <li> <p class="first">By default, s3fs doesn't use SSL, but you can get that to work by using the -o url option to specify <a href="" title=""></a> instead of <a href="" title=""></a></p> <p>It's not documented very well of course. Cough. Proprietary version. Cough.</p> </li> </ul> <h3>S3Backer</h3> <p><a href="">S3Backer</a> is a true open source project under active development, which has a very clever design.</p> <p>Also based on FUSE but instead of implementing usable filesystem on top of S3 it implements a virtual loopback device on top of S3:</p> <pre class="literal-block"> mountpoint/ file # (e.g., can be used as a virtual loopback) stats # human readable statistics </pre> <p>Except for this simple virtual filesystem S3Backer doesn't know anything about filesystems itself. It just maps that one virtual file to a series of dynamically allocated blocks inside S3.</p> <p>The advantage to this approach is that it allows you to leverage well tested code built into the kernel to take care of the higher level business of storing files. For all intents and purposes, it's just a special block device (e.g., use any filesystem, LVM, software Raid, kernel encryption, etc.).</p> <p>In practice it seems to work extremely well, thanks to a few clever performance optimizations. For example:</p> <ul class="simple"> <li>In-memory block cache: so rereads of the same unchanged block don't have to go across the network if it's still cacehd.</li> <li>Delayed, multi-threaded write queue: dirty blocks aren't written immediately to S3 because that can be very inefficient (e.g., the smallest operation would update the entire block). Instead changes seem to be accumulated for a couple of seconds and then written out in parallel to S3.</li> <li>Read-ahead algorithm: will detect and try to predict sequential block reads so the data is available in your cache before you actually ask for it.</li> </ul> <p>Bottom line, under the right configuration S3Backer works well enough that it can easily saturate our Linode's 100Mbit network connection. Impressive.</p> <p>In my testing with Reiserfs I found the performance good enough so that it would be conceivable to use it as an extension of working storage (e.g., as an EBS alternative for a system outside of EC2).</p> <p>There are a few gotchas however:</p> <ul> <li> <p class="first">High risk for data corruption, due to the delayed writes (e.g., your system or the connection to AWS fails). Journaling doesn't help because as far as the filesystem is concerned the blocks have already been written (I.e., to S3Backer's cache).</p> <p>In other words, I wouldn't use this as a back up drive. You can reduce the risk by turning off some of the performance optimizations to minimize the amount of data in limbo for write to Amazon.</p> </li> <li> <p class="first">too small block sizes (e.g., the 4K default) can add significant extra costs (e.g., $130 for 50GB with 4K blocks worth of storage)</p> </li> <li> <p class="first">too large block sizes can add significant data transfer and storage fees.</p> </li> <li> <p class="first">memory usage can be prohibitive: by default it caches 1000 blocks. With the default 4K block size that's not an issue but most users will probably want to increase block size.</p> <p>So watch out 1000 x 256KB block = 256MB.</p> <p>You can adjust the amount of cached blocks to control memory usage.</p> </li> </ul> <p>Future versions of S3Backer will probably include disk based caching which will mitigate data corruption and memory usage issues.</p> <p>Tips:</p> <ul> <li> <p class="first">Use Reiserfs. It can store multiple small files in the same blocks.</p> <p>It also doesn't populate the filesystem with too many empty blocks when the filesystem is created which makes filesystem creation faster and more efficient.</p> <p>Also, with Reiserfs I tested expanding and shrinking off the filesystem (after increasing/decreasing the size of the virtual block device) and it seemed to work just fine.</p> </li> <li> <p class="first">Supports storing multiple block devices in the same bucket using prefixes.</p> </li> </ul> <h2>Conclusions</h2> <ul class="simple"> <li><a href="">s3fs</a>: safe, efficient storage of medium-large files. Perfect for backup / archiving purposes.</li> <li><a href="">S3Backer</a>: high performing live storage on top of S3. EBS alternative outside of EC2. Not safe for backups at this stage.</li> </ul> aws backup cloud research s3 Mon, 11 Jan 2010 21:58:00 +0000 Liraz Siri 926 at