cloudtask en TurnKey Hub: Not just adding random features <p> I started writing a review for cloudtask&nbsp;as a comment on the <a href="">announcement</a>, but decided it would be better to address a <a href="">topic</a> that was raised by Jeremiah when we launched the <a href="">Hub API</a>:</p> <blockquote> <p> I&#39;m impressed with the way all the pieces of TurnKey just fit together like puzzle pieces. Even as new functionality is added it never seems like something gets patched onto the existing framework. &nbsp;Instead, it just integrates smoothly with what&#39;s already there.</p> </blockquote> <p> Liraz&#39;s <a href="">reply</a>:</p> <blockquote> <p> While the new Hub features are coming out incrementally, they really are part of a bigger unifying vision. I don&#39;t like to hype up vaporware, but we have a lot of great ideas we&#39;re working on and we like to think we&#39;re just getting started. There&#39;s a big gap between what the Hub does right now and what we envision it doing in the future.</p> <p> Also note that we&#39;re using the Hub internally to help develop and test TurnKey, so we&#39;re probably using it more than the typical user and run into its limits sooner. We&#39;re scratching our own itch, and the community benefits from that.</p> </blockquote> <p> To recap, some of the features Liraz was referring to in his comment, and some that came after his comment are:</p> <ul> <li> <a href="">Hub 1.0 follow-up: improved server status</a></li> <li> <a href="">New Hub feature: Auto-Restore TKLBAM backup to a new cloud server</a></li> <li> <a href="">Announcing public API for TurnKey Hub</a></li> <li> <a href="">TurnKey Domain management &amp; Dynamic DNS</a></li> <li> <a href="">TurnKey 11.2, free micro instances, EBS backed cloud servers</a></li> <li> <a href="">Introducing CloudTask - a cloud batch execution tool</a></li> </ul> <p> More features are coming, but this a great milestone to stop, reflect, and talk a little about a pain point we&#39;ve had, and how the features we&#39;ve released to this point are helping us solve it.</p> <h2> Building and maintaining exponentially growing library</h2> <p> TurnKey currently has 45 appliances, available in ISO, VMDK, OVF, EC2 S3 and EBS backed, Xen and Eucalyptus.</p> <p> To sum up, thats <strong>585 images</strong> we need to build and maintain.&nbsp;</p> <p> The (long overdue) TKL 11 part 2 will double the appliance library. Additionally we are working on supporting more build targets, as well as the new Tokyo EC2 region. Throw 64bit support into the equasion and the number of images to build and maintain will grow to about <strong>3,300</strong>!</p> <h2> Automation and a glimpse into a pain point</h2> <p> We try to automate everything we do, but even executing and managing the stuff thats already automated needs automation when you reach scale.</p> <p> To give you a little glimps into what I mean. When building the Amazon EC2 S3 backed images (in the past), I couldn&#39;t do it locally as my upstream kinda sucks are would take me about 2 weeks to build and upload.</p> <p> So do it in the cloud, right? Correct. But building and uploading from an EC2 instance still takes a couple of days, mainly due to subsequent builds and uploading images built for different regions is quiet slow.</p> <p> OK, so lets just spawn more servers in the cloud, in the different regions. Yep, this allows us to perform the relative region builds so upload is much faster, and they run in parallel.</p> <p> But manually launching each instance from the Hub&#39;s web UI (pre-&nbsp;<a href="">hub-api</a>), updating my /etc/hosts file to keep track of the different instances (pre-&nbsp;<a href="">hub-domains</a>), logging into each instance and transfering the build infrastructure (pre-&nbsp;<a href="">restore-on-launch</a>), and finally starting the automated build process while making sure nothing breaks is painful to say the least.</p> <p> It would also be much faster (and cost the same) if we could split the appliance builds amoungst more instances in each region, but try managing that manually and keep your hair - argh!</p> <p> To throw some salt on an already bloody wound, remember we also have to build the other target formats (EBS, VMDK, OVF, etc...). Try do that without going bald.</p> <h2> Parallel batch execution with auto-launched cloud servers</h2> <p> This is where integrating the features mentioned above, and <a href="">cloudtask</a> comes into play.</p> <p> Instead of pulling my hair out and waiting days, yesterday I threw a couple of tasks at cloudtask, which launched about 50 instances spanning the globe, our build infrastructure was automatically setup on each instance, and cloudtask started dishing out jobs to each of them.</p> <p> Within about 1 hour, I received reports via email that all the builds completed successfully and were published. Pain point gone.</p> <p> Now thats automation on steriods! And best of all, I get to keep my hair!<br /> &nbsp;</p> <p> As mentioned above, we are just getting started. Lots more to come - it&#39;s getting exciting...</p> aws batch cloud cloudtask ec2 hub parallel Fri, 19 Aug 2011 12:50:25 +0000 Alon Swartz 2595 at Introducing CloudTask - a cloud batch execution tool <p> <em>The cloud</em>. Isn&#39;t that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you&#39;d forgive the cynics for thinking there&#39;s no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren&#39;t practical a few years back.</p> <p> Like renting an armada of servers you could never afford to buy to parallelize a computational task and get results in an hour instead of days, weeks or even months, for exactly the same cost. Now that&#39;s kind of exciting if you can pull it off.</p> <p> We came across a modest version of this problem for various routine TurnKey related maintenance tasks such as rebuilding appliances. It sure was nice to be able to fire up a server on-demand, run a batch job and be able upload new images to sourceforge at 100MB/s. Usually we would leave the batch job running overnight and terminate the server the next day or so. That wasn&#39;t so bad for non-frequent tasks, but we realized we could do better. On Amazon EC2 running 10 servers for 1 hour costs the same as running 1 server for 10 hours. That&#39;s the theory anyhow. In practice launching and controlling many servers by hand can be painful.</p> <p> Obviously, a bit of clever automation would be just the ticket. I hate reinventing the wheel so I first tried surveying existing solutions, but I couldn&#39;t find any that fit our needs.</p> <p> So I rolled up my sleeves, and about a month or so later cloudtask was born. It&#39;s kind of neat. If you&#39;ve ever had to do this sort of thing by hand or put together an ugly mess of scripts you&#39;ll probably find cloudtask an easier and more reliable primitive to build on.</p> <p> Here&#39;s the <a href="">documentation</a>.</p> <p> Here&#39;s a tacky video demo I whipped up (best viewed full screen):</p> <p> <embed allowfullscreen="true" allowscriptaccess="always" height="530" src="" type="application/x-shockwave-flash" width="670" wmode="transparent"></embed></p> aws batch cloud cloudtask ec2 hub news parallel Fri, 12 Aug 2011 13:17:48 +0000 Liraz Siri 2585 at