{"action":"created","issue":{"url":"https://api.github.com/repos/buildkite/buildkite-aws-stack/issues/74","repository_url":"https://api.github.com/repos/buildkite/buildkite-aws-stack","labels_url":"https://api.github.com/repos/buildkite/buildkite-aws-stack/issues/74/labels{/name}","comments_url":"https://api.github.com/repos/buildkite/buildkite-aws-stack/issues/74/comments","events_url":"https://api.github.com/repos/buildkite/buildkite-aws-stack/issues/74/events","html_url":"https://github.com/buildkite/buildkite-aws-stack/issues/74","id":159629391,"number":74,"title":"RFC: Improving cold cache for large no. of parallel jobs w/ autoscaled instances","user":{"login":"toolmantim","id":153,"avatar_url":"https://avatars.githubusercontent.com/u/153?v=3","gravatar_id":"","url":"https://api.github.com/users/toolmantim","html_url":"https://github.com/toolmantim","followers_url":"https://api.github.com/users/toolmantim/followers","following_url":"https://api.github.com/users/toolmantim/following{/other_user}","gists_url":"https://api.github.com/users/toolmantim/gists{/gist_id}","starred_url":"https://api.github.com/users/toolmantim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/toolmantim/subscriptions","organizations_url":"https://api.github.com/users/toolmantim/orgs","repos_url":"https://api.github.com/users/toolmantim/repos","events_url":"https://api.github.com/users/toolmantim/events{/privacy}","received_events_url":"https://api.github.com/users/toolmantim/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-06-10T12:57:14Z","updated_at":"2016-06-13T11:10:01Z","closed_at":null,"body":"If you have a Rails app that takes a long time to do a `docker build` on a fresh machine, and a pipeline with 50+ parallel spec steps, and an auto-scaling group that goes from 0 to x, having cold Docker caches can be a total bummer. This is because each of the `test` steps ends up running `docker-compose build` which can take a huge amount of time for non-trivial apps… basically creating these massive outlier build times that developers would tear that hair out over.\r\n\r\nNow you can use the beta buildkite agent and https://github.com/buildkite/docker-compose-buildkite-plugin you're able to add a \"build\" step at the start of the pipeline and have all subsequent steps pick up this built docker image, instead of having to build it again on all your machines. This is very similar to how most people would structure their own pipelines anyhow, and it means you end up pushing a Docker image somewhere for each build which is also nice for later debugging. And it works with the docker hub credentials support that's already built into this stack. For example:\r\n\r\n```yml\r\n- command: build-and-push.sh\r\n- wait\r\n- command: parallel_spec.sh\r\n  name: \"Spec %n\"\r\n  parallelism: 50\r\n```\r\n\r\nThe way I'm doing this at the moment is by creating two separate CloudFormation stacks: `buildkite-builders` and `buildkite-runners`. `buildkite-builders` is a t2.medium with a min size of 1 and max size of 2 and scale-in/scale-out of 0 to disable auto-scaling (you can't have it min 1 max 1, or else stack updates fail because of the stack update policy that requires at least one instance to be in service). `buildkite-runners` is a c4.medium with spot pricing, min size of 0, max size of 50, scale-out/in of +-25. This setup means you get an always-on warm cache for building+pushing, and relatively fast spin up for runners for the parallel tests (fast in both auto-scaling, and in terms starting the job on a new instance). This keeps most builds < 5 minutes if there's enough agents, or an extra few minutes if new agents need to be spun up.\r\n\r\nI think this sort of thing is a pretty common pattern, and it'd be nice to build this into the stack somehow.\r\n\r\nOne idea would be to enable you to set a fixed size of \"builders\" instances which stay on at all times, which end up with their own queue which you can target using the new docker plugin. It could be that for a given stack you end up with `{stack-name}-builders` and `{stack-name}-runners`. There would be no auto-scaling of builders (but they could still be managed via an ASG). If we built this into the own stack, you'd end up with parameters to configure the two different ASGs, builders and runners."},"comment":{"url":"https://api.github.com/repos/buildkite/buildkite-aws-stack/issues/comments/225552217","html_url":"https://github.com/buildkite/buildkite-aws-stack/issues/74#issuecomment-225552217","issue_url":"https://api.github.com/repos/buildkite/buildkite-aws-stack/issues/74","id":225552217,"user":{"login":"lox","id":15758,"avatar_url":"https://avatars.githubusercontent.com/u/15758?v=3","gravatar_id":"","url":"https://api.github.com/users/lox","html_url":"https://github.com/lox","followers_url":"https://api.github.com/users/lox/followers","following_url":"https://api.github.com/users/lox/following{/other_user}","gists_url":"https://api.github.com/users/lox/gists{/gist_id}","starred_url":"https://api.github.com/users/lox/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lox/subscriptions","organizations_url":"https://api.github.com/users/lox/orgs","repos_url":"https://api.github.com/users/lox/repos","events_url":"https://api.github.com/users/lox/events{/privacy}","received_events_url":"https://api.github.com/users/lox/received_events","type":"User","site_admin":false},"created_at":"2016-06-13T11:10:01Z","updated_at":"2016-06-13T11:10:01Z","body":"> But, I not sure they need to be in the same stack, when spinning up multiple stacks is so easy.\r\n\r\nI keep coming back to this. If we could offer some nested stacks to automate this, sure, but I'd be reluctant to add a lot of complexity to the basic stack."}}
{"actor":{"display_login":"lox"}}