{"action":"created","issue":{"url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145","repository_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws","labels_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145/labels{/name}","comments_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145/comments","events_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145/events","html_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145","id":177724130,"number":145,"title":"Implement shared docker layer cache","user":{"login":"Vektah","id":2247982,"avatar_url":"https://avatars.githubusercontent.com/u/2247982?v=3","gravatar_id":"","url":"https://api.github.com/users/Vektah","html_url":"https://github.com/Vektah","followers_url":"https://api.github.com/users/Vektah/followers","following_url":"https://api.github.com/users/Vektah/following{/other_user}","gists_url":"https://api.github.com/users/Vektah/gists{/gist_id}","starred_url":"https://api.github.com/users/Vektah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vektah/subscriptions","organizations_url":"https://api.github.com/users/Vektah/orgs","repos_url":"https://api.github.com/users/Vektah/repos","events_url":"https://api.github.com/users/Vektah/events{/privacy}","received_events_url":"https://api.github.com/users/Vektah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2016-09-19T08:01:39Z","updated_at":"2016-11-02T02:23:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145","html_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145","diff_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145.diff","patch_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145.patch"},"body":"@porty and I have been hacking on a fix for https://github.com/buildkite/elastic-ci-stack-for-aws/issues/81\n\nIt adds an s3 bucket that contains a tarball for each pipeline that can be `docker load`'ed. Its very much batteries included, all a project should do is nominate a step to save all built layers by setting `DOCKER_CACHE_IMAGES_PUSH=true`. Ideally this is done on master.\n\npre-commit:\n- every step will fetch s3:://cache-bucket/$pipeline.images, which is a list of layer hashes contained in s3://cache-bucket/$pipeline.tar.gz\n- if any of the layer hashes are missing, fetch the tarball and `docker load`\n\npost-commit in one step, probably on master only\n- work out all the layers used by all docker containers tagged with the buildkite id (should be everything built in this step)\n- compare that with the contents of s3://cache-bucket/$pipeline.images\n- if the image list is different `docker save` all of the images into s3://cache-bucket/$pipeline.tar.gz and update s3://cache-bucket/$pipeline.images\n\nOn both sides if the machine already has fetched all of the required layers both the save and load steps skip, so the fast path is really fast. The slow path, where the cache needs to be fetched is quite a bit faster, fetching from s3 is pretty speedy compared to pulling and building.\n\n_Work in progress_: Still need to do a bunch of testing.\n"},"comment":{"url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/comments/257754756","html_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145#issuecomment-257754756","issue_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145","id":257754756,"user":{"login":"Vektah","id":2247982,"avatar_url":"https://avatars.githubusercontent.com/u/2247982?v=3","gravatar_id":"","url":"https://api.github.com/users/Vektah","html_url":"https://github.com/Vektah","followers_url":"https://api.github.com/users/Vektah/followers","following_url":"https://api.github.com/users/Vektah/following{/other_user}","gists_url":"https://api.github.com/users/Vektah/gists{/gist_id}","starred_url":"https://api.github.com/users/Vektah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vektah/subscriptions","organizations_url":"https://api.github.com/users/Vektah/orgs","repos_url":"https://api.github.com/users/Vektah/repos","events_url":"https://api.github.com/users/Vektah/events{/privacy}","received_events_url":"https://api.github.com/users/Vektah/received_events","type":"User","site_admin":false},"created_at":"2016-11-02T02:23:00Z","updated_at":"2016-11-02T02:23:00Z","body":"@gugahoi this PR shares the cache across all hosts by dumping the layers into an s3 bucket and loading them on the other machines as needed"}}
{"actor":{"display_login":"Vektah"}}