{"action":"closed","number":145,"pull_request":{"url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145","id":85750976,"html_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145","diff_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145.diff","patch_url":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145.patch","issue_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145","number":145,"state":"closed","locked":false,"title":"Implement shared docker layer cache","user":{"login":"Vektah","id":2247982,"avatar_url":"https://avatars.githubusercontent.com/u/2247982?v=3","gravatar_id":"","url":"https://api.github.com/users/Vektah","html_url":"https://github.com/Vektah","followers_url":"https://api.github.com/users/Vektah/followers","following_url":"https://api.github.com/users/Vektah/following{/other_user}","gists_url":"https://api.github.com/users/Vektah/gists{/gist_id}","starred_url":"https://api.github.com/users/Vektah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vektah/subscriptions","organizations_url":"https://api.github.com/users/Vektah/orgs","repos_url":"https://api.github.com/users/Vektah/repos","events_url":"https://api.github.com/users/Vektah/events{/privacy}","received_events_url":"https://api.github.com/users/Vektah/received_events","type":"User","site_admin":false},"body":"@porty and I have been hacking on a fix for https://github.com/buildkite/elastic-ci-stack-for-aws/issues/81\n\nIt adds an s3 bucket that contains a tarball for each pipeline that can be `docker load`'ed. Its very much batteries included, all a project should do is nominate a step to save all built layers by setting `DOCKER_CACHE_IMAGES_PUSH=true`. Ideally this is done on master.\n\npre-commit:\n- every step will fetch s3:://cache-bucket/$pipeline.images, which is a list of layer hashes contained in s3://cache-bucket/$pipeline.tar.gz\n- if any of the layer hashes are missing, fetch the tarball and `docker load`\n\npost-commit in one step, probably on master only\n- work out all the layers used by all docker containers tagged with the buildkite id (should be everything built in this step)\n- compare that with the contents of s3://cache-bucket/$pipeline.images\n- if the image list is different `docker save` all of the images into s3://cache-bucket/$pipeline.tar.gz and update s3://cache-bucket/$pipeline.images\n\nOn both sides if the machine already has fetched all of the required layers both the save and load steps skip, so the fast path is really fast. The slow path, where the cache needs to be fetched is quite a bit faster, fetching from s3 is pretty speedy compared to pulling and building.\n\n_Work in progress_: Still need to do a bunch of testing.\n","created_at":"2016-09-19T08:01:39Z","updated_at":"2016-11-14T23:22:24Z","closed_at":"2016-11-14T23:22:24Z","merged_at":null,"merge_commit_sha":"0c6847af3bb5f2a7dc0d6ee8ee6c70f4842736ad","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145/commits","review_comments_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145/comments","review_comment_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/comments{/number}","comments_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145/comments","statuses_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/statuses/20dac5cb0feca6b05a312db9a423dea5b9d89e85","head":{"label":"99designs:image-caching","ref":"image-caching","sha":"20dac5cb0feca6b05a312db9a423dea5b9d89e85","user":{"login":"99designs","id":17182,"avatar_url":"https://avatars.githubusercontent.com/u/17182?v=3","gravatar_id":"","url":"https://api.github.com/users/99designs","html_url":"https://github.com/99designs","followers_url":"https://api.github.com/users/99designs/followers","following_url":"https://api.github.com/users/99designs/following{/other_user}","gists_url":"https://api.github.com/users/99designs/gists{/gist_id}","starred_url":"https://api.github.com/users/99designs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/99designs/subscriptions","organizations_url":"https://api.github.com/users/99designs/orgs","repos_url":"https://api.github.com/users/99designs/repos","events_url":"https://api.github.com/users/99designs/events{/privacy}","received_events_url":"https://api.github.com/users/99designs/received_events","type":"Organization","site_admin":false},"repo":{"id":68554483,"name":"elastic-ci-stack-for-aws","full_name":"99designs/elastic-ci-stack-for-aws","owner":{"login":"99designs","id":17182,"avatar_url":"https://avatars.githubusercontent.com/u/17182?v=3","gravatar_id":"","url":"https://api.github.com/users/99designs","html_url":"https://github.com/99designs","followers_url":"https://api.github.com/users/99designs/followers","following_url":"https://api.github.com/users/99designs/following{/other_user}","gists_url":"https://api.github.com/users/99designs/gists{/gist_id}","starred_url":"https://api.github.com/users/99designs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/99designs/subscriptions","organizations_url":"https://api.github.com/users/99designs/orgs","repos_url":"https://api.github.com/users/99designs/repos","events_url":"https://api.github.com/users/99designs/events{/privacy}","received_events_url":"https://api.github.com/users/99designs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/99designs/elastic-ci-stack-for-aws","description":"A simple, flexible, auto-scaling cluster of build agents running in your own AWS VPC","fork":true,"url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws","forks_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/forks","keys_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/keys{/key_id}","collaborators_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/teams","hooks_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/hooks","issue_events_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/issues/events{/number}","events_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/events","assignees_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/assignees{/user}","branches_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/branches{/branch}","tags_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/tags","blobs_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/git/refs{/sha}","trees_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/git/trees{/sha}","statuses_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/statuses/{sha}","languages_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/languages","stargazers_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/stargazers","contributors_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/contributors","subscribers_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/subscribers","subscription_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/subscription","commits_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/commits{/sha}","git_commits_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/git/commits{/sha}","comments_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/comments{/number}","issue_comment_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/issues/comments{/number}","contents_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/contents/{+path}","compare_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/compare/{base}...{head}","merges_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/merges","archive_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/downloads","issues_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/issues{/number}","pulls_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/pulls{/number}","milestones_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/milestones{/number}","notifications_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/labels{/name}","releases_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/releases{/id}","deployments_url":"https://api.github.com/repos/99designs/elastic-ci-stack-for-aws/deployments","created_at":"2016-09-19T00:09:49Z","updated_at":"2016-09-19T00:09:50Z","pushed_at":"2016-11-14T23:09:59Z","git_url":"git://github.com/99designs/elastic-ci-stack-for-aws.git","ssh_url":"git@github.com:99designs/elastic-ci-stack-for-aws.git","clone_url":"https://github.com/99designs/elastic-ci-stack-for-aws.git","svn_url":"https://github.com/99designs/elastic-ci-stack-for-aws","homepage":"https://buildkite.com/docs/agent","size":678,"stargazers_count":0,"watchers_count":0,"language":"Shell","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"buildkite:master","ref":"master","sha":"b477a7c2fa5201b732ea852ba686ef17fa296a35","user":{"login":"buildkite","id":5055988,"avatar_url":"https://avatars.githubusercontent.com/u/5055988?v=3","gravatar_id":"","url":"https://api.github.com/users/buildkite","html_url":"https://github.com/buildkite","followers_url":"https://api.github.com/users/buildkite/followers","following_url":"https://api.github.com/users/buildkite/following{/other_user}","gists_url":"https://api.github.com/users/buildkite/gists{/gist_id}","starred_url":"https://api.github.com/users/buildkite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buildkite/subscriptions","organizations_url":"https://api.github.com/users/buildkite/orgs","repos_url":"https://api.github.com/users/buildkite/repos","events_url":"https://api.github.com/users/buildkite/events{/privacy}","received_events_url":"https://api.github.com/users/buildkite/received_events","type":"Organization","site_admin":false},"repo":{"id":36488517,"name":"elastic-ci-stack-for-aws","full_name":"buildkite/elastic-ci-stack-for-aws","owner":{"login":"buildkite","id":5055988,"avatar_url":"https://avatars.githubusercontent.com/u/5055988?v=3","gravatar_id":"","url":"https://api.github.com/users/buildkite","html_url":"https://github.com/buildkite","followers_url":"https://api.github.com/users/buildkite/followers","following_url":"https://api.github.com/users/buildkite/following{/other_user}","gists_url":"https://api.github.com/users/buildkite/gists{/gist_id}","starred_url":"https://api.github.com/users/buildkite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buildkite/subscriptions","organizations_url":"https://api.github.com/users/buildkite/orgs","repos_url":"https://api.github.com/users/buildkite/repos","events_url":"https://api.github.com/users/buildkite/events{/privacy}","received_events_url":"https://api.github.com/users/buildkite/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/buildkite/elastic-ci-stack-for-aws","description":"A simple, flexible, auto-scaling cluster of build agents running in your own AWS VPC","fork":false,"url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws","forks_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/forks","keys_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/keys{/key_id}","collaborators_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/teams","hooks_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/hooks","issue_events_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/events{/number}","events_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/events","assignees_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/assignees{/user}","branches_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/branches{/branch}","tags_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/tags","blobs_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/git/refs{/sha}","trees_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/git/trees{/sha}","statuses_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/statuses/{sha}","languages_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/languages","stargazers_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/stargazers","contributors_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/contributors","subscribers_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/subscribers","subscription_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/subscription","commits_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/commits{/sha}","git_commits_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/git/commits{/sha}","comments_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/comments{/number}","issue_comment_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/comments{/number}","contents_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/contents/{+path}","compare_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/compare/{base}...{head}","merges_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/merges","archive_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/downloads","issues_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues{/number}","pulls_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls{/number}","milestones_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/milestones{/number}","notifications_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/labels{/name}","releases_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/releases{/id}","deployments_url":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/deployments","created_at":"2015-05-29T07:28:16Z","updated_at":"2016-11-14T16:14:29Z","pushed_at":"2016-11-13T22:33:07Z","git_url":"git://github.com/buildkite/elastic-ci-stack-for-aws.git","ssh_url":"git@github.com:buildkite/elastic-ci-stack-for-aws.git","clone_url":"https://github.com/buildkite/elastic-ci-stack-for-aws.git","svn_url":"https://github.com/buildkite/elastic-ci-stack-for-aws","homepage":"https://buildkite.com/docs/agent","size":777,"stargazers_count":96,"watchers_count":96,"language":"Shell","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":44,"mirror_url":null,"open_issues_count":18,"forks":44,"open_issues":18,"watchers":96,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145"},"html":{"href":"https://github.com/buildkite/elastic-ci-stack-for-aws/pull/145"},"issue":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145"},"comments":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/issues/145/comments"},"review_comments":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145/comments"},"review_comment":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/pulls/145/commits"},"statuses":{"href":"https://api.github.com/repos/buildkite/elastic-ci-stack-for-aws/statuses/20dac5cb0feca6b05a312db9a423dea5b9d89e85"}},"merged":false,"mergeable":true,"mergeable_state":"clean","merged_by":null,"comments":24,"review_comments":5,"commits":1,"additions":65,"deletions":1,"changed_files":3}}
{"actor":{"display_login":"Vektah"}}