{"action":"created","comment":{"url":"https://api.github.com/repos/containers/image/pulls/comments/199584493","pull_request_review_id":133732601,"id":199584493,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5OTU4NDQ5Mw==","diff_hunk":"@@ -95,14 +98,20 @@ func Schema1Clone(src *Schema1) *Schema1 {\n \treturn &copy\n }\n \n-// initialize verifies invariants so that the rest of this code can assume a minimally healthy manifest.\n+// initialize initializes ExtracedV1Compatibility and verifies invariants, so that the rest of this code can assume a minimally healthy manifest.","path":"manifest/docker_schema1.go","position":null,"original_position":49,"commit_id":"1a0a68618d9512919168715fbd81a41e5ee1d7c4","original_commit_id":"6853458f509e18c79d88912fc3ba2c8c76f711be","user":{"login":"nalind","id":1200159,"node_id":"MDQ6VXNlcjEyMDAxNTk=","avatar_url":"https://avatars0.githubusercontent.com/u/1200159?v=4","gravatar_id":"","url":"https://api.github.com/users/nalind","html_url":"https://github.com/nalind","followers_url":"https://api.github.com/users/nalind/followers","following_url":"https://api.github.com/users/nalind/following{/other_user}","gists_url":"https://api.github.com/users/nalind/gists{/gist_id}","starred_url":"https://api.github.com/users/nalind/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nalind/subscriptions","organizations_url":"https://api.github.com/users/nalind/orgs","repos_url":"https://api.github.com/users/nalind/repos","events_url":"https://api.github.com/users/nalind/events{/privacy}","received_events_url":"https://api.github.com/users/nalind/received_events","type":"User","site_admin":false},"body":"Has an extra \"initialize\" in this comment.","created_at":"2018-07-02T18:31:55Z","updated_at":"2018-07-02T18:34:54Z","html_url":"https://github.com/containers/image/pull/472#discussion_r199584493","pull_request_url":"https://api.github.com/repos/containers/image/pulls/472","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/containers/image/pulls/comments/199584493"},"html":{"href":"https://github.com/containers/image/pull/472#discussion_r199584493"},"pull_request":{"href":"https://api.github.com/repos/containers/image/pulls/472"}}},"pull_request":{"url":"https://api.github.com/repos/containers/image/pulls/472","id":198470032,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk4NDcwMDMy","html_url":"https://github.com/containers/image/pull/472","diff_url":"https://github.com/containers/image/pull/472.diff","patch_url":"https://github.com/containers/image/pull/472.patch","issue_url":"https://api.github.com/repos/containers/image/issues/472","number":472,"state":"open","locked":false,"title":"Don't physically store “empty layers” in v2s1 images in containers-storage:","user":{"login":"mtrmac","id":7052838,"node_id":"MDQ6VXNlcjcwNTI4Mzg=","avatar_url":"https://avatars2.githubusercontent.com/u/7052838?v=4","gravatar_id":"","url":"https://api.github.com/users/mtrmac","html_url":"https://github.com/mtrmac","followers_url":"https://api.github.com/users/mtrmac/followers","following_url":"https://api.github.com/users/mtrmac/following{/other_user}","gists_url":"https://api.github.com/users/mtrmac/gists{/gist_id}","starred_url":"https://api.github.com/users/mtrmac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mtrmac/subscriptions","organizations_url":"https://api.github.com/users/mtrmac/orgs","repos_url":"https://api.github.com/users/mtrmac/repos","events_url":"https://api.github.com/users/mtrmac/events{/privacy}","received_events_url":"https://api.github.com/users/mtrmac/received_events","type":"User","site_admin":false},"body":"`buildah` image export uses the layer tree to build a list of DiffID values; these correspond to non-empty layers in the manifest (in v2s1, which represents empty layers; v2s2 and OCI don't list them in manifests at all).  So far we have been storing the empty layers as well, which broke buildah assumptions ( https://github.com/projectatomic/buildah/issues/684 ).\r\n\r\nTo make `buildah` work, don't store empty layers to the underlying storage. Then, as an `ImageSource`, replace the discarded empty layers with a canonical version, which is served by hard-coded code in `GetBlob`.\r\n\r\n**NOTE**: This will make the current problems with v2s1 images A BIT WORSE for pre-existing containers/storage stores; users with such images will have to delete (or at least untag) the v2s1 images, and pull them again:\r\n\r\nIn principle, we could support images which don't physically store any empty layers, or images which physically store all of them, in a single codebase and be compatible with the pre-existing stores.  (In the intermediate case, where only a few empty layers are stored, matching the physical layout (which is uncompressed, i.e. has DiffIDs) to the v2s1 manifest (which only contains compressed blob digests) is not possible in general because `storage.Layer.CompressedDigest` is optional.  So, it's either all or nothing.)\r\n\r\nAll of the code needs to support the model of not physically storing empty layers, because that's how v2s2/OCI always work.  As for supporting also the other model with physical empty layers (i.e. existing stores with pre-existing v2s1 images), it seems not worth it overall:\r\n\r\n- So far, copying v2s1 images to `containers-storage:` has created physical versions of the empty layers, and such images could be run; but they couldn't be copied out (`LayerInfosForCopy` returned a different number of entries, making `UpdateLayerInfos` abort) and they couldn't be used as `buildah` base images.  So, they were never all that well supported anyway.\r\n\r\n- Supporting both formats would commit us to maintain two different code paths in anything that walked the layer tree and expected some correspondence with config `RootFS.DiffIDs`; we expect the proportion of v2s1 images to trend to zero over time, so the long-term maintenance burden (and risk of v2s1 being broken again in new code because noone notices) is not proportionate.\r\n\r\n- `storageImage.Commit()` enforces that storing the same image (~same manifest) always ends up with the same top layer.  That code would have to be taught to support both possible top layer IDs - to compute, without physically building the layer tree, the layer ID sequence for both alternative layouts, check for a pre-existing image and top layer ID match, determine which of the layouts to use, and finally build it.  That would be _even more_ brittle and difficult-to-test code to support pre-existing images which never worked all that well.\r\n\r\nSo, anyone copying v2s1 images out of `containers-storage:` will need to delete them and create/pull them again.  This should primarily (only?) impact `buildah` users, who haven’t been able to work with such images until now anyway.\r\n\r\nPlain deployment-only uses of images (e.g. k8s nodes with CRI-O) can continue to use the previously pulled images, and they will continue to work as well as they ever did.  BUT, force-pulling the same v2s1 image again, like `buildah from --pull-always`, will fail with a top layer mismatch; again, the workaround is to delete/untag the v2s1 image, and pull it again using the new codebase.\r\n\r\n(The above describes the last commit; the previous ones are minor cleanups and building infrastructure to make it possible.  See individual commit messages for details.)\r\n\r\n@nalind @runcom PTAL","created_at":"2018-06-30T10:11:15Z","updated_at":"2018-07-02T18:34:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9debf5950009d192845be951c69cff88548f9ca1","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/containers/image/pulls/472/commits","review_comments_url":"https://api.github.com/repos/containers/image/pulls/472/comments","review_comment_url":"https://api.github.com/repos/containers/image/pulls/comments{/number}","comments_url":"https://api.github.com/repos/containers/image/issues/472/comments","statuses_url":"https://api.github.com/repos/containers/image/statuses/1a0a68618d9512919168715fbd81a41e5ee1d7c4","head":{"label":"mtrmac:storage-emptylayers","ref":"storage-emptylayers","sha":"1a0a68618d9512919168715fbd81a41e5ee1d7c4","user":{"login":"mtrmac","id":7052838,"node_id":"MDQ6VXNlcjcwNTI4Mzg=","avatar_url":"https://avatars2.githubusercontent.com/u/7052838?v=4","gravatar_id":"","url":"https://api.github.com/users/mtrmac","html_url":"https://github.com/mtrmac","followers_url":"https://api.github.com/users/mtrmac/followers","following_url":"https://api.github.com/users/mtrmac/following{/other_user}","gists_url":"https://api.github.com/users/mtrmac/gists{/gist_id}","starred_url":"https://api.github.com/users/mtrmac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mtrmac/subscriptions","organizations_url":"https://api.github.com/users/mtrmac/orgs","repos_url":"https://api.github.com/users/mtrmac/repos","events_url":"https://api.github.com/users/mtrmac/events{/privacy}","received_events_url":"https://api.github.com/users/mtrmac/received_events","type":"User","site_admin":false},"repo":{"id":62076230,"node_id":"MDEwOlJlcG9zaXRvcnk2MjA3NjIzMA==","name":"image","full_name":"mtrmac/image","owner":{"login":"mtrmac","id":7052838,"node_id":"MDQ6VXNlcjcwNTI4Mzg=","avatar_url":"https://avatars2.githubusercontent.com/u/7052838?v=4","gravatar_id":"","url":"https://api.github.com/users/mtrmac","html_url":"https://github.com/mtrmac","followers_url":"https://api.github.com/users/mtrmac/followers","following_url":"https://api.github.com/users/mtrmac/following{/other_user}","gists_url":"https://api.github.com/users/mtrmac/gists{/gist_id}","starred_url":"https://api.github.com/users/mtrmac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mtrmac/subscriptions","organizations_url":"https://api.github.com/users/mtrmac/orgs","repos_url":"https://api.github.com/users/mtrmac/repos","events_url":"https://api.github.com/users/mtrmac/events{/privacy}","received_events_url":"https://api.github.com/users/mtrmac/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mtrmac/image","description":null,"fork":true,"url":"https://api.github.com/repos/mtrmac/image","forks_url":"https://api.github.com/repos/mtrmac/image/forks","keys_url":"https://api.github.com/repos/mtrmac/image/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mtrmac/image/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mtrmac/image/teams","hooks_url":"https://api.github.com/repos/mtrmac/image/hooks","issue_events_url":"https://api.github.com/repos/mtrmac/image/issues/events{/number}","events_url":"https://api.github.com/repos/mtrmac/image/events","assignees_url":"https://api.github.com/repos/mtrmac/image/assignees{/user}","branches_url":"https://api.github.com/repos/mtrmac/image/branches{/branch}","tags_url":"https://api.github.com/repos/mtrmac/image/tags","blobs_url":"https://api.github.com/repos/mtrmac/image/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mtrmac/image/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mtrmac/image/git/refs{/sha}","trees_url":"https://api.github.com/repos/mtrmac/image/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mtrmac/image/statuses/{sha}","languages_url":"https://api.github.com/repos/mtrmac/image/languages","stargazers_url":"https://api.github.com/repos/mtrmac/image/stargazers","contributors_url":"https://api.github.com/repos/mtrmac/image/contributors","subscribers_url":"https://api.github.com/repos/mtrmac/image/subscribers","subscription_url":"https://api.github.com/repos/mtrmac/image/subscription","commits_url":"https://api.github.com/repos/mtrmac/image/commits{/sha}","git_commits_url":"https://api.github.com/repos/mtrmac/image/git/commits{/sha}","comments_url":"https://api.github.com/repos/mtrmac/image/comments{/number}","issue_comment_url":"https://api.github.com/repos/mtrmac/image/issues/comments{/number}","contents_url":"https://api.github.com/repos/mtrmac/image/contents/{+path}","compare_url":"https://api.github.com/repos/mtrmac/image/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mtrmac/image/merges","archive_url":"https://api.github.com/repos/mtrmac/image/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mtrmac/image/downloads","issues_url":"https://api.github.com/repos/mtrmac/image/issues{/number}","pulls_url":"https://api.github.com/repos/mtrmac/image/pulls{/number}","milestones_url":"https://api.github.com/repos/mtrmac/image/milestones{/number}","notifications_url":"https://api.github.com/repos/mtrmac/image/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mtrmac/image/labels{/name}","releases_url":"https://api.github.com/repos/mtrmac/image/releases{/id}","deployments_url":"https://api.github.com/repos/mtrmac/image/deployments","created_at":"2016-06-27T17:54:58Z","updated_at":"2018-07-02T17:26:09Z","pushed_at":"2018-07-02T17:26:08Z","git_url":"git://github.com/mtrmac/image.git","ssh_url":"git@github.com:mtrmac/image.git","clone_url":"https://github.com/mtrmac/image.git","svn_url":"https://github.com/mtrmac/image","homepage":null,"size":4305,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"containers:master","ref":"master","sha":"e0206e46ecb4dc5bee406f174f4641271d6be951","user":{"login":"containers","id":5874934,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU4NzQ5MzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5874934?v=4","gravatar_id":"","url":"https://api.github.com/users/containers","html_url":"https://github.com/containers","followers_url":"https://api.github.com/users/containers/followers","following_url":"https://api.github.com/users/containers/following{/other_user}","gists_url":"https://api.github.com/users/containers/gists{/gist_id}","starred_url":"https://api.github.com/users/containers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/containers/subscriptions","organizations_url":"https://api.github.com/users/containers/orgs","repos_url":"https://api.github.com/users/containers/repos","events_url":"https://api.github.com/users/containers/events{/privacy}","received_events_url":"https://api.github.com/users/containers/received_events","type":"Organization","site_admin":false},"repo":{"id":61739430,"node_id":"MDEwOlJlcG9zaXRvcnk2MTczOTQzMA==","name":"image","full_name":"containers/image","owner":{"login":"containers","id":5874934,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU4NzQ5MzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5874934?v=4","gravatar_id":"","url":"https://api.github.com/users/containers","html_url":"https://github.com/containers","followers_url":"https://api.github.com/users/containers/followers","following_url":"https://api.github.com/users/containers/following{/other_user}","gists_url":"https://api.github.com/users/containers/gists{/gist_id}","starred_url":"https://api.github.com/users/containers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/containers/subscriptions","organizations_url":"https://api.github.com/users/containers/orgs","repos_url":"https://api.github.com/users/containers/repos","events_url":"https://api.github.com/users/containers/events{/privacy}","received_events_url":"https://api.github.com/users/containers/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/containers/image","description":"Work with containers' images","fork":false,"url":"https://api.github.com/repos/containers/image","forks_url":"https://api.github.com/repos/containers/image/forks","keys_url":"https://api.github.com/repos/containers/image/keys{/key_id}","collaborators_url":"https://api.github.com/repos/containers/image/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/containers/image/teams","hooks_url":"https://api.github.com/repos/containers/image/hooks","issue_events_url":"https://api.github.com/repos/containers/image/issues/events{/number}","events_url":"https://api.github.com/repos/containers/image/events","assignees_url":"https://api.github.com/repos/containers/image/assignees{/user}","branches_url":"https://api.github.com/repos/containers/image/branches{/branch}","tags_url":"https://api.github.com/repos/containers/image/tags","blobs_url":"https://api.github.com/repos/containers/image/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/containers/image/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/containers/image/git/refs{/sha}","trees_url":"https://api.github.com/repos/containers/image/git/trees{/sha}","statuses_url":"https://api.github.com/repos/containers/image/statuses/{sha}","languages_url":"https://api.github.com/repos/containers/image/languages","stargazers_url":"https://api.github.com/repos/containers/image/stargazers","contributors_url":"https://api.github.com/repos/containers/image/contributors","subscribers_url":"https://api.github.com/repos/containers/image/subscribers","subscription_url":"https://api.github.com/repos/containers/image/subscription","commits_url":"https://api.github.com/repos/containers/image/commits{/sha}","git_commits_url":"https://api.github.com/repos/containers/image/git/commits{/sha}","comments_url":"https://api.github.com/repos/containers/image/comments{/number}","issue_comment_url":"https://api.github.com/repos/containers/image/issues/comments{/number}","contents_url":"https://api.github.com/repos/containers/image/contents/{+path}","compare_url":"https://api.github.com/repos/containers/image/compare/{base}...{head}","merges_url":"https://api.github.com/repos/containers/image/merges","archive_url":"https://api.github.com/repos/containers/image/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/containers/image/downloads","issues_url":"https://api.github.com/repos/containers/image/issues{/number}","pulls_url":"https://api.github.com/repos/containers/image/pulls{/number}","milestones_url":"https://api.github.com/repos/containers/image/milestones{/number}","notifications_url":"https://api.github.com/repos/containers/image/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/containers/image/labels{/name}","releases_url":"https://api.github.com/repos/containers/image/releases{/id}","deployments_url":"https://api.github.com/repos/containers/image/deployments","created_at":"2016-06-22T17:46:41Z","updated_at":"2018-07-02T16:21:50Z","pushed_at":"2018-07-02T17:20:15Z","git_url":"git://github.com/containers/image.git","ssh_url":"git@github.com:containers/image.git","clone_url":"https://github.com/containers/image.git","svn_url":"https://github.com/containers/image","homepage":"","size":4214,"stargazers_count":164,"watchers_count":164,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":82,"mirror_url":null,"archived":false,"open_issues_count":51,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":82,"open_issues":51,"watchers":164,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/containers/image/pulls/472"},"html":{"href":"https://github.com/containers/image/pull/472"},"issue":{"href":"https://api.github.com/repos/containers/image/issues/472"},"comments":{"href":"https://api.github.com/repos/containers/image/issues/472/comments"},"review_comments":{"href":"https://api.github.com/repos/containers/image/pulls/472/comments"},"review_comment":{"href":"https://api.github.com/repos/containers/image/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/containers/image/pulls/472/commits"},"statuses":{"href":"https://api.github.com/repos/containers/image/statuses/1a0a68618d9512919168715fbd81a41e5ee1d7c4"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"nalind"}}