{"action":"created","issue":{"url":"https://api.github.com/repos/openSUSE/umoci/issues/256","repository_url":"https://api.github.com/repos/openSUSE/umoci","labels_url":"https://api.github.com/repos/openSUSE/umoci/issues/256/labels{/name}","comments_url":"https://api.github.com/repos/openSUSE/umoci/issues/256/comments","events_url":"https://api.github.com/repos/openSUSE/umoci/issues/256/events","html_url":"https://github.com/openSUSE/umoci/issues/256","id":358302187,"node_id":"MDU6SXNzdWUzNTgzMDIxODc=","number":256,"title":"[rfc] OCIv2 implementation","user":{"login":"cyphar","id":2888411,"node_id":"MDQ6VXNlcjI4ODg0MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/2888411?v=4","gravatar_id":"","url":"https://api.github.com/users/cyphar","html_url":"https://github.com/cyphar","followers_url":"https://api.github.com/users/cyphar/followers","following_url":"https://api.github.com/users/cyphar/following{/other_user}","gists_url":"https://api.github.com/users/cyphar/gists{/gist_id}","starred_url":"https://api.github.com/users/cyphar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyphar/subscriptions","organizations_url":"https://api.github.com/users/cyphar/orgs","repos_url":"https://api.github.com/users/cyphar/repos","events_url":"https://api.github.com/users/cyphar/events{/privacy}","received_events_url":"https://api.github.com/users/cyphar/received_events","type":"User","site_admin":false},"labels":[{"id":876563275,"node_id":"MDU6TGFiZWw4NzY1NjMyNzU=","url":"https://api.github.com/repos/openSUSE/umoci/labels/oci-spec","name":"oci-spec","color":"34beed","default":false},{"id":476003437,"node_id":"MDU6TGFiZWw0NzYwMDM0Mzc=","url":"https://api.github.com/repos/openSUSE/umoci/labels/upstream","name":"upstream","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/openSUSE/umoci/milestones/10","html_url":"https://github.com/openSUSE/umoci/milestone/10","labels_url":"https://api.github.com/repos/openSUSE/umoci/milestones/10/labels","id":2899340,"node_id":"MDk6TWlsZXN0b25lMjg5OTM0MA==","number":10,"title":"0.5.0","description":"","creator":{"login":"cyphar","id":2888411,"node_id":"MDQ6VXNlcjI4ODg0MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/2888411?v=4","gravatar_id":"","url":"https://api.github.com/users/cyphar","html_url":"https://github.com/cyphar","followers_url":"https://api.github.com/users/cyphar/followers","following_url":"https://api.github.com/users/cyphar/following{/other_user}","gists_url":"https://api.github.com/users/cyphar/gists{/gist_id}","starred_url":"https://api.github.com/users/cyphar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyphar/subscriptions","organizations_url":"https://api.github.com/users/cyphar/orgs","repos_url":"https://api.github.com/users/cyphar/repos","events_url":"https://api.github.com/users/cyphar/events{/privacy}","received_events_url":"https://api.github.com/users/cyphar/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":1,"state":"open","created_at":"2017-11-09T04:38:40Z","updated_at":"2018-09-11T05:13:55Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2018-09-08T14:58:02Z","updated_at":"2018-10-11T04:30:24Z","closed_at":null,"author_association":"MEMBER","body":"I have some proposal ideas for the OCIv2 image specification (it would actually be OCIv1.1 but that is a less-cool name for the idea) and they primarily involve swapping out the lower levels of the archive format to better designed (along the same lines as [restic](https://restic.net/) or borgbackup).\r\n\r\nWe need to implement this as a PoC in umoci before it's proposed to the image-spec proper so that we don't get stuck in debates over whether it has been tested \"in the wild\" -- which is something that I imagine any OCI extension is going to go through."},"comment":{"url":"https://api.github.com/repos/openSUSE/umoci/issues/comments/428817098","html_url":"https://github.com/openSUSE/umoci/issues/256#issuecomment-428817098","issue_url":"https://api.github.com/repos/openSUSE/umoci/issues/256","id":428817098,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODgxNzA5OA==","user":{"login":"cyphar","id":2888411,"node_id":"MDQ6VXNlcjI4ODg0MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/2888411?v=4","gravatar_id":"","url":"https://api.github.com/users/cyphar","html_url":"https://github.com/cyphar","followers_url":"https://api.github.com/users/cyphar/followers","following_url":"https://api.github.com/users/cyphar/following{/other_user}","gists_url":"https://api.github.com/users/cyphar/gists{/gist_id}","starred_url":"https://api.github.com/users/cyphar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyphar/subscriptions","organizations_url":"https://api.github.com/users/cyphar/orgs","repos_url":"https://api.github.com/users/cyphar/repos","events_url":"https://api.github.com/users/cyphar/events{/privacy}","received_events_url":"https://api.github.com/users/cyphar/received_events","type":"User","site_admin":false},"created_at":"2018-10-11T04:30:24Z","updated_at":"2018-10-11T04:30:24Z","author_association":"MEMBER","body":"@flx42 \r\n\r\n> what was the argument against doing simply file-level deduplication?\r\n\r\nSmall modifications of large files, or files that are *substantially* similar but not identical (think man pages, shared libraries and binaries shipped by multiple distributions, and so on) would be entirely duplicated. So for the image format I think that using file-level deduplication is flawed, for the same reasons that file-level deduplication in backup systems is flawed.\r\n\r\n*But* for storage deduplication this is a different story. My main reason for wanting to use reflinks is to be able to use less disk space. Unfortunately (as I discovered above) this is not possible for variable-size chunks (unless they are all multiples of the chunk size).\r\n\r\nUsing file-based deduplication for storage does make some sense (though it does naively double your storage requirement out-of-the gate). My idea for this would be that when you download all of the chunks and metadata into your OCI store, you set up a separate content-addressed store which has files that correspond to each file represented in your OCI store. Then, when constructing a rootfs, you can just reflink (or hardlink if you want) all of the files from the file store into a rootfs (overlayfs would have to be used to make sure you couldn't touch any of the underlying files). Of course, it might be necessary (for fast container \"boot\" times) to pre-generate the rootfs for any given image -- but benchmarks would have to be done to see if it's necessary.\r\n\r\nMy main interest in reflinks was to see whether it was possible to use them to remove the need for the copies for the \"file store\", but given that you cannot easily map CDC chunks to filesystem chunks (the latter being fixed-size) we are pretty much required to make copies I think. You could play with a FUSE filesystem to do it, but that is still slow (though some recent proposals to use eBPF could make it natively fast).\r\n\r\nAs for the page-cache I'm not sure. Reflinks work by referencing the same extents in the filesystem, so it depends on how the page-cache interacts with extents or whether the page-cache is entirely tied to the particular inode."}}
{"actor":{"display_login":"cyphar"}}