{"action":"created","issue":{"url":"https://api.github.com/repos/borgbackup/borg/issues/1031","repository_url":"https://api.github.com/repos/borgbackup/borg","labels_url":"https://api.github.com/repos/borgbackup/borg/issues/1031/labels{/name}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/1031/comments","events_url":"https://api.github.com/repos/borgbackup/borg/issues/1031/events","html_url":"https://github.com/borgbackup/borg/issues/1031","id":153620731,"number":1031,"title":"add flexible encryption chunk header","user":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/borgbackup/borg/milestones/17","html_url":"https://github.com/borgbackup/borg/milestones/1.2%20sequential%20threading","labels_url":"https://api.github.com/repos/borgbackup/borg/milestones/17/labels","id":1695570,"number":17,"title":"1.2 sequential threading","description":"do not put non-trivial issues in here that are not related to #929, so we can concentrate on that.","creator":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":0,"state":"open","created_at":"2016-04-09T22:18:46Z","updated_at":"2016-05-08T12:44:40Z","due_on":null,"closed_at":null},"comments":14,"created_at":"2016-05-07T22:15:47Z","updated_at":"2016-05-09T16:26:48Z","closed_at":null,"body":"I've looked at the experimental branch, thinking about what to get from there and how.\r\n\r\nCompression\r\n------------------\r\nThe flexible compression first done there was meanwhile implemented in borg - a bit differently. The idea here was adding a 2-byte header in front of the compressed data that tells the (de)compression algorithm. Plus the small hack that we can keep existing zlib chunks without header, because we can determine zlib from first 2 zlib-output bytes also. We just must not use zlib-looking 2 type bytes for all other compression algorithms.\r\n```\r\nOld: N bytes zlib,6 output (always, attic heritage)\r\n\r\nNew: 2 bytes compression type (except for zlib) + N bytes compressor output\r\n```\r\nThis is nice for layering, we just give the decompressor the whole bytestring and it can make up everything from that (choosing the right algo internally). The compressor just prepends the right 2 bytes when returning its output.\r\n\r\nEncryption\r\n--------------\r\nI'ld like to add flexible encryption in a similar way - see link in next post.\r\n"},"comment":{"url":"https://api.github.com/repos/borgbackup/borg/issues/comments/217914867","html_url":"https://github.com/borgbackup/borg/issues/1031#issuecomment-217914867","issue_url":"https://api.github.com/repos/borgbackup/borg/issues/1031","id":217914867,"user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"created_at":"2016-05-09T16:26:48Z","updated_at":"2016-05-09T16:26:48Z","body":"> Since the \"DKID\" is in plain-text I don't feel to comfy deriving them directly from the crypto keys. Maybe encrypt a well-known phrase with the initial state when creating new keys and use the result as the ID? [I.e. in the proposed API CipherSuite.create() would make up the ID in a pseudo-random way the cipher suite sees fit].\r\n> \r\n> Alternatively: ID = HMAC_of_master_key(packed_dek). We use that function all the time anyway.\r\n\r\nHm. Having deterministic, repo-independent DKIDs can have some nice uses (e.g. fast copies between repos sharing the same set of keys). So, let's say we make them deterministic. We don't set the scheme, but have CipherSuite.create (referring to the draft spec) do it. Contract:\r\n\r\n1. DKID is deterministic, only depends on the DEK (and all constant fields of it)\r\n1. The implementation / cipher suite must be able to\r\n  1. Verify the DKID against a DEK later\r\n  1. Ensure that no key data is leaked through the DKID, since the DKID is considered public knowledge. This includes the algorithms used.\r\n\r\nAES-CTR...HMAC... could then do something like mentioned above: Pack the DEK, do the normal payload encryption with IV=very_very_large (remember that the IV has 16 bytes, but only 8 are stored in normal chunks -- at least currently, but I see no reason to change that -- so using a very large, well-known IV deters IV reuse attacks if they would even be possible somehow here), then the DKID = envelope HMAC.\r\n\r\nFor other cipher suites this might look totally different (aes-gcm, chacha+poly is pretty much the same story as aes+hmac)."}}