{"action":"created","issue":{"url":"https://api.github.com/repos/borgbackup/borg/issues/1031","repository_url":"https://api.github.com/repos/borgbackup/borg","labels_url":"https://api.github.com/repos/borgbackup/borg/issues/1031/labels{/name}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/1031/comments","events_url":"https://api.github.com/repos/borgbackup/borg/issues/1031/events","html_url":"https://github.com/borgbackup/borg/issues/1031","id":153620731,"number":1031,"title":"add flexible encryption chunk header","user":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-05-07T22:15:47Z","updated_at":"2016-05-08T12:12:29Z","closed_at":null,"body":"I've looked at the experimental branch, thinking about what to get from there and how.\r\n\r\nCompression\r\n------------------\r\nThe flexible compression first done there was meanwhile implemented in borg - a bit differently. The idea here was adding a 2-byte header in front of the compressed data that tells the (de)compression algorithm. Plus the small hack that we can keep existing zlib chunks without header, because we can determine zlib from first 2 zlib-output bytes also. We just must not use zlib-looking 2 type bytes for all other compression algorithms.\r\n```\r\nOld: N bytes zlib,6 output (always, attic heritage)\r\n\r\nNew: 2 bytes compression type (except for zlib) + N bytes compressor output\r\n```\r\nThis is nice for layering, we just give the decompressor the whole bytestring and it can make up everything from that (choosing the right algo internally). The compressor just prepends the right 2 bytes when returning its output.\r\n\r\nEncryption\r\n--------------\r\nI'ld like to add flexible encryption in a similar way.\r\n\r\n```\r\nkey_type = none, keyfile, repokey, passphrase (maps to repokey code), flexible\r\n\r\nOld:\r\nkey_type(1) + cleartext payload(N)\r\nkey_type(1) + HMAC(32) + IV(8) + encrypted payload(N)\r\n\r\nNew:\r\nas above, plus additionally for \"flexible\" key_type:\r\nkey_type(1) + msgpack((mac, meta_tuple, payload))\r\n\r\nmeta_tuple = namedtuple(\"meta\", \"key_type_real cipher_suite iv\")\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/borgbackup/borg/issues/comments/217712764","html_url":"https://github.com/borgbackup/borg/issues/1031#issuecomment-217712764","issue_url":"https://api.github.com/repos/borgbackup/borg/issues/1031","id":217712764,"user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"created_at":"2016-05-08T12:12:29Z","updated_at":"2016-05-08T12:12:29Z","body":"Since the \"DKID\" is in plain-text I don't feel to comfy deriving them directly from the crypto keys. Maybe encrypt a well-known phrase with the initial state when creating new keys and use the result as the ID? \r\n\r\nmsgpack vs readable: For the keyblob it doesn't matter, since it's an encrypted|authenticated blob anyway. When we (later) want to be able to import/export them JSON makes sense (most of it would still be a blob, but the ID + cipher suite would be readily readable, and it's much easier to integrate with higher-level tools)."}}