{"action":"opened","issue":{"url":"https://api.github.com/repos/spacejam/sled/issues/325","repository_url":"https://api.github.com/repos/spacejam/sled","labels_url":"https://api.github.com/repos/spacejam/sled/issues/325/labels{/name}","comments_url":"https://api.github.com/repos/spacejam/sled/issues/325/comments","events_url":"https://api.github.com/repos/spacejam/sled/issues/325/events","html_url":"https://github.com/spacejam/sled/issues/325","id":338371681,"node_id":"MDU6SXNzdWUzMzgzNzE2ODE=","number":325,"title":"PageCache raw_buf's and Invertible Transforms","user":{"login":"davidrusu","id":1832378,"node_id":"MDQ6VXNlcjE4MzIzNzg=","avatar_url":"https://avatars3.githubusercontent.com/u/1832378?v=4","gravatar_id":"","url":"https://api.github.com/users/davidrusu","html_url":"https://github.com/davidrusu","followers_url":"https://api.github.com/users/davidrusu/followers","following_url":"https://api.github.com/users/davidrusu/following{/other_user}","gists_url":"https://api.github.com/users/davidrusu/gists{/gist_id}","starred_url":"https://api.github.com/users/davidrusu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidrusu/subscriptions","organizations_url":"https://api.github.com/users/davidrusu/orgs","repos_url":"https://api.github.com/users/davidrusu/repos","events_url":"https://api.github.com/users/davidrusu/events{/privacy}","received_events_url":"https://api.github.com/users/davidrusu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-04T20:39:55Z","updated_at":"2018-07-04T20:39:55Z","closed_at":null,"author_association":"NONE","body":"Hey there, I have a need for encryption at rest and I was hoping I could use sled for my storage layer.\r\n\r\nI was looking through the code base for areas where it would make sense to wedge encryption in and it seemed like it should be right around where the compression is done since compressing an encrypted message is not something you'd want to do.\r\n\r\nWe could augment the `LogReader::read_message` and `IoBufs::encapsulate` methods to check for a `config.use_encryption` and then do the encryption, but between this encryption and compression, it looks like a pattern is beginning, ie. both compression and encryption are an invertible transforms acting on the raw message buffer to be written to the log.\r\n\r\nI'm curious what your thoughts would be on exposing some `InvertibleTransform` trait e.g:\r\n```rust\r\ntrait InvertibleTransform {\r\n    fn forward(&[u8]) -> Vec<u8>;\r\n    fn backward(&[u8]) -> Vec<u8>;\r\n}\r\n```\r\nand replace the `use_compression` field in Config with a `transforms` field:\r\n``` rust\r\nstruct Config {\r\n    ...\r\n    transforms: Vec<dyn InvertibleTransform>\r\n    ...\r\n}\r\n```\r\nthen for the forward pass, `IoBufs::encapsulate` would do something like:\r\n``` rust\r\nlet mut buf = raw_buf;\r\nfor transform in transforms.iter() {\r\n    let buf = transform.forward(&buf)\r\n}\r\n```\r\nand for the backwards pass, `LogReader::read_message` would iterate the transforms starting with the last transform and move backwards through the list:\r\n``` rust\r\nlet mut buf = raw_buf;\r\nfor transform in transforms.iter().rev() {\r\n    let buf = transform.backward(&buf)\r\n}\r\n```\r\nThis would allow us to move the compression out of the PageCache crate if that's desirable (I noticed a few `#[cfg(feature = \"zstd\")]` macros which can be removed),  and even allow users to plug in their own compression algorithms that may work better for their application e.g. Word-level compression if they are working with a bunch of text.\r\n\r\nAlso, more important for my use-case, encryption can be handled entirely outside of the database, this is important for cases where you want to build a distributed database on top of sled, distributed encryption with a same secret keys needs tight control over how a block cipher nonce is generated as a reused nonce can often mean catastrophic failure and a leaked secret key.\r\n\r\nI'm curious to hear your thoughts on this change, I would be happy to do the work since I need an encrypted datastore for my own project, is this something you would be open to?"}}
{"actor":{"display_login":"davidrusu"}}