{"action":"created","issue":{"url":"https://api.github.com/repos/fullcontact/lmdb-zero/issues/2","repository_url":"https://api.github.com/repos/fullcontact/lmdb-zero","labels_url":"https://api.github.com/repos/fullcontact/lmdb-zero/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/fullcontact/lmdb-zero/issues/2/comments","events_url":"https://api.github.com/repos/fullcontact/lmdb-zero/issues/2/events","html_url":"https://github.com/fullcontact/lmdb-zero/issues/2","id":183534200,"number":2,"title":"Persistence","user":{"login":"KenanSulayman","id":280212,"avatar_url":"https://avatars.githubusercontent.com/u/280212?v=3","gravatar_id":"","url":"https://api.github.com/users/KenanSulayman","html_url":"https://github.com/KenanSulayman","followers_url":"https://api.github.com/users/KenanSulayman/followers","following_url":"https://api.github.com/users/KenanSulayman/following{/other_user}","gists_url":"https://api.github.com/users/KenanSulayman/gists{/gist_id}","starred_url":"https://api.github.com/users/KenanSulayman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KenanSulayman/subscriptions","organizations_url":"https://api.github.com/users/KenanSulayman/orgs","repos_url":"https://api.github.com/users/KenanSulayman/repos","events_url":"https://api.github.com/users/KenanSulayman/events{/privacy}","received_events_url":"https://api.github.com/users/KenanSulayman/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-10-17T21:35:21Z","updated_at":"2016-10-24T22:04:06Z","closed_at":"2016-10-17T22:15:13Z","body":"Hey,\r\n\r\nI'm trying to add this as a cache to one of our toolchain components, but I have problems with data not being persisted. I have 500 keys that contain json data and I am writing a u64 and a [u8] for each of them. Additionally, after each commit I am calling env#sync(true), but after restarting the tool, only half of the keys are there.\r\n\r\nSpecifically, this is a snippet from the impl that is used to persist data with lmdb-zero:\r\n\r\n```\r\npub struct Memory<'a> {\r\n    lmdb_env: &'a lmdb::Environment,\r\n    lmdb_db: &'a lmdb::Database<'a>\r\n}\r\n\r\nimpl<'a> Memory<'a> {\r\n    #[inline]\r\n    pub fn new (lmdb_env: &'a lmdb::Environment, lmdb_db: &'a lmdb::Database) -> Memory<'a> {\r\n        Memory {\r\n            lmdb_env: lmdb_env,\r\n            lmdb_db: lmdb_db\r\n        }\r\n    }\r\n\r\n    pub fn sync (&self) {\r\n        self.lmdb_env.sync(true);\r\n    }\r\n\r\n    #[inline]\r\n    pub fn put (&mut self, key: &str, data: &Vec<u8>) {\r\n        let tx = lmdb::WriteTransaction::new(&self.lmdb_env).unwrap();\r\n\r\n        {\r\n            let mut access = tx.access();\r\n\r\n            access.put(&self.lmdb_db, key, data.as_lmdb_bytes(), lmdb::put::Flags::empty()).unwrap();\r\n        }\r\n\r\n        tx.commit().unwrap();\r\n\r\n        self.sync();\r\n    }\r\n\r\n    #[inline]\r\n    pub fn put_u64 (&mut self, key: &str, data: &u64) {\r\n        let tx = lmdb::WriteTransaction::new(&self.lmdb_env).unwrap();\r\n\r\n        {\r\n            let mut access = tx.access();\r\n\r\n            access.put(&self.lmdb_db, key, data, lmdb::put::Flags::empty()).unwrap();\r\n        }\r\n\r\n        tx.commit().unwrap();\r\n\r\n        self.sync();\r\n    }\r\n\r\n    /* other methods here */\r\n}\r\n```\r\n\r\nThere's one method used that effectively does a 7200 second cache per URL:\r\n\r\n```\r\n    fn client_cond_get (url: &str,\r\n                        memory: &mut memory::Memory,\r\n                        client: &client::HTTP\r\n                    ) -> Vec<u8> {\r\n        /* unixtime */\r\n        let now = time::now().to_timespec().sec as u64;\r\n\r\n        let mut url_ts_id = url.to_string();\r\n                url_ts_id.push_str(&\"_ts\");\r\n\r\n        let known_url_ts = memory.get_u64(&url_ts_id);\r\n\r\n        if known_url_ts.is_some() {\r\n            if (now - known_url_ts.unwrap()) < 7200 {\r\n                return memory.get(&url).unwrap();\r\n            }\r\n        }\r\n\r\n        let res = client.get(url);\r\n\r\n        let _ = memory.put_u64(&url_ts_id, &now);\r\n\r\n        memory.put(&url, &res);\r\n\r\n        res\r\n    }\r\n```\r\n\r\nSo per http \"get\" operation there are at worst 2 reads and 2 puts, at best 2 reads (when data is obtained from the cache).\r\n\r\nCould you maybe help me brainstorm possible issues I should be aware of when doing a commit? I am aware of the fact that a commit is writing to disk, but I don't know how to deal best with the OS disk buffering.\r\n\r\nThanks a lot.\r\n\r\nTake this as a gift: 🍻"},"comment":{"url":"https://api.github.com/repos/fullcontact/lmdb-zero/issues/comments/255879695","html_url":"https://github.com/fullcontact/lmdb-zero/issues/2#issuecomment-255879695","issue_url":"https://api.github.com/repos/fullcontact/lmdb-zero/issues/2","id":255879695,"user":{"login":"AltSysrq","id":1392512,"avatar_url":"https://avatars.githubusercontent.com/u/1392512?v=3","gravatar_id":"","url":"https://api.github.com/users/AltSysrq","html_url":"https://github.com/AltSysrq","followers_url":"https://api.github.com/users/AltSysrq/followers","following_url":"https://api.github.com/users/AltSysrq/following{/other_user}","gists_url":"https://api.github.com/users/AltSysrq/gists{/gist_id}","starred_url":"https://api.github.com/users/AltSysrq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AltSysrq/subscriptions","organizations_url":"https://api.github.com/users/AltSysrq/orgs","repos_url":"https://api.github.com/users/AltSysrq/repos","events_url":"https://api.github.com/users/AltSysrq/events{/privacy}","received_events_url":"https://api.github.com/users/AltSysrq/received_events","type":"User","site_admin":false},"created_at":"2016-10-24T22:04:06Z","updated_at":"2016-10-24T22:04:06Z","body":"Version 0.3.0 has some changes that will help with this kind of problem:\r\n\r\n- Value rejection errors now have error messages indicating the exact problem and pointing to relevant documentation.\r\n\r\n- This particular instance — trying to get a bare `&u64` — is now a hard compile-time error as the bare primitive types other than `i8` and `u8` are no longer `LmdbRaw`, but _must_ be wrapped with `Unaligned`."}}
{"actor":{"display_login":"AltSysrq"}}