{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/4056","repository_url":"https://api.github.com/repos/facebook/rocksdb","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/4056/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/4056/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/4056/events","html_url":"https://github.com/facebook/rocksdb/issues/4056","id":335732167,"node_id":"MDU6SXNzdWUzMzU3MzIxNjc=","number":4056,"title":"Feature Request: Allowing write to memtable fail","user":{"login":"rockeet","id":1574991,"node_id":"MDQ6VXNlcjE1NzQ5OTE=","avatar_url":"https://avatars3.githubusercontent.com/u/1574991?v=4","gravatar_id":"","url":"https://api.github.com/users/rockeet","html_url":"https://github.com/rockeet","followers_url":"https://api.github.com/users/rockeet/followers","following_url":"https://api.github.com/users/rockeet/following{/other_user}","gists_url":"https://api.github.com/users/rockeet/gists{/gist_id}","starred_url":"https://api.github.com/users/rockeet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rockeet/subscriptions","organizations_url":"https://api.github.com/users/rockeet/orgs","repos_url":"https://api.github.com/users/rockeet/repos","events_url":"https://api.github.com/users/rockeet/events{/privacy}","received_events_url":"https://api.github.com/users/rockeet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-06-26T09:24:26Z","updated_at":"2018-06-29T21:54:33Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### Expected behavior\r\nWhen write to a memtable, it may failed because reached memory limit, in this failed case, rocksdb should create a new memtable and write to the new memtable.\r\n\r\n### Actual behavior\r\nNow rocksdb memtable does not allow write fail, but checks the memtable memory usage and to dicide wheather need to create a new memtable. This mechenism has many defects:\r\n\r\n#### 1. Waste memory\r\nFor default `SkipList` memtable, maybe the memtable is marked imutable just after it allocated a new arena and inserted few entries: if we set `write_buffer_size` to `1GB`, area size should be `128M`, the last `128M` arena may be wasted.\r\n\r\n#### 2. Makes hard effort on third party memtable implementation\r\nWe implementead a `PatriciaTrie` based memtable, aimed to faster speed and lower memory usage, we preallocate a large block equals to `write_buffer_size`(e.g. `1GB`), to minimize memory usage, we using 32 bit offset instead of using 64 bit pointer for `PatriciaTrie` nodes, and we implemented copy on write to avoid using mutex lock in multi thread, by this approach, enlarge the memory block is impossible(e.g. doubling realloc the memory block, realloc may invalidate all pointers).\r\n\r\nNow our solution is to create a vector of `PatriciaTrie` in one memtable, if the **last** `PatriciaTrie` is full, we add new PatriciaTrie to the vector, new KeyValues are inserted into  new PatriciaTrie, and implement its iterator by `MergingIterator`..... but this is very ugly and greatly kill performance.\r\n"},"comment":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/comments/401483689","html_url":"https://github.com/facebook/rocksdb/issues/4056#issuecomment-401483689","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/4056","id":401483689,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTQ4MzY4OQ==","user":{"login":"riversand963","id":20094703,"node_id":"MDQ6VXNlcjIwMDk0NzAz","avatar_url":"https://avatars0.githubusercontent.com/u/20094703?v=4","gravatar_id":"","url":"https://api.github.com/users/riversand963","html_url":"https://github.com/riversand963","followers_url":"https://api.github.com/users/riversand963/followers","following_url":"https://api.github.com/users/riversand963/following{/other_user}","gists_url":"https://api.github.com/users/riversand963/gists{/gist_id}","starred_url":"https://api.github.com/users/riversand963/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riversand963/subscriptions","organizations_url":"https://api.github.com/users/riversand963/orgs","repos_url":"https://api.github.com/users/riversand963/repos","events_url":"https://api.github.com/users/riversand963/events{/privacy}","received_events_url":"https://api.github.com/users/riversand963/received_events","type":"User","site_admin":false},"created_at":"2018-06-29T21:54:33Z","updated_at":"2018-06-29T21:54:33Z","author_association":"CONTRIBUTOR","body":"@rockeet , Thanks for clarification of \"reaching memory limit\".\r\nHave you take a look at `WriteCallback` passed to `WriteImpl`? Is it possible to implement your feature here? This callback will be called before writing to the memtables, maybe you can check whether the current memtable is full."}}
{"actor":{"display_login":"riversand963"}}