{"action":"opened","issue":{"url":"https://api.github.com/repos/spacejam/sled/issues/432","repository_url":"https://api.github.com/repos/spacejam/sled","labels_url":"https://api.github.com/repos/spacejam/sled/issues/432/labels{/name}","comments_url":"https://api.github.com/repos/spacejam/sled/issues/432/comments","events_url":"https://api.github.com/repos/spacejam/sled/issues/432/events","html_url":"https://github.com/spacejam/sled/issues/432","id":381703388,"node_id":"MDU6SXNzdWUzODE3MDMzODg=","number":432,"title":"lock-free SegmentAccountant","user":{"login":"spacejam","id":1505488,"node_id":"MDQ6VXNlcjE1MDU0ODg=","avatar_url":"https://avatars0.githubusercontent.com/u/1505488?v=4","gravatar_id":"","url":"https://api.github.com/users/spacejam","html_url":"https://github.com/spacejam","followers_url":"https://api.github.com/users/spacejam/followers","following_url":"https://api.github.com/users/spacejam/following{/other_user}","gists_url":"https://api.github.com/users/spacejam/gists{/gist_id}","starred_url":"https://api.github.com/users/spacejam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spacejam/subscriptions","organizations_url":"https://api.github.com/users/spacejam/orgs","repos_url":"https://api.github.com/users/spacejam/repos","events_url":"https://api.github.com/users/spacejam/events{/privacy}","received_events_url":"https://api.github.com/users/spacejam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-16T18:02:17Z","updated_at":"2018-11-16T18:02:17Z","closed_at":null,"author_association":"OWNER","body":"All pagecache mutations block on taking out a `SegmentAccountant` Mutex.\r\n\r\nDesired effects:\r\n* make SA less contentious\r\n* properly track segment space utilization in bytes\r\n* avoid thundering herd issues with page rewrites\r\n\r\nSegments can be seen as having a lifespan that traverses the following stages:\r\n\r\n1. Active: the Segment has just been allocated, and threads are busy making reservations that will fill up the segment. Primary concern: low latency write access. Space is not an issue for Active segments, because there will only be <# io bufs> of them. A naive wait-free `Radix` tree is fine for this. We must wait in the Active state until all threads that may have created a Reservation corresponding to this Segment have called `SegmentAccountant::mark_replace` and exited the Reservation, possibly writing and fsyncing it.\r\n2. Inactive: the segment has been fsync'd and is now stable. no more pages will ever be written. the segment is now in a state of decay until it is eventually Free. We can give segments in this state a Vec of <# pages present> / 64  AtomicUsize's, and use `fetch_or` to mark pages as no-longer-present. If our `fetch_or` succeeds in flipping the desired bit, we can decrement a `AtomicUsize`. It will remain in this state until it falls below its minimum desired utilization. There are going to be a lot of these. We need to minimize \r\n3. Draining: When a node falls below a certain space utilization, we need to start copying resident pages to other segments. This includes the same functionality as above, with the added complexity of needing to mark pages as requiring relocation. This is done by asking the pagecache to rewrite the page. It should also tell pagecache which segment it's trying to clean up for that page, so the pagecache can avoid rewriting the page if there was a race and the page is no longer resident there. We need to keep track of which segments are draining, which pages need to be rewritten, and which pages have possible open requests out to rewrite them. Care needs to be taken to manage the increased latency between buffer reservations, calls to mark_replace, and a later call to actually deactivate the segment from IoBufs::make_stable.\r\n4. Free: all pages that have migrated away from this segment are now stable elsewhere."}}
{"actor":{"display_login":"spacejam"}}