{"action":"opened","issue":{"url":"https://api.github.com/repos/influxdata/platform/issues/1983","repository_url":"https://api.github.com/repos/influxdata/platform","labels_url":"https://api.github.com/repos/influxdata/platform/issues/1983/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/platform/issues/1983/comments","events_url":"https://api.github.com/repos/influxdata/platform/issues/1983/events","html_url":"https://github.com/influxdata/platform/issues/1983","id":391406563,"node_id":"MDU6SXNzdWUzOTE0MDY1NjM=","number":1983,"title":"tsm1: issues found investigating deletes for the TSM index","user":{"login":"zeebo","id":96925,"node_id":"MDQ6VXNlcjk2OTI1","avatar_url":"https://avatars2.githubusercontent.com/u/96925?v=4","gravatar_id":"","url":"https://api.github.com/users/zeebo","html_url":"https://github.com/zeebo","followers_url":"https://api.github.com/users/zeebo/followers","following_url":"https://api.github.com/users/zeebo/following{/other_user}","gists_url":"https://api.github.com/users/zeebo/gists{/gist_id}","starred_url":"https://api.github.com/users/zeebo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeebo/subscriptions","organizations_url":"https://api.github.com/users/zeebo/orgs","repos_url":"https://api.github.com/users/zeebo/repos","events_url":"https://api.github.com/users/zeebo/events{/privacy}","received_events_url":"https://api.github.com/users/zeebo/received_events","type":"User","site_admin":false},"labels":[{"id":1071471878,"node_id":"MDU6TGFiZWwxMDcxNDcxODc4","url":"https://api.github.com/repos/influxdata/platform/labels/area/performance","name":"area/performance","color":"4adb11","default":false},{"id":983113072,"node_id":"MDU6TGFiZWw5ODMxMTMwNzI=","url":"https://api.github.com/repos/influxdata/platform/labels/area/storage","name":"area/storage","color":"8adcfc","default":false},{"id":948324435,"node_id":"MDU6TGFiZWw5NDgzMjQ0MzU=","url":"https://api.github.com/repos/influxdata/platform/labels/difficulty/medium","name":"difficulty/medium","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":{"login":"zeebo","id":96925,"node_id":"MDQ6VXNlcjk2OTI1","avatar_url":"https://avatars2.githubusercontent.com/u/96925?v=4","gravatar_id":"","url":"https://api.github.com/users/zeebo","html_url":"https://github.com/zeebo","followers_url":"https://api.github.com/users/zeebo/followers","following_url":"https://api.github.com/users/zeebo/following{/other_user}","gists_url":"https://api.github.com/users/zeebo/gists{/gist_id}","starred_url":"https://api.github.com/users/zeebo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeebo/subscriptions","organizations_url":"https://api.github.com/users/zeebo/orgs","repos_url":"https://api.github.com/users/zeebo/repos","events_url":"https://api.github.com/users/zeebo/events{/privacy}","received_events_url":"https://api.github.com/users/zeebo/received_events","type":"User","site_admin":false},"assignees":[{"login":"zeebo","id":96925,"node_id":"MDQ6VXNlcjk2OTI1","avatar_url":"https://avatars2.githubusercontent.com/u/96925?v=4","gravatar_id":"","url":"https://api.github.com/users/zeebo","html_url":"https://github.com/zeebo","followers_url":"https://api.github.com/users/zeebo/followers","following_url":"https://api.github.com/users/zeebo/following{/other_user}","gists_url":"https://api.github.com/users/zeebo/gists{/gist_id}","starred_url":"https://api.github.com/users/zeebo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeebo/subscriptions","organizations_url":"https://api.github.com/users/zeebo/orgs","repos_url":"https://api.github.com/users/zeebo/repos","events_url":"https://api.github.com/users/zeebo/events{/privacy}","received_events_url":"https://api.github.com/users/zeebo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-12-15T19:49:08Z","updated_at":"2018-12-15T19:49:08Z","closed_at":null,"author_association":"MEMBER","body":"While investigating deletes and performance for the TSM index, a number of bugs and inefficiencies were found. Fixing these issues will help with performance, robustness, ease of implementation, and maintainability for #1837.\r\n\r\n- [ ] `indirectIndex` keeps the offsets slice in an anonymous mmap, which does nothing other than hide the allocation from the Go profiling tools.\r\n- [ ] `directIndex` keeps track of the size as a `uint32`, but `indirectIndex` uses `int32` or even `int` to record offsets.\r\n- [ ] `directIndex` doesn't ensure that the uint32 doesn't overflow. if it does, the `indirectIndex` is hopeless.\r\n- [ ] Incorrect locking during `(*indirectIndex).DeleteRange`\r\n- [ ] Suboptimal tracking of if tombstones cover the full range of a series, in that it requires the tombstones be contiguous and covering, rather than just fully covering the series.\r\n- [ ] Tombstones are tracked in a `map[string]...` instead of a `map[uint32]...` where the `uint32` uniquely identifies the key by offset.\r\n- [ ] Seeks require reading the tsm index for finding a key, when keeping some extra bytes in ram would greatly reduce the disk hits (will add benchmarks/tracking for hits to the mmap buffer to check this).\r\n- [ ] Many places use external iteration into the index by indexing into the offsets, which become silently invalid if they are deleted. Some iterators check this by making sure `KeyCount` didn't decrease, but I think we can do better.\r\n- [ ] `DeleteRange` and `Delete` on the `*indirectIndex` don't attempt to do any binary searches past the initial key, so non-contiguous key deletes are problematic.\r\n- [ ] Multiple methods doing the same thing but with different interfaces for efficiency. They can be unified behind an iterator type.\r\n- [ ] Interface types where only one concrete representation exists, needlessly making the codebase harder to follow.\r\n- [ ] Using a pattern like `x = search(i, &x)` to reuse a slice stored in `x` rather than the more familiar `x = search(i, x)` pattern that `append` uses.\r\n- [ ] Errors when reading the index are ignored in multiple places. The callers should decide what the appropriate response is (return an error, ignore it for that key, etc.)\r\n\r\n"}}
{"actor":{"display_login":"zeebo"}}