{"action":"created","issue":{"url":"https://api.github.com/repos/zarr-developers/zarr/issues/278","repository_url":"https://api.github.com/repos/zarr-developers/zarr","labels_url":"https://api.github.com/repos/zarr-developers/zarr/issues/278/labels{/name}","comments_url":"https://api.github.com/repos/zarr-developers/zarr/issues/278/comments","events_url":"https://api.github.com/repos/zarr-developers/zarr/issues/278/events","html_url":"https://github.com/zarr-developers/zarr/issues/278","id":343014230,"node_id":"MDU6SXNzdWUzNDMwMTQyMzA=","number":278,"title":"LRU cache for decoded chunks","user":{"login":"alimanfoo","id":703554,"node_id":"MDQ6VXNlcjcwMzU1NA==","avatar_url":"https://avatars1.githubusercontent.com/u/703554?v=4","gravatar_id":"","url":"https://api.github.com/users/alimanfoo","html_url":"https://github.com/alimanfoo","followers_url":"https://api.github.com/users/alimanfoo/followers","following_url":"https://api.github.com/users/alimanfoo/following{/other_user}","gists_url":"https://api.github.com/users/alimanfoo/gists{/gist_id}","starred_url":"https://api.github.com/users/alimanfoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alimanfoo/subscriptions","organizations_url":"https://api.github.com/users/alimanfoo/orgs","repos_url":"https://api.github.com/users/alimanfoo/repos","events_url":"https://api.github.com/users/alimanfoo/events{/privacy}","received_events_url":"https://api.github.com/users/alimanfoo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-07-20T08:15:41Z","updated_at":"2018-08-28T11:19:30Z","closed_at":null,"author_association":"MEMBER","body":"Via @calclavia from https://github.com/zarr-developers/zarr/issues/265#issuecomment-405452022 breaking this out as a separate issue:\r\n\r\n> I've a related use case that I believe would benefit from introducing chunk-based decompressed cache. I use zarr for storing data that will be used for training neural networks. For this use case, often times you want to sample random (or almost random) rows from the dataset. If the sampling is mostly localized within a chunk, it would be great if the LRUCache could cache an entire chunk so we can take advantage of spatial locality.\r\n> \r\n> For example, I would like to same data points [1, 5, 8, 3, 2], and because these all reside in the same compressed chunk (cached by LRU), only reading the first sample should be slow, and the rest should be already cached in memory.\r\n\r\nN.B., this differs from the LRUStoreCache class already implemented, because that caches the *encoded* chunk data, but the proposal here is to add a layer for caching the *decoded* chunk data (and therefore avoid having to decode the chunk multiple times)."},"comment":{"url":"https://api.github.com/repos/zarr-developers/zarr/issues/comments/416548032","html_url":"https://github.com/zarr-developers/zarr/issues/278#issuecomment-416548032","issue_url":"https://api.github.com/repos/zarr-developers/zarr/issues/278","id":416548032,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjU0ODAzMg==","user":{"login":"shikharsg","id":8643775,"node_id":"MDQ6VXNlcjg2NDM3NzU=","avatar_url":"https://avatars0.githubusercontent.com/u/8643775?v=4","gravatar_id":"","url":"https://api.github.com/users/shikharsg","html_url":"https://github.com/shikharsg","followers_url":"https://api.github.com/users/shikharsg/followers","following_url":"https://api.github.com/users/shikharsg/following{/other_user}","gists_url":"https://api.github.com/users/shikharsg/gists{/gist_id}","starred_url":"https://api.github.com/users/shikharsg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shikharsg/subscriptions","organizations_url":"https://api.github.com/users/shikharsg/orgs","repos_url":"https://api.github.com/users/shikharsg/repos","events_url":"https://api.github.com/users/shikharsg/events{/privacy}","received_events_url":"https://api.github.com/users/shikharsg/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T11:19:30Z","updated_at":"2018-08-28T11:19:30Z","author_association":"NONE","body":"If I've understood you correct, do you mean that the mutable mapping(either the `store` or the `chunk_cache`) won't implement encode/decode at all? And in `_chunk_getitem()`, before trying to get from store and decode, we would first try to get from `chunk_cache`(if one has been provided) and if we get a KeyError, we try getting from the store and decode?"}}
{"actor":{"display_login":"shikharsg"}}