{"action":"created","issue":{"url":"https://api.github.com/repos/frerich/clcache/issues/173","repository_url":"https://api.github.com/repos/frerich/clcache","labels_url":"https://api.github.com/repos/frerich/clcache/issues/173/labels{/name}","comments_url":"https://api.github.com/repos/frerich/clcache/issues/173/comments","events_url":"https://api.github.com/repos/frerich/clcache/issues/173/events","html_url":"https://github.com/frerich/clcache/pull/173","id":168331268,"number":173,"title":"Manifest sections","user":{"login":"frerich","id":533425,"avatar_url":"https://avatars.githubusercontent.com/u/533425?v=3","gravatar_id":"","url":"https://api.github.com/users/frerich","html_url":"https://github.com/frerich","followers_url":"https://api.github.com/users/frerich/followers","following_url":"https://api.github.com/users/frerich/following{/other_user}","gists_url":"https://api.github.com/users/frerich/gists{/gist_id}","starred_url":"https://api.github.com/users/frerich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frerich/subscriptions","organizations_url":"https://api.github.com/users/frerich/orgs","repos_url":"https://api.github.com/users/frerich/repos","events_url":"https://api.github.com/users/frerich/events{/privacy}","received_events_url":"https://api.github.com/users/frerich/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-29T14:00:27Z","updated_at":"2016-07-30T16:04:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/frerich/clcache/pulls/173","html_url":"https://github.com/frerich/clcache/pull/173","diff_url":"https://github.com/frerich/clcache/pull/173.diff","patch_url":"https://github.com/frerich/clcache/pull/173.patch"},"body":"This PR introduces the notion of 'manifest sections'. The `ManifestsManager` class provides access to one of up to 256 `ManifestSection` objects, each of which representing one of the up to 256 subdirectories of the `manifests` directory. A manifest section is then the object which provides access to individual manifests.\r\n\r\nThe PR makes the code base use `ManifestSection` objects throughout. The purpose of this is to make it easy to introduce section-specific locks in a future commit, by adding a single `with manifestSection.lock:` in the `processDirect` function and then having the rest of the code operate on that single locked section, instead of locking the entire cache.\r\n\r\nThe PR should not introduce any functional change, it's just preparation for adding more fine-grained locking in order to address #160 .\r\n\r\n@webmaster128 can you have a look? The `ManifestsManager` is kinda your baby, I'd rather not botch it right away."},"comment":{"url":"https://api.github.com/repos/frerich/clcache/issues/comments/236373435","html_url":"https://github.com/frerich/clcache/pull/173#issuecomment-236373435","issue_url":"https://api.github.com/repos/frerich/clcache/issues/173","id":236373435,"user":{"login":"webmaster128","id":2603011,"avatar_url":"https://avatars.githubusercontent.com/u/2603011?v=3","gravatar_id":"","url":"https://api.github.com/users/webmaster128","html_url":"https://github.com/webmaster128","followers_url":"https://api.github.com/users/webmaster128/followers","following_url":"https://api.github.com/users/webmaster128/following{/other_user}","gists_url":"https://api.github.com/users/webmaster128/gists{/gist_id}","starred_url":"https://api.github.com/users/webmaster128/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/webmaster128/subscriptions","organizations_url":"https://api.github.com/users/webmaster128/orgs","repos_url":"https://api.github.com/users/webmaster128/repos","events_url":"https://api.github.com/users/webmaster128/events{/privacy}","received_events_url":"https://api.github.com/users/webmaster128/received_events","type":"User","site_admin":false},"created_at":"2016-07-30T16:04:51Z","updated_at":"2016-07-30T16:04:51Z","body":"Turns out the manifest operation is slow compared to restoring an object.\r\n\r\nI took 3 manifests with approx. 1000, 500 and 100 includes with objects of size 650, 570 and 80 kB. In all cases, the manifest operation results in multiples on copying time.\r\n\r\nThis is the result with hardlinks enabled.\r\n\r\n```\r\nManifest operation\r\n\r\nL 942 includes\r\nM 504 includes\r\nS 127 includes\r\n\r\nL 314.64345000000003 ms\r\nM 166.14678000000004 ms\r\nS 48.97450000000001 ms\r\n\r\n\r\nCopy and hit count operation\r\nL 645 kB\r\nM 565 kB\r\nS 82 kB\r\nL 4.750289999999957 ms\r\nM 4.19571000000003 ms\r\nS 4.198900000000005 ms\r\n```\r\n\r\nWithout hardlinks, the copying takes twice the time:\r\n\r\n```\r\nManifest operation\r\n\r\nL 942 includes\r\nM 504 includes\r\nS 127 includes\r\n\r\nL 323.68877000000003 ms\r\nM 162.54767999999996 ms\r\nS 48.13206000000001 ms\r\n\r\n\r\nCopy and hit count operation\r\nL 645 kB\r\nM 565 kB\r\nS 82 kB\r\nL 8.776709999999976 ms\r\nM 8.943410000000007 ms\r\nS 6.7199200000000125 ms\r\n```\r\n\r\nSo the resulting factors are 12x-66x for hardlink mode and 7x-37x.\r\n\r\nGiven those numbers, the question is if we should touch objects and stats locks at all.\r\n\r\n-----------------------------\r\n\r\nRegarding this PR: Could you rebase it after merging #174, please?"}}
{"actor":{"display_login":"webmaster128"}}