{"action":"closed","issue":{"url":"https://api.github.com/repos/borgbackup/borg/issues/1452","repository_url":"https://api.github.com/repos/borgbackup/borg","labels_url":"https://api.github.com/repos/borgbackup/borg/issues/1452/labels{/name}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/1452/comments","events_url":"https://api.github.com/repos/borgbackup/borg/issues/1452/events","html_url":"https://github.com/borgbackup/borg/issues/1452","id":169986284,"number":1452,"title":"per archive total item count limit","user":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/borgbackup/borg/labels/documentation","name":"documentation","color":"bfe5bf"},{"url":"https://api.github.com/repos/borgbackup/borg/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"assignees":[{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/borgbackup/borg/milestones/28","html_url":"https://github.com/borgbackup/borg/milestones/1.0.7rc2","labels_url":"https://api.github.com/repos/borgbackup/borg/milestones/28/labels","id":1925877,"number":28,"title":"1.0.7rc2","description":"","creator":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":16,"state":"open","created_at":"2016-08-05T16:36:45Z","updated_at":"2016-08-12T22:16:17Z","due_on":"2016-08-13T00:00:00Z","closed_at":null},"comments":9,"created_at":"2016-08-08T18:01:51Z","updated_at":"2016-08-12T22:16:18Z","closed_at":"2016-08-12T22:16:17Z","body":"the archive item (which has a list of \"item metadata stream\" chunks) is currently stored as a single object.\r\n\r\nwe limit size of repo objects to MAX_OBJ_SIZE (20MB).\r\nnote: we did only enfore the limit on get() - not on put().\r\n\r\neach list entry takes ~40 bytes (32 sha256 id + 2 * 4 for the size / csize), msgpacked a bit less maybe.\r\n\r\nso, that means we can reference ~500.000 item metadata stream chunks per archive.\r\n\r\neach item metadata stream chunk is ~16kiB (due to special, more fine grained and hardcoded item metadata chunker settings).\r\n\r\nso that means the whole item metadata stream is limited to ~8GB.\r\n\r\nthe item entry size is determined by:\r\n- file size (number of chunks * 40B)\r\n- path lengths\r\n- amount of ACLs, xattrs\r\n- a constant small amount of file metadata (timestamps, uid/gid, mode, ...)\r\n\r\nif the medium size of an item entry is 100B (small size file, no ACLs/xattrs), that means we have a limit of 80M files/directories per archive. if the medium size of an item entry is 2kB (~100MB size files), that means we have a limit of 4M files/directories per archive.\r\n\r\nwe need a short term change to relax this limit and also a long term change to resolve it completely, this issue is about the short term change.\r\n\r\nalso, this limit should be documented."}}
{"actor":{"display_login":"ThomasWaldmann"}}