{"action":"created","issue":{"url":"https://api.github.com/repos/s3ql/s3ql/issues/52","repository_url":"https://api.github.com/repos/s3ql/s3ql","labels_url":"https://api.github.com/repos/s3ql/s3ql/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/s3ql/s3ql/issues/52/comments","events_url":"https://api.github.com/repos/s3ql/s3ql/issues/52/events","html_url":"https://github.com/s3ql/s3ql/issues/52","id":394630853,"node_id":"MDU6SXNzdWUzOTQ2MzA4NTM=","number":52,"title":"Support storing multiple files in the same backend object (\"fragments\")","user":{"login":"Nikratio","id":701625,"node_id":"MDQ6VXNlcjcwMTYyNQ==","avatar_url":"https://avatars1.githubusercontent.com/u/701625?v=4","gravatar_id":"","url":"https://api.github.com/users/Nikratio","html_url":"https://github.com/Nikratio","followers_url":"https://api.github.com/users/Nikratio/followers","following_url":"https://api.github.com/users/Nikratio/following{/other_user}","gists_url":"https://api.github.com/users/Nikratio/gists{/gist_id}","starred_url":"https://api.github.com/users/Nikratio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Nikratio/subscriptions","organizations_url":"https://api.github.com/users/Nikratio/orgs","repos_url":"https://api.github.com/users/Nikratio/repos","events_url":"https://api.github.com/users/Nikratio/events{/privacy}","received_events_url":"https://api.github.com/users/Nikratio/received_events","type":"User","site_admin":false},"labels":[{"id":235435719,"node_id":"MDU6TGFiZWwyMzU0MzU3MTk=","url":"https://api.github.com/repos/s3ql/s3ql/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":235435723,"node_id":"MDU6TGFiZWwyMzU0MzU3MjM=","url":"https://api.github.com/repos/s3ql/s3ql/labels/wontfix","name":"wontfix","color":"ffffff","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2018-12-28T12:53:44Z","updated_at":"2019-01-04T20:05:26Z","closed_at":"2019-01-04T13:15:04Z","author_association":"COLLABORATOR","body":"[migrated from BitBucket]\r\n\r\nStoring lots of small files is very inefficient, since every file requires its own block.\r\n\r\nWe should add support for fragments, so that multiple files can be stored in the same block.\r\n\r\nWith the new bucket interface, we should be able to implement this relatively easily:\r\n\r\n-    Upload workers get list of cache entries, new blocks may be coalesced into single object\r\n-    CommitThread() and expire() only call to worker threads once they have a reasonably big chunk of data ready\r\n-    We keep objects until reference count of all contained blocks is zero\r\n-    Therefore, blocks may continue to exist with refcount=0 and can possibly be reused\r\n-    s3qladm may need a \"cleanup\" function to get rid of these blocks\r\n-    When downloading object, db can be used to determine which blocks in the object belong to files (and should be added to cache) and which ones can be discarded\r\n-    Minimum size of cache entries passed to workers could be adjusted dynamically based on upload bandwith, latency, and compression ratio of previous uploads\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/s3ql/s3ql/issues/comments/451553874","html_url":"https://github.com/s3ql/s3ql/issues/52#issuecomment-451553874","issue_url":"https://api.github.com/repos/s3ql/s3ql/issues/52","id":451553874,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MTU1Mzg3NA==","user":{"login":"segator","id":2348131,"node_id":"MDQ6VXNlcjIzNDgxMzE=","avatar_url":"https://avatars2.githubusercontent.com/u/2348131?v=4","gravatar_id":"","url":"https://api.github.com/users/segator","html_url":"https://github.com/segator","followers_url":"https://api.github.com/users/segator/followers","following_url":"https://api.github.com/users/segator/following{/other_user}","gists_url":"https://api.github.com/users/segator/gists{/gist_id}","starred_url":"https://api.github.com/users/segator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/segator/subscriptions","organizations_url":"https://api.github.com/users/segator/orgs","repos_url":"https://api.github.com/users/segator/repos","events_url":"https://api.github.com/users/segator/events{/privacy}","received_events_url":"https://api.github.com/users/segator/received_events","type":"User","site_admin":false},"created_at":"2019-01-04T20:05:26Z","updated_at":"2019-01-04T20:05:26Z","author_association":"NONE","body":"> Neither GDrive nor Backblaze are currently supported by S3QL, so I don't think that rate limits on their end should influence this decision\r\nhope gdrive it will be suported soon.\r\n\r\ngdrive list object is slow, can take 2-10s per request\r\nabout parallel is not possible at least with gdrive, you get a pagination token to go the next page, I cannot jump between the whole list. I need to interate in order.\r\nProxyFS I talked with one of the developers in their slack to know more about how it works, but I finally See that have exactly same problem as s3ql, only single mount at a time is possible.\r\n\r\n"}}
{"actor":{"display_login":"segator"}}