{"action":"created","issue":{"url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse/issues/808","repository_url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse","labels_url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse/issues/808/labels{/name}","comments_url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse/issues/808/comments","events_url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse/issues/808/events","html_url":"https://github.com/s3fs-fuse/s3fs-fuse/issues/808","id":348859524,"node_id":"MDU6SXNzdWUzNDg4NTk1MjQ=","number":808,"title":"Random write behavior","user":{"login":"kunallillaney","id":1380214,"node_id":"MDQ6VXNlcjEzODAyMTQ=","avatar_url":"https://avatars0.githubusercontent.com/u/1380214?v=4","gravatar_id":"","url":"https://api.github.com/users/kunallillaney","html_url":"https://github.com/kunallillaney","followers_url":"https://api.github.com/users/kunallillaney/followers","following_url":"https://api.github.com/users/kunallillaney/following{/other_user}","gists_url":"https://api.github.com/users/kunallillaney/gists{/gist_id}","starred_url":"https://api.github.com/users/kunallillaney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kunallillaney/subscriptions","organizations_url":"https://api.github.com/users/kunallillaney/orgs","repos_url":"https://api.github.com/users/kunallillaney/repos","events_url":"https://api.github.com/users/kunallillaney/events{/privacy}","received_events_url":"https://api.github.com/users/kunallillaney/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-08-08T19:25:16Z","updated_at":"2018-09-17T20:34:45Z","closed_at":null,"author_association":"NONE","body":"I had a few questions on how s3fs handles random writes since I seeing some weird network stats (given the disclaimer in the README and this subsequent post #607)\r\n\r\nBackground:\r\n* Setup: Existing file on S3 - 100MB. I am using fio for random writes on this and monitoring the network I/O using slurm.\r\n* From my understanding, S3FS prefetches data in 40MB (multipart block size* threads) blocks into the cache when a region is a read. This is verified by slurm and the sparse file in the cache.\r\n* However, when I perform random writes on S3FS, say changing a single byte, no data is fetched either into the cache or shown by slurm (double checked with /proc/net/dev/) but file size worth of data (100MB in this case) does get transmitted to S3. Here s3fs is not overwriting the other parts of the file with garbage data but correctly updating the single byte.\r\n\r\nQuestion:\r\n* In the random write case, my assumption was s3fs did a multipart-download, merge changes and multipart-upload. Is this not the case? Does it perform multipart-copy for unchanged regions for S3 (similar to the operation for rename) but the network stats do not support this since it should have only transmitted 40MB worth data and copied the other 60MB. Am I missing something?\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse/issues/comments/422161255","html_url":"https://github.com/s3fs-fuse/s3fs-fuse/issues/808#issuecomment-422161255","issue_url":"https://api.github.com/repos/s3fs-fuse/s3fs-fuse/issues/808","id":422161255,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjE2MTI1NQ==","user":{"login":"gaul","id":848247,"node_id":"MDQ6VXNlcjg0ODI0Nw==","avatar_url":"https://avatars1.githubusercontent.com/u/848247?v=4","gravatar_id":"","url":"https://api.github.com/users/gaul","html_url":"https://github.com/gaul","followers_url":"https://api.github.com/users/gaul/followers","following_url":"https://api.github.com/users/gaul/following{/other_user}","gists_url":"https://api.github.com/users/gaul/gists{/gist_id}","starred_url":"https://api.github.com/users/gaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gaul/subscriptions","organizations_url":"https://api.github.com/users/gaul/orgs","repos_url":"https://api.github.com/users/gaul/repos","events_url":"https://api.github.com/users/gaul/events{/privacy}","received_events_url":"https://api.github.com/users/gaul/received_events","type":"User","site_admin":false},"created_at":"2018-09-17T20:34:45Z","updated_at":"2018-09-17T20:34:45Z","author_association":"CONTRIBUTOR","body":"The boto3 documentation for part size is incorrect.  This phrase seems to originate from some AWS code generation which I cannot find the source for so I opened aws/aws-cli#3577 which has the same issue and hope they can direct to the proper location.\r\n\r\nDid you reproduce your data loss issue?  Please provide instructions so I can investigate this otherwise can you close the issue?"}}
{"actor":{"display_login":"gaul"}}