{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5114","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5114/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5114/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5114/events","html_url":"https://github.com/zfsonlinux/zfs/issues/5114","id":177266729,"number":5114,"title":"zvol slow on overwrites","user":{"login":"mheubach","id":8254426,"avatar_url":"https://avatars.githubusercontent.com/u/8254426?v=3","gravatar_id":"","url":"https://api.github.com/users/mheubach","html_url":"https://github.com/mheubach","followers_url":"https://api.github.com/users/mheubach/followers","following_url":"https://api.github.com/users/mheubach/following{/other_user}","gists_url":"https://api.github.com/users/mheubach/gists{/gist_id}","starred_url":"https://api.github.com/users/mheubach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mheubach/subscriptions","organizations_url":"https://api.github.com/users/mheubach/orgs","repos_url":"https://api.github.com/users/mheubach/repos","events_url":"https://api.github.com/users/mheubach/events{/privacy}","received_events_url":"https://api.github.com/users/mheubach/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-15T19:29:06Z","updated_at":"2016-09-16T18:21:34Z","closed_at":null,"body":"Hello,\r\n\r\nI think I either hit a bug or a basic misunderstanding. In a testing environment I get good write performance when writing to a zvol for the first time. All subsequent writes are becoming terribly slow.\r\nA closer look reveals that there are a lot of read operations going on when overwriting a zvol.\r\n\r\nMy pool is a striped mirror of 6 sas drives. I have a 10GB file of arbitrary data which I copy with ddrescue to the zvol. ZFS is 0.6.5.6 on Ubuntu 16.04.\r\n\r\n`zfs create -V 100G -o volblocksize=64k testpool/testvol\r\nddrescue /root/ddfile /dev/zvol/testpool/testvol -f `\r\n\r\nzpool iostat -v 5 in a second session shows mainly something like this:\r\n\r\n`                               capacity     operations    bandwidth\r\npool                        alloc   free   read  write   read  write\r\n--------------------------  -----  -----  -----  -----  -----  -----\r\ntestpool                    6.49G  1.35T      0  4.90K      0   293M\r\n  mirror                    2.16G   462G      0  1.68K      0   100M\r\n    scsi-35000c500100c3a73      -      -      0    842      0  99.8M\r\n    scsi-35000c5002639c333      -      -      0    846      0   100M\r\n  mirror                    2.16G   462G      0  1.60K      0  95.4M\r\n    scsi-35000c500100c418f      -      -      0    822      0  97.7M\r\n    scsi-35000c50010358383      -      -      0    802      0  95.4M\r\n  mirror                    2.17G   462G      0  1.62K      0  97.3M\r\n    scsi-35000c500103584f3      -      -      0    829      0  98.2M\r\n    scsi-35000c50026395057      -      -      0    821      0  97.3M\r\n--------------------------  -----  -----  -----  -----  -----  -----\r\n`\r\n\r\nIf I run ddrescue for a second time the zvol slows down and I can see read activity on the zvol:\r\n\r\n`                               capacity     operations    bandwidth\r\npool                        alloc   free   read  write   read  write\r\n--------------------------  -----  -----  -----  -----  -----  -----\r\ntestpool                    9.13G  1.35T    420    419  23.5M  19.5M\r\n  mirror                    3.04G   461G    106    135  5.94M  6.24M\r\n    scsi-35000c500100c3a73      -      -     52     63  2.96M  6.24M\r\n    scsi-35000c5002639c333      -      -     53     63  2.98M  6.24M\r\n  mirror                    3.04G   461G    166    140  9.18M  6.77M\r\n    scsi-35000c500100c418f      -      -     99     67  5.40M  6.77M\r\n    scsi-35000c50010358383      -      -     66     66  3.78M  6.77M\r\n  mirror                    3.04G   461G    147    142  8.40M  6.44M\r\n    scsi-35000c500103584f3      -      -     97     61  5.54M  6.45M\r\n    scsi-35000c50026395057      -      -     49     62  2.87M  6.45M\r\n--------------------------  -----  -----  -----  -----  -----  -----\r\n`\r\n\r\nThis is reproducible. First writes are fast second writes to the same blocks are slow. I can delete the zvol and recreate it - always the same. If I use snapshots and create a snapshot immediately after creating the zvol I can just rollback the zvol and have the same story. First writes are fast second writes to same blocks are slow.\r\nIf I rerun ddrescue with e.g. opos=20G on a zvol which already became slow I have a fast write again. However if I overwrite the data on the same position it is slow again.\r\nWhat really amazes me is that there is so much reading going on which is the reason for the slow writes. But I don't now why that is and how to stop it.\r\n\r\nThis only happens when writing directly into the zvol. If I create an ext4 filesystem on the zvol and mount it, I can copy and overwrite my 10GB file as many times as I like it is always fast and there is nearly no reading going on.\r\n\r\nI also tested this with Ubuntu 14.04 and ZFS 0.6.5.3 and PVE (Proxmox) ZFS 0.6.5.7 - it's all the same although there are differences in overall performance.\r\n\r\nSo what is happening here? Does anyone have an idea?\r\n\r\nBest regards\r\nManfred"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/247671782","html_url":"https://github.com/zfsonlinux/zfs/issues/5114#issuecomment-247671782","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5114","id":247671782,"user":{"login":"behlendorf","id":148917,"avatar_url":"https://avatars.githubusercontent.com/u/148917?v=3","gravatar_id":"","url":"https://api.github.com/users/behlendorf","html_url":"https://github.com/behlendorf","followers_url":"https://api.github.com/users/behlendorf/followers","following_url":"https://api.github.com/users/behlendorf/following{/other_user}","gists_url":"https://api.github.com/users/behlendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/behlendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behlendorf/subscriptions","organizations_url":"https://api.github.com/users/behlendorf/orgs","repos_url":"https://api.github.com/users/behlendorf/repos","events_url":"https://api.github.com/users/behlendorf/events{/privacy}","received_events_url":"https://api.github.com/users/behlendorf/received_events","type":"User","site_admin":false},"created_at":"2016-09-16T18:21:33Z","updated_at":"2016-09-16T18:21:33Z","body":"This is absolutely because ZFS is a CoW filesystem and the problem is described in #361.  The effect is mitigated somewhat when there's a filesystem mounted on the volume because it will be batching it's IO in to larger chunks.\r\n\r\nI'd be interested to hear what kind of results you get when testing with the [0.7.0-rc1 ](\r\nhttps://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.0-rc1) release candidate.  Commit 755065f3dce1123eac03e2b25c81647026b8e49b significantly improved the prefetch logic which may help your ddrescue workload."}}
{"actor":{"display_login":"behlendorf"}}