{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4265","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4265/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4265/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4265/events","html_url":"https://github.com/zfsonlinux/zfs/issues/4265","id":128381187,"number":4265,"title":"ZVOLs 3X slower than recordsets at same recordsize/volblocksize for sequential writes","user":{"login":"sempervictus","id":1331084,"avatar_url":"https://avatars.githubusercontent.com/u/1331084?v=3","gravatar_id":"","url":"https://api.github.com/users/sempervictus","html_url":"https://github.com/sempervictus","followers_url":"https://api.github.com/users/sempervictus/followers","following_url":"https://api.github.com/users/sempervictus/following{/other_user}","gists_url":"https://api.github.com/users/sempervictus/gists{/gist_id}","starred_url":"https://api.github.com/users/sempervictus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sempervictus/subscriptions","organizations_url":"https://api.github.com/users/sempervictus/orgs","repos_url":"https://api.github.com/users/sempervictus/repos","events_url":"https://api.github.com/users/sempervictus/events{/privacy}","received_events_url":"https://api.github.com/users/sempervictus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2016-01-24T08:27:02Z","updated_at":"2016-02-02T17:40:00Z","closed_at":null,"body":"As we do more work on SSD-only pools we're starting to see consistent slowdowns across the board on ZVOLs vs normal data sets. Our use case is primarily iSCSI from ZVOLs, so this is a bit of a problem for us.\r\n\r\nTo prove this out, i wrote a simple test script for tiotest which creates zvols and data sets, benchmarks them, and destroys them:\r\n```\r\nMPOINT=$(zfs get mountpoint $POOL | awk '/mountp/{print$3}')\r\n\r\nfor i in 4 8 64 128 ; do\r\n  echo \"Testing ZVOL ${i}K blocksize on $POOL\"\r\n  zfs create $POOL/testvol-${i}k -o volblocksize=${i}k -V10G\r\n  tiotest -f 256 -R -r 30000 -d /dev/zvol/$POOL/testvol-${i}k\r\n  zfs destroy $POOL/testvol-${i}k\r\n  CMD=\"zfs create $POOL/testset-${i}k -o recordsize=${i}k\"\r\n  if [[ \"$MPOINT\" == \"none\" ]]; then\r\n    CMD=\"$CMD -o mountpoint=/testset-${i}k\"\r\n  fi\r\n  sh -c \"$CMD\"\r\n  LMP=$(zfs get mountpoint $POOL/testset-${i}k | awk '/mountp/{print$3}')\r\n  echo \"Testing DS ${i}K blocksize on $POOL\"\r\n  tiotest -f 256 -r 3000 -d $LMP\r\n  zfs destroy $POOL/testset-${i}k  \r\ndone\r\n```\r\nThe results (http://pastebin.com/AfU5VYuG) are odd - ~3X performance penalty on linear writes to ZVOLs, and somehow inverse for random writes. In neither case does the underlying disk show full-load on iostat. Thoughts?"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/178707263","html_url":"https://github.com/zfsonlinux/zfs/issues/4265#issuecomment-178707263","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4265","id":178707263,"user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"created_at":"2016-02-02T17:40:00Z","updated_at":"2016-02-02T17:40:00Z","body":"@sempervictus I've not given up yet :)  The problem seems to be that the zvol_request->dmu_write_uio->dmu_write_uio_dnode path is slower than the zfs_write_dmu_write_uio_dbuf->dmu_write_uio_dnode path (even though they both ultimately use ```dmu_write_uio_dnode()```).  In my particular test case, the slowdown can be summarized as follows:  a call to ```zfs_write()``` to write a 4K block takes about 0.214 ms. whereas a call to ```zvol_request()``` to write a 4K block takes about 0.312 ms.  My test involves 20 streams each writing 256MB 4K at at time to sequential parts of the zvol.  This is using the same tiotest parameters as the original poster.  The only difference is that I've hacked tiotest to run in O_DIRECT mode for zvols.\r\n\r\nI'm following all the various call paths of ```zvol_request()``` one at a time.  My initial suspicion was that the ```dmu_tx_count_write()``` path was the bottleneck but I've stubbed it out and it didn't help much.  I suspect I'll be able to track down the cause of the increased latency but it's going to take time because there are a lot of call paths and I'm looking for a 0.1 ms. difference."}}