{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4880","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4880/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4880/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4880/events","html_url":"https://github.com/zfsonlinux/zfs/issues/4880","id":167625277,"number":4880,"title":"Reduced performance for 0.6.5 in virtualized environment","user":{"login":"koplover","id":2776880,"avatar_url":"https://avatars.githubusercontent.com/u/2776880?v=3","gravatar_id":"","url":"https://api.github.com/users/koplover","html_url":"https://github.com/koplover","followers_url":"https://api.github.com/users/koplover/followers","following_url":"https://api.github.com/users/koplover/following{/other_user}","gists_url":"https://api.github.com/users/koplover/gists{/gist_id}","starred_url":"https://api.github.com/users/koplover/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koplover/subscriptions","organizations_url":"https://api.github.com/users/koplover/orgs","repos_url":"https://api.github.com/users/koplover/repos","events_url":"https://api.github.com/users/koplover/events{/privacy}","received_events_url":"https://api.github.com/users/koplover/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Performance","name":"Performance","color":"DDDDDD"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2016-07-26T14:43:38Z","updated_at":"2016-10-06T05:40:12Z","closed_at":null,"body":"We have been using zfs for a couple of years in our xen virtualized environment. Initially we used 0.6.3 before moving to 0.6.4.2 running within our disk driver domain.\r\n\r\nWe have been experiencing various lockup issues in these releases, impacted by memory pressure, and impacting our guest workloads and end user experience, and have been trying to move to the 0.6.5 stream for the last 6 months to improve stability. Unfortunately, we experience performance issues when moving to 0.6.5\r\n\r\nWith two identical systems (hardware and software) except zfs version (0.6.4.2 and 0.6.5.7), I run a benchmark (Microsoft JetStress) on a Windows Xen guest to establish disk subsystem performance to understand performance for an Exchange DB. Running this benchmark against 0.6.4.2 (Achieved Transactional I/O per Second 464.083) shows a far better performance figure than the configuration using 0.6.5.7 (Achieved Transactional I/O per Second 226.203)\r\n\r\nIn these benchmarks an Exchange Database is created and manipulated in a separate volume of the Windows guest VM, formatted as 64K block NTFS, supported underlying by a 64k block zvol. From monitoring this underlying zvol using iostat, it seems something has changed in the implementation of the zvols which means that concurrency at the disk layer is impaired (the queue size in the 0.6.5.7 deployment for the actual measurement stage is just below 1 when viewed with iostat).\r\n\r\nLooking at the various stages of the benchmark and the impact on the zvol via iostat I see the following:\r\n\r\nDatabase creation phase (synchronous writes, no reads)\r\n0.6.5.7 - example sample\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nzd112             0.00     0.00    0.00  536.30     0.00 152144.00   567.38     0.70    1.31    0.00    1.31   1.31  70.27\r\n\r\n0.6.4.2 - example sample\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nzd4640            0.00     0.00    0.00  347.87     0.00 94684.40   544.37     0.24    0.68    0.00    0.68   0.48  16.53\r\n\r\nSo write performance on this benchmark initialisation stage significantly better for 0.6.5.7 compared to 0.6.4.2 deployment\r\n\r\nBenchmark measurement phase (mix read / write)\r\n0.6.5.7 - example sample\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nzd112             0.00     0.00  119.33  107.20  5596.00  3315.50    78.68     0.80    3.53    6.67    0.03   3.53  79.99\r\n\r\n0.6.4.2 - example sample\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nzd4640            0.00     0.00  243.40  204.57  9823.20  6188.03    71.48     1.39    3.10    5.56    0.18   1.61  72.28\r\n\r\nRead / write throughput significantly lower here, I suspect read difference leading the impact on write. Average queue size not huge for 0.6.4.2 but > 1\r\n\r\nChecksum checking phase (read only phase, checking db integrity at the end of the benchmark)\r\n0.6.5.7 - example sample\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nzd112             0.00     0.00  619.87    0.00 39671.47     0.00   128.00     0.98    1.57    1.57    0.00   1.58  98.08\r\n\r\n0.6.4.2 - example sample\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nzd4640            0.00     0.00  817.23    0.00 52302.93     0.00   128.00    27.61   34.17   34.17    0.00   1.22  99.83\r\n\r\nStats for 0.6.4.2 looks worse for 0.6.4.2, with the exception of read KB/s which is better. This is a sequentual read (mostly) 128 blocks at a time through the db to perform the verification. I assume the average queue size is allow for greater concurrency lower down the stack - and for some reason in 0.6.5.7 there is some form of serialization maintaining the queue size below 1 here?\r\n\r\nSome further information on the deployments:\r\n0.6.5.7\r\nVM running zfs:\r\n8GB, 4 vcpus\r\n\r\nzpool status:\r\n  pool: diskconvm\r\n state: ONLINE\r\n  scan: none requested\r\nconfig:\r\n\r\n\tNAME                                    STATE     READ WRITE CKSUM\r\n\tdiskconvm                               ONLINE       0     0     0\r\n\t  mirror-0                              ONLINE       0     0     0\r\n\t    ata-MB4000GCVBU_WMC130027610-part6  ONLINE       0     0     0\r\n\t    ata-MB4000GCVBU_WMC130027304-part6  ONLINE       0     0     0\r\n\tlogs\r\n\t  nvme0n1p1                             ONLINE       0     0     0\r\n\tcache\r\n\t  nvme0n1p2                             ONLINE       0     0     0\r\n\r\nerrors: No known data errors\r\n\r\nSLOG is 5GB\r\nL2-cache is 40GB\r\n\r\narcstats (note not when running, taking later)\r\n\r\n##cat /proc/spl/kstat/zfs/arcstats\r\nname                            type data\r\nhits                            4    800938102\r\nmisses                          4    95584272\r\ndemand_data_hits                4    680089930\r\ndemand_data_misses              4    24056278\r\ndemand_metadata_hits            4    98553094\r\ndemand_metadata_misses          4    31939474\r\nprefetch_data_hits              4    22183275\r\nprefetch_data_misses            4    39487999\r\nprefetch_metadata_hits          4    111803\r\nprefetch_metadata_misses        4    100521\r\nmru_hits                        4    291564434\r\nmru_ghost_hits                  4    7763753\r\nmfu_hits                        4    490582956\r\nmfu_ghost_hits                  4    4077487\r\ndeleted                         4    141253300\r\nmutex_miss                      4    8202\r\nevict_skip                      4    3399460\r\nevict_not_enough                4    20995\r\nevict_l2_cached                 4    938505652224\r\nevict_l2_eligible               4    598995230720\r\nevict_l2_ineligible             4    47057618944\r\nevict_l2_skip                   4    157936\r\nhash_elements                   4    3954384\r\nhash_elements_max               4    11437702\r\nhash_collisions                 4    510396727\r\nhash_chains                     4    926524\r\nhash_chain_max                  4    33\r\np                               4    3183601582\r\nc                               4    5115232256\r\nc_min                           4    2557616128\r\nc_max                           4    5115232256\r\nsize                            4    5076138536\r\nhdr_size                        4    716308416\r\ndata_size                       4    3611287552\r\nmetadata_size                   4    276021248\r\nother_size                      4    261026608\r\nanon_size                       4    3710976\r\nanon_evictable_data             4    0\r\nanon_evictable_metadata         4    0\r\nmru_size                        4    1790918656\r\nmru_evictable_data              4    1595752448\r\nmru_evictable_metadata          4    6385664\r\nmru_ghost_size                  4    3185651200\r\nmru_ghost_evictable_data        4    2179940352\r\nmru_ghost_evictable_metadata    4    1005710848\r\nmfu_size                        4    2092679168\r\nmfu_evictable_data              4    2015510528\r\nmfu_evictable_metadata          4    18325504\r\nmfu_ghost_size                  4    1926894080\r\nmfu_ghost_evictable_data        4    1571565568\r\nmfu_ghost_evictable_metadata    4    355328512\r\nl2_hits                         4    10246487\r\nl2_misses                       4    54510480\r\nl2_feeds                        4    1186238\r\nl2_rw_clash                     4    45\r\nl2_read_bytes                   4    131037714432\r\nl2_write_bytes                  4    628141021184\r\nl2_writes_sent                  4    834651\r\nl2_writes_done                  4    834651\r\nl2_writes_error                 4    0\r\nl2_writes_lock_retry            4    654\r\nl2_evict_lock_retry             4    244\r\nl2_evict_reading                4    39\r\nl2_evict_l1cached               4    1608607\r\nl2_free_on_write                4    829258\r\nl2_cdata_free_on_write          4    3708\r\nl2_abort_lowmem                 4    29\r\nl2_cksum_bad                    4    0\r\nl2_io_error                     4    0\r\nl2_size                         4    39083837952\r\nl2_asize                        4    35688606720\r\nl2_hdr_size                     4    211494712\r\nl2_compress_successes           4    26986228\r\nl2_compress_zeros               4    0\r\nl2_compress_failures            4    31411278\r\nmemory_throttle_count           4    0\r\nduplicate_buffers               4    0\r\nduplicate_buffers_size          4    0\r\nduplicate_reads                 4    3675\r\nmemory_direct_count             4    67864\r\nmemory_indirect_count           4    542404\r\narc_no_grow                     4    0\r\narc_tempreserve                 4    0\r\narc_loaned_bytes                4    0\r\narc_prune                       4    0\r\narc_meta_used                   4    1464850984\r\narc_meta_limit                  4    4603709030\r\narc_meta_max                    4    2473087208\r\narc_meta_min                    4    16777216\r\narc_need_free                   4    0\r\narc_sys_free                    4    130256896\r\n\r\nThe zfs parameters are as follows:\r\n## for i in *; do echo $i $(cat $i); done\r\nl2arc_feed_again 1\r\nl2arc_feed_min_ms 200\r\nl2arc_feed_secs 1\r\nl2arc_headroom 2\r\nl2arc_headroom_boost 200\r\nl2arc_nocompress 0\r\nl2arc_noprefetch 1\r\nl2arc_norw 0\r\nl2arc_write_boost 8388608\r\nl2arc_write_max 8388608\r\nmetaslab_aliquot 524288\r\nmetaslab_bias_enabled 1\r\nmetaslab_debug_load 0\r\nmetaslab_debug_unload 0\r\nmetaslab_fragmentation_factor_enabled 1\r\nmetaslab_lba_weighting_enabled 1\r\nmetaslab_preload_enabled 1\r\nmetaslabs_per_vdev 200\r\nspa_asize_inflation 24\r\nspa_config_path /etc/zfs/zpool.cache\r\nspa_load_verify_data 1\r\nspa_load_verify_maxinflight 10000\r\nspa_load_verify_metadata 1\r\nspa_slop_shift 5\r\nzfetch_array_rd_sz 1048576\r\nzfetch_block_cap 256\r\nzfetch_max_streams 8\r\nzfetch_min_sec_reap 2\r\nzfs_admin_snapshot 0\r\nzfs_arc_average_blocksize 8192\r\nzfs_arc_grow_retry 0\r\nzfs_arc_lotsfree_percent 0\r\nzfs_arc_max 5115232256\r\nzfs_arc_meta_adjust_restarts 4096\r\nzfs_arc_meta_limit 4603709030\r\nzfs_arc_meta_min 0\r\nzfs_arc_meta_prune 10000\r\nzfs_arc_meta_strategy 1\r\nzfs_arc_min 2557616128\r\nzfs_arc_min_prefetch_lifespan 0\r\nzfs_arc_num_sublists_per_state 4\r\nzfs_arc_p_aggressive_disable 1\r\nzfs_arc_p_dampener_disable 1\r\nzfs_arc_p_min_shift 0\r\nzfs_arc_shrink_shift 0\r\nzfs_arc_sys_free 0\r\nzfs_autoimport_disable 1\r\nzfs_dbgmsg_enable 1\r\nzfs_dbgmsg_maxsize 4194304\r\nzfs_dbuf_state_index 0\r\nzfs_deadman_enabled 1\r\nzfs_deadman_synctime_ms 1000000\r\nzfs_dedup_prefetch 0\r\nzfs_delay_min_dirty_percent 60\r\nzfs_delay_scale 500000\r\nzfs_dirty_data_max 833645772\r\nzfs_dirty_data_max_max 2084114432\r\nzfs_dirty_data_max_max_percent 25\r\nzfs_dirty_data_max_percent 10\r\nzfs_dirty_data_sync 67108864\r\nzfs_disable_dup_eviction 0\r\nzfs_expire_snapshot 300\r\nzfs_flags 0\r\nzfs_free_leak_on_eio 0\r\nzfs_free_max_blocks 100000\r\nzfs_free_min_time_ms 1000\r\nzfs_immediate_write_sz 32768\r\nzfs_max_recordsize 1048576\r\nzfs_mdcomp_disable 0\r\nzfs_metaslab_fragmentation_threshold 70\r\nzfs_mg_fragmentation_threshold 85\r\nzfs_mg_noalloc_threshold 0\r\nzfs_nocacheflush 0\r\nzfs_nopwrite_enabled 1\r\nzfs_no_scrub_io 0\r\nzfs_no_scrub_prefetch 0\r\nzfs_object_mutex_size 64\r\nzfs_pd_bytes_max 52428800\r\nzfs_prefetch_disable 0\r\nzfs_read_chunk_size 1048576\r\nzfs_read_history 0\r\nzfs_read_history_hits 0\r\nzfs_recover 0\r\nzfs_resilver_delay 2\r\nzfs_resilver_min_time_ms 3000\r\nzfs_scan_idle 50\r\nzfs_scan_min_time_ms 1000\r\nzfs_scrub_delay 4\r\nzfs_send_corrupt_data 0\r\nzfs_sync_pass_deferred_free 2\r\nzfs_sync_pass_dont_compress 5\r\nzfs_sync_pass_rewrite 2\r\nzfs_top_maxinflight 32\r\nzfs_txg_history 0\r\nzfs_txg_timeout 5\r\nzfs_vdev_aggregation_limit 131072\r\nzfs_vdev_async_read_max_active 3\r\nzfs_vdev_async_read_min_active 1\r\nzfs_vdev_async_write_active_max_dirty_percent 60\r\nzfs_vdev_async_write_active_min_dirty_percent 30\r\nzfs_vdev_async_write_max_active 10\r\nzfs_vdev_async_write_min_active 1\r\nzfs_vdev_cache_bshift 16\r\nzfs_vdev_cache_max 16384\r\nzfs_vdev_cache_size 0\r\nzfs_vdev_max_active 1000\r\nzfs_vdev_mirror_switch_us 10000\r\nzfs_vdev_read_gap_limit 32768\r\nzfs_vdev_scheduler noop\r\nzfs_vdev_scrub_max_active 2\r\nzfs_vdev_scrub_min_active 1\r\nzfs_vdev_sync_read_max_active 10\r\nzfs_vdev_sync_read_min_active 10\r\nzfs_vdev_sync_write_max_active 10\r\nzfs_vdev_sync_write_min_active 10\r\nzfs_vdev_write_gap_limit 4096\r\nzfs_zevent_cols 80\r\nzfs_zevent_console 0\r\nzfs_zevent_len_max 64\r\nzil_replay_disable 0\r\nzil_slog_limit 1048576\r\nzio_delay_max 30000\r\nzio_requeue_io_start_cut_in_line 1\r\nzio_taskq_batch_pct 75\r\nzvol_inhibit_dev 0\r\nzvol_major 230\r\nzvol_max_discard_blocks 16384\r\nzvol_prefetch_bytes 131072\r\n\r\n##cat /proc/spl/kstat/zfs/dmu_tx\r\n5 1 0x01 11 528 3239285416 1723566213675618\r\nname                            type data\r\ndmu_tx_assigned                 4    121905724\r\ndmu_tx_delay                    4    0\r\ndmu_tx_error                    4    0\r\ndmu_tx_suspended                4    0\r\ndmu_tx_group                    4    0\r\ndmu_tx_memory_reserve           4    0\r\ndmu_tx_memory_reclaim           4    0\r\ndmu_tx_dirty_throttle           4    0\r\ndmu_tx_dirty_delay              4    19266866\r\ndmu_tx_dirty_over_max           4    82\r\ndmu_tx_quota                    4    0\r\nNote: the dmy_tx_dirty_delay is increased during the initial database creation write phase as far as I could monitor, when performance was actually better than 0.6.4.2\r\n\r\n0.6.4.2\r\nVM running zfs:\r\n8GB, 4 vcpus\r\n\r\nzpool status:\r\n  pool: diskconvm\r\n state: ONLINE\r\n  scan: none requested\r\nconfig:\r\n\r\n\tNAME                                    STATE     READ WRITE CKSUM\r\n\tdiskconvm                               ONLINE       0     0     0\r\n\t  mirror-0                              ONLINE       0     0     0\r\n\t    ata-MB2000GDUNV_Y597Y0GTF1BA-part6  ONLINE       0     0     0\r\n\t    ata-MB2000GDUNV_Y597Y0GVF1BA-part6  ONLINE       0     0     0\r\n\tlogs\r\n\t  6f3a40b4-173b-4aa9-90b1-17db58231105  ONLINE       0     0     0\r\n\tcache\r\n\t  891a8189-6020-4045-92c4-8a090b34495c  ONLINE       0     0     0\r\n\r\nerrors: No known data errors\r\n\r\nSLOG is 5GB\r\nL2-cache is 40GB\r\n\r\nThe zfs parameters are as follows:\r\n## for i in *; do echo $i $(cat $i); done\r\nl2arc_feed_again 1\r\nl2arc_feed_min_ms 200\r\nl2arc_feed_secs 1\r\nl2arc_headroom 2\r\nl2arc_headroom_boost 200\r\nl2arc_nocompress 0\r\nl2arc_noprefetch 1\r\nl2arc_norw 0\r\nl2arc_write_boost 8388608\r\nl2arc_write_max 8388608\r\nmetaslab_bias_enabled 1\r\nmetaslab_debug_load 0\r\nmetaslab_debug_unload 0\r\nmetaslab_fragmentation_factor_enabled 1\r\nmetaslab_lba_weighting_enabled 1\r\nmetaslab_preload_enabled 1\r\nmetaslabs_per_vdev 200\r\nspa_asize_inflation 24\r\nspa_config_path /etc/zfs/zpool.cache\r\nspa_load_verify_data 1\r\nspa_load_verify_maxinflight 10000\r\nspa_load_verify_metadata 1\r\nzfetch_array_rd_sz 1048576\r\nzfetch_block_cap 256\r\nzfetch_max_streams 8\r\nzfetch_min_sec_reap 2\r\nzfs_arc_average_blocksize 8192\r\nzfs_arc_grow_retry 5\r\nzfs_arc_max 12568722432\r\nzfs_arc_memory_throttle_disable 1\r\nzfs_arc_meta_adjust_restarts 4096\r\nzfs_arc_meta_limit 11311850188\r\nzfs_arc_meta_prune 10000\r\nzfs_arc_min 6284361216\r\nzfs_arc_min_prefetch_lifespan 250\r\nzfs_arc_p_aggressive_disable 1\r\nzfs_arc_p_dampener_disable 1\r\nzfs_arc_shrink_shift 5\r\nzfs_autoimport_disable 1\r\nzfs_dbuf_state_index 0\r\nzfs_deadman_enabled 1\r\nzfs_deadman_synctime_ms 1000000\r\nzfs_dedup_prefetch 0\r\nzfs_delay_min_dirty_percent 60\r\nzfs_delay_scale 500000\r\nzfs_dirty_data_max 1675829657\r\nzfs_dirty_data_max_max 4189574144\r\nzfs_dirty_data_max_max_percent 25\r\nzfs_dirty_data_max_percent 10\r\nzfs_dirty_data_sync 67108864\r\nzfs_disable_dup_eviction 0\r\nzfs_expire_snapshot 300\r\nzfs_flags 0\r\nzfs_free_leak_on_eio 0\r\nzfs_free_max_blocks 100000\r\nzfs_free_min_time_ms 1000\r\nzfs_immediate_write_sz 32768\r\nzfs_mdcomp_disable 0\r\nzfs_metaslab_fragmentation_threshold 70\r\nzfs_mg_fragmentation_threshold 85\r\nzfs_mg_noalloc_threshold 0\r\nzfs_nocacheflush 0\r\nzfs_nopwrite_enabled 1\r\nzfs_no_scrub_io 0\r\nzfs_no_scrub_prefetch 0\r\nzfs_pd_bytes_max 52428800\r\nzfs_prefetch_disable 0\r\nzfs_read_chunk_size 1048576\r\nzfs_read_history 0\r\nzfs_read_history_hits 0\r\nzfs_recover 0\r\nzfs_resilver_delay 2\r\nzfs_resilver_min_time_ms 3000\r\nzfs_scan_idle 50\r\nzfs_scan_min_time_ms 1000\r\nzfs_scrub_delay 4\r\nzfs_send_corrupt_data 0\r\nzfs_sync_pass_deferred_free 2\r\nzfs_sync_pass_dont_compress 5\r\nzfs_sync_pass_rewrite 2\r\nzfs_top_maxinflight 32\r\nzfs_txg_history 0\r\nzfs_txg_timeout 5\r\nzfs_vdev_aggregation_limit 131072\r\nzfs_vdev_async_read_max_active 3\r\nzfs_vdev_async_read_min_active 1\r\nzfs_vdev_async_write_active_max_dirty_percent 60\r\nzfs_vdev_async_write_active_min_dirty_percent 30\r\nzfs_vdev_async_write_max_active 10\r\nzfs_vdev_async_write_min_active 1\r\nzfs_vdev_cache_bshift 16\r\nzfs_vdev_cache_max 16384\r\nzfs_vdev_cache_size 0\r\nzfs_vdev_max_active 1000\r\nzfs_vdev_mirror_switch_us 10000\r\nzfs_vdev_read_gap_limit 32768\r\nzfs_vdev_scheduler noop\r\nzfs_vdev_scrub_max_active 2\r\nzfs_vdev_scrub_min_active 1\r\nzfs_vdev_sync_read_max_active 10\r\nzfs_vdev_sync_read_min_active 10\r\nzfs_vdev_sync_write_max_active 10\r\nzfs_vdev_sync_write_min_active 10\r\nzfs_vdev_write_gap_limit 4096\r\nzfs_zevent_cols 80\r\nzfs_zevent_console 0\r\nzfs_zevent_len_max 64\r\nzil_replay_disable 0\r\nzil_slog_limit 1048576\r\nzio_delay_max 30000\r\nzio_requeue_io_start_cut_in_line 1\r\nzvol_inhibit_dev 0\r\nzvol_major 230\r\nzvol_max_discard_blocks 16384\r\nzvol_threads 32\r\n\r\nUnfortunately, the machine was redeployed so I lost this information, I'll rerun the benchmark and post again when the details below are available\r\n##cat /proc/spl/kstat/zfs/dmu_tx\r\narcstats \r\n##cat /proc/spl/kstat/zfs/arcstats\r\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/251871678","html_url":"https://github.com/zfsonlinux/zfs/issues/4880#issuecomment-251871678","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4880","id":251871678,"user":{"login":"koplover","id":2776880,"avatar_url":"https://avatars.githubusercontent.com/u/2776880?v=3","gravatar_id":"","url":"https://api.github.com/users/koplover","html_url":"https://github.com/koplover","followers_url":"https://api.github.com/users/koplover/followers","following_url":"https://api.github.com/users/koplover/following{/other_user}","gists_url":"https://api.github.com/users/koplover/gists{/gist_id}","starred_url":"https://api.github.com/users/koplover/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koplover/subscriptions","organizations_url":"https://api.github.com/users/koplover/orgs","repos_url":"https://api.github.com/users/koplover/repos","events_url":"https://api.github.com/users/koplover/events{/privacy}","received_events_url":"https://api.github.com/users/koplover/received_events","type":"User","site_admin":false},"created_at":"2016-10-06T05:40:12Z","updated_at":"2016-10-06T05:40:12Z","body":"@dweezil I've had a look at perf while the performance logging is going on, and pretty much 90% of the time is sat in the Xen scheduler.\r\n\r\nLocking does not seem to be an issue in this scenario, the greatest I saw was 0.15% in the highest spinlock routine.\r\n\r\nIt would be really great to backport the new zpool iostat options back to 0.6.4.2 to compare what was going on there, I'll take a look at the feasibility.\r\n\r\nAs I say, I've been trying to nail it by running fio tests, but have so far been unsuccessful with an array of sync / async and random / sequential read and write fio tests. I have a combined read / write job which shows an approximate match to the benchmark traffic when viewed through blktrace on a 0.6.4.2 system and this is showing some promise although I can't get the queue size anywhere near where it gets with the real benchmark (with fio in 0.6.4.2 I'm getting to 3-4 avg queue size, a long way short of what I see during the benchmark)"}}
{"actor":{"display_login":"koplover"}}