{"action":"closed","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5441","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5441/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5441/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5441/events","html_url":"https://github.com/zfsonlinux/zfs/issues/5441","id":192843621,"number":5441,"title":"zfs 0.6.5.8 hangs during large rsync / high I/O","user":{"login":"danyel2014","id":7780216,"avatar_url":"https://avatars.githubusercontent.com/u/7780216?v=3","gravatar_id":"","url":"https://api.github.com/users/danyel2014","html_url":"https://github.com/danyel2014","followers_url":"https://api.github.com/users/danyel2014/followers","following_url":"https://api.github.com/users/danyel2014/following{/other_user}","gists_url":"https://api.github.com/users/danyel2014/gists{/gist_id}","starred_url":"https://api.github.com/users/danyel2014/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danyel2014/subscriptions","organizations_url":"https://api.github.com/users/danyel2014/orgs","repos_url":"https://api.github.com/users/danyel2014/repos","events_url":"https://api.github.com/users/danyel2014/events{/privacy}","received_events_url":"https://api.github.com/users/danyel2014/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-01T13:14:59Z","updated_at":"2016-12-11T16:14:51Z","closed_at":"2016-12-11T16:14:51Z","body":"while copying around 5 TB of data between two zpools by using multiple rsync sessions, I've manually started scrub for one of the zpools. Later on I've noticed that any zfs/zpool commands are stalling and my logs are spammed with:\r\n\r\nDec  1 11:06:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:06:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:06:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:06:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:06:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff897eff034780\r\nDec  1 11:06:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff887b35ab07a8 0000000000000001\r\nDec  1 11:06:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:06:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:10:01 srv_cs1 systemd: Started Session 89 of user root.\r\nDec  1 11:10:01 srv_cs1 systemd: Starting Session 89 of user root.\r\nDec  1 11:10:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:10:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:10:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:10:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:10:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff897eff274780\r\nDec  1 11:10:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff896381b29058 0000000000000001\r\nDec  1 11:10:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:10:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:14:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:14:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:14:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:14:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:14:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff88beffa14780\r\nDec  1 11:14:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff8979cad96b90 0000000000000001\r\nDec  1 11:14:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:14:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:18:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:18:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:18:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:18:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:18:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff88beffa54780\r\nDec  1 11:18:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff8972a73a8c00 0000000000000001\r\nDec  1 11:18:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:18:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:20:01 srv_cs1 systemd: Started Session 90 of user root.\r\nDec  1 11:20:01 srv_cs1 systemd: Starting Session 90 of user root.\r\nDec  1 11:22:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:22:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:22:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:22:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:22:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff88beff894780\r\nDec  1 11:22:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff88c886ca55d8 0000000000000001\r\nDec  1 11:22:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:22:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:26:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:26:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:26:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:26:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:26:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff88beffab4780\r\nDec  1 11:26:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff8889aba5c478 0000000000000001\r\nDec  1 11:26:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:26:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:30:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:30:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:30:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:30:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:30:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff88beff9f4780\r\nDec  1 11:30:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff88c8fde6c8d0 0000000000000001\r\nDec  1 11:30:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810a6b0b>] ? autoremove_wake_function+0x2b/0x40\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:30:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:34:50 srv_cs1 kernel: INFO: task txg_sync:17368 blocked for more than 120 seconds.\r\nDec  1 11:34:50 srv_cs1 kernel: \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\nDec  1 11:34:50 srv_cs1 kernel: txg_sync        D 0000000000000001     0 17368      2 0x00000080\r\nDec  1 11:34:50 srv_cs1 kernel: ffff885e43b6bac0 0000000000000046 ffff885e41c03980 ffff885e43b6bfd8\r\nDec  1 11:34:50 srv_cs1 kernel: ffff885e43b6bfd8 ffff885e43b6bfd8 ffff885e41c03980 ffff885eff954780\r\nDec  1 11:34:50 srv_cs1 kernel: 0000000000000000 7fffffffffffffff ffff88b322117440 0000000000000001\r\nDec  1 11:34:50 srv_cs1 kernel: Call Trace:\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff8163a909>] schedule+0x29/0x70\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810b8c22>] ? default_wake_function+0x12/0x20\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810af028>] ? __wake_up_common+0x58/0x90\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff81639f3e>] io_schedule_timeout+0xae/0x130\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810a6896>] ? prepare_to_wait_exclusive+0x56/0x90\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff81639fd8>] io_schedule+0x18/0x20\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa051d627>] cv_wait_common+0xa7/0x130 [spl]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa051d708>] __cv_wait_io+0x18/0x20 [spl]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa64f15ab>] zio_wait+0x10b/0x1f0 [zfs]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa647ad2f>] dsl_pool_sync+0xbf/0x440 [zfs]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa64958d8>] spa_sync+0x388/0xb50 [zfs]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa64a7555>] txg_sync_thread+0x3c5/0x620 [zfs]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa64a7190>] ? txg_init+0x280/0x280 [zfs]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa0518861>] thread_generic_wrapper+0x71/0x80 [spl]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffffa05187f0>] ? __thread_exit+0x20/0x20 [spl]\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810a5aef>] kthread+0xcf/0xe0\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff81645858>] ret_from_fork+0x58/0x90\r\nDec  1 11:34:50 srv_cs1 kernel: [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140\r\n\r\nI am using RHEL 7.2 with kmod-zfs-0.6.5.8.\r\n\r\n[root@srv_cs1 ~]# rpm -qa | grep zfs\r\nzfs-release-1-3.el7.centos.noarch\r\nlibzfs2-0.6.5.8-1.el7.centos.x86_64\r\nzfs-0.6.5.8-1.el7.centos.x86_64\r\nkmod-zfs-0.6.5.8-1.el7.centos.x86_64\r\n[root@srv_cs1 ~]# rpm -qa | grep zfs\r\nzfs-release-1-3.el7.centos.noarch\r\nlibzfs2-0.6.5.8-1.el7.centos.x86_64\r\nzfs-0.6.5.8-1.el7.centos.x86_64\r\nkmod-zfs-0.6.5.8-1.el7.centos.x86_64\r\n\r\n$ uname -a\r\nLinux srv_cs1 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n"}}
{"actor":{"display_login":"kpande"}}