{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4267","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4267/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4267/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4267/events","html_url":"https://github.com/zfsonlinux/zfs/issues/4267","id":128416560,"number":4267,"title":"PANIC at metaslab.c:2677:metaslab_fastwrite_unmark()","user":{"login":"janlam7","id":5459235,"avatar_url":"https://avatars.githubusercontent.com/u/5459235?v=3","gravatar_id":"","url":"https://api.github.com/users/janlam7","html_url":"https://github.com/janlam7","followers_url":"https://api.github.com/users/janlam7/followers","following_url":"https://api.github.com/users/janlam7/following{/other_user}","gists_url":"https://api.github.com/users/janlam7/gists{/gist_id}","starred_url":"https://api.github.com/users/janlam7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/janlam7/subscriptions","organizations_url":"https://api.github.com/users/janlam7/orgs","repos_url":"https://api.github.com/users/janlam7/repos","events_url":"https://api.github.com/users/janlam7/events{/privacy}","received_events_url":"https://api.github.com/users/janlam7/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/11","html_url":"https://github.com/zfsonlinux/zfs/milestones/0.7.0","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/11/labels","id":7013,"number":11,"title":"0.7.0","description":"ZFS on Linux v0.7.0 - Memory management\r\n\r\n","creator":{"login":"behlendorf","id":148917,"avatar_url":"https://avatars.githubusercontent.com/u/148917?v=3","gravatar_id":"","url":"https://api.github.com/users/behlendorf","html_url":"https://github.com/behlendorf","followers_url":"https://api.github.com/users/behlendorf/followers","following_url":"https://api.github.com/users/behlendorf/following{/other_user}","gists_url":"https://api.github.com/users/behlendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/behlendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behlendorf/subscriptions","organizations_url":"https://api.github.com/users/behlendorf/orgs","repos_url":"https://api.github.com/users/behlendorf/repos","events_url":"https://api.github.com/users/behlendorf/events{/privacy}","received_events_url":"https://api.github.com/users/behlendorf/received_events","type":"User","site_admin":false},"open_issues":65,"closed_issues":204,"state":"open","created_at":"2011-05-03T17:38:46Z","updated_at":"2016-05-20T18:11:13Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2016-01-24T18:47:10Z","updated_at":"2016-05-22T23:04:13Z","closed_at":null,"body":"While trying to remove a log device from my pool, assisted by @ilovezfs eventually the following stacktrace occured and the system hung:\r\n\r\n[ 3904.485505] VERIFY3(vd->vdev_pending_fastwrite >= psize) failed (0 >= 4096)\r\n[ 3904.485511] PANIC at metaslab.c:2677:metaslab_fastwrite_unmark()\r\n[ 3904.485514] Showing stack for process 3508\r\n[ 3904.485523] CPU: 0 PID: 3508 Comm: txg_sync Tainted: P           O    4.1.12-gentoo #11\r\n[ 3904.485526] Hardware name: System manufacturer System Product Name/M5A78L-M/USB3, BIOS 2001    09/11/2014\r\n[ 3904.485530]  0000000000000a75 ffff88003564b858 ffffffff814aed8e 0000000000000007\r\n[ 3904.485537]  ffffffffa03823c9 ffff88003564b868 ffffffffa0189664 ffff88003564b9e8\r\n[ 3904.485543]  ffffffffa018971c ffff88003564b8a8 ffffffff00000030 ffff88003564b9f8\r\n[ 3904.485549] Call Trace:\r\n[ 3904.485566]  [<ffffffff814aed8e>] dump_stack+0x45/0x57\r\n[ 3904.485615]  [<ffffffffa0189664>] spl_dumpstack+0x44/0x50 [spl]\r\n[ 3904.485635]  [<ffffffffa018971c>] spl_panic+0xac/0xf0 [spl]\r\n[ 3904.485642]  [<ffffffff814b2476>] ? mutex_lock+0x16/0x40\r\n[ 3904.485646]  [<ffffffff814b2476>] ? mutex_lock+0x16/0x40\r\n[ 3904.485744]  [<ffffffffa02b015b>] ? refcount_add_many+0xab/0x150 [zfs]\r\n[ 3904.485819]  [<ffffffffa02b0216>] ? refcount_add+0x16/0x20 [zfs]\r\n[ 3904.485929]  [<ffffffffa02ca47e>] ? spa_config_enter+0x11e/0x230 [zfs]\r\n[ 3904.486010]  [<ffffffffa02ca344>] ? spa_config_held+0x94/0xb0 [zfs]\r\n[ 3904.486088]  [<ffffffffa02ad859>] metaslab_fastwrite_unmark+0x109/0x1e0 [zfs]\r\n[ 3904.486182]  [<ffffffffa0328fad>] zil_sync+0x4dd/0x640 [zfs]\r\n[ 3904.486267]  [<ffffffffa032ce66>] ? zio_taskq_dispatch+0xa6/0x150 [zfs]\r\n[ 3904.486349]  [<ffffffffa032cf22>] ? zio_issue_async+0x12/0x20 [zfs]\r\n[ 3904.486430]  [<ffffffffa0333aeb>] ? zio_nowait+0xdb/0x290 [zfs]\r\n[ 3904.486480]  [<ffffffffa026baa0>] dmu_objset_sync+0x300/0x490 [zfs]\r\n[ 3904.486527]  [<ffffffffa0269570>] ? primary_cache_changed_cb+0x60/0x60 [zfs]\r\n[ 3904.486575]  [<ffffffffa026bc30>] ? dmu_objset_sync+0x490/0x490 [zfs]\r\n[ 3904.486631]  [<ffffffffa028e6fb>] dsl_dataset_sync+0x7b/0x170 [zfs]\r\n[ 3904.486687]  [<ffffffffa029965c>] dsl_pool_sync+0xec/0x660 [zfs]\r\n[ 3904.486755]  [<ffffffffa02bd6b0>] spa_sync+0x360/0xe10 [zfs]\r\n[ 3904.486760]  [<ffffffff8107fe15>] ? __wake_up_common+0x55/0x90\r\n[ 3904.486832]  [<ffffffffa02d34f2>] txg_sync_thread+0x3c2/0x6c0 [zfs]\r\n[ 3904.486839]  [<ffffffff8107ac8e>] ? pick_next_task_fair+0xfe/0x4d0\r\n[ 3904.486911]  [<ffffffffa02d3130>] ? txg_thread_wait+0xf0/0xf0 [zfs]\r\n[ 3904.486928]  [<ffffffffa01852ca>] thread_generic_wrapper+0x7a/0xc0 [spl]\r\n[ 3904.486945]  [<ffffffffa0185250>] ? __thread_exit+0x20/0x20 [spl]\r\n[ 3904.486950]  [<ffffffff81063ba9>] kthread+0xc9/0xe0\r\n[ 3904.486955]  [<ffffffff81063ae0>] ? kthread_create_on_node+0x180/0x180\r\n[ 3904.486959]  [<ffffffff814b4552>] ret_from_fork+0x42/0x70\r\n[ 3904.486963]  [<ffffffff81063ae0>] ? kthread_create_on_node+0x180/0x180\r\nmediacenter ~ # dmesg | grep 3904\r\n[ 3904.485505] VERIFY3(vd->vdev_pending_fastwrite >= psize) failed (0 >= 4096)\r\n[ 3904.485511] PANIC at metaslab.c:2677:metaslab_fastwrite_unmark()\r\n[ 3904.485514] Showing stack for process 3508\r\n[ 3904.485523] CPU: 0 PID: 3508 Comm: txg_sync Tainted: P           O    4.1.12-gentoo #11\r\n[ 3904.485526] Hardware name: System manufacturer System Product Name/M5A78L-M/USB3, BIOS 2001    09/11/2014\r\n[ 3904.485530]  0000000000000a75 ffff88003564b858 ffffffff814aed8e 0000000000000007\r\n[ 3904.485537]  ffffffffa03823c9 ffff88003564b868 ffffffffa0189664 ffff88003564b9e8\r\n[ 3904.485543]  ffffffffa018971c ffff88003564b8a8 ffffffff00000030 ffff88003564b9f8\r\n[ 3904.485549] Call Trace:\r\n[ 3904.485566]  [<ffffffff814aed8e>] dump_stack+0x45/0x57\r\n[ 3904.485615]  [<ffffffffa0189664>] spl_dumpstack+0x44/0x50 [spl]\r\n[ 3904.485635]  [<ffffffffa018971c>] spl_panic+0xac/0xf0 [spl]\r\n[ 3904.485642]  [<ffffffff814b2476>] ? mutex_lock+0x16/0x40\r\n[ 3904.485646]  [<ffffffff814b2476>] ? mutex_lock+0x16/0x40\r\n[ 3904.485744]  [<ffffffffa02b015b>] ? refcount_add_many+0xab/0x150 [zfs]\r\n[ 3904.485819]  [<ffffffffa02b0216>] ? refcount_add+0x16/0x20 [zfs]\r\n[ 3904.485929]  [<ffffffffa02ca47e>] ? spa_config_enter+0x11e/0x230 [zfs]\r\n[ 3904.486010]  [<ffffffffa02ca344>] ? spa_config_held+0x94/0xb0 [zfs]\r\n[ 3904.486088]  [<ffffffffa02ad859>] metaslab_fastwrite_unmark+0x109/0x1e0 [zfs]\r\n[ 3904.486182]  [<ffffffffa0328fad>] zil_sync+0x4dd/0x640 [zfs]\r\n[ 3904.486267]  [<ffffffffa032ce66>] ? zio_taskq_dispatch+0xa6/0x150 [zfs]\r\n[ 3904.486349]  [<ffffffffa032cf22>] ? zio_issue_async+0x12/0x20 [zfs]\r\n[ 3904.486430]  [<ffffffffa0333aeb>] ? zio_nowait+0xdb/0x290 [zfs]\r\n[ 3904.486480]  [<ffffffffa026baa0>] dmu_objset_sync+0x300/0x490 [zfs]\r\n[ 3904.486527]  [<ffffffffa0269570>] ? primary_cache_changed_cb+0x60/0x60 [zfs]\r\n[ 3904.486575]  [<ffffffffa026bc30>] ? dmu_objset_sync+0x490/0x490 [zfs]\r\n[ 3904.486631]  [<ffffffffa028e6fb>] dsl_dataset_sync+0x7b/0x170 [zfs]\r\n[ 3904.486687]  [<ffffffffa029965c>] dsl_pool_sync+0xec/0x660 [zfs]\r\n[ 3904.486755]  [<ffffffffa02bd6b0>] spa_sync+0x360/0xe10 [zfs]\r\n[ 3904.486760]  [<ffffffff8107fe15>] ? __wake_up_common+0x55/0x90\r\n[ 3904.486832]  [<ffffffffa02d34f2>] txg_sync_thread+0x3c2/0x6c0 [zfs]\r\n[ 3904.486839]  [<ffffffff8107ac8e>] ? pick_next_task_fair+0xfe/0x4d0\r\n[ 3904.486911]  [<ffffffffa02d3130>] ? txg_thread_wait+0xf0/0xf0 [zfs]\r\n[ 3904.486928]  [<ffffffffa01852ca>] thread_generic_wrapper+0x7a/0xc0 [spl]\r\n[ 3904.486945]  [<ffffffffa0185250>] ? __thread_exit+0x20/0x20 [spl]\r\n[ 3904.486950]  [<ffffffff81063ba9>] kthread+0xc9/0xe0\r\n[ 3904.486955]  [<ffffffff81063ae0>] ? kthread_create_on_node+0x180/0x180\r\n[ 3904.486959]  [<ffffffff814b4552>] ret_from_fork+0x42/0x70\r\n[ 3904.486963]  [<ffffffff81063ae0>] ? kthread_create_on_node+0x180/0x180\r\n\r\nI was running gentoo zfs-0.6.5.3-r3 with a patch for issue 4112 and the recursive-lock-failure.patch at the time with a debug kernel.\r\n\r\nmediacenter / # zpool status\r\n  pool: pool4\r\n state: ONLINE\r\n  scan: resilvered 144K in 0h37m with 0 errors on Sun Jan 24 17:04:07 2016\r\nconfig:\r\n\r\n\tNAME        STATE     READ WRITE CKSUM\r\n\tpool4       ONLINE       0     0     0\r\n\t  raidz1-0  ONLINE       0     0     0\r\n\t    sdb3    ONLINE       0     0     0\r\n\t    sdc3    ONLINE       0     0     0\r\n\t    sdd2    ONLINE       0     0     0\r\n\tlogs\r\n\t  sda3      ONLINE       0     0     0\r\n\tcache\r\n\t  sda5      ONLINE       0     0     0\r\n\r\nerrors: No known data errors\r\n\r\nmediacenter / # zpool remove pool4 sda3\r\ncannot remove sda3: pool already exists\r\n\r\nIn the process of trying to remove the log device the existing log was extended to be a mirror. After detaching /dev/sda3 the above stacktrace occured.\r\n\r\nI'll try to remove the log device using FreeBSD."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/220862129","html_url":"https://github.com/zfsonlinux/zfs/issues/4267#issuecomment-220862129","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4267","id":220862129,"user":{"login":"dechamps","id":866859,"avatar_url":"https://avatars.githubusercontent.com/u/866859?v=3","gravatar_id":"","url":"https://api.github.com/users/dechamps","html_url":"https://github.com/dechamps","followers_url":"https://api.github.com/users/dechamps/followers","following_url":"https://api.github.com/users/dechamps/following{/other_user}","gists_url":"https://api.github.com/users/dechamps/gists{/gist_id}","starred_url":"https://api.github.com/users/dechamps/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dechamps/subscriptions","organizations_url":"https://api.github.com/users/dechamps/orgs","repos_url":"https://api.github.com/users/dechamps/repos","events_url":"https://api.github.com/users/dechamps/events{/privacy}","received_events_url":"https://api.github.com/users/dechamps/received_events","type":"User","site_admin":false},"created_at":"2016-05-22T23:04:12Z","updated_at":"2016-05-22T23:04:12Z","body":"Even when using the specific parameters in your failure (`-m 0 -r 0 -R 2 -v 5 -a 9 -T 62 -P 24 -s 128m`), I still can't reproduce it after a few hours.\r\n\r\nAnyway, the error would seem to indicate a bug with fastwrite bookkeeping where a call to `metaslab_fastwrite_unmark()` is not matched to a call to `metaslab_fastwrite_mark()`. The spurious unmark call occurs in `zil_sync()`, which is interesting - it either means that there is a logic bug where it's unmarking writes that have never actually been marked, or there is a race between that function and `zio_done()` which results in double unmark.\r\n\r\nI'll try to investigate some more in the next few days, but I wouldn't get my hopes up. Troubleshooting subtle, hard-to-reproduce bugs/races in low-level complex code that I have not even looked at in years (literally), on my free time and with very limited resources for experimentation, that's not exactly a recipe for success."}}
{"actor":{"display_login":"dechamps"}}