{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2831","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2831/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2831/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2831/events","html_url":"https://github.com/zfsonlinux/zfs/issues/2831","id":46826962,"number":2831,"title":"cannot import pool: I/O error","user":{"login":"simonbuehler","id":78061,"avatar_url":"https://avatars.githubusercontent.com/u/78061?v=3","gravatar_id":"","url":"https://api.github.com/users/simonbuehler","html_url":"https://github.com/simonbuehler","followers_url":"https://api.github.com/users/simonbuehler/followers","following_url":"https://api.github.com/users/simonbuehler/following{/other_user}","gists_url":"https://api.github.com/users/simonbuehler/gists{/gist_id}","starred_url":"https://api.github.com/users/simonbuehler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonbuehler/subscriptions","organizations_url":"https://api.github.com/users/simonbuehler/orgs","repos_url":"https://api.github.com/users/simonbuehler/repos","events_url":"https://api.github.com/users/simonbuehler/events{/privacy}","received_events_url":"https://api.github.com/users/simonbuehler/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Bug%20-%20Major","name":"Bug - Major","color":"e11d21"},{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Difficulty%20-%20Medium","name":"Difficulty - Medium","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":66,"created_at":"2014-10-25T23:39:55Z","updated_at":"2016-01-07T11:10:22Z","closed_at":null,"body":"\r\nhi,\r\n\r\nafter this crash \r\n![2014-10-25 19 55 47](https://cloud.githubusercontent.com/assets/78061/4781481/669d5b3a-5ca1-11e4-8b48-750725f77310.jpg)\r\nmy pool is gone, can't reimport although the devices are there (dmcrypt decryption works)\r\n\r\nwhat can i do to get my data back, import -fF failed ?\r\ndestroy and recreate wtf? this was supposed to be the backup server"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/169632550","html_url":"https://github.com/zfsonlinux/zfs/issues/2831#issuecomment-169632550","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2831","id":169632550,"user":{"login":"sempervictus","id":1331084,"avatar_url":"https://avatars.githubusercontent.com/u/1331084?v=3","gravatar_id":"","url":"https://api.github.com/users/sempervictus","html_url":"https://github.com/sempervictus","followers_url":"https://api.github.com/users/sempervictus/followers","following_url":"https://api.github.com/users/sempervictus/following{/other_user}","gists_url":"https://api.github.com/users/sempervictus/gists{/gist_id}","starred_url":"https://api.github.com/users/sempervictus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sempervictus/subscriptions","organizations_url":"https://api.github.com/users/sempervictus/orgs","repos_url":"https://api.github.com/users/sempervictus/repos","events_url":"https://api.github.com/users/sempervictus/events{/privacy}","received_events_url":"https://api.github.com/users/sempervictus/received_events","type":"User","site_admin":false},"created_at":"2016-01-07T11:10:22Z","updated_at":"2016-01-07T11:10:22Z","body":"This is kind of a hateful problem...\r\nWas running heavy synchronous IO workloads over the last couple of weeks and suddenly managed to kill a pool this way. Z2 pool consisting of 5 disks didn't come back on reboot, one drive is completely missing from dmesg (likely disk-controller death) and one comes up but stalls the system with IO errors. 3 disks remain, and they cant seem to agree on what they want to do.\r\nCurrent build is based on #4133 with some of the newer PRs cherry-picked in. Older builds fail to load the pool too, results in a hard lock of the system.\r\nI'll try building out with the spacemap ignorance patch and see where that goes after testing the zvol-related stack in the build (since i'm seeing zvol_create_minor in the mix).\r\nStack traces look like this:\r\n```\r\n[ 1718.382574] VERIFY3(0 == dmu_bonus_hold(os, object, dl, &dl->dl_dbuf)) failed (0 == 52)\r\n[ 1718.383047] PANIC at dsl_deadlist.c:106:dsl_deadlist_open()\r\n[ 1718.383355] Showing stack for process 13208\r\n[ 1718.383359] CPU: 7 PID: 13208 Comm: zpool Tainted: P           OE   4.3.0-ssv-i7 #ssv\r\n[ 1718.383361] Hardware name: SGI S5000PSL/S5000PSL, BIOS S5000.86B.15.00.0101.110920101604 11/09/2010\r\n[ 1718.383364]  000000000000006a ffff8807f596f6d0 ffffffff8239632d ffffffffc09b15d9\r\n[ 1718.383368]  ffff8807f596f6e0 ffffffffc0691f94 ffff8807f596f868 ffffffffc069205f\r\n[ 1718.383371]  ffff8807f596f7a0 ffffffff00000030 ffff8807f596f878 ffff8807f596f818\r\n[ 1718.383374] Call Trace:\r\n[ 1718.383386]  [<ffffffff8239632d>] dump_stack+0x44/0x57\r\n[ 1718.383402]  [<ffffffffc0691f94>] spl_dumpstack+0x44/0x50 [spl]\r\n[ 1718.383408]  [<ffffffffc069205f>] spl_panic+0xbf/0xf0 [spl]\r\n[ 1718.383466]  [<ffffffffc08e8de6>] ? dbuf_rele+0x36/0x40 [zfs]\r\n[ 1718.383497]  [<ffffffffc0902488>] ? dnode_hold_impl+0x308/0x4c0 [zfs]\r\n[ 1718.383546]  [<ffffffffc0983951>] ? zio_destroy+0xc1/0xd0 [zfs]\r\n[ 1718.383586]  [<ffffffffc098757d>] ? zio_wait+0x10d/0x150 [zfs]\r\n[ 1718.383616]  [<ffffffffc090dc61>] dsl_deadlist_open+0x101/0x180 [zfs]\r\n[ 1718.383622]  [<ffffffffc068d0db>] ? spl_kmem_zalloc+0xbb/0x160 [spl]\r\n[ 1718.383651]  [<ffffffffc0906f2d>] dsl_dataset_hold_obj+0x1fd/0x890 [zfs]\r\n[ 1718.383677]  [<ffffffffc08f5b84>] dmu_objset_find_impl+0x224/0x3e0 [zfs]\r\n[ 1718.383718]  [<ffffffffc09923c0>] ? zvol_create_minor+0x70/0x70 [zfs]\r\n[ 1718.383723]  [<ffffffff821c7430>] ? kfree+0xf0/0x120\r\n[ 1718.383749]  [<ffffffffc08f5b17>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]\r\n[ 1718.383789]  [<ffffffffc09923c0>] ? zvol_create_minor+0x70/0x70 [zfs]\r\n[ 1718.383815]  [<ffffffffc08f5b17>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]\r\n[ 1718.383855]  [<ffffffffc09923c0>] ? zvol_create_minor+0x70/0x70 [zfs]\r\n[ 1718.383881]  [<ffffffffc08f5b17>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]\r\n[ 1718.383920]  [<ffffffffc09923c0>] ? zvol_create_minor+0x70/0x70 [zfs]\r\n[ 1718.383946]  [<ffffffffc08f5b17>] dmu_objset_find_impl+0x1b7/0x3e0 [zfs]\r\n[ 1718.383986]  [<ffffffffc09923c0>] ? zvol_create_minor+0x70/0x70 [zfs]\r\n[ 1718.384026]  [<ffffffffc09923c0>] ? zvol_create_minor+0x70/0x70 [zfs]\r\n[ 1718.384069]  [<ffffffffc08f5d89>] dmu_objset_find+0x49/0x70 [zfs]\r\n[ 1718.384113]  [<ffffffffc0992569>] zvol_create_minors+0x29/0x30 [zfs]\r\n[ 1718.384153]  [<ffffffffc0935444>] spa_import+0x464/0x730 [zfs]\r\n[ 1718.384195]  [<ffffffffc0963efc>] zfs_ioc_pool_import+0xdc/0x110 [zfs]\r\n[ 1718.384237]  [<ffffffffc0966718>] zfsdev_ioctl+0x458/0x490 [zfs]\r\n[ 1718.384243]  [<ffffffff82203104>] ? mntput+0x24/0x40\r\n[ 1718.384250]  [<ffffffff821f6c8d>] do_vfs_ioctl+0x2cd/0x4b0\r\n[ 1718.384257]  [<ffffffff821f2c23>] ? putname+0x53/0x60\r\n[ 1718.384264]  [<ffffffff821f6ee9>] SyS_ioctl+0x79/0x90\r\n[ 1718.384273]  [<ffffffff827a4f36>] entry_SYSCALL_64_fastpath+0x16/0x75\r\n```\r\nOf course this had to happen on the pool which works as the lowest basin of the backup waterfall... simply adorable. Several T of data in there, and although there shouldn't be any single instance bits not present elsewhere, re-aggregating all of this will be a PITA.\r\nThis is the second time space maps have bitten us, first one was when we figured out that scrubs cant validate free space maps, now this. SM implementation may be a bit flawed... \r\nAnyone know a good exorcist, or have thoughts on how we would address this past whats in this thread?"}}