{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4912","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4912/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4912/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4912/events","html_url":"https://github.com/zfsonlinux/zfs/issues/4912","id":168603034,"number":4912,"title":"PANIC at arc.c:2493:arc_evict_state_impl() VERIFY failed","user":{"login":"JuliaVixen","id":19146207,"avatar_url":"https://avatars.githubusercontent.com/u/19146207?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaVixen","html_url":"https://github.com/JuliaVixen","followers_url":"https://api.github.com/users/JuliaVixen/followers","following_url":"https://api.github.com/users/JuliaVixen/following{/other_user}","gists_url":"https://api.github.com/users/JuliaVixen/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaVixen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaVixen/subscriptions","organizations_url":"https://api.github.com/users/JuliaVixen/orgs","repos_url":"https://api.github.com/users/JuliaVixen/repos","events_url":"https://api.github.com/users/JuliaVixen/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaVixen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-01T09:24:17Z","updated_at":"2016-08-01T10:52:24Z","closed_at":null,"body":"Well, I did this:\r\n```\r\nzpool create -O atime=off -O compression=lz4 -O exec=off -O devices=off \\\r\n -O recordsize=1M -O setuid=off -O checksum=sha256 \\\r\n -o ashift=12 -o feature@lz4_compress=enabled -o feature@embedded_data=enabled \\\r\nswappy /dev/disk/by-id/ata-ST4000DM000-1F2168_W300KAQ5\r\n```\r\nAnd then I did this:\r\n```\r\nlocalhost ~ # zfs snapshot l/Lightroom@2016_Jul_30\r\nlocalhost ~ # zfs snapshot l/Jul_29_2016@2016_Jul_30\r\nlocalhost ~ # zfs send -eLv -R l/Lightroom@2016_Jul_30  | zfs recv -evsF swappy\r\nfull send of l/Lightroom@2016_Jul_30 estimated size is 357G\r\ntotal estimated size is 357G\r\nTIME        SENT   SNAPSHOT\r\nreceiving full stream of l/Lightroom@2016_Jul_30 into swappy/Lightroom@2016_Jul_30\r\n09:22:10    771K   l/Lightroom@2016_Jul_30\r\n[...]\r\n09:23:05   1.89G   l/Lightroom@2016_Jul_30\r\n09:23:06   1.93G   l/Lightroom@2016_Jul_30\r\n```\r\nAnd then while that `zfs send|recv` was going, I did this in another terminal:\r\n```\r\nlocalhost ~ # zfs create swappy/recovered\r\nlocalhost ~ # zfs set copies=2 swappy/recovered\r\nlocalhost ~ # cp -avi  \"/mnt/temp/STUFF.ISO\" /swappy/re <tab>\r\n```\r\n\r\n... And the tab completion never completed, and everything doing any I/O on the 'swappy' pool blocked. The other pool, \"l\", was not blocking, and I could access it just fine.\r\n\r\n```\r\n[261664.320743] VERIFY(!(((*({ __attribute__((unused)) typeof((hash_lock)->m_owner) __var = ( typeof((hash_lock)->m_owner)) 0; (volatile typeof((hash_lock)->m_owner) *)&((hash_lock)->m_owner); }))) == get_current())) failed\r\n[261664.320906] PANIC at arc.c:2493:arc_evict_state_impl()\r\n[261664.321001] Showing stack for process 7428\r\n[261664.321003] CPU: 2 PID: 7428 Comm: send_traverse Tainted: P           O    4.4.6-gentoo-debug #1\r\n[261664.321004] Hardware name: Supermicro X10SLL-F/X10SLL-SF/X10SLL-F/X10SLL-SF, BIOS 1.0a 06/11/2013\r\n[261664.321005]  0000000000000000 ffff8804a2bceec8 ffffffff8130fa9d ffffffffa1325817\r\n[261664.321007]  ffffffffa13257ff ffff8804a2bceed8 ffffffffa0a7c2d6 ffff8804a2bcf068\r\n[261664.321009]  ffffffffa0a7c48e ffff8804a2bceef8 ffffffff00000028 ffff8804a2bcf078\r\n[261664.321010] Call Trace:\r\n[261664.321015]  [<ffffffff8130fa9d>] dump_stack+0x4d/0x63\r\n[261664.321021]  [<ffffffffa0a7c2d6>] spl_dumpstack+0x3d/0x3f [spl]\r\n[261664.321023]  [<ffffffffa0a7c48e>] spl_panic+0xa9/0xdc [spl]\r\n[261664.321039]  [<ffffffffa1225420>] arc_evict_state+0x2df/0x9c4 [zfs]\r\n[261664.321048]  [<ffffffffa1225b89>] arc_adjust_impl.constprop.25+0x2a/0x31 [zfs]\r\n[261664.321056]  [<ffffffffa1225ed3>] arc_adjust+0x343/0x4b8 [zfs]\r\n[261664.321064]  [<ffffffffa1227f96>] arc_shrink+0x10d/0x10f [zfs]\r\n[261664.321072]  [<ffffffffa1228042>] __arc_shrinker_func.isra.24+0xaa/0x118 [zfs]\r\n[261664.321079]  [<ffffffffa12280bc>] arc_shrinker_func_scan_objects+0xc/0x1d [zfs]\r\n[261664.321082]  [<ffffffff810c5962>] shrink_slab.part.56.constprop.61+0x16b/0x1fc\r\n[261664.321083]  [<ffffffff810c787e>] shrink_zone+0x6e/0x144\r\n[261664.321085]  [<ffffffff810c8173>] do_try_to_free_pages+0x1b3/0x2bc\r\n[261664.321086]  [<ffffffff810c8380>] try_to_free_pages+0x89/0x90\r\n[261664.321088]  [<ffffffff810c000f>] __alloc_pages_nodemask+0x57c/0x8d4\r\n[261664.321090]  [<ffffffff81059377>] ? ttwu_do_wakeup+0x12/0x7f\r\n[261664.321092]  [<ffffffff8105f7ea>] ? set_next_entity+0x26/0x52f\r\n[261664.321094]  [<ffffffff810e9b58>] cache_alloc_refill+0x26d/0x4a3\r\n[261664.321096]  [<ffffffff810e9de6>] kmem_cache_alloc+0x58/0xae\r\n[261664.321098]  [<ffffffffa0a793e9>] spl_kmem_cache_alloc+0x6f/0x65a [spl]\r\n[261664.321101]  [<ffffffff814d0c84>] ? mutex_lock+0xf/0x21\r\n[261664.321103]  [<ffffffffa0a7def0>] ? cv_wait_common+0xcc/0x108 [spl]\r\n[261664.321111]  [<ffffffffa1221cb4>] ? arc_space_consume+0x8f/0x8f [zfs]\r\n[261664.321123]  [<ffffffffa12e9575>] zio_buf_alloc+0x53/0x57 [zfs]\r\n[261664.321130]  [<ffffffffa122211b>] arc_get_data_buf+0x355/0x47c [zfs]\r\n[261664.321137]  [<ffffffffa1220c7f>] ? add_reference+0xef/0x1ed [zfs]\r\n[261664.321145]  [<ffffffffa1227b42>] ? arc_read+0xcc6/0xcc6 [zfs]\r\n[261664.321152]  [<ffffffffa122762a>] arc_read+0x7ae/0xcc6 [zfs]\r\n[261664.321154]  [<ffffffff814d0a10>] ? mutex_unlock+0x9/0xb\r\n[261664.321164]  [<ffffffffa124b6ad>] traverse_visitbp+0x4ab/0x783 [zfs]\r\n[261664.321174]  [<ffffffffa124b600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[261664.321184]  [<ffffffffa124b600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[261664.321194]  [<ffffffffa124b600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[261664.321204]  [<ffffffffa124b600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[261664.321213]  [<ffffffffa124b600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[261664.321222]  [<ffffffffa124b600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[261664.321232]  [<ffffffffa124be90>] traverse_dnode+0xb3/0x161 [zfs]\r\n[261664.321241]  [<ffffffffa124b843>] traverse_visitbp+0x641/0x783 [zfs]\r\n[261664.321243]  [<ffffffffa0a7a27f>] ? taskq_dispatch+0x16b/0x17d [spl]\r\n[261664.321252]  [<ffffffffa124bcd4>] traverse_impl+0x34f/0x458 [zfs]\r\n[261664.321254]  [<ffffffff8106176d>] ? enqueue_task_fair+0xb83/0xbe3\r\n[261664.321264]  [<ffffffffa124729b>] ? dump_bytes_cb+0x13e/0x13e [zfs]\r\n[261664.321273]  [<ffffffffa124c15b>] traverse_dataset_resume+0x4d/0x4f [zfs]\r\n[261664.321282]  [<ffffffffa1247403>] ? dmu_recv_existing_end+0x6f/0x6f [zfs]\r\n[261664.321291]  [<ffffffffa12472d3>] send_traverse_thread+0x38/0x75 [zfs]\r\n[261664.321294]  [<ffffffffa0a79cf0>] thread_generic_wrapper+0x6c/0x79 [spl]\r\n[261664.321296]  [<ffffffffa0a79c84>] ? __thread_create+0x112/0x112 [spl]\r\n[261664.321298]  [<ffffffff8105533a>] kthread+0xcd/0xd5\r\n[261664.321299]  [<ffffffff8105526d>] ? kthread_freezable_should_stop+0x43/0x43\r\n[261664.321300]  [<ffffffff814d265f>] ret_from_fork+0x3f/0x70\r\n[261664.321302]  [<ffffffff8105526d>] ? kthread_freezable_should_stop+0x43/0x43\r\n\r\n```\r\n\r\nThis is the GIT version from around July 4, 2016. I have ECC memory in this computer, and no disk errors have been reported."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/236549740","html_url":"https://github.com/zfsonlinux/zfs/issues/4912#issuecomment-236549740","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4912","id":236549740,"user":{"login":"JuliaVixen","id":19146207,"avatar_url":"https://avatars.githubusercontent.com/u/19146207?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaVixen","html_url":"https://github.com/JuliaVixen","followers_url":"https://api.github.com/users/JuliaVixen/followers","following_url":"https://api.github.com/users/JuliaVixen/following{/other_user}","gists_url":"https://api.github.com/users/JuliaVixen/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaVixen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaVixen/subscriptions","organizations_url":"https://api.github.com/users/JuliaVixen/orgs","repos_url":"https://api.github.com/users/JuliaVixen/repos","events_url":"https://api.github.com/users/JuliaVixen/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaVixen/received_events","type":"User","site_admin":false},"created_at":"2016-08-01T10:52:24Z","updated_at":"2016-08-01T10:52:24Z","body":"I rebooted, and then did this:\r\n```\r\nlocalhost ~ # zfs send -eLv -t 1-c23ec650b-d8-789c636064000310a500c4ec50360710e72765a526973030ec1581a8c1904f4b2b4e2d618003903c1b927c5265496a3190be11b7a50a9bfe92fcf4d2cc14060683dd4cde26fff31b0c90e439c1f27989b9a90c0c39fa3e99e9192545f9f9b90e46068666f15ea539f1c6060c12507360ee4fcd4d4a4dc9cf06f3015cdc1e64  | zfs recv -evsF swappy\r\nresume token contents:\r\nnvlist version: 0\r\n\tobject = 0x14bd\r\n\toffset = 0x0\r\n\tbytes = 0x7ab45ed8\r\n\ttoguid = 0x806fff344b02bb30\r\n\ttoname = l/Lightroom@2016_Jul_30\r\n\tembedok = 1\r\nfull send of l/Lightroom@2016_Jul_30 estimated size is 355G\r\nTIME        SENT   SNAPSHOT\r\nreceiving full stream of l/Lightroom@2016_Jul_30 into swappy/Lightroom@2016_Jul_30\r\n10:07:14    157M   l/Lightroom@2016_Jul_30\r\n10:07:15    376M   l/Lightroom@2016_Jul_30\r\n10:07:16    615M   l/Lightroom@2016_Jul_30\r\n10:07:17    847M   l/Lightroom@2016_Jul_30\r\n10:07:18   1.04G   l/Lightroom@2016_Jul_30\r\n10:07:19   1.25G   l/Lightroom@2016_Jul_30\r\n[...]\r\n```\r\nAnd everything seemed to be working fine... until...\r\n```\r\n[...]\r\n10:54:28    258G   l/Lightroom@2016_Jul_30\r\n10:54:29    258G   l/Lightroom@2016_Jul_30\r\n10:54:30    258G   l/Lightroom@2016_Jul_30\r\n10:54:31    258G   l/Lightroom@2016_Jul_30\r\n10:54:32    258G   l/Lightroom@2016_Jul_30\r\n```\r\n... And then this happened, and all I/O blocked again.\r\n```\r\n[ 4356.035411] VERIFY(!(((*({ __attribute__((unused)) typeof((hash_lock)->m_owner) __var = ( typeof((hash_lock)->m_owner)) 0; (volatile typeof((hash_lock)->m_owner) *)&((hash_lock)->m_owner); }))) == get_current())) failed\r\n[ 4356.035588] PANIC at arc.c:2493:arc_evict_state_impl()\r\n[ 4356.035687] Showing stack for process 11485\r\n[ 4356.035689] CPU: 1 PID: 11485 Comm: send_traverse Tainted: P           O    4.4.6-gentoo-debug #1\r\n[ 4356.035690] Hardware name: Supermicro X10SLL-F/X10SLL-SF/X10SLL-F/X10SLL-SF, BIOS 1.0a 06/11/2013\r\n[ 4356.035691]  0000000000000000 ffff8804ecc9ae28 ffffffff8130fa9d ffffffffa0fc2817\r\n[ 4356.035693]  ffffffffa0fc27ff ffff8804ecc9ae38 ffffffffa0a692d6 ffff8804ecc9afc8\r\n[ 4356.035694]  ffffffffa0a6948e ffff88082c1fe800 ffff880800000028 ffff8804ecc9afd8\r\n[ 4356.035695] Call Trace:\r\n[ 4356.035699]  [<ffffffff8130fa9d>] dump_stack+0x4d/0x63\r\n[ 4356.035705]  [<ffffffffa0a692d6>] spl_dumpstack+0x3d/0x3f [spl]\r\n[ 4356.035708]  [<ffffffffa0a6948e>] spl_panic+0xa9/0xdc [spl]\r\n[ 4356.035723]  [<ffffffffa0ec2420>] arc_evict_state+0x2df/0x9c4 [zfs]\r\n[ 4356.035726]  [<ffffffff814d0a10>] ? mutex_unlock+0x9/0xb\r\n[ 4356.035733]  [<ffffffffa0ec2b89>] arc_adjust_impl.constprop.25+0x2a/0x31 [zfs]\r\n[ 4356.035741]  [<ffffffffa0ec2eea>] arc_adjust+0x35a/0x4b8 [zfs]\r\n[ 4356.035748]  [<ffffffffa0ec4f96>] arc_shrink+0x10d/0x10f [zfs]\r\n[ 4356.035755]  [<ffffffffa0ec5042>] __arc_shrinker_func.isra.24+0xaa/0x118 [zfs]\r\n[ 4356.035762]  [<ffffffffa0ec50bc>] arc_shrinker_func_scan_objects+0xc/0x1d [zfs]\r\n[ 4356.035765]  [<ffffffff810c5962>] shrink_slab.part.56.constprop.61+0x16b/0x1fc\r\n[ 4356.035766]  [<ffffffff810c787e>] shrink_zone+0x6e/0x144\r\n[ 4356.035767]  [<ffffffff810c8173>] do_try_to_free_pages+0x1b3/0x2bc\r\n[ 4356.035768]  [<ffffffff810c8380>] try_to_free_pages+0x89/0x90\r\n[ 4356.035770]  [<ffffffff810c000f>] __alloc_pages_nodemask+0x57c/0x8d4\r\n[ 4356.035783]  [<ffffffffa0f46db8>] ? vdev_raidz_io_start+0x1f6/0x33f [zfs]\r\n[ 4356.035785]  [<ffffffff810e9b58>] cache_alloc_refill+0x26d/0x4a3\r\n[ 4356.035787]  [<ffffffff810e9de6>] kmem_cache_alloc+0x58/0xae\r\n[ 4356.035789]  [<ffffffffa0a663e9>] spl_kmem_cache_alloc+0x6f/0x65a [spl]\r\n[ 4356.035802]  [<ffffffffa0f43ea8>] ? vdev_mirror_io_start+0xda/0x1ac [zfs]\r\n[ 4356.035813]  [<ffffffffa0f435f4>] ? vdev_config_sync+0x1b5/0x1b5 [zfs]\r\n[ 4356.035820]  [<ffffffffa0ebecfb>] ? buf_cons+0x47/0x4d [zfs]\r\n[ 4356.035827]  [<ffffffffa0ebecb4>] ? arc_space_consume+0x8f/0x8f [zfs]\r\n[ 4356.035839]  [<ffffffffa0f86575>] zio_buf_alloc+0x53/0x57 [zfs]\r\n[ 4356.035846]  [<ffffffffa0ebf11b>] arc_get_data_buf+0x355/0x47c [zfs]\r\n[ 4356.035853]  [<ffffffffa0ec462a>] arc_read+0x7ae/0xcc6 [zfs]\r\n[ 4356.035860]  [<ffffffffa0ec4b42>] ? arc_read+0xcc6/0xcc6 [zfs]\r\n[ 4356.035871]  [<ffffffffa0ee7e13>] traverse_prefetch_metadata+0xa5/0xa7 [zfs]\r\n[ 4356.035882]  [<ffffffffa0ee7e6a>] prefetch_dnode_metadata+0x55/0xa7 [zfs]\r\n[ 4356.035891]  [<ffffffffa0ee8700>] traverse_visitbp+0x4fe/0x783 [zfs]\r\n[ 4356.035901]  [<ffffffffa0ee8600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[ 4356.035910]  [<ffffffffa0ee8600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[ 4356.035912]  [<ffffffff814d0a10>] ? mutex_unlock+0x9/0xb\r\n[ 4356.035921]  [<ffffffffa0ee8600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[ 4356.035923]  [<ffffffff814d0a10>] ? mutex_unlock+0x9/0xb\r\n[ 4356.035932]  [<ffffffffa0ee8600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[ 4356.035933]  [<ffffffff814d0a10>] ? mutex_unlock+0x9/0xb\r\n[ 4356.035943]  [<ffffffffa0ee8600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[ 4356.035944]  [<ffffffff814d0a10>] ? mutex_unlock+0x9/0xb\r\n[ 4356.035953]  [<ffffffffa0ee8600>] traverse_visitbp+0x3fe/0x783 [zfs]\r\n[ 4356.035960]  [<ffffffffa0ec4b2d>] ? arc_read+0xcb1/0xcc6 [zfs]\r\n[ 4356.035970]  [<ffffffffa0ee8e90>] traverse_dnode+0xb3/0x161 [zfs]\r\n[ 4356.035979]  [<ffffffffa0ee8843>] traverse_visitbp+0x641/0x783 [zfs]\r\n[ 4356.035981]  [<ffffffffa0a6727f>] ? taskq_dispatch+0x16b/0x17d [spl]\r\n[ 4356.035990]  [<ffffffffa0ee8cd4>] traverse_impl+0x34f/0x458 [zfs]\r\n[ 4356.035992]  [<ffffffff8106176d>] ? enqueue_task_fair+0xb83/0xbe3\r\n[ 4356.036001]  [<ffffffffa0ee429b>] ? dump_bytes_cb+0x13e/0x13e [zfs]\r\n[ 4356.036010]  [<ffffffffa0ee915b>] traverse_dataset_resume+0x4d/0x4f [zfs]\r\n[ 4356.036020]  [<ffffffffa0ee4403>] ? dmu_recv_existing_end+0x6f/0x6f [zfs]\r\n[ 4356.036029]  [<ffffffffa0ee42d3>] send_traverse_thread+0x38/0x75 [zfs]\r\n[ 4356.036031]  [<ffffffffa0a66cf0>] thread_generic_wrapper+0x6c/0x79 [spl]\r\n[ 4356.036033]  [<ffffffffa0a66c84>] ? __thread_create+0x112/0x112 [spl]\r\n[ 4356.036035]  [<ffffffff8105533a>] kthread+0xcd/0xd5\r\n[ 4356.036036]  [<ffffffff8105526d>] ? kthread_freezable_should_stop+0x43/0x43\r\n[ 4356.036038]  [<ffffffff814d265f>] ret_from_fork+0x3f/0x70\r\n[ 4356.036039]  [<ffffffff8105526d>] ? kthread_freezable_should_stop+0x43/0x43\r\n```\r\n\r\nI've been doing this kind of stuff for a month, and there have been no crashes until now. (Although I was using raidz pools, and not a single drive like this time. I'm trying to move some data between two large pools, which I don't have enough SATA ports to plug everything in altogether at the same time.)"}}
{"actor":{"display_login":"JuliaVixen"}}