{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3937","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3937/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3937/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3937/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3937","id":112132290,"number":3937,"title":"PANIC: metaslab_free_dva(): bad DVA with zfs 0.6.5.2","user":{"login":"xhernandez","id":1781528,"avatar_url":"https://avatars.githubusercontent.com/u/1781528?v=3","gravatar_id":"","url":"https://api.github.com/users/xhernandez","html_url":"https://github.com/xhernandez","followers_url":"https://api.github.com/users/xhernandez/followers","following_url":"https://api.github.com/users/xhernandez/following{/other_user}","gists_url":"https://api.github.com/users/xhernandez/gists{/gist_id}","starred_url":"https://api.github.com/users/xhernandez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xhernandez/subscriptions","organizations_url":"https://api.github.com/users/xhernandez/orgs","repos_url":"https://api.github.com/users/xhernandez/repos","events_url":"https://api.github.com/users/xhernandez/events{/privacy}","received_events_url":"https://api.github.com/users/xhernandez/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Bug%20-%20Major","name":"Bug - Major","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/12","html_url":"https://github.com/zfsonlinux/zfs/milestones/0.8.0","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/12/labels","id":7014,"number":12,"title":"0.8.0","description":"ZFS on Linux v0.8.0 - ZFS+SPL Consolidation\r\n","creator":{"login":"behlendorf","id":148917,"avatar_url":"https://avatars.githubusercontent.com/u/148917?v=3","gravatar_id":"","url":"https://api.github.com/users/behlendorf","html_url":"https://github.com/behlendorf","followers_url":"https://api.github.com/users/behlendorf/followers","following_url":"https://api.github.com/users/behlendorf/following{/other_user}","gists_url":"https://api.github.com/users/behlendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/behlendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behlendorf/subscriptions","organizations_url":"https://api.github.com/users/behlendorf/orgs","repos_url":"https://api.github.com/users/behlendorf/repos","events_url":"https://api.github.com/users/behlendorf/events{/privacy}","received_events_url":"https://api.github.com/users/behlendorf/received_events","type":"User","site_admin":false},"open_issues":58,"closed_issues":14,"state":"open","created_at":"2011-05-03T17:39:03Z","updated_at":"2016-05-12T21:45:48Z","due_on":null,"closed_at":null},"comments":34,"created_at":"2015-10-19T12:16:43Z","updated_at":"2016-05-30T12:10:42Z","closed_at":null,"body":"I've found what seems an in-memory corruption of zfs.\r\n\r\nI have created a pool using raidz1 with 6 disks and two child datasets. I've configured the following properties:\r\n\r\nxattr=sa\r\nacltype=posixacl\r\ncompression=lz4\r\n\r\nAnd I've set a quota on each child dataset. I use each dataset as a brick for gluster.\r\n\r\nWhen gluster rebuilds a brick (self-heal operation), it copies data inside zfs that contains extended attributes and acl's. While it was doing this, zfs detected a problem (see below).\r\n\r\nAfter restarting the server and accessing the volume, no problem has been detected, so I assume that on-disk data is healthy. I'm currently trying to reproduce the problem on another test server because this one is in production.\r\n\r\nThe bad DVA is 201326592:14468632831362131968:0. I only have two vdevs (0 and 1), so 201326592 is clearly wrong (in hex it's 0xC000000, not sure if it means anything). The DVA's offset is more interesting: in hex it's 0xC8CAE8E6EAE4E800. Divided by two is 0x6465747375727400 and \"trusted\" in ascii. Many of the extended attributes that gluster uses start by \"trusted\" so I'm guessing that some extended attribute manipulation has corrupted some memory block.\r\n\r\n```\r\nkernel: [5292444.782956] PANIC: metaslab_free_dva(): bad DVA 201326592:14468632831362131968:0\r\nkernel: [5292444.783282] Showing stack for process 61687\r\nkernel: [5292444.783286] CPU: 6 PID: 61687 Comm: z_wr_int_4 Tainted: P           O--------------   3.10.0-5-pve #1\r\nkernel: [5292444.783288] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.03.0003.041920141333 04/19/2014\r\nkernel: [5292444.783290]  0000000000000000 ffff8808e85b3728 ffffffff816125e0 ffff8808e85b3738\r\nkernel: [5292444.783296]  ffffffffa062e784 ffff8808e85b3868 ffffffffa062e81e ffff8808e85b37b8\r\nkernel: [5292444.783299]  62616c736174656d 76645f656572665f 646162203a292861 3130322041564420\r\nkernel: [5292444.783302] Call Trace:\r\nkernel: [5292444.783313]  [<ffffffff816125e0>] dump_stack+0x19/0x1b\r\nkernel: [5292444.783323]  [<ffffffffa062e784>] spl_dumpstack+0x44/0x50 [spl]\r\nkernel: [5292444.783328]  [<ffffffffa062e81e>] vcmn_err+0x8e/0x130 [spl]\r\nkernel: [5292444.783355]  [<ffffffffa0262454>] ? isci_task_execute_task+0x204/0x320 [isci]\r\nkernel: [5292444.783360]  [<ffffffff8118ed02>] ? kmem_cache_alloc+0x1b2/0x1e0\r\nkernel: [5292444.783363]  [<ffffffff816161ed>] ? mutex_lock+0x1d/0x41\r\nkernel: [5292444.783367]  [<ffffffffa062ae19>] ? spl_kmem_cache_alloc+0x69/0x150 [spl]\r\nkernel: [5292444.783410]  [<ffffffffa0b00342>] zfs_panic_recover+0x52/0x60 [zfs]\r\nkernel: [5292444.783414]  [<ffffffffa062ae19>] ? spl_kmem_cache_alloc+0x69/0x150 [spl]\r\nkernel: [5292444.783435]  [<ffffffffa0ae3ff8>] metaslab_free_dva+0x1e8/0x3b0 [zfs]\r\nkernel: [5292444.783456]  [<ffffffffa0ae6fdc>] metaslab_free+0x9c/0xe0 [zfs]\r\nkernel: [5292444.783482]  [<ffffffffa0b4a1bc>] zio_dva_free+0x1c/0x30 [zfs]\r\nkernel: [5292444.783504]  [<ffffffffa0b4e012>] zio_wait+0xd2/0x210 [zfs]\r\nkernel: [5292444.783524]  [<ffffffffa0b4e21b>] zio_free+0xcb/0x120 [zfs]\r\nkernel: [5292444.783544]  [<ffffffffa0addb21>] dsl_free+0x11/0x20 [zfs]\r\nkernel: [5292444.783562]  [<ffffffffa0ac7e88>] dsl_dataset_block_kill+0x278/0x4c0 [zfs]\r\nkernel: [5292444.783576]  [<ffffffffa0aa74ca>] dbuf_write_done+0x19a/0x240 [zfs]\r\nkernel: [5292444.783588]  [<ffffffffa0a9e5fe>] arc_write_done+0x25e/0x3f0 [zfs]\r\nkernel: [5292444.783609]  [<ffffffffa0b4ff59>] zio_done.part.11+0x259/0xed0 [zfs]\r\nkernel: [5292444.783613]  [<ffffffffa06298ca>] ? spl_kmem_free+0x2a/0x40 [spl]\r\nkernel: [5292444.783616]  [<ffffffff8118dfbd>] ? kfree+0xfd/0x130\r\nkernel: [5292444.783618]  [<ffffffff816161ed>] ? mutex_lock+0x1d/0x41\r\nkernel: [5292444.783638]  [<ffffffffa0b50c4a>] zio_done+0x7a/0x80 [zfs]\r\nkernel: [5292444.783658]  [<ffffffffa0b506fc>] zio_done.part.11+0x9fc/0xed0 [zfs]\r\nkernel: [5292444.783677]  [<ffffffffa0b50c4a>] zio_done+0x7a/0x80 [zfs]\r\nkernel: [5292444.783696]  [<ffffffffa0b506fc>] zio_done.part.11+0x9fc/0xed0 [zfs]\r\nkernel: [5292444.783716]  [<ffffffffa0ad7830>] ? dsl_pool_undirty_space+0xd0/0xe0 [zfs]\r\nkernel: [5292444.783735]  [<ffffffffa0b50c4a>] zio_done+0x7a/0x80 [zfs]\r\nkernel: [5292444.783755]  [<ffffffffa0b506fc>] zio_done.part.11+0x9fc/0xed0 [zfs]\r\nkernel: [5292444.783774]  [<ffffffffa0b50c4a>] zio_done+0x7a/0x80 [zfs]\r\nkernel: [5292444.783793]  [<ffffffffa0b4ad68>] zio_execute+0xc8/0x180 [zfs]\r\nkernel: [5292444.783798]  [<ffffffffa062caee>] taskq_thread+0x1fe/0x3f0 [spl]\r\nkernel: [5292444.783803]  [<ffffffff81094450>] ? try_to_wake_up+0x2a0/0x2a0\r\nkernel: [5292444.783807]  [<ffffffffa062c8f0>] ? taskq_thread_spawn+0x70/0x70 [spl]\r\nkernel: [5292444.783812]  [<ffffffff81083080>] kthread+0xc0/0xd0\r\nkernel: [5292444.783815]  [<ffffffff81082fc0>] ? flush_kthread_worker+0x80/0x80\r\nkernel: [5292444.783820]  [<ffffffff8162262c>] ret_from_fork+0x7c/0xb0\r\nkernel: [5292444.783822]  [<ffffffff81082fc0>] ? flush_kthread_worker+0x80/0x80\r\n```"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/222479081","html_url":"https://github.com/zfsonlinux/zfs/issues/3937#issuecomment-222479081","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3937","id":222479081,"user":{"login":"javenwu","id":2182079,"avatar_url":"https://avatars.githubusercontent.com/u/2182079?v=3","gravatar_id":"","url":"https://api.github.com/users/javenwu","html_url":"https://github.com/javenwu","followers_url":"https://api.github.com/users/javenwu/followers","following_url":"https://api.github.com/users/javenwu/following{/other_user}","gists_url":"https://api.github.com/users/javenwu/gists{/gist_id}","starred_url":"https://api.github.com/users/javenwu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javenwu/subscriptions","organizations_url":"https://api.github.com/users/javenwu/orgs","repos_url":"https://api.github.com/users/javenwu/repos","events_url":"https://api.github.com/users/javenwu/events{/privacy}","received_events_url":"https://api.github.com/users/javenwu/received_events","type":"User","site_admin":false},"created_at":"2016-05-30T12:10:42Z","updated_at":"2016-05-30T12:10:42Z","body":"@xhernandez I did some experiments and tests. I think I have some findings today.\r\n1. The  size calculation for bonus/spill has no problem. \r\n2. sa_build_layout has no problem when panic happens.\r\n3. There is no bonus buffer copy back to dn_phys between spill block pointer being assigned in dbuf_check_blkptr  and panic. I added some log there, and the results proves the dn_spill is not be overwrote after assignment.\r\n4. I think I found a problem in the code which might be relevant to the panic and I am trying a quick fix. The fixing is in testing. So far, the testing runs well. \r\n\r\nMy fixing against 0.6.5.6 is as below:\r\n\r\n--- dbuf.c\t2016-03-22 18:08:04.000000000 -0700\r\n+++ dbuf.c.new\t2016-05-30 04:45:45.777248524 -0700\r\n@@ -2499,14 +2499,15 @@\r\n \t/* ASSERT(dmu_tx_is_syncing(tx) */\r\n \tASSERT(MUTEX_HELD(&db->db_mtx));\r\n \r\n-\tif (db->db_blkptr != NULL)\r\n-\t\treturn;\r\n-\r\n \tif (db->db_blkid == DMU_SPILL_BLKID) {\r\n \t\tdb->db_blkptr = &dn->dn_phys->dn_spill;\r\n \t\tBP_ZERO(db->db_blkptr);\r\n \t\treturn;\r\n \t}\r\n+\r\n+\tif (db->db_blkptr != NULL)\r\n+\t\treturn;\r\n+\r\n \tif (db->db_level == dn->dn_phys->dn_nlevels-1) {\r\n \t\t/*\r\n \t\t * This buffer was allocated at a time when there was\r\n\r\nLet me explain a little bit on the fixing.\r\nThe fixing is actually switch the two clauses of dbuf_check_blkptr().\r\n\r\nThe dbuf  can be used for the same spill block of a dnode, so when the dbuf_sync_leaf()->dbf_check_blkptr(), the db->db_blkptr can be not NULL, so it will return directly without calling BP_ZERO, and the content could be the invalid if the bonus buffer just copy back to the dn_phys before. The bonus content (xattr) overwrote the spill blockptr. So without my fixing, the dbuf just returns with an invalid db_blkptr. Usually, if the dbuf enters ZIO pipeline, once zio_dva_allocated() is invoked, the blkptr can be set to correct value. Unfortunately, there is possibilities zio_pipeline take the ZIO_INTERLOCK_PIPELINE and skip the zio_dva_allocate with the \"old/invalid\" blkptr. The panic could happen when following reference of the invalid blkptr.\r\n\r\nAs long as we clean the blkptr for spill block in dbuf_check_blkptr(), the problem can be avoided.  So I switched the two clauses.\r\n\r\nI am not confident about the fixing so far, and not sure if the fixing can bring any negative impact. \r\nI will do more testings and let you folks know result. If you would like to try the fixing, please let me know the testing result.\r\n\r\n\r\n"}}
{"actor":{"display_login":"javenwu"}}