{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4809","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4809/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4809/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4809/events","html_url":"https://github.com/zfsonlinux/zfs/issues/4809","id":162342263,"number":4809,"title":"Silently corrupted file in snapshots after send/receive","user":{"login":"pwolny","id":3493884,"avatar_url":"https://avatars.githubusercontent.com/u/3493884?v=3","gravatar_id":"","url":"https://api.github.com/users/pwolny","html_url":"https://github.com/pwolny","followers_url":"https://api.github.com/users/pwolny/followers","following_url":"https://api.github.com/users/pwolny/following{/other_user}","gists_url":"https://api.github.com/users/pwolny/gists{/gist_id}","starred_url":"https://api.github.com/users/pwolny/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwolny/subscriptions","organizations_url":"https://api.github.com/users/pwolny/orgs","repos_url":"https://api.github.com/users/pwolny/repos","events_url":"https://api.github.com/users/pwolny/events{/privacy}","received_events_url":"https://api.github.com/users/pwolny/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Bug%20-%20Major","name":"Bug - Major","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/38","html_url":"https://github.com/zfsonlinux/zfs/milestones/0.6.5.8","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/38/labels","id":1765413,"number":38,"title":"0.6.5.8","description":"","creator":{"login":"behlendorf","id":148917,"avatar_url":"https://avatars.githubusercontent.com/u/148917?v=3","gravatar_id":"","url":"https://api.github.com/users/behlendorf","html_url":"https://github.com/behlendorf","followers_url":"https://api.github.com/users/behlendorf/followers","following_url":"https://api.github.com/users/behlendorf/following{/other_user}","gists_url":"https://api.github.com/users/behlendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/behlendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behlendorf/subscriptions","organizations_url":"https://api.github.com/users/behlendorf/orgs","repos_url":"https://api.github.com/users/behlendorf/repos","events_url":"https://api.github.com/users/behlendorf/events{/privacy}","received_events_url":"https://api.github.com/users/behlendorf/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":14,"state":"open","created_at":"2016-05-13T18:41:50Z","updated_at":"2016-08-18T19:50:02Z","due_on":null,"closed_at":null},"comments":96,"created_at":"2016-06-26T19:09:11Z","updated_at":"2016-08-21T08:53:39Z","closed_at":null,"body":"I have experienced a silently corrupted snapshot by send/receive – a checksum of a single file on source and target do not match on the filesystem and on the snapshots.\r\nRepeated scrubs of the tested pools do not show any errors. \r\nTrying to replicate the source filesystem results in a modification of single file on the target pool, in the received  filesystem and all snapshots.\r\nSource pool is a 6x1TB RAIDZ2 array on Debian 8.2.0 (Jessie, kernel 3.16.0-4-amd64, installed from DVDs, no additional updates) with version 0.6.5.3 of ZFS/SPL built from source (standard configure and no patches).\r\n\r\nMy source pool (“nas1FS’) was created on a non-ECC RAM machine and after filling up with data through a samba share (standard samba server, on the source fs sharesmb=off) was moved to a different computer with ECC RAM (I have used the same operating system on both machines). \r\nI can understand non-ECC RAM in the first computer causing permanent corruption of data that is visible on both source and target pool, but in this case the data is only silently changed during the transfer on a new computer with ECC RAM and source pool data seems to be fine. \r\nThis corruption of data during send/receive is repeatable.\r\n\r\n\r\nTo better explain what I have done:\r\nFirst I have created a 6x1TB raidz2 pool on my old computer (\"nas1FS\"). \r\nAfter filling  this pool with data I have moved the array  from old computer to a new one and I have tried to backup the data on “nas1FS” pool to a different pool (“backup_raidz“).\r\n\r\n“nas1FS” pool contained following snapshots that are of interest in this issue: \r\n```\r\n# zfs list -t snapshot |grep “nas1FS/backup@”\r\nnas1FS/backup@20151121_1                                  59.6G      -   726G  -\r\nnas1FS/backup@20151124                                     594M      -   758G  -\r\nnas1FS/backup@20151129                                     945M      -   758G  -\r\nnas1FS/backup@20151210                                     111M      -   765G  -\r\nnas1FS/backup@20160220                                     134M      -   772G  -\r\nnas1FS/backup@20160306                                     128M      -   779G  -\r\nnas1FS/backup@20160306_1                                  27.7M      -   780G  -\r\nnas1FS/backup@20160618                                    16.0K      -   866G  -\r\n```\r\n \r\nI have created a “backup_raidz” pool for backup (with compression turned on): \r\n`# zpool create -o ashift=12 -O compression=gzip-9 backup_raidz raidz1 /dev/disk/by-id/ata-SAMSUNG_HD103UJ_SerialNo1 /dev/disk/by-id/ata-SAMSUNG_HD103UJ_SerialNo2 /dev/disk/by-id/ata-HGST_HTS721010A9E630_SerialNo3 -f`\r\n\r\nAfterwards I have tried to replicate “nas1FS” pool.\r\n`# zfs send -R -vvvv \"nas1FS@20160618\" |zfs receive -vvvv -F \"backup_raidz/nas1FS_bak\"`\r\nthis command finished successfully without any error.\r\n\r\nI have executed following commands to get a list of file checksums on both source and target:\r\n```\r\n# md5deep -o f -e -r /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20160618/* > sums_1.txt\r\n# md5deep -o f -e -r /nas1FS/backup/.zfs/snapshot/20160618/* > sums_2.txt\r\n```\r\nand compared the resulting files\r\n\r\nI have found a checksum mismatch on a single file:\r\n```\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20160618/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20160618/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n\r\n```\r\nCorrect checksum is “3178c03d3205ac148372a71d75a835ec”, it was verified on the source used to populate the “nas1FS” filesystem.\r\n\r\nThis checksum mismatch was propagated through all snapshots in which the file was present on target pool:\r\n \r\n```\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20151124/samba_share/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20151129/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20151210/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20160220/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20160306/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20160306_1/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n 9917ceb0f20ca6f23a06857327bd9940  /backup_raidz/nas1FS_bak/backup/.zfs/snapshot/20160618/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n```\r\nSource pool has shown correct checksum on all snapshot that the offending file was accessible\r\n\r\n```\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20160618/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20160306_1/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20160306/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20160220/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20151210/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20151129/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20151124/samba_share/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n```\r\n\r\nTrying to access this file on a snapshot when it did not exist (“backup@20151121_1”) results in a “No such file or directory” on target pool (“backup_raidz” or “backup_raidz_test”).\r\n\r\nWhen I have tried to access offending file on “nas1FS” with a command:\r\n`# md5sum /nas1FS/backup/.zfs/snapshot/20151121_1/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb`\r\nit resulted in a very hard system lockup, I could not get a system reaction on “Ctrl-Alt-SysRq-h” and similar key combinations, any IO to disks stopped completely and immediately, system stopped responding to ping and only a hard reset achieved any reaction out of the system. \r\nAfter hard reset everything was working, above mentioned file checksum results were unchanged.\r\n\r\n\r\nI have also tried a send/recieve to a different target pool (a single 1TB HGST disk):\r\n`# zfs send -R -vvvv \"nas1FS/backup@20160618\" |zfs receive -vvvv -F \"backup_raidz_test/nas1FS_bak\"`\r\nresulted with same md5sum mismatches.\r\n\r\nWhen sending  only the latest snapshot with:\r\n`# zfs send -vvvv \"nas1FS/backup@20160618\" |zfs receive -vvvv -F \"backup_raidz_test/backup\"`\r\nI get  a correct md5sum on the target filesystem.\r\n\r\nWhen trying to do an incremental send receive from the first available snapshot on source pool: \r\n `# zfs send -vvvv \"nas1FS/backup@20151121_1\" |zfs receive -vvvv -F \"backup_raidz_test/backup\"`\r\nOffending file not present on target and source pool and trying to access it on target pool does not cause any issues.\r\n` # zfs send -vvvv -I \"nas1FS/backup@20151121_1\" \"nas1FS/backup@20151124\" |zfs receive -vvvv -F \"backup_raidz_test/backup\"`\r\nI get again a checksum mismatch.\r\n\r\nWhen trying to do an incremental send receive from the second available snapshot on source pool I get correct checksums on both snapshots on target pool .... \r\n```\r\n# zfs send -vvvv \"nas1FS/backup@20151124\" |zfs receive -vvvv -F \"backup_raidz_test/backup\"\r\n# md5sum /nas1FS/backup/.zfs/snapshot/20151124/samba_share/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20151124/samba_share/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n# md5sum /backup_raidz_test/backup/.zfs/snapshot/20151124/samba_share/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw2opgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /backup_raidz_test/backup/.zfs/snapshot/20151124/samba_share/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n\r\n# zfs send -vvvv -I \"nas1FS/backup@20151124\" \"nas1FS/backup@20151129\" |zfs receive -vvvv -F \"backup_raidz_test/backup\"\r\n# md5sum /backup_raidz_test/backup/.zfs/snapshot/20151129/samba_share/a/home//wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /backup_raidz_test/backup/.zfs/snapshot/20151129/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n\r\n#md5sum /nas1FS/backup/.zfs/snapshot/20151129/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n3178c03d3205ac148372a71d75a835ec  /nas1FS/backup/.zfs/snapshot/20151129/samba_share/a/home/bak/aa/wx/wxWidgets-2.8.12/additions/lib/vc_lib/wxmsw28ud_propgrid.pdb\r\n```\r\n\r\n\r\nIt is interesting to note that only a single block of 4096 bytes of data is corrupted at the end of the file (that has a size of 321 x 4096 bytes ) and only when transferring data with the first source snapshot (\"nas1FS/backup@20151121_1\") .\r\n\r\nBinary comparison of the offending file:\r\n```\r\n# vbindiff wxmsw28ud_propgrid.pdb.nas1FS wxmsw28ud_propgrid.pdb.backup_raidz \r\nwxmsw28ud_propgrid.pdb.nas1FS                                                   \r\n0013 FFD0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0013 FFE0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0013 FFF0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0000: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0010: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0050: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0080: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0090: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 00A0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n\r\nwxmsw28ud_propgrid.pdb.backup_raidz                                             \r\n0013 FFD0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0013 FFE0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0013 FFF0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0000: F3 4A 37 0C B1 20 72 48  CF 6F FE BD 3E 76 03 61  .J7.. rH .o..>v.a  \r\n0014 0010: 77 91 72 CE 19 83 73 81  D6 95 DF AE 3A 74 3F 9D  w.r...s. ....:t?.  \r\n0014 0020: 4D DB DD B0 1E 43 2A 2C  63 77 0D B9 E3 EA 8F 1B  M....C*, cw......  \r\n0014 0030: 75 CF E7 48 C9 85 65 8E  35 0A 41 6C 06 E8 33 9E  u..H..e. 5.Al..3.  \r\n0014 0040: FD FD 69 01 00 62 41 52  CE A0 7C C1 0E 32 71 FE  ..i..bAR ..|..2q.  \r\n0014 0050: 3D E9 A0 E4 19 11 80 6D  EB BD 1C 31 1B 7F 3E B4  =......m ...1..>.  \r\n0014 0060: 09 AB D8 8C E4 2B 10 A4  BA B7 CC 31 B7 39 3C F5  .....+.. ...1.9<.  \r\n0014 0070: A5 8B 0C 49 C1 1E A3 DC  7A FE 74 0C 9C F4 19 C9  ...I.... z.t.....  \r\n0014 0080: F4 1D 47 34 83 E6 C6 49  07 04 60 70 40 39 CF 3F  ..G4...I ..`p@9.?  \r\n0014 0090: 8F 7A 00 70 53 16 DD 80  85 C9 DA 64 7F 30 91 DF  .z.pS... ...d.0..  \r\n0014 00A0: 93 9C F5 3D 69 CE FB 86  57 21 79 C7 F9 3F D6 80  ...=i... W!y..?..  \r\n\r\n\r\nwxmsw28ud_propgrid.pdb.nas1FS                                                   \r\n0014 0F30: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0F40: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0F50: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0F60: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0F70: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0F80: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0F90: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0FA0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0FB0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0FC0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0FD0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0FE0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 0FF0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........  \r\n0014 1000:                                                                      \r\n\r\nwxmsw28ud_propgrid.pdb.backup_raidz                                             \r\n0014 0F30: B7 60 DB C7 7E 77 FF 00  9F 5A 75 E5 FD E5 94 3E  .`..~w.. .Zu....>  \r\n0014 0F40: 7C 16 86 F5 60 63 98 77  06 3C F5 CF AD 33 32 2D  |...`c.w .<...32-  \r\n0014 0F50: 23 50 B7 BC B7 BA 76 84  43 2E 76 84 27 95 24 FC  #P....v. C.v.'.$.  \r\n0014 0F60: E0 FF 00 F5 EA AE A3 79  AB D9 C2 64 B6 B5 8E 55  .......y ...d...U  \r\n0014 0F70: 12 65 27 8F 2C A2 3F 42  0F 7A CA 4E 49 BD 59 51  .e'.,.?B .z.NI.YQ  \r\n0014 0F80: F8 91 AB 65 7B 1D E5 BC  57 13 03 0D C2 AE D5 66  ...e{... W......f  \r\n0014 0F90: 1E 5E F2 73 BF 8E FF 00  5A C8 D5 AE B5 6B 79 61  .^.s.... Z....kya  \r\n0014 0FA0: 78 12 39 2D 55 F6 BF 95  93 27 27 AF FF 00 5E A0  x.9-U... .''...^.  \r\n0014 0FB0: D8 DD 56 33 DB 11 E5 4D  F6 97 55 29 9F BA 54 F5  ..V3...M ..U)..T.  \r\n0014 0FC0: 39 3D 6B 02 FD 75 6B 3D  5A 1B AB 3B A3 34 09 0C  9=k..uk= Z..;.4..  \r\n0014 0FD0: 71 8C 42 24 F2 8B 7D F0  C3 A1 FA 9A 00 DE B8 7B  q.B$..}. .......{  \r\n0014 0FE0: 99 A7 84 E6 39 05 C4 3D  15 02 30 90 7F 52 4D 73  ....9..= ..0..RMs  \r\n0014 0FF0: D6 56 3A C5 A6 A5 B8 98  DA 32 CC 6E 15 E6 19 41  .V:..... .2.n...A  \r\n0014 1000:                                                                      \r\n```\r\n\r\n\r\n\r\nI have also run zdb on the source pool, commands I did check did not find any errors:\r\n```\r\n# zdb -ccc  nas1FS\r\n\r\nTraversing all blocks to verify checksums and verify nothing leaked ...\r\n\r\nloading space map for vdev 0 of 1, metaslab 173 of 174 ...\r\n1.95T completed (  94MB/s) estimated time remaining: 0hr 00min 00sec         \r\n\tNo leaks (block sum matches space maps exactly)\r\n\r\n\tbp count:        16048635\r\n\tganged count:           0\r\n\tbp logical:    1868775708160      avg: 116444\r\n\tbp physical:   1395945222144      avg:  86982     compression:   1.34\r\n\tbp allocated:  2147719213056      avg: 133825     compression:   0.87\r\n\tbp deduped:             0    ref>1:      0   deduplication:   1.00\r\n\tSPA allocated: 2147719213056     used: 35.92%\r\n\r\n\tadditional, non-pointer bps of type 0:     178850\r\n\tDittoed blocks on same vdev: 473244\r\n\r\n```\r\n\r\n```\r\n# zdb -mc nas1FS\r\n\r\nMetaslabs:\r\n\tvdev          0\r\n\tmetaslabs   174   offset                spacemap          free      \r\n\t---------------   -------------------   ---------------   -------------\r\n\tmetaslab      0   offset            0   spacemap     38   free     138M\r\n\tmetaslab      1   offset    800000000   spacemap     39   free    22.8G\r\n\tmetaslab      2   offset   1000000000   spacemap     52   free    75.2M\r\n\tmetaslab      3   offset   1800000000   spacemap     68   free    30.3G\r\n\tmetaslab      4   offset   2000000000   spacemap     53   free     104M\r\n\tmetaslab      5   offset   2800000000   spacemap     67   free    30.3M\r\n\tmetaslab      6   offset   3000000000   spacemap     69   free    30.3M\r\n\tmetaslab      7   offset   3800000000   spacemap     70   free    30.6M\r\n\tmetaslab      8   offset   4000000000   spacemap     71   free    24.6G\r\n\tmetaslab      9   offset   4800000000   spacemap     72   free    30.6G\r\n\tmetaslab     10   offset   5000000000   spacemap     73   free    31.5G\r\n\tmetaslab     11   offset   5800000000   spacemap     74   free    30.3G\r\n\tmetaslab     12   offset   6000000000   spacemap     77   free    32.0G\r\n\tmetaslab     13   offset   6800000000   spacemap     78   free    32.0G\r\n\tmetaslab     14   offset   7000000000   spacemap     79   free    32.0G\r\n\tmetaslab     15   offset   7800000000   spacemap     80   free    32.0G\r\n\tmetaslab     16   offset   8000000000   spacemap     81   free      32G\r\n\tmetaslab     17   offset   8800000000   spacemap     82   free      32G\r\n\tmetaslab     18   offset   9000000000   spacemap     83   free    32.0G\r\n\tmetaslab     19   offset   9800000000   spacemap     84   free    32.0G\r\n\tmetaslab     20   offset   a000000000   spacemap     85   free    32.0G\r\n\tmetaslab     21   offset   a800000000   spacemap     86   free      32G\r\n\tmetaslab     22   offset   b000000000   spacemap     87   free      32G\r\n\tmetaslab     23   offset   b800000000   spacemap     88   free    32.0G\r\n\tmetaslab     24   offset   c000000000   spacemap     90   free    32.0G\r\n\tmetaslab     25   offset   c800000000   spacemap     91   free    12.1G\r\n\tmetaslab     26   offset   d000000000   spacemap     92   free    2.66M\r\n\tmetaslab     27   offset   d800000000   spacemap     93   free    4.55M\r\n\tmetaslab     28   offset   e000000000   spacemap     94   free    8.17G\r\n\tmetaslab     29   offset   e800000000   spacemap    113   free    38.0M\r\n\tmetaslab     30   offset   f000000000   spacemap    114   free    5.46G\r\n\tmetaslab     31   offset   f800000000   spacemap    115   free    2.07G\r\n\tmetaslab     32   offset  10000000000   spacemap    116   free    2.56G\r\n\tmetaslab     33   offset  10800000000   spacemap     37   free    28.0G\r\n\tmetaslab     34   offset  11000000000   spacemap    117   free     451M\r\n\tmetaslab     35   offset  11800000000   spacemap    118   free     298M\r\n\tmetaslab     36   offset  12000000000   spacemap    119   free    10.8G\r\n\tmetaslab     37   offset  12800000000   spacemap    120   free     9.8M\r\n\tmetaslab     38   offset  13000000000   spacemap    121   free    9.49G\r\n\tmetaslab     39   offset  13800000000   spacemap    131   free    23.9G\r\n\tmetaslab     40   offset  14000000000   spacemap    132   free    19.4G\r\n\tmetaslab     41   offset  14800000000   spacemap    134   free    25.2G\r\n\tmetaslab     42   offset  15000000000   spacemap    135   free    23.1G\r\n\tmetaslab     43   offset  15800000000   spacemap    136   free    23.2G\r\n\tmetaslab     44   offset  16000000000   spacemap    137   free    23.2G\r\n\tmetaslab     45   offset  16800000000   spacemap     76   free    23.2G\r\n\tmetaslab     46   offset  17000000000   spacemap    138   free    18.2G\r\n\tmetaslab     47   offset  17800000000   spacemap    139   free    20.8G\r\n\tmetaslab     48   offset  18000000000   spacemap    140   free    18.1G\r\n\tmetaslab     49   offset  18800000000   spacemap    141   free    22.8G\r\n\tmetaslab     50   offset  19000000000   spacemap    142   free    23.7G\r\n\tmetaslab     51   offset  19800000000   spacemap    143   free    24.5G\r\n\tmetaslab     52   offset  1a000000000   spacemap    144   free    23.2G\r\n\tmetaslab     53   offset  1a800000000   spacemap    156   free    22.8G\r\n\tmetaslab     54   offset  1b000000000   spacemap    158   free    23.7G\r\n\tmetaslab     55   offset  1b800000000   spacemap    159   free    19.3G\r\n\tmetaslab     56   offset  1c000000000   spacemap    160   free    23.0G\r\n\tmetaslab     57   offset  1c800000000   spacemap     89   free    23.4G\r\n\tmetaslab     58   offset  1d000000000   spacemap    161   free    23.2G\r\n\tmetaslab     59   offset  1d800000000   spacemap    162   free    23.2G\r\n\tmetaslab     60   offset  1e000000000   spacemap    174   free    23.4G\r\n\tmetaslab     61   offset  1e800000000   spacemap    175   free    18.6G\r\n\tmetaslab     62   offset  1f000000000   spacemap    176   free    18.5G\r\n\tmetaslab     63   offset  1f800000000   spacemap    177   free    23.0G\r\n\tmetaslab     64   offset  20000000000   spacemap    178   free    23.2G\r\n\tmetaslab     65   offset  20800000000   spacemap    202   free    55.4M\r\n\tmetaslab     66   offset  21000000000   spacemap     36   free    12.2G\r\n\tmetaslab     67   offset  21800000000   spacemap    203   free    2.75G\r\n\tmetaslab     68   offset  22000000000   spacemap    204   free    50.7M\r\n\tmetaslab     69   offset  22800000000   spacemap    206   free     9.7G\r\n\tmetaslab     70   offset  23000000000   spacemap    210   free    4.62M\r\n\tmetaslab     71   offset  23800000000   spacemap    211   free     839M\r\n\tmetaslab     72   offset  24000000000   spacemap    226   free     901M\r\n\tmetaslab     73   offset  24800000000   spacemap    279   free    93.2M\r\n\tmetaslab     74   offset  25000000000   spacemap    106   free    31.9G\r\n\tmetaslab     75   offset  25800000000   spacemap    146   free    32.0G\r\n\tmetaslab     76   offset  26000000000   spacemap    147   free    32.0G\r\n\tmetaslab     77   offset  26800000000   spacemap    148   free    32.0G\r\n\tmetaslab     78   offset  27000000000   spacemap     75   free    31.9G\r\n\tmetaslab     79   offset  27800000000   spacemap    157   free    31.9G\r\n\tmetaslab     80   offset  28000000000   spacemap    149   free    32.0G\r\n\tmetaslab     81   offset  28800000000   spacemap    151   free    32.0G\r\n\tmetaslab     82   offset  29000000000   spacemap    164   free      32G\r\n\tmetaslab     83   offset  29800000000   spacemap    165   free    32.0G\r\n\tmetaslab     84   offset  2a000000000   spacemap    166   free    32.0G\r\n\tmetaslab     85   offset  2a800000000   spacemap    167   free    32.0G\r\n\tmetaslab     86   offset  2b000000000   spacemap    168   free    32.0G\r\n\tmetaslab     87   offset  2b800000000   spacemap    169   free    32.0G\r\n\tmetaslab     88   offset  2c000000000   spacemap    170   free    32.0G\r\n\tmetaslab     89   offset  2c800000000   spacemap    180   free    32.0G\r\n\tmetaslab     90   offset  2d000000000   spacemap    133   free    31.9G\r\n\tmetaslab     91   offset  2d800000000   spacemap    181   free    32.0G\r\n\tmetaslab     92   offset  2e000000000   spacemap    182   free    32.0G\r\n\tmetaslab     93   offset  2e800000000   spacemap    184   free    32.0G\r\n\tmetaslab     94   offset  2f000000000   spacemap    185   free    32.0G\r\n\tmetaslab     95   offset  2f800000000   spacemap    186   free    32.0G\r\n\tmetaslab     96   offset  30000000000   spacemap    191   free    32.0G\r\n\tmetaslab     97   offset  30800000000   spacemap    192   free    32.0G\r\n\tmetaslab     98   offset  31000000000   spacemap    193   free    32.0G\r\n\tmetaslab     99   offset  31800000000   spacemap    194   free    32.0G\r\n\tmetaslab    100   offset  32000000000   spacemap    195   free    32.0G\r\n\tmetaslab    101   offset  32800000000   spacemap    196   free    32.0G\r\n\tmetaslab    102   offset  33000000000   spacemap    205   free    32.0G\r\n\tmetaslab    103   offset  33800000000   spacemap    197   free    32.0G\r\n\tmetaslab    104   offset  34000000000   spacemap    209   free    32.0G\r\n\tmetaslab    105   offset  34800000000   spacemap    245   free    32.0G\r\n\tmetaslab    106   offset  35000000000   spacemap    246   free    32.0G\r\n\tmetaslab    107   offset  35800000000   spacemap    247   free    32.0G\r\n\tmetaslab    108   offset  36000000000   spacemap    248   free    32.0G\r\n\tmetaslab    109   offset  36800000000   spacemap    255   free    32.0G\r\n\tmetaslab    110   offset  37000000000   spacemap    256   free    31.9G\r\n\tmetaslab    111   offset  37800000000   spacemap    257   free    32.0G\r\n\tmetaslab    112   offset  38000000000   spacemap    258   free    32.0G\r\n\tmetaslab    113   offset  38800000000   spacemap    259   free    32.0G\r\n\tmetaslab    114   offset  39000000000   spacemap    150   free    32.0G\r\n\tmetaslab    115   offset  39800000000   spacemap    260   free    32.0G\r\n\tmetaslab    116   offset  3a000000000   spacemap    261   free    31.9G\r\n\tmetaslab    117   offset  3a800000000   spacemap    268   free    30.7G\r\n\tmetaslab    118   offset  3b000000000   spacemap    270   free    31.3G\r\n\tmetaslab    119   offset  3b800000000   spacemap    272   free    31.9G\r\n\tmetaslab    120   offset  3c000000000   spacemap    273   free    31.5G\r\n\tmetaslab    121   offset  3c800000000   spacemap    281   free    30.9G\r\n\tmetaslab    122   offset  3d000000000   spacemap    282   free    31.2G\r\n\tmetaslab    123   offset  3d800000000   spacemap    283   free    31.0G\r\n\tmetaslab    124   offset  3e000000000   spacemap    284   free    31.2G\r\n\tmetaslab    125   offset  3e800000000   spacemap    285   free    31.1G\r\n\tmetaslab    126   offset  3f000000000   spacemap    183   free    31.2G\r\n\tmetaslab    127   offset  3f800000000   spacemap    299   free    30.9G\r\n\tmetaslab    128   offset  40000000000   spacemap    300   free    31.0G\r\n\tmetaslab    129   offset  40800000000   spacemap    301   free    31.0G\r\n\tmetaslab    130   offset  41000000000   spacemap    310   free    30.9G\r\n\tmetaslab    131   offset  41800000000   spacemap    365   free    30.9G\r\n\tmetaslab    132   offset  42000000000   spacemap    399   free    29.8G\r\n\tmetaslab    133   offset  42800000000   spacemap    400   free    30.7G\r\n\tmetaslab    134   offset  43000000000   spacemap    434   free    31.0G\r\n\tmetaslab    135   offset  43800000000   spacemap    435   free    30.9G\r\n\tmetaslab    136   offset  44000000000   spacemap    436   free    30.9G\r\n\tmetaslab    137   offset  44800000000   spacemap    437   free    31.8G\r\n\tmetaslab    138   offset  45000000000   spacemap    244   free    32.0G\r\n\tmetaslab    139   offset  45800000000   spacemap    438   free      32G\r\n\tmetaslab    140   offset  46000000000   spacemap    439   free    32.0G\r\n\tmetaslab    141   offset  46800000000   spacemap    448   free    32.0G\r\n\tmetaslab    142   offset  47000000000   spacemap    530   free    12.9G\r\n\tmetaslab    143   offset  47800000000   spacemap    628   free    41.7M\r\n\tmetaslab    144   offset  48000000000   spacemap    629   free    60.0M\r\n\tmetaslab    145   offset  48800000000   spacemap    630   free    59.7M\r\n\tmetaslab    146   offset  49000000000   spacemap    631   free    41.5M\r\n\tmetaslab    147   offset  49800000000   spacemap    632   free    63.8M\r\n\tmetaslab    148   offset  4a000000000   spacemap    633   free    13.1G\r\n\tmetaslab    149   offset  4a800000000   spacemap    634   free     238M\r\n\tmetaslab    150   offset  4b000000000   spacemap    267   free    18.9G\r\n\tmetaslab    151   offset  4b800000000   spacemap    269   free    18.1G\r\n\tmetaslab    152   offset  4c000000000   spacemap    271   free    1.15G\r\n\tmetaslab    153   offset  4c800000000   spacemap    635   free    6.17G\r\n\tmetaslab    154   offset  4d000000000   spacemap    636   free    10.8G\r\n\tmetaslab    155   offset  4d800000000   spacemap    637   free    5.34G\r\n\tmetaslab    156   offset  4e000000000   spacemap    638   free    54.3M\r\n\tmetaslab    157   offset  4e800000000   spacemap    639   free    3.82G\r\n\tmetaslab    158   offset  4f000000000   spacemap    332   free    22.3M\r\n\tmetaslab    159   offset  4f800000000   spacemap    343   free    76.9M\r\n\tmetaslab    160   offset  50000000000   spacemap    344   free    7.76G\r\n\tmetaslab    161   offset  50800000000   spacemap    345   free    7.33G\r\n\tmetaslab    162   offset  51000000000   spacemap    346   free    7.86G\r\n\tmetaslab    163   offset  51800000000   spacemap    347   free     110M\r\n\tmetaslab    164   offset  52000000000   spacemap    364   free    2.27G\r\n\tmetaslab    165   offset  52800000000   spacemap    101   free     127M\r\n\tmetaslab    166   offset  53000000000   spacemap     51   free     176M\r\n\tmetaslab    167   offset  53800000000   spacemap    198   free    64.2M\r\n\tmetaslab    168   offset  54000000000   spacemap    200   free     9.7G\r\n\tmetaslab    169   offset  54800000000   spacemap    240   free    9.27G\r\n\tmetaslab    170   offset  55000000000   spacemap    251   free    41.7M\r\n\tmetaslab    171   offset  55800000000   spacemap    249   free    1.94G\r\n\tmetaslab    172   offset  56000000000   spacemap    278   free    5.11M\r\n\tmetaslab    173   offset  56800000000   spacemap    291   free    4.97G\r\n\r\n\r\nTraversing all blocks to verify metadata checksums and verify nothing leaked ...\r\n\r\n\r\n\tNo leaks (block sum matches space maps exactly)\r\n\r\n\tbp count:        16048574\r\n\tganged count:           0\r\n\tbp logical:    1868774123008      avg: 116444\r\n\tbp physical:   1395943846400      avg:  86982     compression:   1.34\r\n\tbp allocated:  2147716362240      avg: 133825     compression:   0.87\r\n\tbp deduped:             0    ref>1:      0   deduplication:   1.00\r\n\tSPA allocated: 2147716362240     used: 35.92%\r\n\r\n\tadditional, non-pointer bps of type 0:     178850\r\n\tDittoed blocks on same vdev: 473223\r\n\r\n\r\n```\r\n\r\n\r\n\r\nTo summarize used  system configurations:\r\n\r\nSystem used for “nas1FS” data fillup (old computer):\r\nMotherboard MSI X58 Pro (MSI MS-7522/MSI X58 Gold)  with Intel Quad Core i7 965 3.2GHz and 14GB non-ECC RAM (MB, CPU and RAM and PS are about 6 years old).\r\n“/boot” :  INTEL SSDMCEAW080A4  80GB SSD\r\n“nas1FS” pool:  6x1TB  HGST Travelstar 7K1000  in a RAIDZ2 array (HDDs are ~6months old). \r\n\r\nNew system to which “nas1FS” was moved (all disks):\r\nMotherboard Supermicro A1SA7-2750F (8 core Intel Atom) with 32GB ECC RAM (MB and RAM and PS are new).\r\n“/boot” :  INTEL SSDMCEAW080A4  80GB SSD\r\n“nas1FS” pool:  6x1TB  HGST Travelstar 7K1000  in a RAIDZ2 array (moved from the old computer).\r\n“backup_raidz” pool:  2x1TB Samsung HD103UJ + 1x1TB  HGST Travelstar 7K1000  (pool used for backup)\r\n“backup_raidz_test” pool:  1TB Samsung HD103UJ (pool with no parity, for additional tests)\r\n\r\nBoth systems were tested with memtest, cpuburn etc. without errors.\r\nI am using Debian Jessie booted from zfs pool (with a separate boot partition), same operating system on both machines used with “nas1FS” pool.\r\n  \r\nKernel Command line: \r\n`BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=ZFS=/rootFS/1_jessie ro rpool=nas1FS bootfs=nas1FS/rootFS/1_jessie root=ZFS=nas1FS/rootFS/1_jessie rootfstype=zfs boot=zfs quiet\r\n`\r\n```\r\n# uname -a  \r\nLinux nas1.ip 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64 GNU/Linux\r\n\r\n```\r\nSPL and ZFS was built from source. \r\n```\r\n# dpkg -l |grep zfs\r\nii  kmod-zfs-3.16.0-4-amd64              0.6.5.3-1                            amd64        zfs kernel module(s) for 3.16.0-4-amd64\r\nii  kmod-zfs-devel                       0.6.5.3-1                            amd64        zfs kernel module(s) devel common\r\nii  kmod-zfs-devel-3.16.0-4-amd64        0.6.5.3-1                            amd64        zfs kernel module(s) devel for 3.16.0-4-amd64\r\nii  libzfs2                              0.6.5.3-1                            amd64        Native ZFS filesystem library for Linux\r\nii  libzfs2-devel                        0.6.5.3-1                            amd64        Development headers\r\nii  zfs                                  0.6.5.3-1                            amd64        Commands to control the kernel modules and libraries\r\nii  zfs-dracut                           0.6.5.3-1                            amd64        Dracut module\r\nii  zfs-initramfs                        0.6.5.3-1                            amd64        Initramfs module\r\nii  zfs-test                             0.6.5.3-1                            amd64        Test infrastructure\r\n\r\n```\r\n\r\n\r\n```\r\n# cat /etc/modprobe.d/zfs-arc-max.conf \r\noptions zfs zfs_arc_max=4294967296\r\n```\r\n\r\n```\r\n# zfs get all nas1FS/backup      \r\nNAME           PROPERTY              VALUE                  SOURCE\r\nnas1FS/backup  type                  filesystem             -\r\nnas1FS/backup  creation              Mon Nov  9  1:50 2015  -\r\nnas1FS/backup  used                  1.02T                  -\r\nnas1FS/backup  available             2.21T                  -\r\nnas1FS/backup  referenced            866G                   -\r\nnas1FS/backup  compressratio         1.39x                  -\r\nnas1FS/backup  mounted               yes                    -\r\nnas1FS/backup  quota                 none                   default\r\nnas1FS/backup  reservation           none                   default\r\nnas1FS/backup  recordsize            128K                   default\r\nnas1FS/backup  mountpoint            /nas1FS/backup         inherited from nas1FS\r\nnas1FS/backup  sharenfs              off                    default\r\nnas1FS/backup  checksum              on                     default\r\nnas1FS/backup  compression           gzip-9                 local\r\nnas1FS/backup  atime                 off                    inherited from nas1FS\r\nnas1FS/backup  devices               on                     default\r\nnas1FS/backup  exec                  on                     default\r\nnas1FS/backup  setuid                on                     default\r\nnas1FS/backup  readonly              off                    default\r\nnas1FS/backup  zoned                 off                    default\r\nnas1FS/backup  snapdir               hidden                 default\r\nnas1FS/backup  aclinherit            restricted             default\r\nnas1FS/backup  canmount              on                     default\r\nnas1FS/backup  xattr                 sa                     local\r\nnas1FS/backup  copies                1                      default\r\nnas1FS/backup  version               5                      -\r\nnas1FS/backup  utf8only              off                    -\r\nnas1FS/backup  normalization         none                   -\r\nnas1FS/backup  casesensitivity       sensitive              -\r\nnas1FS/backup  vscan                 off                    default\r\nnas1FS/backup  nbmand                off                    default\r\nnas1FS/backup  sharesmb              off                    default\r\nnas1FS/backup  refquota              none                   default\r\nnas1FS/backup  refreservation        none                   default\r\nnas1FS/backup  primarycache          all                    default\r\nnas1FS/backup  secondarycache        all                    default\r\nnas1FS/backup  usedbysnapshots       63.4G                  -\r\nnas1FS/backup  usedbydataset         866G                   -\r\nnas1FS/backup  usedbychildren        113G                   -\r\nnas1FS/backup  usedbyrefreservation  0                      -\r\nnas1FS/backup  logbias               latency                default\r\nnas1FS/backup  dedup                 off                    default\r\nnas1FS/backup  mlslabel              none                   default\r\nnas1FS/backup  sync                  standard               default\r\nnas1FS/backup  refcompressratio      1.45x                  -\r\nnas1FS/backup  written               16.0K                  -\r\nnas1FS/backup  logicalused           1.39T                  -\r\nnas1FS/backup  logicalreferenced     1.19T                  -\r\nnas1FS/backup  filesystem_limit      none                   default\r\nnas1FS/backup  snapshot_limit        none                   default\r\nnas1FS/backup  filesystem_count      none                   default\r\nnas1FS/backup  snapshot_count        none                   default\r\nnas1FS/backup  snapdev               hidden                 default\r\nnas1FS/backup  acltype               off                    default\r\nnas1FS/backup  context               none                   default\r\nnas1FS/backup  fscontext             none                   default\r\nnas1FS/backup  defcontext            none                   default\r\nnas1FS/backup  rootcontext           none                   default\r\nnas1FS/backup  relatime              off                    default\r\nnas1FS/backup  redundant_metadata    all                    default\r\nnas1FS/backup  overlay               off                    default\r\n```\r\n\r\n\r\n```\r\n# zfs get all nas1FS       \r\nNAME    PROPERTY              VALUE                  SOURCE\r\nnas1FS  type                  filesystem             -\r\nnas1FS  creation              Sun Nov  8 18:14 2015  -\r\nnas1FS  used                  1.30T                  -\r\nnas1FS  available             2.21T                  -\r\nnas1FS  referenced            298M                   -\r\nnas1FS  compressratio         1.33x                  -\r\nnas1FS  mounted               yes                    -\r\nnas1FS  quota                 none                   default\r\nnas1FS  reservation           none                   default\r\nnas1FS  recordsize            128K                   default\r\nnas1FS  mountpoint            /nas1FS                local\r\nnas1FS  sharenfs              off                    default\r\nnas1FS  checksum              on                     default\r\nnas1FS  compression           off                    default\r\nnas1FS  atime                 off                    local\r\nnas1FS  devices               on                     default\r\nnas1FS  exec                  on                     default\r\nnas1FS  setuid                on                     default\r\nnas1FS  readonly              off                    default\r\nnas1FS  zoned                 off                    default\r\nnas1FS  snapdir               hidden                 default\r\nnas1FS  aclinherit            restricted             default\r\nnas1FS  canmount              on                     default\r\nnas1FS  xattr                 on                     default\r\nnas1FS  copies                1                      default\r\nnas1FS  version               5                      -\r\nnas1FS  utf8only              off                    -\r\nnas1FS  normalization         none                   -\r\nnas1FS  casesensitivity       sensitive              -\r\nnas1FS  vscan                 off                    default\r\nnas1FS  nbmand                off                    default\r\nnas1FS  sharesmb              off                    default\r\nnas1FS  refquota              none                   default\r\nnas1FS  refreservation        none                   default\r\nnas1FS  primarycache          all                    default\r\nnas1FS  secondarycache        all                    default\r\nnas1FS  usedbysnapshots       144K                   -\r\nnas1FS  usedbydataset         298M                   -\r\nnas1FS  usedbychildren        1.30T                  -\r\nnas1FS  usedbyrefreservation  0                      -\r\nnas1FS  logbias               latency                default\r\nnas1FS  dedup                 off                    default\r\nnas1FS  mlslabel              none                   default\r\nnas1FS  sync                  standard               default\r\nnas1FS  refcompressratio      1.00x                  -\r\nnas1FS  written               298M                   -\r\nnas1FS  logicalused           1.70T                  -\r\nnas1FS  logicalreferenced     298M                   -\r\nnas1FS  filesystem_limit      none                   default\r\nnas1FS  snapshot_limit        none                   default\r\nnas1FS  filesystem_count      none                   default\r\nnas1FS  snapshot_count        none                   default\r\nnas1FS  snapdev               hidden                 default\r\nnas1FS  acltype               off                    default\r\nnas1FS  context               none                   default\r\nnas1FS  fscontext             none                   default\r\nnas1FS  defcontext            none                   default\r\nnas1FS  rootcontext           none                   default\r\nnas1FS  relatime              off                    default\r\nnas1FS  redundant_metadata    all                    default\r\nnas1FS  overlay               off                    default\r\n```\r\n\r\n\r\n```\r\n# zpool get all nas1FS\r\nNAME    PROPERTY                    VALUE                       SOURCE\r\nnas1FS  size                        5.44T                       -\r\nnas1FS  capacity                    35%                         -\r\nnas1FS  altroot                     -                           default\r\nnas1FS  health                      ONLINE                      -\r\nnas1FS  guid                        12764045164742778073        default\r\nnas1FS  version                     -                           default\r\nnas1FS  bootfs                      nas1FS/rootFS/1_jessie      local\r\nnas1FS  delegation                  on                          default\r\nnas1FS  autoreplace                 off                         default\r\nnas1FS  cachefile                   -                           default\r\nnas1FS  failmode                    wait                        default\r\nnas1FS  listsnapshots               off                         default\r\nnas1FS  autoexpand                  off                         default\r\nnas1FS  dedupditto                  0                           default\r\nnas1FS  dedupratio                  1.00x                       -\r\nnas1FS  free                        3.48T                       -\r\nnas1FS  allocated                   1.95T                       -\r\nnas1FS  readonly                    off                         -\r\nnas1FS  ashift                      12                          local\r\nnas1FS  comment                     -                           default\r\nnas1FS  expandsize                  -                           -\r\nnas1FS  freeing                     0                           default\r\nnas1FS  fragmentation               12%                         -\r\nnas1FS  leaked                      0                           default\r\nnas1FS  feature@async_destroy       enabled                     local\r\nnas1FS  feature@empty_bpobj         active                      local\r\nnas1FS  feature@lz4_compress        active                      local\r\nnas1FS  feature@spacemap_histogram  active                      local\r\nnas1FS  feature@enabled_txg         active                      local\r\nnas1FS  feature@hole_birth          active                      local\r\nnas1FS  feature@extensible_dataset  enabled                     local\r\nnas1FS  feature@embedded_data       active                      local\r\nnas1FS  feature@bookmarks           enabled                     local\r\nnas1FS  feature@filesystem_limits   enabled                     local\r\nnas1FS  feature@large_blocks        enabled                     local\r\n```\r\n\r\n\r\n\r\nSome excerpts from dmesg (blocked messages are not connected to the hard lockup of the system):\r\n\r\n```\r\n[14051.194118] INFO: task txg_sync:6287 blocked for more than 120 seconds.\r\n[14051.194125]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[14051.194127] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[14051.194130] txg_sync        D ffff880785cb2ef8     0  6287      2 0x00000000\r\n[14051.194136]  ffff880785cb2aa0 0000000000000046 0000000000012f00 ffff880782d1ffd8\r\n[14051.194140]  0000000000012f00 ffff880785cb2aa0 ffff88087fc537b0 ffff8807a1088850\r\n[14051.194143]  ffff8807a1088890 0000000000000001 ffff88085afed000 0000000000000000\r\n[14051.194147] Call Trace:\r\n[14051.194156]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[14051.194175]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[14051.194189]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[14051.194210]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[14051.194229]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[14051.194249]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[14051.194278]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[14051.194298]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[14051.194305]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[14051.194311]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[14051.194316]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[14051.194320]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[14051.194323]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[14051.194334]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n\r\n\r\n```\r\n[172938.829534] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[172938.829549]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[172938.829551] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[172938.829554] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[172938.829559]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[172938.829563]  0000000000012f00 ffff8807876a2a60 ffff88087fd537b0 ffff88065adb1110\r\n[172938.829567]  ffff88065adb1150 0000000000000001 ffff88085afed000 0000000000000000\r\n[172938.829570] Call Trace:\r\n[172938.829579]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[172938.829599]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[172938.829604]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[172938.829625]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[172938.829652]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[172938.829673]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[172938.829693]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[172938.829720]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[172938.829727]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[172938.829733]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[172938.829737]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[172938.829741]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[172938.829745]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[172938.829749]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[173539.308306] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[173539.308312]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[173539.308314] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[173539.308317] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[173539.308322]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[173539.308326]  0000000000012f00 ffff8807876a2a60 ffff88087fc537b0 ffff8807fd2523b0\r\n[173539.308329]  ffff8807fd2523f0 0000000000000001 ffff88085afed000 0000000000000000\r\n[173539.308333] Call Trace:\r\n[173539.308341]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[173539.308367]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[173539.308372]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[173539.308392]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[173539.308411]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[173539.308431]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[173539.308450]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[173539.308470]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[173539.308484]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[173539.308490]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[173539.308494]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[173539.308498]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[173539.308502]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[173539.308506]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n\r\n```\r\n[173539.308512] INFO: task zfs:28953 blocked for more than 120 seconds.\r\n[173539.308515]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[173539.308516] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[173539.308518] zfs             D ffff8807fd460ef8     0 28953  14132 0x00000000\r\n[173539.308522]  ffff8807fd460aa0 0000000000000086 0000000000012f00 ffff88068360bfd8\r\n[173539.308525]  0000000000012f00 ffff8807fd460aa0 ffff88085afed368 ffff88085afed220\r\n[173539.308529]  ffff88085afed370 0000000000000000 ffff88085afed320 ffff88064b160000\r\n[173539.308532] Call Trace:\r\n[173539.308539]  [<ffffffffa02026cd>] ? cv_wait_common+0xcd/0x100 [spl]\r\n[173539.308543]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[173539.308562]  [<ffffffffa02e61eb>] ? txg_wait_open+0xbb/0x100 [zfs]\r\n[173539.308585]  [<ffffffffa02a3280>] ? dmu_tx_wait+0x370/0x380 [zfs]\r\n[173539.308601]  [<ffffffffa02a332b>] ? dmu_tx_assign+0x9b/0x510 [zfs]\r\n[173539.308617]  [<ffffffffa029ee19>] ? restore_write+0xd9/0x1c0 [zfs]\r\n[173539.308633]  [<ffffffffa02a0b39>] ? dmu_recv_stream+0x429/0xb00 [zfs]\r\n[173539.308640]  [<ffffffffa021d4e4>] ? nvlist_common.part.102+0xe4/0x200 [znvpair]\r\n[173539.308659]  [<ffffffffa030e225>] ? zfs_ioc_recv+0x1f5/0xb10 [zfs]\r\n[173539.308671]  [<ffffffffa00a817d>] ? avl_find+0x5d/0xa0 [zavl]\r\n[173539.308677]  [<ffffffffa020ea9e>] ? zprop_name_to_prop_cb+0x5e/0x70 [zcommon]\r\n[173539.308696]  [<ffffffffa02bc90a>] ? dsl_prop_get_dd+0x18a/0x240 [zfs]\r\n[173539.308703]  [<ffffffffa02033df>] ? tsd_hash_dtor+0x6f/0x80 [spl]\r\n[173539.308709]  [<ffffffffa0203ab5>] ? tsd_set+0x165/0x4d0 [spl]\r\n[173539.308727]  [<ffffffffa02bcb3c>] ? dsl_prop_get_ds+0x17c/0x230 [zfs]\r\n[173539.308782]  [<ffffffffa02cc5d8>] ? rrw_exit+0x58/0x160 [zfs]\r\n[173539.308812]  [<ffffffffa030d019>] ? zfsdev_ioctl+0x489/0x4c0 [zfs]\r\n[173539.308818]  [<ffffffff811ba4df>] ? do_vfs_ioctl+0x2cf/0x4b0\r\n[173539.308821]  [<ffffffff8150d8c1>] ? __schedule+0x2b1/0x710\r\n[173539.308842]  [<ffffffff811ba741>] ? SyS_ioctl+0x81/0xa0\r\n[173539.308857]  [<ffffffff8151158d>] ? system_call_fast_compare_end+0x10/0x15\r\n```\r\n\r\n\r\n```\r\n[187471.366228] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[187471.366235]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[187471.366237] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[187471.366239] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[187471.366245]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[187471.366249]  0000000000012f00 ffff8807876a2a60 ffff88087fcd37b0 ffff88071be29d70\r\n[187471.366252]  ffff88071be29db0 0000000000000001 ffff88085afed000 0000000000000000\r\n[187471.366256] Call Trace:\r\n[187471.366264]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[187471.366291]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[187471.366296]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[187471.366317]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[187471.366336]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[187471.366356]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[187471.366383]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[187471.366403]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[187471.366409]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[187471.366415]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[187471.366420]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[187471.366424]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[187471.366428]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[187471.366431]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n\r\n```\r\n[201523.040835] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[201523.040842]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[201523.040844] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[201523.040846] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[201523.040852]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[201523.040855]  0000000000012f00 ffff8807876a2a60 ffff88087fc137b0 ffff880727ac8330\r\n[201523.040859]  ffff880727ac8370 0000000000000001 ffff88085afed000 0000000000000000\r\n[201523.040862] Call Trace:\r\n[201523.040871]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[201523.040889]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[201523.040894]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[201523.040915]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[201523.040942]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[201523.040962]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[201523.040982]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[201523.041002]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[201523.041016]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[201523.041022]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[201523.041026]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[201523.041030]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[201523.041034]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[201523.041037]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n\r\n```\r\n[219777.534318] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[219777.534325]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[219777.534326] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[219777.534329] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[219777.534334]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[219777.534338]  0000000000012f00 ffff8807876a2a60 ffff88087fcd37b0 ffff8808557c3950\r\n[219777.534341]  ffff8808557c3990 0000000000000001 ffff88085afed000 0000000000000000\r\n[219777.534345] Call Trace:\r\n[219777.534353]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[219777.534372]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[219777.534385]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[219777.534405]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[219777.534425]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[219777.534445]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[219777.534465]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[219777.534491]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[219777.534498]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[219777.534504]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[219777.534509]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[219777.534512]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[219777.534516]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[219777.534520]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n\r\n```\r\n[220257.909086] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[220257.909092]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[220257.909094] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[220257.909097] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[220257.909109]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[220257.909113]  0000000000012f00 ffff8807876a2a60 ffff88087fdd37b0 ffff8807c2be5050\r\n[220257.909117]  ffff8807c2be5090 0000000000000001 ffff88085afed000 0000000000000000\r\n[220257.909120] Call Trace:\r\n[220257.909129]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[220257.909149]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[220257.909154]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[220257.909175]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[220257.909202]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[220257.909222]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[220257.909242]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[220257.909261]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[220257.909274]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[220257.909280]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[220257.909285]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[220257.909289]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[220257.909292]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[220257.909296]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n```\r\n[220257.909303] INFO: task zfs:597 blocked for more than 120 seconds.\r\n[220257.909306]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[220257.909307] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[220257.909309] zfs             D ffff88077f5cd8c8     0   597    584 0x00000000\r\n[220257.909313]  ffff88077f5cd470 0000000000000082 0000000000012f00 ffff880765093fd8\r\n[220257.909316]  0000000000012f00 ffff88077f5cd470 ffff88085afed368 ffff88085afed220\r\n[220257.909319]  ffff88085afed370 0000000000000000 ffff88085afed320 ffff88064b160000\r\n[220257.909323] Call Trace:\r\n[220257.909330]  [<ffffffffa02026cd>] ? cv_wait_common+0xcd/0x100 [spl]\r\n[220257.909341]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[220257.909360]  [<ffffffffa02e61eb>] ? txg_wait_open+0xbb/0x100 [zfs]\r\n[220257.909377]  [<ffffffffa02a3280>] ? dmu_tx_wait+0x370/0x380 [zfs]\r\n[220257.909393]  [<ffffffffa02a332b>] ? dmu_tx_assign+0x9b/0x510 [zfs]\r\n[220257.909409]  [<ffffffffa029ee19>] ? restore_write+0xd9/0x1c0 [zfs]\r\n[220257.909432]  [<ffffffffa02a0b39>] ? dmu_recv_stream+0x429/0xb00 [zfs]\r\n[220257.909439]  [<ffffffffa021d4e4>] ? nvlist_common.part.102+0xe4/0x200 [znvpair]\r\n[220257.909458]  [<ffffffffa030e225>] ? zfs_ioc_recv+0x1f5/0xb10 [zfs]\r\n[220257.909464]  [<ffffffffa00a817d>] ? avl_find+0x5d/0xa0 [zavl]\r\n[220257.909470]  [<ffffffffa020ea9e>] ? zprop_name_to_prop_cb+0x5e/0x70 [zcommon]\r\n[220257.909489]  [<ffffffffa02bc90a>] ? dsl_prop_get_dd+0x18a/0x240 [zfs]\r\n[220257.909502]  [<ffffffffa02033df>] ? tsd_hash_dtor+0x6f/0x80 [spl]\r\n[220257.909509]  [<ffffffffa0203ab5>] ? tsd_set+0x165/0x4d0 [spl]\r\n[220257.909527]  [<ffffffffa02bcb3c>] ? dsl_prop_get_ds+0x17c/0x230 [zfs]\r\n[220257.909548]  [<ffffffffa02cc5d8>] ? rrw_exit+0x58/0x160 [zfs]\r\n[220257.909565]  [<ffffffffa030d019>] ? zfsdev_ioctl+0x489/0x4c0 [zfs]\r\n[220257.909576]  [<ffffffff811ba4df>] ? do_vfs_ioctl+0x2cf/0x4b0\r\n[220257.909579]  [<ffffffff8150d8c1>] ? __schedule+0x2b1/0x710\r\n[220257.909583]  [<ffffffff811ba741>] ? SyS_ioctl+0x81/0xa0\r\n[220257.909587]  [<ffffffff8151158d>] ? system_call_fast_compare_end+0x10/0x15\r\n```\r\n\r\n\r\n```\r\n[221218.788723] INFO: task txg_sync:13384 blocked for more than 120 seconds.\r\n[221218.788730]       Tainted: P           O  3.16.0-4-amd64 #1\r\n[221218.788732] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[221218.788734] txg_sync        D ffff8807876a2eb8     0 13384      2 0x00000000\r\n[221218.788747]  ffff8807876a2a60 0000000000000046 0000000000012f00 ffff880622707fd8\r\n[221218.788750]  0000000000012f00 ffff8807876a2a60 ffff88087fc937b0 ffff8807f5abe3f0\r\n[221218.788754]  ffff8807f5abe430 0000000000000001 ffff88085afed000 0000000000000000\r\n[221218.788757] Call Trace:\r\n[221218.788766]  [<ffffffff8150e019>] ? io_schedule+0x99/0x120\r\n[221218.788784]  [<ffffffffa0202690>] ? cv_wait_common+0x90/0x100 [spl]\r\n[221218.788789]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0\r\n[221218.788810]  [<ffffffffa032d73b>] ? zio_wait+0x10b/0x1e0 [zfs]\r\n[221218.788829]  [<ffffffffa02bb75a>] ? dsl_pool_sync+0xaa/0x460 [zfs]\r\n[221218.788856]  [<ffffffffa02d5646>] ? spa_sync+0x366/0xb30 [zfs]\r\n[221218.788877]  [<ffffffffa02e70c1>] ? txg_sync_thread+0x3d1/0x680 [zfs]\r\n[221218.788896]  [<ffffffffa02e6cf0>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]\r\n[221218.788903]  [<ffffffffa01fdd3b>] ? thread_generic_wrapper+0x6b/0x80 [spl]\r\n[221218.788909]  [<ffffffffa01fdcd0>] ? __thread_exit+0x20/0x20 [spl]\r\n[221218.788913]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0\r\n[221218.788917]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n[221218.788921]  [<ffffffff815114d8>] ? ret_from_fork+0x58/0x90\r\n[221218.788931]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180\r\n```\r\n\r\n\r\nI hope this information will be helpful, but feel free to let me know what other tests I can perform to diagnose this issue. I will be happy to provide any other info.  \r\n \r\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/241246196","html_url":"https://github.com/zfsonlinux/zfs/issues/4809#issuecomment-241246196","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4809","id":241246196,"user":{"login":"datiscum","id":14824345,"avatar_url":"https://avatars.githubusercontent.com/u/14824345?v=3","gravatar_id":"","url":"https://api.github.com/users/datiscum","html_url":"https://github.com/datiscum","followers_url":"https://api.github.com/users/datiscum/followers","following_url":"https://api.github.com/users/datiscum/following{/other_user}","gists_url":"https://api.github.com/users/datiscum/gists{/gist_id}","starred_url":"https://api.github.com/users/datiscum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datiscum/subscriptions","organizations_url":"https://api.github.com/users/datiscum/orgs","repos_url":"https://api.github.com/users/datiscum/repos","events_url":"https://api.github.com/users/datiscum/events{/privacy}","received_events_url":"https://api.github.com/users/datiscum/received_events","type":"User","site_admin":false},"created_at":"2016-08-21T08:53:39Z","updated_at":"2016-08-21T08:53:39Z","body":"I do not know if it belongs here. I have the script used to test from above. And find the result very strange.\r\n\r\n#!/bin/sh\r\nZFSPOOL='DS1'\r\n\r\nzfs destroy $ZFSPOOL/backup -r\r\nzfs destroy $ZFSPOOL/hb -r\r\nzfs create -o recordsize=4k $ZFSPOOL/hb\r\ntruncate -s 1G /$ZFSPOOL/hb/large_file\r\ndd if=/dev/urandom of=/$ZFSPOOL/hb/large_file bs=4k count=11264 seek=1152\r\nzfs snapshot $ZFSPOOL/hb@2_1\r\ntruncate -s 4194304 /$ZFSPOOL/hb/large_file\r\ndd if=/dev/urandom of=/$ZFSPOOL/hb/large_file bs=4k count=152 seek=384 conv=notrunc \r\n\r\n**sleep 5  # If sleep >= 5 seconds is the checksum differently.** \r\n# I do not know why 5 seconds maybe there needs to be more on other systems.\r\n\r\ndd if=/dev/urandom of=/$ZFSPOOL/hb/large_file bs=4k count=10 seek=1408 conv=notrunc \r\nzfs snapshot $ZFSPOOL/hb@2_2\r\nzfs send $ZFSPOOL/hb@2_1 | zfs recv $ZFSPOOL/backup\r\nzfs send -i $ZFSPOOL/hb@2_1 $ZFSPOOL/hb@2_2 | zfs recv $ZFSPOOL/backup\r\nmd5sum /$ZFSPOOL/hb/large_file /$ZFSPOOL/backup/large_file\r\n\r\nI have called it on two different systems, more than 50 times. Sleep <5 seconds everything OK> = 5 seconds error.\r\n\r\nSystem 1 Debian 8 ZFS 0.6.5.7-8-jessie\r\nSystem 2 Ubuntu 16.04 ZFS 0.6.5.6-0ubuntu10\r\n\r\n\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"datiscum"}}