{"push_id":2540624972,"size":7,"distinct_size":7,"ref":"refs/heads/illumos-9075-next","head":"a61b0cc459236040845ada9058d94699ce4fba6b","before":"ef4c6c63839fab0024726d6726f5213d8e3d9875","commits":[{"sha":"5e097c67f14e32de67c62752e728e9fbabb920dc","author":{"name":"Matthew Ahrens","email":"0962b5d2f5c1117073cf6f982c188f029b03d89e@delphix.com"},"message":"OpenZFS 9443 - panic when scrub a v10 pool\n\nWhile expanding stored pools, we ran into a panic using an old pool.\n\nSteps to reproduce:\n\n    $ sudo zpool create -o version=2 test c2t1d0\n    $ sudo cp /etc/passwd /test/foo\n    $ sudo zpool attach test c2t1d0 c2t2d0\n\nWe'll get this panic:\n\n    ffffff000fc0e5e0 unix:real_mode_stop_cpu_stage2_end+b27c ()\n    ffffff000fc0e6f0 unix:trap+dc8 ()\n    ffffff000fc0e700 unix:cmntrap+e6 ()\n    ffffff000fc0e860 zfs:dsl_scan_visitds+1ff ()\n    ffffff000fc0ea20 zfs:dsl_scan_visit+fe ()\n    ffffff000fc0ea80 zfs:dsl_scan_sync+1b3 ()\n    ffffff000fc0eb60 zfs:spa_sync+435 ()\n    ffffff000fc0ec20 zfs:txg_sync_thread+23f ()\n    ffffff000fc0ec30 unix:thread_start+8 ()\n\nThe problem is a bad trap accessing a NULL pointer. We're looking for\nthe dp_origin_snap of a dsl_pool_t, but version 2 didn't have that. The\nsystem will go into a reboot loop at this point, and the dump won't be\naccessible except by removing the cache file from within the recovery\nenvironment.\n\nThis impacts any sort of scrub or resilver on version <11 pools, e.g.:\n\n    $ zpool create -o version=10 test c2t1d0\n    $ zpool scrub test\n\nAuthored by: Matthew Ahrens <mahrens@delphix.com>\nReviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>\nReviewed by: George Wilson <george.wilson@delphix.com>\nReviewed by: Andriy Gapon <avg@FreeBSD.org>\nReviewed by: Igor Kozhukhov <igor@dilos.org>\nApproved by: Dan McDonald <danmcd@joyent.com>\nPorted-by: Brian Behlendorf <behlendorf1@llnl.gov>\n\nOpenZFS-issue: https://www.illumos.org/issues/9443\nOpenZFS-commit: https://github.com/openzfs/openzfs/commit/010eed29\nCloses #7501","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/5e097c67f14e32de67c62752e728e9fbabb920dc"},{"sha":"64c1dcefe38a62d304177475a2ef77111cec612a","author":{"name":"Paul Dagnelie","email":"250714fe6017fdb0fb7ffa578021c44333317c9a@delphix.com"},"message":"OpenZFS 9421, 9422 - zdb show possibly leaked objects\n\n9421 zdb should detect and print out the number of \"leaked\" objects\n9422 zfs diff and zdb should explicitly mark objects that are on\n     the deleted queue\n\nIt is possible for zfs to \"leak\" objects in such a way that they are not\nfreed, but are also not accessible via the POSIX interface. As the only\nway to know that this is happened is to see one of them directly in a\nzdb run, or by noting unaccounted space usage, zdb should be enhanced to\ncount these objects and return failure if some are detected.\n\nWe have access to the delete queue through the zfs_get_deleteq function;\nwe should call it in dump_znode to determine if the object is on the\ndelete queue. This is not the most efficient possible method, but it is\nthe simplest to implement, and should suffice for the common case where\nthere few objects on the delete queue.\n\nAlso zfs diff and zdb currently traverse every single dnode in a dataset\nand tries to figure out the path of the object by following it's parent.\nWhen an object is placed on the delete queue, for all practical purposes\nit's already discarded, it's parent might not exist anymore, and another\nobject might now have the object number that belonged to the parent.\nWhile all of the above makes sense, when trying to figure out the path\nof an object that is on the delete queue, we can run into issues where\neither it is impossible to determine the path because the parent is\ngone, or another dnode has taken it's place and thus we are returned a\nwrong path.\n\nWe should therefore avoid trying to determine the path of an object on\nthe delete queue and mark the object itself as being on the delete queue\nto avoid confusion. To achieve this, we currently have two ideas:\n\n1. When putting an object on the delete queue, change it's parent object\n   number to a known constant that means NULL.\n\n2. When displaying objects, first check if it is present on the delete\n   queue.\n\nAuthored by: Paul Dagnelie <pcd@delphix.com>\nReviewed by: Matt Ahrens <matt@delphix.com>\nReviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>\nApproved by: Matt Ahrens <mahrens@delphix.com>\nPorted-by: Brian Behlendorf <behlendorf1@llnl.gov>\n\nOpenZFS-issue: https://illumos.org/issues/9421\nOpenZFS-issue: https://illumos.org/issues/9422\nOpenZFS-commit: https://github.com/openzfs/openzfs/commit/45ae0dd9ca\nCloses #7500","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/64c1dcefe38a62d304177475a2ef77111cec612a"},{"sha":"2663d69176ab02c302db1f9d9b9ba2e415736290","author":{"name":"Pavel Zakharov","email":"fda3bfdda442bcfad2d20b42ba5f02aa3b7a2f96@delphix.com"},"message":"OpenZFS 8961 - SPA load/import should tell us...\n\n why it failed\n\nReviewed by: George Wilson <george.wilson@delphix.com>\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nReviewed by: Andrew Stormont <andyjstormont@gmail.com>\nApproved by: Dan McDonald <danmcd@joyent.com>\nOpenZFS-issue: https://illumos.org/issues/8961\nOpenZFS-commit: https://github.com/openzfs/openzfs/commit/418079e0\nPorted-by: Tim Chase <tim@chase2k.com>\nSigned-off-by: Tim Chase <tim@chase2k.com>\n\nProblem\n=======\n\nWhen we fail to open or import a storage pool, we typically don't\nget any additional diagnostic information, just \"no pool found\" or\n\"can not import\".\n\nWhile there may be no additional user-consumable information, we should\nat least make this situation easier to debug/diagnose for developers\nand support.  For example, we could start by using `zfs_dbgmsg()`\nto log each thing that we try when importing, and which things\nfailed. E.g. \"tried uberblock of txg X from label Y of device Z\". Also,\nwe could log each of the stages that we go through in `spa_load_impl()`.\n\nSolution\n========\n\nFollowing the cleanup to `spa_load_impl()`, debug messages have been\nadded to every point of failure in that function. Additionally,\ndebug messages have been added to strategic places, such as\n`vdev_disk_open()`.","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/2663d69176ab02c302db1f9d9b9ba2e415736290"},{"sha":"f97b0af7d847432a35e3c8e843e170c0d6befc20","author":{"name":"Pavel Zakharov","email":"fda3bfdda442bcfad2d20b42ba5f02aa3b7a2f96@delphix.com"},"message":"OpenZFS 8962 - zdb should work on non-idle pools\n\nCurrently `zdb` consistently fails to examine non-idle pools as it\nfails during the `spa_load()` process. The main problem seems to be\nthat `spa_load_verify()` fails as can be seen below:\n\n    $ sudo zdb -d -G dcenter\n    zdb: can't open 'dcenter': I/O error\n\n    ZFS_DBGMSG(zdb):\n    spa_open_common: opening dcenter\n    spa_load(dcenter): LOADING\n    disk vdev '/dev/dsk/c4t11d0s0': best uberblock found for spa dcenter. txg 40824950\n    spa_load(dcenter): using uberblock with txg=40824950\n    spa_load(dcenter): UNLOADING\n    spa_load(dcenter): RELOADING\n    spa_load(dcenter): LOADING\n    disk vdev '/dev/dsk/c3t10d0s0': best uberblock found for spa dcenter. txg 40824952\n    spa_load(dcenter): using uberblock with txg=40824952\n    spa_load(dcenter): FAILED: spa_load_verify failed [error=5]\n    spa_load(dcenter): UNLOADING\n\nThis change makes `spa_load_verify()` a dryrun when ran from\n`zdb`. This is done by creating a global flag in zfs and then setting\nit in `zdb`.\n\nReviewed by: George Wilson <george.wilson@delphix.com>\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nReviewed by: Andy Stormont <astormont@racktopsystems.com>\nOpenZFS-issue: https://illumos.org/issues/8962\nOpenZFS-commit: https://github.com/openzfs/openzfs/commit/180ad792\nPorted-by: Tim Chase <tim@chase2k.com>\nSigned-off-by: Tim Chase <tim@chase2k.com>","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/f97b0af7d847432a35e3c8e843e170c0d6befc20"},{"sha":"724e56e3af8a067229a2c378a5d4a5c42f1752fe","author":{"name":"Pavel Zakharov","email":"fda3bfdda442bcfad2d20b42ba5f02aa3b7a2f96@delphix.com"},"message":"OpenZFS 9075 - Improve ZFS pool import/load...\n\n process and corrupted pool recovery\n\nReviewed by: George Wilson <george.wilson@delphix.com>\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nReviewed by: Andrew Stormont <andyjstormont@gmail.com>\nOpenZFS-issue: https://illumos.org/issues/9075\nOpenZFS-commit: https://github.com/openzfs/openzfs/commit/619c0123\nPorted-by: Tim Chase <tim@chase2k.com>\nSigned-off-by: Tim Chase <tim@chase2k.com>\n\nSome work has been done lately to improve the debugability of the ZFS pool\nload (and import) process. This includes:\n\n\t7638 Refactor spa_load_impl into several functions\n\t8961 SPA load/import should tell us why it failed\n\t7277 zdb should be able to print zfs_dbgmsg's\n\nTo iterate on top of that, there's a few changes that were made to make the\nimport process more resilient and crash free. One of the first tasks during the\npool load process is to parse a config provided from userland that describes\nwhat devices the pool is composed of. A vdev tree is generated from that config,\nand then all the vdevs are opened.\n\nThe Meta Object Set (MOS) of the pool is accessed, and several metadata objects\nthat are necessary to load the pool are read. The exact configuration of the\npool is also stored inside the MOS. Since the configuration provided from\nuserland is external and might not accurately describe the vdev tree\nof the pool at the txg that is being loaded, it cannot be relied upon to safely\noperate the pool. For that reason, the configuration in the MOS is read early\non. In the past, the two configurations were compared together and if there was\na mismatch then the load process was aborted and an error was returned.\n\nThe latter was a good way to ensure a pool does not get corrupted, however it\nmade the pool load process needlessly fragile in cases where the vdev\nconfiguration changed or the userland configuration was outdated. Since the MOS\nis stored in 3 copies, the configuration provided by userland doesn't have to be\nperfect in order to read its contents. Hence, a new approach has been adopted:\nThe pool is first opened with the untrusted userland configuration just so that\nthe real configuration can be read from the MOS. The trusted MOS configuration\nis then used to generate a new vdev tree and the pool is re-opened.\n\nWhen the pool is opened with an untrusted configuration, writes are disabled\nto avoid accidentally damaging it. During reads, some sanity checks are\nperformed on block pointers to see if each DVA points to a known vdev;\nwhen the configuration is untrusted, instead of panicking the system if those\nchecks fail we simply avoid issuing reads to the invalid DVAs.\n\nThis new two-step pool load process now allows rewinding pools accross\nvdev tree changes such as device replacement, addition, etc. Loading a pool\nfrom an external config file in a clustering environment also becomes much\nsafer now since the pool will import even if the config is outdated and didn't,\nfor instance, register a recent device addition.\n\nWith this code in place, it became relatively easy to implement a\nlong-sought-after feature: the ability to import a pool with missing top level\n(i.e. non-redundant) devices. Note that since this almost guarantees some loss\nof data, this feature is for now restricted to a read-only import.\n\nPorting notes:\n\nZTS:\n    Fix 'make dist' target in zpool_import\n\n    The maximum path length allowed by tar is 99 characters.  Several\n    of the new test cases exceeded this limit resulting in them not\n    being included in the tarball.  Shorten the names slightly.\n\n    Set/get tunables using accessor functions.\n\n    Get last synced txg via the \"zfs_txg_history\" mechanism.\n\n    Clear zinject handlers in cleanup for import_cache_device_replaced\n    and import_rewind_device_replaced in order that the zpool can be\n    exported if there is an error.\n\n    Increase FILESIZE to 8G in zfs-test.sh to allow for a larger\n    ext4 file system to be created on ZFS_DISK2.  Also, there's\n    no need to partition ZFS_DISK2 at all.  The partitioning had\n    already been disabled for multipath devices.  Among other things,\n    the partitioning steals some space from the ext4 file system,\n    makes it difficult to accurately calculate the paramters to\n    parted and can make some of the tests fail.\n\n    Increase FS_SIZE and FILE_SIZE in the zpool_import test\n    configuration now that FILESIZE is larger.\n\n    Write more data in order that device evacuation take lonnger in\n    a couple tests.\n\n    Use mkdir -p to avoid errors when the directory already exists.\n\n    Remove use of sudo in import_rewind_config_changed.","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/724e56e3af8a067229a2c378a5d4a5c42f1752fe"},{"sha":"e34a76fa8f38e8a5ff86b411fde8d9ea25a32aaa","author":{"name":"Tim Chase","email":"5ee0edb9e2229c0838f1959779f1949031de0123@chase2k.com"},"message":"Clear vdev_faulted if ZPOOL_CONFIG_AUX_STATE...\n\n is not set to \"external\".\n\nZoL supports \"zpool export -f\" (force fault), which can be combined\nwith \"-t\" (temporary fault; don't persist across export/import) and\ncauses a MOS configuration to be set with ZPOOL_CONFIG_FAULTED=1\nand without ZFS_CONFIG_AUX_STATE set at all.  In this case, the\npreviously-offlined vdev should be imported in an on-line state and.\nClearing the \"vdev_faulted\" flag causes the import to treat the\ndevice as on-line.  Typically, resilver will catch it up based on\nits DTL.\n\nSigned-off-by: Tim Chase <tim@chase2k.com>","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/e34a76fa8f38e8a5ff86b411fde8d9ea25a32aaa"},{"sha":"a61b0cc459236040845ada9058d94699ce4fba6b","author":{"name":"Tim Chase","email":"5ee0edb9e2229c0838f1959779f1949031de0123@chase2k.com"},"message":"Streamline the zpool_import tests\n\nDon't create an ext4 file system atop $DEV_DISKDIR/$DISK2.\nThere's likely to not be sufficient space for it to succeed.\nInstead, simply create the vdev files in the directory where it\nwould have been mounted.\n\nSigned-off-by: Tim Chase <tim@chase2k.com>","distinct":true,"url":"https://api.github.com/repos/dweeezil/zfs/commits/a61b0cc459236040845ada9058d94699ce4fba6b"}]}
{"actor":{"display_login":"dweeezil"}}