{"push_id":2412907776,"size":62,"distinct_size":1,"ref":"refs/heads/issue116_lite","head":"71de46e28428b759fb7050f9304bca2b8807bfdd","before":"2181d91960bc2511232a732a41507e1bf5720b78","commits":[{"sha":"53287e83a5a4ffa7d6c5d20b831856e690e569fb","author":{"name":"Brendon Humphrey","email":"fbc49bc0e77fc903b8f183fd0ec4b681a5647ce8@mac.com"},"message":"Eliminate use of uninitialized variable.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/53287e83a5a4ffa7d6c5d20b831856e690e569fb"},{"sha":"3867c0cf1d21056c8b5353ef6ea9bd176a97baf3","author":{"name":"Tom Caputi","email":"102fdea76bf4221d11361f9f327e7c452294a203@datto.com"},"message":"Encryption Stability and On-Disk Format Fixes\n\nThe on-disk format for encrypted datasets protects not only the\nencrypted and authenticated blocks themselves, but also the order and\ninterpretation of these blocks. In order to make this work while\nmaintaining the ability to do raw sends, the indirect bps maintain a\nsecure checksum of all the MACs in the block below it along with a few\nother fields that determine how the data is interpreted.\n\nUnfortunately, the current on-disk format erroneously includes some\nfields which are not portable and thus cannot support raw sends. It is\nnot possible to easily work around this issue due to a separate and\nmuch smaller bug which causes indirect blocks for encrypted dnodes to\nnot be compressed, which conflicts with the previous bug. In addition,\nthe current code generates incompatible on-disk formats on big endian\nand little endian systems due to an issue with how block pointers are\nauthenticated. Finally, raw send streams do not currently include\ndn_maxblkid when sending both the metadnode and normal dnodes which\nare needed in order to ensure that we are correctly maintaining the\nportable objset MAC.\n\nThis patch zero's out the offending fields when computing the bp MAC\nand ensures that these MACs are always calculated in little endian\norder (regardless of the host system's byte order). This patch also\nregisters an errata for the old on-disk format, which we detect by\nadding a \"version\" field to newly created DSL Crypto Keys. We allow\ndatasets without a version (version 0) to only be mounted for read so\nthat they can easily be migrated. We also now include dn_maxblkid in\nraw send streams to ensure the MAC can be maintained correctly.\n\nFixes ZOL # 6845\n\nSigned-off-by: Tom Caputi <tcaputi@datto.com>\n\nFix for ZOL # 6916\n\nWhen performing zil_claim() at pool import time, it is important that\nencrypted datasets set os_next_write_raw before writing to the\nzil_header_t. This prevents the code from attempting to\nre-authenticate the objset_phys_t when it writes it out, which is\nunnecessary because the zil_header_t is not protected by either objset\nMAC and impossible since the keys aren't loaded yet. Unfortunately,\none of the code paths did not set this flag, which causes failed\nASSERTs during 'zpool import -F'. This patch corrects this issue.\n\nSigned-off-by: Tom Caputi <tcaputi@datto.com>\n\nNow with data in the test pool\n\nCorrect readonly during mount\n\nRaw sends must be able to decrease nlevels\n\nCurrently, when a raw zfs send file includes a DRR_OBJECT record\nthat would decrease the number of levels of an existing object,\nthe object is reallocated with dmu_object_reclaim() which\ncreates the new dnode using the old object's nlevels. For non-raw\nsends this doesn't really matter, but raw sends require that\nnlevels on the receive side match that of the send side so that\nthe checksum-of-MAC tree can be properly maintained. This patch\ncorrects the issue by freeing the object completely before\nallocating it again in this case.\n\nThis patch also corrects several issues with dnode_hold_impl()\nand related functions that prevented dnodes (particularly\nmulti-slot dnodes) from being reallocated properly due to\nthe fact that existing dnodes were not being fully cleaned up\nwhen they were freed.\n\nSigned-off-by: Tom Caputi <tcaputi@datto.com>\nSigned-off-by: Brian Behlendorf <behlendorf1@llnl.gov>\n\nChange os->os_next_write_raw to work per txg\n\nCurrently, os_next_write_raw is a single boolean used for determining\nwhether or not the next call to dmu_objset_sync() should write out the\nobjset_phys_t as a raw buffer. Since the boolean is not associated\nwith a txg, the work simply happens during the next txg, which is not\nnecessarily the correct one. In the current implementation this issue\nwas misdiagnosed, resulting in a small hack in dmu_objset_sync() which\nseemed to resolve the problem.\n\nThis patch changes os_next_write_raw to be an array of booleans, one\nfor each txg in TXG_OFF and removes the hack.\n\nSigned-off-by: Tom Caputi <tcaputi@datto.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/3867c0cf1d21056c8b5353ef6ea9bd176a97baf3"},{"sha":"09114377ba3b7f32d95fbfad0df5ed9be47d814c","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Allow zfs mount -o readonly=on option\n\nMore options could be supported beyond readonly.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/09114377ba3b7f32d95fbfad0df5ed9be47d814c"},{"sha":"fc444cc6ee78458ebc99e0e750400f2af1bcbfb8","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Revert \"pageoutv2 must clear the dirty bit when committing the UPL\"\n\nThis reverts commit 771dad6c9f0461345f68cff4ad99b79273c84d4e.\n\nThis causes Assertion failed: !(flags & UPL_PAGEOUT)\n\nvery early in testing.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/fc444cc6ee78458ebc99e0e750400f2af1bcbfb8"},{"sha":"d98dc24b88eefece19a0007b7f7c80c6bf6c506e","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Add missing tests files.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/d98dc24b88eefece19a0007b7f7c80c6bf6c506e"},{"sha":"4ca968ca867b89f47c45321e93deb2f5288c2020","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Handle compressed buffers in __dbuf_hold_impl()\n\n    In __dbuf_hold_impl(), if a buffer is currently syncing and is still\n    referenced from db_data, a copy is made in case it is dirtied again in\n    the txg.  Previously, the buffer for the copy was simply allocated with\n    arc_alloc_buf() which doesn't handle compressed or encrypted buffers\n    (which are a special case of a compressed buffer).  The result was\n    typically an invalid memory access because the newly-allocated buffer\n    was of the uncompressed size.\n\n    This commit fixes the problem by handling the 2 compressed cases,\n    encrypted and unencrypted, respectively, with arc_alloc_raw_buf() and\n    arc_alloc_compressed_buf().\n\n    Although using the proper allocation functions fixes the invalid memory\n    access by allocating a buffer of the compressed size, another unrelated\n    issue made it impossible to properly detect compressed buffers in the\n    first place.  The header's compression flag was set to ZIO_COMPRESS_OFF\n    in arc_write() when it was possible that an attached buffer was actually\n    compressed.  This commit adds logic to only set ZIO_COMPRESS_OFF in\n    the non-ZIO_RAW case which wil handle both cases of compressed buffers\n    (encrypted or unencrypted).\n\n    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>\n    Signed-off-by: Tim Chase <tim@chase2k.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/4ca968ca867b89f47c45321e93deb2f5288c2020"},{"sha":"81c6857644d2d24b2356c5e51d8ebe3e4459df9b","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Last surviving hardlink would forget it was once a hardlink\n\nOnce the zp->z_links has returned back to 1, we would no longer correct the\nparentid of the item, and it would be returned as \"last used\" parentid\nwhich may have been deleted.\n\nThis corrects the panic from:\n\ndate > s ; mkdir a ; ln s a/d.lock ; rm a/d.lock ; rmdir a ; rm s\n\nIt is interesting to note that SA_ZPL_PARENT can be left \"incorrect\" on\ndisk by ZFS.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/81c6857644d2d24b2356c5e51d8ebe3e4459df9b"},{"sha":"e364899f30929ba23f9823651a1035b479ae56b0","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Correct l2arc_write_buffers\n\nThe merge of l2arc_write_buffers and crypto was not entirely clean, so\nwe need to correct it here.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/e364899f30929ba23f9823651a1035b479ae56b0"},{"sha":"fb28036ba20be9edcd2dccccd42918b0293dd35c","author":{"name":"ilovezfs","email":"ae0e3b7da98ed54394b6ebcb2d19e7f25d54d7b1@icloud.com"},"message":"Tag zfs-1.7.1\n\nMETA file updated.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/fb28036ba20be9edcd2dccccd42918b0293dd35c"},{"sha":"36f9b649964d4a8a9fa0c6a63817485290462fd7","author":{"name":"George Melikov","email":"3b8e10480754ba93523db5dc4626a5f001e54bb7@gmelikov.ru"},"message":"8435 zpool.1m and zfs.1m: minor cleanup\n\nReviewed by: Matt Ahrens <mahrens@delphix.com>\nReviewed by: Yuri Pankov <yuripv@gmx.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/36f9b649964d4a8a9fa0c6a63817485290462fd7"},{"sha":"dd4d0de1de5a4aa9ac9c79eb1ffcf8b9decb2124","author":{"name":"Prakash Surya","email":"904e529b3e1d9a5db38845125535b2686730807f@delphix.com"},"message":"8558 lwp_create() returns EAGAIN on system with more than 80K ZFS filesystems\n\nReviewed by: George Wilson <george.wilson@delphix.com>\nReviewed by: Sebastien Roy <sebastien.roy@delphix.com>\n\nOn a system with more than 80K ZFS filesystems, we've seen cases where\nlwp_create() will start to fail by returning EAGAIN. The problem being,\nfor each of those 80K ZFS filesystems, a taskq will be created for each\ndataset as part of the ZIL for each dataset.\n\nFor each of these taskq's, a kernel thread will be created which results\nin 24KB being allocated for each thread. With enough of these 24KB\nallocations, we eventually exhaust the memory region set aside for these\nallocations. Currently, segkpsize is set to a value of 2GB, which means\nwe can only support about 80K filesystems; 2GB / 24KB = ~80K.\n\nThe lwp_create() failure comes into play due to the fact that LWP\ncreation also allocates 24KB from this same region of memory. Thus, if\nwe've exhausted this region of memory due to the number of ZIL taskq's,\nthere won't be any memory avaible to allow the call to lwp_create() to\nsucceed.\n\nThe solution taken by this patch is to modify the taskq used by the ZIL\nto be a per-pool taskq, instead of a per-dataset taskq.\n\nAdditionally, to increase observability of when we fail to dispatch to\nthe taskq from within zil_clean(), the taskq internals have also been\nslightly reworked to track and expose the \"nomem\" kstat field for\nnon-dynamic taskqs; previously this field only existed for dynamic\ntaskq's. This should allow one to relatively easily determine if\nzil_clean() is unable to successfully dispatch, which could impact the\nperformance of spa_sync().","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/dd4d0de1de5a4aa9ac9c79eb1ffcf8b9decb2124"},{"sha":"2fe73d18bc0d7e84e3098231557c8c719f51e024","author":{"name":"Andriy Gapon","email":"b24d11afb16644e3ff52469bebd6e863d3282d4e@FreeBSD.org"},"message":"8569 problem with inline functions in abd.h\n\nReviewed by: Matt Ahrens <mahrens@delphix.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/2fe73d18bc0d7e84e3098231557c8c719f51e024"},{"sha":"2319fca29f7d092956ef4a66fea0e8cb8d42d1e0","author":{"name":"Prakash Surya","email":"904e529b3e1d9a5db38845125535b2686730807f@delphix.com"},"message":"8585 improve batching done in zil_commit()\n\nReviewed by: Brad Lewis <brad.lewis@delphix.com>\nReviewed by: Matt Ahrens <mahrens@delphix.com>\nReviewed by: George Wilson <george.wilson@delphix.com>\nApproved by: Dan McDonald <danmcd@joyent.com>\n\nThis removes the OsX specific ZIL changes, and updates our zil.c to that\nof IllumOS. If the deadlocks return, we may need to re-instate the OsX\nspecific work around.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/2319fca29f7d092956ef4a66fea0e8cb8d42d1e0"},{"sha":"2c25be8f8fb39a575abd1dc80cdc36ee2f27d05e","author":{"name":"Jorgen Lundman","email":"d35e73bea9a417812828c4aa53bf599fbd2eccd6@lundman.net"},"message":"Remove python2.6 specific path from zfs-tester","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/2c25be8f8fb39a575abd1dc80cdc36ee2f27d05e"},{"sha":"5379044bedcb14c4fc9e348653b4ad94e69ca9d6","author":{"name":"Alan Somers","email":"98dace73f9f4aeb631994692d06377058a93a391@gmail.com"},"message":"8473 scrub does not detect errors on active spares\nReviewed by: Andy Stormont <astormont@racktopsystems.com>\nReviewed by: Matt Ahrens <mahrens@delphix.com>\nReviewed by: George Wilson <george.wilson@delphix.com>\nApproved by: Richard Lowe <richlowe@richlowe.net>\n\nThis updates vdev_mirror.c to IllumOS at this point.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/5379044bedcb14c4fc9e348653b4ad94e69ca9d6"},{"sha":"698b3472527d8e20b24d907a38be9741556d2a78","author":{"name":"Alan Somers","email":"98dace73f9f4aeb631994692d06377058a93a391@gmail.com"},"message":"8567 Inconsistent return value in zpool_read_label\nReviewed by: Prakash Surya <prakash.surya@delphix.com>\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nApproved by: Robert Mustacchi <rm@joyent.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/698b3472527d8e20b24d907a38be9741556d2a78"},{"sha":"19555c874c563b96db8a7691028f631d2ebe571f","author":{"name":"Prakash Surya","email":"904e529b3e1d9a5db38845125535b2686730807f@delphix.com"},"message":"8602 remove unused \"dp_early_sync_tasks\" field from \"dsl_pool\" structure\nReviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nApproved by: Robert Mustacchi <rm@joyent.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/19555c874c563b96db8a7691028f631d2ebe571f"},{"sha":"3bc41864c377de00ef9afa8dc3d528dbf6b542c4","author":{"name":"LOLi","email":"4fc42463de6b5982a0025d9fdef0bc7dc8d328df@users.noreply.github.com"},"message":"8648 Fix range locking in ZIL commit codepath\n\nReviewed by: Igor Kozhukhov <igor@dilos.org>\nReviewed by: Matt Ahrens <mahrens@delphix.com>\nReviewed by: Andriy Gapon <avg@FreeBSD.org>\nReviewed by: Alexander Motin <mav@FreeBSD.org>\n\nThis is a port of the following ZFS on Linux commit:\n\n    commit f763c3d1df569a8d6b60bcb5e95cf07aa7a189e6\n    Author: LOLi <loli10K@users.noreply.github.com>\n    Date:   Mon Aug 21 17:59:48 2017 +0200\n\n        Fix range locking in ZIL commit codepath\n\n        Since OpenZFS 7578 (1b7c1e5) if we have a ZVOL with logbias=throughput\n        we will force WR_INDIRECT itxs in zvol_log_write() setting itx->itx_lr\n        offset and length to the offset and length of the BIO from\n        zvol_write()->zvol_log_write(): these offset and length are later used\n        to take a range lock in zillog->zl_get_data function: zvol_get_data().\n\n        Now suppose we have a ZVOL with blocksize=8K and push 4K writes to\n        offset 0: we will only be range-locking 0-4096. This means the\n        ASSERTion we make in dbuf_unoverride() is no longer valid because now\n        dmu_sync() is called from zilog's get_data functions holding a partial\n        lock on the dbuf.\n\n        Fix this by taking a range lock on the whole block in zvol_get_data().\n\n        Reviewed-by: Chunwei Chen <tuxoko@gmail.com>\n        Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>\n        Signed-off-by: loli10K <ezomori.nozomu@gmail.com>\n\nDifferences with ZFS on Linux commit:\n\n 1. This port doesn't pull over the new \"zfstest\" test that was added to\n    Linux. I wasn't able to get the test script to reproduce the issue,\n    so I opted to pull in the fix, and manually test it instead.\n\n 2. The `ASSERT3U`'s added in `dmu_sync` in the Linux version were\n    removed for this port. It appears as though the `struct rl`\n    definition is not available in userspace, which cause compilation\n    errors.\n\n 3. The ZFS on Linux patch had changes to `zvol_discard` and\n    `zvol_request`; these functions don't exist on delphixos/illumos, so\n    these changes were dropped when porting this over.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/3bc41864c377de00ef9afa8dc3d528dbf6b542c4"},{"sha":"e1012b7a836ea5a042406664689e6620efd2a443","author":{"name":"Alan Somers","email":"98dace73f9f4aeb631994692d06377058a93a391@gmail.com"},"message":"8081 Compiler warnings in zdb\n\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nReviewed by: Prakash Surya <prakash.surya@delphix.com>\n\nFix compiler warnings in zdb.  With these changes, FreeBSD can compile\nzdb with all compiler warnings enabled save -Wunused-parameter.\n\nusr/src/cmd/zdb/Makefile.com\n\tEnable some compiler warnings\n\nusr/src/cmd/zdb/zdb.c\nusr/src/cmd/zdb/zdb_il.c\nusr/src/uts/common/fs/zfs/sys/sa.h\nusr/src/uts/common/fs/zfs/sys/spa.h\n\tFix numerous warnings, including:\n\t* const-correctness\n\t* shadowing global definitions\n\t* signed vs unsigned comparisons\n\t* missing prototypes, or missing static declarations\n\t* unused variables and functions\n\t* Unreadable array initializations\n\t* Missing struct initializers\n\nusr/src/cmd/zdb/zdb.h\n\tAdd a header file to declare common symbols\n\nusr/src/lib/libzpool/common/sys/zfs_context.h\nusr/src/uts/common/fs/zfs/arc.c\nusr/src/uts/common/fs/zfs/dbuf.c\nusr/src/uts/common/fs/zfs/spa.c\nusr/src/uts/common/fs/zfs/txg.c\n\tAdd a function prototype for zk_thread_create, and ensure that every\n\tcallback supplied to this function actually matches the prototype.\n\nusr/src/cmd/ztest/ztest.c\nusr/src/uts/common/fs/zfs/sys/zil.h\nusr/src/uts/common/fs/zfs/zfs_replay.c\nusr/src/uts/common/fs/zfs/zvol.c\n\tAdd a function prototype for zil_replay_func_t, and ensure that\n\tevery function of this type actually matches the prototype.\n\nusr/src/uts/common/fs/zfs/sys/refcount.h\n\tChange FTAG so it discards any constness of __func__, necessary\n\tsince existing APIs expect it passed as void *.","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/e1012b7a836ea5a042406664689e6620efd2a443"},{"sha":"f6b6a4d16c5f6f4b7caaea9611dd4ebb84cfe681","author":{"name":"loli10K","email":"d09de01b90020805bb8815e2eb117437d0a32572@gmail.com"},"message":"8713 Buffer overflow in dsl_dataset_name()\n\nReviewed by: Prakash Surya <prakash.surya@delphix.com>\nReviewed by: Matthew Ahrens <mahrens@delphix.com>\nApproved by: Dan McDonald <danmcd@joyent.com>","distinct":false,"url":"https://api.github.com/repos/openzfsonosx/zfs/commits/f6b6a4d16c5f6f4b7caaea9611dd4ebb84cfe681"}]}
{"actor":{"display_login":"lundman"}}