{"push_id":1452799748,"size":16,"distinct_size":16,"ref":"refs/heads/rotor_vector","head":"e39f69127fd383d829600dfea1f12b272feb4ba3","before":"1a8fcb4634fa9494cc6e8a35bbf0ba400635f8c9","commits":[{"sha":"02730c333c4fce8c0ead1f7f3404caca5cdb3936","author":{"name":"Brian Behlendorf","email":"a858f2314cfaacf8f584edaf19860167d645669b@llnl.gov"},"message":"Use cstyle -cpP in `make cstyle` check\n\nEnable picky cstyle checks and resolve the new warnings.  The vast\r\nmajority of the changes needed were to handle minor issues with\r\nwhitespace formatting.  This patch contains no functional changes.\r\n\r\nNon-whitespace changes are as follows:\r\n\r\n* 8 times ; to { } in for/while loop\r\n* fix missing ; in cmd/zed/agents/zfs_diagnosis.c\r\n* comment (confim -> confirm)\r\n* change endline , to ; in cmd/zpool/zpool_main.c\r\n* a number of /* BEGIN CSTYLED */ /* END CSTYLED */ blocks\r\n* /* CSTYLED */ markers\r\n* change == 0 to !\r\n* ulong to unsigned long in module/zfs/dsl_scan.c\r\n* rearrangement of module_param lines in module/zfs/metaslab.c\r\n* add { } block around statement after for_each_online_node\r\n\r\nReviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>\r\nReviewed-by: Håkan Johansson <f96hajo@chalmers.se>\r\nSigned-off-by: Brian Behlendorf <behlendorf1@llnl.gov>\r\nCloses #5465","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/02730c333c4fce8c0ead1f7f3404caca5cdb3936"},{"sha":"bc2324624c5700c4363b9f777ddb667d9cdc7cc5","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Turn metaslab class rotor into a vector, in preparation for selecting\nmetaslab depending on allocation size.\n\nRenaming of mc_rotor -> mc_rotorv and mc_aliquot -> mc_aliquotv just to\nmake sure all references are found.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/bc2324624c5700c4363b9f777ddb667d9cdc7cc5"},{"sha":"ca86df0d3889b0f0c10d5b939c1c88c36f809070","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Allocate using vector of rotors.\n\nIn a pool that consist of e.g. a small but fast SSD-based mirror and\na large but long-latency HDD-based RAIDZn, it is useful to have the\nmetadata, as well as very small files, stored on the SSD.  This\nis handled in this patch by selecting the storage based on the size\nof the allocation.\n\nThis is done by using a vector of rotors, each of which is associated with\nmetaslab groups of each kind of storage.  If the preferred group is full,\nattempts are made to fill slower groups.  Better groups are not attempted -\nrationale is that an almost full filesystem shall not spill large-size\ndata into the expensive SSD vdev, since that will not be reclaimable\nwithout deleting the files.  Better then to consider the filesystem full\nwhen the large-size storage is full.\n\nOne could also think of having e.g. a 3-level storage:  Mirror SSD for\nreally small records, mirror HDD for medium-size records and raidzn HDD\nfor the bulk of data.\n\nSome performance numbers:\n\nTested on three separate pools each consisting of a 20 GB SSD\npartition and a 100 GB HDD partition, from the same disks.  The HDD is\n2 TB in total.)  SSD raw reads: 350 MB/s, HDD raw reads 132 MB/s.\n\nThe filesystems were filled to ~60 % with a random directory tree,\neach with random 0-6 subdirectories and 0-100 files, maximum depth 8.\nThe filesize was random 0-400 kB.  The fill script was run with 10\ninstances in parallel, aborted at ~the same size.  The performance\nvariations below are much larger than the filesystem fill differences.\n\nThe patch does not handle the 'inactive' case very well (it begins by\nfilling nonrotating storage).  Setting 0 is actually the original\n7a27ad0 commit.  Setting 8000 and 16000 is the value for\nzfs_mixed_slowsize_threshold, i.e. below which size data is stored\nusing rotor[0] (nonrotating SSD), instead of rotor[1] (rotating HDD).\n\n-              Setting 8000  Setting 16000  Setting 0\n-              ------------  -------------  ------------\n\nTotal # files  305666        304439         308962\nTotal size     75334 kB      75098 kB       75231 kB\n\nAs per 'zfs iostat -v':\n\nTotal alloc    71.8 G        71.6 G         71.7 G\nSSD alloc      3.34 G        3.41 G         3.71 G\nHDD alloc      68.5 G        68.2 G         68.0\n\nTime for 'find' and 'zpool scrub' after fresh 'zfs import':\n\nfind           5.6 s         5.5 s          42 s\nscrub          560 s         560 s          1510 s\n\nTime for serial 'find | xargs -P 1 md5sum' and\nparallell 'find | xargs -P 4 -n 10 md5sum'.\n(Only first 10000 files each)\n\n-P 1 md5sum    129 s         122 s          168 s\n-P 4 md5sum    182 s         150 s          187 s\n(size summed)  2443 MB       2499 MB        2423 MB\n\n---\n\nMust decide on rotor vector index earlier, in order to do space accounting\nper rotor category.\n\nSet metaslab group rotor category at end of vdev_open().\nThen we do it before the group is activated.\nCan then get rid of metaslab_group_rotor_insert/remove also.\n\nFixups:\n\nMoving the metaslab_group_set_rotor_category() up.\n\nztest did an (unreproducible) failure with stack trace of\nspa_tryimport() -> ... vdev_load() -> ... vdev_metaslab_init() ->\nmetaslab_init() -> ... metaslab_class_space_update() that failed its\nASSERT.  Inspection showed that vdev_metaslab_init() would soon call\nmetaslab_group_activate(), i.e. we need to assign mg_nrot.\n(Hopefully, vdev_open was called earlier...?)\n\nAssign mg_nrot no matter how vdev_open() fails.\n\nIn dealing with yet another spa_tryimport failure.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/ca86df0d3889b0f0c10d5b939c1c88c36f809070"},{"sha":"28cdf899d6123357b4e5c701650906e7a2125a2a","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Adjust the dspace reported from metaslab_class such that we look full when\none rotor vector category (SSD or HDD) becomes full.  This since large\ncontent for the HDD cannot spill into the SSD, and as we do not want to\nspill small content that should be on the SSD onto the HDD.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/28cdf899d6123357b4e5c701650906e7a2125a2a"},{"sha":"d1e91e1e5b0535ee15552f605549a8db6d57a7de","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Instead of refusing to at all use better rotor categories than selected, try them when all other failed.\nThe implementation is not very pretty.  And the better ones should be tried in reverse order.\nAt least does get it a bit further through ztest.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/d1e91e1e5b0535ee15552f605549a8db6d57a7de"},{"sha":"319cd3f42dff4f4e17c524c392de5172eafee88d","author":{"name":"Haakan Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Keep track of mixed nonrotational (ssd+hdd) devices.\n\nOnly mirrors are mixed.  If a pool consist of several mixed vdevs, it is\nmixed if all vdevs are either mixed, or fully nonrotational.\n\nDo not mark as mixed when fully nonrotational.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/319cd3f42dff4f4e17c524c392de5172eafee88d"},{"sha":"ce8d9d28c22cd21d4554a60a29f19ddf140ed06c","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Introducing a pool property 'rotorvector'.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/ce8d9d28c22cd21d4554a60a29f19ddf140ed06c"},{"sha":"6f2a94728434d241a9c8ef0d096a3017b9b21dd8","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Pass the value of the pool property 'rotorvector' to metaslab_class_create().","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/6f2a94728434d241a9c8ef0d096a3017b9b21dd8"},{"sha":"f04c3e1cc6fd015096a57ade7131ca95866a9e46","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Keep information in the metaslab class to know in which rotor vector to place vdevs.\nTodo: The guid list should not have a fixed length but be dynamic.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/f04c3e1cc6fd015096a57ade7131ca95866a9e46"},{"sha":"b5ff11447d5a5140ba609a1419aa6474b6f01773","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Routine to parse a rotor vector configuration.\n\nProbably the configuration should be split into multiple properties, one per rotor, and limit separate from types.\n\nFormat:\n\n[spec<=limit];spec\n\nwith spec being a comma-separated list of vdev-guids and generic type specifiers: ssd, ssd-raidz, mixed, hdd, and hdd-raidz.\n\nlimit gives the limit in kbytes of which allocations are allowed within the rotor vector.  The last rotor has no limit.  (And vdevs which are not matched by the guids or the generic types are placed in the last rotor.)\n\nFor the moment things are a bit difficult due to the 32 character limit of the comment field.  That is also why the limit is implicitly expressed in units of 1024 bytes.\n\nExample:\n\nssd<=4;mixed<=64;123,hdd\n\nAllocations less than 4 kbytes are allocated on an ssd-only vdev (mirror or not).\nAllocations less than 64 kbytes end up on ssd/hdd mixed (mirrors, such raidz makes no sense).\nOther allocations end up on remaining disks.\n123 represents av vdev guid.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/b5ff11447d5a5140ba609a1419aa6474b6f01773"},{"sha":"fd4c6ed54bb20282940788d0644f37711a51dd65","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"More rotors, such that more categories of allocation targets can be differentiated.\nAssign rotor vector categories based on pool property config instead of module parameters.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/fd4c6ed54bb20282940788d0644f37711a51dd65"},{"sha":"40219d1f43350c59c8109da39259fb2cf32ebcfa","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Capability to give different allocation thresholds for data and metadata.\n\nAlso include blocks with level > 0 i metadata category (from #5182).\n\nExample:\n\nzpool set \"rotorvector=ssd<=meta:4;mixed<=64;123,hdd\" <poolname>\n\npure ssd drive takes metadata <= 4 kB\nmixed (mirror) takes data (or metadata) <= 64 kB\nothers (hdd) takes remainder\n\nExample II:\n\nzpool set \"rotorvector=ssd<=meta:128,4;mixed<=64;123,hdd\" <poolname>\n\npure ssd drive takes metadata <= 128 kB and data <= 4 kB\nmixed (mirror) takes data <= 64 kB  [this metadata already taken by ssd]\nothers (hdd) takes remainder","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/40219d1f43350c59c8109da39259fb2cf32ebcfa"},{"sha":"845e720688d8ede9f66758e156ebc21b8eb6526c","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"zloop testing by randomly assign vdevs as rotational.\n\nThe random assignment is a dirty hack.  Reason for this is that it has to be\nset before the device open finishes, or the rotor vector index will be\nassigned on whatever initial nonrot value the device has.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/845e720688d8ede9f66758e156ebc21b8eb6526c"},{"sha":"69f6c7b1d4750b86a1d415106fd208f100fbb19b","author":{"name":"Haakan Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"Pass info to zpool cmd whether devices are solid state, or rotational.\n\nInfo is passed in ZPOOL_CONFIG_VDEV_STATS_EX -> ZPOOL_CONFIG_VDEV_KIND.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/69f6c7b1d4750b86a1d415106fd208f100fbb19b"},{"sha":"cb66f3d40ee003e06043b8f0c9b0c06b93df1123","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"zpool iostat -k shows kind of vdevs: ssd, hdd or mixed.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/cb66f3d40ee003e06043b8f0c9b0c06b93df1123"},{"sha":"e39f69127fd383d829600dfea1f12b272feb4ba3","author":{"name":"Haakan T Johansson","email":"53c029c7af7a8e43bcfa5933e6cb4009b5fef48f@chalmers.se"},"message":"zpool iostat tells which rotor vector each vdev is part of.\n\nFor the time being, abusing the -k flag.","distinct":true,"url":"https://api.github.com/repos/inkdot7/zfs/commits/e39f69127fd383d829600dfea1f12b272feb4ba3"}]}
{"actor":{"display_login":"inkdot7"}}