{"push_id":2979653763,"size":26,"distinct_size":26,"ref":"refs/heads/wip","head":"abafc2da6d9d4aa99fba27dd2a0d06318c412252","before":"4c7dd26ea19cbe010ac426d661aa930b68d395c3","commits":[{"sha":"64bc38c37ae07254e92ea4f3eb926acfd561d9bc","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Work around circular imports\n\nInstead of doing second level imports, just import btrfs at the\nbeginning always.\n\nThe trick is to not use things before they get imported. An import\nbtrfs.ctree can fail because ctree is not in there, because the current\nmodule gets imported because of an earlier import in btrfs __init__.py\netc...","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/64bc38c37ae07254e92ea4f3eb926acfd561d9bc"},{"sha":"baa63d9182b35787e383a1461f74e87abc10aa3e","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Work around Python 3.7 struct.format\n\nhttps://docs.python.org/3/whatsnew/3.7.html#changes-in-the-python-api\n  \"The struct.Struct.format type is now str instead of bytes.\"\n\nSo, put in a wrapper function to handle both scenarios.\n\nThanks Holger Hoffstätte for reporting the issue.\n\nGithub: Fixes: #13","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/baa63d9182b35787e383a1461f74e87abc10aa3e"},{"sha":"f7cb2174f63cecc899334127b3800b42f3ff73aa","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"FileSystem: use context manager to close fd\n\nAhem. This is quite useful when doing something with btrfs as part of a\nlonger running program... We would be leaving open file descriptors\nbehind otherwise...\n\nNow we can use the with statement:\n\n    with btrfs.FileSystem('/') as fs:\n        print(fs.fs_info())","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/f7cb2174f63cecc899334127b3800b42f3ff73aa"},{"sha":"a87f06b2ac722fd6ccbb5d0ab7e5fabaeb437253","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Add comment about balance struct names\n\nThe btrfs_balance_args and btrfs_balance_progress are actual structs\nthat exist in the kernel source, they're not just helpers to get the\ndirty work done in here. So, they shouldn't be prefixed with an\nunderscore.\n\nHowever, we also have a function with the name balance_progress in\nhere. Argh! That's what you get when stripping off prefixes and moving\npart of the name of C structs into the python lib hierarchy...","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/a87f06b2ac722fd6ccbb5d0ab7e5fabaeb437253"},{"sha":"361707dd717f081d6628256c9816606d861ca082","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Add the sync ioctl\n\nThis one is useful for test scripts. If we write data to files, we can\ncall sync to be sure the metadata we get back from a search reflects the\nchanges we just did.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/361707dd717f081d6628256c9816606d861ca082"},{"sha":"d2014a343f9b98a0ff68bbdb1432c043fe34408d","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Add the fideduperange ioctl\n\nThis commit implements calling the deduplication ioctl, which started\nout as a btrfs specific ioctl once (FILE_EXTENT_SAME), but was moved to\nthe generic filesystem code and renamed FIDEDUPERANGE. The same number\n(based on the 0x94 btrfs ioctl number) is still used, but others can\nimplement it now. An example is XFS.\n\nWhen using deduplication, the hard part of the story is not really how\nto actually tell btrfs to deduplicate a range of bytes in some files.\nThe hard part is to actually find out where similar parts of files could\nbe, in order to have some ranges to feed to the ioctl.\n\nAn example program is also added. But, in order to use that example, you\nalready have to know which files have identical parts in them.\n\nHere's an example how to use the example:\n\n    dd if=/dev/urandom of=src bs=1MiB count=1\n    dd if=src of=dst1 bs=1MiB count=1\n    dd if=src of=dst2 seek=1 bs=1MiB count=1\n\n    sync\n\n    ./show_file.py src\n    ./show_file.py dst1\n    ./show_file.py dst2\n\nRight now, you should see that while the 1MiB of data in the files is\nthe same, every file has its own extents with data.\n\nNow, let's try to free up 2MiB of disk space by having dst1 and dst2\nreflink to the extent in src instead:\n\n    ./file_dedupe_range.py src 0 1048576 dst1 0 dst2 1048576\n\nThe show_file.py output should now show you that dst1 and dst2 refer to\nthe same data extent that is used by the file src.\n\nThe first version of a patch for dedupe support was submitted by\ndarkbringer via IRC. Thanks!","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/d2014a343f9b98a0ff68bbdb1432c043fe34408d"},{"sha":"532a5e83af525e1b98d319a3bccd1f9ab6bdbd91","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Allow pretty printing ioctl objects\n\nThis small change allows us to also pretty print the content of ioctl\nresult objects like FsInfo, DevStats and SpaceInfo. I'm using this in\nthe new tutorial documentation.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/532a5e83af525e1b98d319a3bccd1f9ab6bdbd91"},{"sha":"5d36a4293022156f090569bd716035a1ad317603","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Introduce ItemNotFoundError\n\nThe handling of fs.block_group(vaddr) was a bit stupid actually, because\nif no block group is found at that vaddr, the return block_groups[0]\nexplodes with the IndexError because the list is empty...\n\nWith a more specific error, the calling code wouldn't accidentally also\ncatch an IndexError because of a bug in other code, like the search_v2\nwhich it's using.\n\nWe have to inherit the ItemNotFoundError from IndexError to prevent\nbreaking checks in existing code that uses this.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/5d36a4293022156f090569bd716035a1ad317603"},{"sha":"6ae1b34775689d0abeaa57d4d11499112b641d45","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"SpaceInfo: hide type, profile from pretty printer\n\nWhile pretty printing SpaceInfo objects with the new object pretty\nprinter, I do not want to see all the derived attributes which are\nnot part of the data we get from the kernel.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/6ae1b34775689d0abeaa57d4d11499112b641d45"},{"sha":"88d719ba6381f5ff339d14d542dddd8ffade6372","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Rename space type / profile descriptions util\n\nblock_group_type_str and block_group_profile_str were confusing, since\nthey did something different than block_group_flags_str (which prints\nflags like usual, e.g. 'DATA|RAID0'. Instead of this, the functions\nprinted the description that is used for SpaceInfo. Also, global reserve\nis not a block group type.\n\nSo, the block_group_*_str handle flags, and the space_info_*\ndescriptions:\n\n    >>> btrfs.utils.block_group_flags_str(0x12)\n    'SYSTEM|RAID1'\n    >>> btrfs.utils.block_group_type_str(0x12)\n    'SYSTEM'\n    >>> btrfs.utils.block_group_profile_str(0x12)\n    'RAID1'\n    >>> btrfs.utils.space_type_description(0x12)\n    'System'\n    >>> btrfs.utils.space_profile_description(0x12)\n    'RAID1'\n    >>> btrfs.utils.space_profile_description(0)\n    'single'\n    >>> btrfs.utils.block_group_profile_str(0)\n    'none'\n\nThe last one says 'none' because when printing flags, single profile is\nnot displayed, e,g 'DATA' instead of 'DATA|SINGLE'.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/88d719ba6381f5ff339d14d542dddd8ffade6372"},{"sha":"aac589fb39c5e32bae31f2132536cf762b22f818","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"flags_str: add unrecognized flags to output\n\nThis makes it easier to spot bugs, and it will at least display\nsomething if new flags are added which this code does not recognize yet.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/aac589fb39c5e32bae31f2132536cf762b22f818"},{"sha":"0c020f7b989532db3b59f95583cfc85ba930dc9d","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"SpaceInfo: Pretty print attributes\n\n    >>> btrfs.utils.pretty_print(fs.space_info())\n    -\n      <btrfs.ioctl.SpaceInfo>\n      used_bytes: 693.42GiB\n      total_bytes: 733.00GiB\n      flags: Data, single\n    -\n      <btrfs.ioctl.SpaceInfo>\n      used_bytes: 112.00KiB\n      total_bytes: 32.00MiB\n      flags: System, DUP\n    -\n      <btrfs.ioctl.SpaceInfo>\n      used_bytes: 1.63GiB\n      total_bytes: 3.50GiB\n      flags: Metadata, DUP\n    -\n      <btrfs.ioctl.SpaceInfo>\n      used_bytes: 0.00B\n      total_bytes: 512.00MiB\n      flags: GlobalReserve, single","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/0c020f7b989532db3b59f95583cfc85ba930dc9d"},{"sha":"1e1757bdee05d8c81d68b70b7394506d7fd7df05","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Add the get features ioctl\n\n    >>> import btrfs\n    >>> fs = btrfs.FileSystem('/mnt/tutorial')\n    >>> features = fs.features()\n    >>> btrfs.utils.pretty_print(features)\n    <btrfs.ioctl.Features>\n    compat_flags: none\n    incompat_flags:\n    mixed_backref|mixed_groups|extended_iref|raid56|skinny_metadata\n    compat_ro_flags: free_space_tree|free_space_tree_valid\n    >>> features.incompat_flags & btrfs.FEATURE_INCOMPAT_MIXED_GROUPS\n    4\n\nMaybe we can add an even more convenient way to check for specific\nfeatures in the future. I couldn't decide on how to do that yet.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/1e1757bdee05d8c81d68b70b7394506d7fd7df05"},{"sha":"b3772feb4c53b873dd5e57ee4fca86d711d5e631","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Also pretty print dictionaries\n\nFor custom stringify of dictionary key values, put a function inside the\nobject having attribute name appended with '_key_str' as name:\n\n    class Foo(object):\n        def __init__(self, sound):\n            self.foo = {'qar' : sound, 'qaz': sound}\n\n        def foo_key_str(k):\n            return k.replace('q', 'b')\n\n    pretty_print(Foo('blaat'))\n\nThis should pretty print as:\n\n    <Foo>\n    foo:\n      bar:\n        blaat\n      baz:\n        blaat","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/b3772feb4c53b873dd5e57ee4fca86d711d5e631"},{"sha":"014f19c4fbd7ac721b8abdaca03a42ad4a037ae6","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Add volumes.py, handling device / chunk logic\n\nIn the kernel, this is in volumes.c, so keep it the same.\n\nThe kernel has an ugly array with attributes per raid type, called\nraid_attrs, or raid_array. It's copied here as-is now, since I don't\nwant to start 'fixing' things in python-btrfs, but keep naming the same\nas in the kernel.\n\nWe need these properties to build a few functions which can calculate\ninteresting things about chunks and block groups.\n\nI added a new field, nparity, which is needed information to make this\nhappen. In the kernel code, the same information and calculations are\nhidden in numerous if statements and open coded special cases all over\nthe place instead. There are plans to clean this up. If this happens,\nthe code in here will adapt to it.\n\nRight now, there's five of them already:\n\n1. chunk_to_dev_extent_length(chunk) and\n2. chunk_length_to_dev_extent_length(flags, num_stripes, chunk_length)\n\nGiven a chunk metadata object, it calculates what the length of the\ndevice extent is (they're all the same size). The second function does\nthe same, when providing the needed data explicitely.\n\n3. dev_extent_length_to_chunk_length(flags, num_stripes, stripe_size)\n\nThis is the inverse of the first two functions.\n\n4. chunk_to_raw_parity_bytes(chunk)\n\nGiven a chunk metadata object, this calculates how many bytes are\nreserved for storing parity data for RAID56 profiles. The number is\nrelevant to understand how much raw disk space that is allocated but not\nused is actually not usable for data, because it's reserved for parity.\n\n5. block_group_profile_raw_data_ratio(flags)\n\nThis function returns how many times the actual data is replicated on\ndisk given block group flags as input. E.g. for RAID1 this is 2, but for\nRAID5, this is just 1. The actual data is stored once in case of RAID5,\nand the redundancy is done using parity blocks, not data itself.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/014f19c4fbd7ac721b8abdaca03a42ad4a037ae6"},{"sha":"3084d124679f3b186738d362a04763050ee5de99","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"ctree: FileSystem: add mixed_groups()\n\nThis is a convenience function to find out if the filesystem has mixed\ngroups (data and metadata in the same block group).","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/3084d124679f3b186738d362a04763050ee5de99"},{"sha":"301de18f7b9d3d0fbd77f339668f5d1e8611777e","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"FsUsage, a detailed usage calculator\n\n--- >8 ---\n\nBackground\n\nThe utils module currently has some functionality to determine space\nusage and disk usage, which is used by the munin and nagios monitoring\nplugin. That code is one of the first things I wrote for this library.\nHowever, it suffers from being very naive, and not supporting the RAID56\ntypes.\n\nI had another look at this, and for RAID56 this means getting an idea\nabout how much allocated space is reserved for parity. For total\nnumbers, there's a quick way to do this, since on device level,\nallocated usage included parity, and totals of 'spaces' do not include\nthe parity bytes. So, taking the difference and assuming it's the bytes\nreserved for parity is enough.\n\n--- >8 ---\n\nBut, I wanted to have a bit more detailed report of usage, since there's\nquite some scenarios in which this is useful.\n\nWhen creating an FsUsage object with btrfs.utils.FsUsage(fs), the\nresulting object will contain:\n\nFsUsage - Total, unallocated, allocated, parity and used disk space for\n |        the btrfs physical address space. Total and used bytes for the\n |        virtual address space.\n \\_ DevUsage - Total and allocated physical bytes per device.\n |  \\_ DevSpaceUsage - Allocated and parity bytes per space per device.\n \\_ RawSpaceUsage - Allocated, parity and used raw bytes per space.\n \\_ BlockGroupTypeUsage - Allocated, parity and used bytes per block\n |                        group type (system, data, metadata)\n \\_ VirtualSpaceUsage - Total and used virtual bytes per space.\n \\_ VirtualBlockGroupTypeUsage - Total and used virtual bytes per block\n                                 group type (system, data, metadata)\n\nWith this info, it's possible to construct output like the btrfs dev\nusage and btrfs fi usage show, and many more overviews. But, it's also\npossible to use it for e.g. creating a smart balance algorithm that does\nthe minimal amount of work to get the filesystem balanced after changing\ndisks.\n\nHint: try creating a FsUsage object and then use\nbtrfs.utils.pretty_print to show it:\n\n    import btrfs\n    fs = btrfs.FileSystem('/')\n    btrfs.utils.pretty_print(fs.usage())\n\nSince the amount of code involved is a bit more than I anticipated, I\nput it in its own new module, fs_usage, instead of in utils.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/301de18f7b9d3d0fbd77f339668f5d1e8611777e"},{"sha":"4d5b37bc04195c104db885331250b045725cbeba","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"utils: add some size constants\n\nAs seen in kernel code. These are a less error prone than using literal\nnumbers, or computing them on the spot.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/4d5b37bc04195c104db885331250b045725cbeba"},{"sha":"fc5cfc07dd4b0d9b7cad1ad0e0ace094ffd602df","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Add wasted space calculator\n\nBy using a slightly simplified copy of the chunk allocator logic from\nthe kernel, we can just simulate allocating all available raw disk\nspace, and see whatever remains as left-over unallocatable space.\n\nThe result is two sets of information:\n1. 'soft' wasted space: This is the unallocatable space when continuing\n   to use the filesystem in its current state, adding new data and\n   metadata.\n2. 'hard' wasted space: The unallocatable space when the filesystem\n   would be optimally balanced.\n\nThe second set of numbers is disk space that should not be displayed as\nunallocated raw disk space by any tool, because it simply can't be used\nwith the current allocation profiles.\n\nThe difference between the soft and hard wasted space is raw disk space\nthan can be reclaimed when using btrfs balance to move data around over\nthe devices in the filesystem.\n\nNote 1: The results are approximates. When using this, don't complain\nabout the algorithm reporting 1 or 2 GiB of unallocatable space. Use it\nto discover that you have many GiBs or TiBs of unallocatable space.\n\nNote 2: An FsUsage object is initialized with a FileSystem object.  But,\nsince python is ducktyped, it's also possible to construct a fake object\nand calculate wasted space for whatever combination of disks we would\nwant to know about.","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/fc5cfc07dd4b0d9b7cad1ad0e0ace094ffd602df"},{"sha":"3788f4e1bae465afca351ae97667f34ad93152dd","author":{"name":"Hans van Kranenburg","email":"8800578b51f022c8d8adb9606a8b3db4fedbdac6@knorrie.org"},"message":"Adapt munin plugin to new FsUsage\n\n* Leverage new FsUsage info, also supporting RAID 0,10,5,6 now\n* Explicitely handle block group types (data, metadata, etc)","distinct":true,"url":"https://api.github.com/repos/knorrie/python-btrfs/commits/3788f4e1bae465afca351ae97667f34ad93152dd"}]}
{"actor":{"display_login":"knorrie"}}