{"action":"opened","issue":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/201","repository_url":"https://api.github.com/repos/markfasheh/duperemove","labels_url":"https://api.github.com/repos/markfasheh/duperemove/issues/201/labels{/name}","comments_url":"https://api.github.com/repos/markfasheh/duperemove/issues/201/comments","events_url":"https://api.github.com/repos/markfasheh/duperemove/issues/201/events","html_url":"https://github.com/markfasheh/duperemove/issues/201","id":326181002,"number":201,"title":"ignores tails of files","user":{"login":"kilobyte","id":48801,"avatar_url":"https://avatars0.githubusercontent.com/u/48801?v=4","gravatar_id":"","url":"https://api.github.com/users/kilobyte","html_url":"https://github.com/kilobyte","followers_url":"https://api.github.com/users/kilobyte/followers","following_url":"https://api.github.com/users/kilobyte/following{/other_user}","gists_url":"https://api.github.com/users/kilobyte/gists{/gist_id}","starred_url":"https://api.github.com/users/kilobyte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kilobyte/subscriptions","organizations_url":"https://api.github.com/users/kilobyte/orgs","repos_url":"https://api.github.com/users/kilobyte/repos","events_url":"https://api.github.com/users/kilobyte/events{/privacy}","received_events_url":"https://api.github.com/users/kilobyte/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-24T15:43:37Z","updated_at":"2018-05-24T15:43:37Z","closed_at":null,"author_association":"NONE","body":"Turns out **duperemove** completely skips blocks smaller than 128KB (or whatever is configured), which results in leaving an undeduped and fragmented part of every file whose size is not an exact multiply of 128KB.\r\n\r\nI have a backup server that holds copies of ~50 systems, 24ish snapshots of each.  Several months ago I switched to zstd, I wanted to recompress old backups to zstd as well.  As a test, I start with `*/*/bin/bash`: a nice compressible binary, only a few distinct versions in the whole set.  Defrag succeeded, after dedupe it (measured with **compsize**) took slightly _more_ than before.  Uh oh.\r\n\r\nThen, I repeated the same with dash:\r\n```\r\n[/srv/backups]# compsize */*/bin/dash */*/*/bin/dash\r\nProcessed 857 files, 70 regular extents (1549 refs), 0 inline.\r\nType       Perc     Disk Usage   Uncompressed Referenced\r\nTOTAL       84%      3.1M         3.6M          93M\r\nnone       100%      2.4M         2.4M          34M\r\nzlib        53%       60K         112K         224K\r\nzstd        56%      680K         1.1M          58M\r\n[/srv/backups]# btrfs fi def -r -czstd */*/bin/dash */*/*/bin/dash\r\n[/srv/backups]# compsize */*/bin/dash */*/*/bin/dash\r\nProcessed 857 files, 857 regular extents (857 refs), 0 inline.\r\nType       Perc     Disk Usage   Uncompressed Referenced\r\nTOTAL       55%       51M          93M          93M\r\nzstd        55%       51M          93M          93M\r\n```\r\nNot a single shared extent!\r\n\r\nCommon versions of dash have sizes of 113KB and 117KB, thus are below the 128KB threshold.  None are small enough to fit into an inline extent which would make them ineligible for dedupe, thus there's no reason they couldn't be — and indeed, manually calling /usr/bin/btrfs-extent-same does the trick.\r\n\r\nNot only these tails take a substantial amount of wasted space (24*50 * half of 128KB on the average, per file, is a lot), but also increase fragmentation as almost every file would naturally be written in contiguous space after deduplication is finished — but now that tail has to be stored separately."}}
{"actor":{"display_login":"kilobyte"}}