I have a compressed tarball (eg foo.tar.gz) that I wish to extract all the files from, but the files within the tarball are not compressed. That is to say, the contents of foo.tar.gz are un-compressed txt files.

There is not enough space on my filesystem to extract the files directly, so I wish to extract these files and immediately compress them as they are written to the disk. I can't simply extract the files and then gzip the extracted files because, as I've said, there's not enough space on the filesystem. I would also like to ensure that the original filenames, including their directories are faithfully preserved on disk. So if one of the files in the tarball is /a/b/c/foo.txt, at the end of the process I would like to have /a/b/c/foo.txt.gz

How can I accomplish this?


It won't be fast, especially for a large tarball with lots of files, but in bash you can do this:

tar -tzf tarball.tgz | while IFS= read -r file; do
    tar --no-recursion -xzf tarball.tgz -- "$file"
    gzip -- "$file"

The first tar command extracts the names of the files in the tarball, and passes those names to a while read ... loop. The file name is then passed to a second tar command that extracts just that file, which is then compressed before the next file is extracted. The --no-recursion flag is used so trying to extract a directory doesn't extract all the files under that directory, which is what tar would normally do.

You'll still need enough free space to store somewhat more than the original size of the compressed tarball.

  • But won't tar -t extract the archive (maybe not saved on the disk) in the first place to get the file names? I was thinking to tell him something like: tar -xzvf foo.tar.gz | parallel gzip, but tar -v output needs to be slightly formatted. – Bichoy Apr 25 '15 at 19:22
  • -t just reads the file names from the tar file. It won't extract anything. I thought about something like your parallel gzip, but I don't recall when in the extraction process tar emits the file name - if it's not completely on disk when the file name is emitted and gzip starts, there's no way to guarantee the output files will have their entire original contents after they're compressed. – Andrew Henle Apr 25 '15 at 19:29
  • for -t I guess if the tarball is zipped, it gets extracted first before the filenames are listed (even though it is not actually written to disk). I am not entirely sure of that, but better check. I find that -t takes very long time on large compressed tars ... maybe I am wrong anyway. – Bichoy Apr 25 '15 at 20:08
  • @Bichoy - Absolutely true - the tarball does have to be extracted multiple times in the solution I posted. Which is why I said it'd be slow in my answer. The original extraction gets piped to a gzip process forked off by tar (at least in the version I have, but there's no reason why tar couldn't handle the decompression internally without forking). And then the tarball has to get uncompressed again for each and every file that needs to be extracted. Your example is much faster. I just don't know offhand if it's safe - it depends on when tar emits the filename, which I don't know. – Andrew Henle Apr 25 '15 at 20:21
  • Makes sense, and looks like it should work perfectly. – John Dibling Apr 25 '15 at 21:14

Your Answer

