Take the 2-minute tour ×

I am pulling data from a bzip2 stream within a C application. As chunks of data come out of the decompressor, they can be written to stdout:

fwrite(buffer, 1, length, stdout);

This works great. I get all the data when it is sent to stdout.

Instead of writing to stdout, I would like to process the output from this statement internally in one-line-chunks: a string that is terminated with a newline character \n.

Do I write the output of the decompressor stream to another buffer, one character at a time, until I hit a newline, and then call the per-line processing function? Is this slow and is there a smarter approach? Thanks for your advice.


Thanks for your suggestions. I ended up creating a pair of buffers that store the remainder (the "stub" at the end of an output buffer) at the beginning of a short line buffer, each time I pass through the output buffer's worth of data.

I loop through the output buffer character by character and process a newline-line's worth of data at a time. The newline-less remainder gets allocated and assigned, and copied to the next stream's line buffer. It seems like realloc is less expensive than repeated malloc-free statements.

Here's the code I came up with:

char bzBuf[BZBUFMAXLEN];
int bzError, bzNBuf;
char *bzBufRemainder = NULL;
int bzBufPosition, bzLineBufPosition;

bzFp = BZ2_bzReadOpen(&bzError, *fp, 0, 0, NULL, 0); /* http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#bzcompress-init */ 

if (bzError != BZ_OK) {
    BZ2_bzReadClose(&bzError, bzFp);   
    fprintf(stderr, "\n\t[gchr2] - Error: Bzip2 data could not be retrieved\n\n");
    return -1;          

bzError = BZ_OK;
bzLineBufPosition = 0;
while (bzError == BZ_OK) {

    bzNBuf = BZ2_bzRead(&bzError, bzFp, bzBuf, sizeof(bzBuf));

    if (bzError == BZ_OK || bzError == BZ_STREAM_END) {
        if (bzBufRemainder != NULL) {
            /* fprintf(stderr, "copying bzBufRemainder to bzLineBuf...\n"); */
            strncpy(bzLineBuf, bzBufRemainder, strlen(bzBufRemainder)); /* leave out \0 */
            bzLineBufPosition = strlen(bzBufRemainder);

        for (bzBufPosition = 0; bzBufPosition < bzNBuf; bzBufPosition++) {
            bzLineBuf[bzLineBufPosition++] = bzBuf[bzBufPosition];
            if (bzBuf[bzBufPosition] == '\n') {
                bzLineBuf[bzLineBufPosition] = '\0'; /* terminate bzLineBuf */

                /* process the line buffer, e.g. print it out or transform it, etc. */
                fprintf(stdout, "%s", bzLineBuf);

                bzLineBufPosition = 0; /* reset line buffer position */
            else if (bzBufPosition == (bzNBuf - 1)) {
                bzLineBuf[bzLineBufPosition] = '\0';
                if (bzBufRemainder != NULL)
                    bzBufRemainder = (char *)realloc(bzBufRemainder, bzLineBufPosition);
                    bzBufRemainder = (char *)malloc(bzLineBufPosition);
                strncpy(bzBufRemainder, bzLineBuf, bzLineBufPosition);

if (bzError != BZ_STREAM_END) {
    BZ2_bzReadClose(&bzError, bzFp);
    fprintf(stderr, "\n\t[gchr2] - Error: Bzip2 data could not be uncompressed\n\n");
    return -1;  
} else {   
    BZ2_bzReadGetUnused(&bzError, bzFp, 0, 0);
    BZ2_bzReadClose(&bzError, bzFp);

bzBufRemainder = NULL;

I really appreciate everyone's help. This is working nicely.

share|improve this question
strtok could save you from checking the newline on every chracter in the buffer. –  Tumas Oct 13 '10 at 11:30
strtok is what I tried previously, but it doesn't work for me and I lose data, because the output buffer coming out of the bzip2 decompressor is split across a line. I lose that bit at the end. Perhaps I'm using it incorrectly. Is there a use of strtok that allows keeping the newline-less "stub", gluing that stub to the start of the next output buffer chunk? –  Alex Reynolds Oct 13 '10 at 11:33
No, you're right, strtok doesn't quite fit here. You lose that bit if you use strtok and it's probably easier to check every char. I don't know any clever way of saving that newline-less 'stub' for later usage except for doing it manually, as Opera has described. Sorry, for misleading you a bit :) –  Tumas Oct 13 '10 at 11:56
add comment

3 Answers

up vote 1 down vote accepted

This would be easy to do using C++'s std::string, but in C it takes some code if you want to do it efficiently (unless you use a dynamic string library).

char *bz_read_line(BZFILE *input)
    size_t offset = 0;
    size_t len = CHUNK;  // arbitrary
    char *output = (char *)xmalloc(len);
    int bzerror;

    while (BZ2_bzRead(&bzerror, input, output + offset, 1) == 1) {
        if (offset+1 == len) {
            len += CHUNK;
            output = xrealloc(output, len);
        if (output[offset] == '\n')

        output[offset] = '\0';  // strip trailing newline
    else if (bzerror != BZ_STREAM_END) {
        return NULL;

    return output;

(Where xmalloc and xrealloc handle errors internally. Don't forget to free the returned string.)

This is almost an order of magnitude slower than bzcat:

lars@zygmunt:/tmp$ wc foo
 1193  5841 42868 foo
lars@zygmunt:/tmp$ bzip2 foo
lars@zygmunt:/tmp$ time bzcat foo.bz2 > /dev/null

real    0m0.010s
user    0m0.008s
sys     0m0.000s
lars@zygmunt:/tmp$ time ./a.out < foo.bz2 > /dev/null

real    0m0.093s
user    0m0.044s
sys     0m0.020s

Decide for yourself whether that's acceptable.

share|improve this answer
I have a bunch of bz2 streams concatenated in one very large file. I'm trying to write a self-contained application to unpack one stream among many. This is very helpful, thanks! –  Alex Reynolds Oct 13 '10 at 18:03
add comment

I don't think there's a smarter approach (except finding an automata library that already does this for you). Be careful with allocating proper size for the "last line" buffer: if it cannot handle arbitrary length and the input comes from something accessible to third parties, it becomes a security risk.

share|improve this answer
add comment

I think you should copy chunks of characters to another buffer until the latest chunk you write contains a new line character. Then you can work on the whole line.

You can save the rest of the buffer (after the '\n') into a temporary and then create a new line from it.

share|improve this answer
add comment

Your Answer


