title: How do I perform a streaming character conversion?
body: <p>I have data stored on disk in files that are far too big to store in main memory.</p>
  
  <p>I want to stream this data from the disk into a data processing pipeline via <code>iconv</code>, like this:</p>
  
  <pre><code>zcat myfile | iconv -f L1 -t UTF-8 | # rest of the pipeline goes here
  </code></pre>
  
  <p>Unfortunately, I'm seeing iconv buffer the entire file in memory until it's exhausted before outputting any data. This means that I'm using up all of my main memory on a blocking operation in a pipeline whose memory footprint is otherwise minimal.</p>
  
  <p>I've tried calling iconv like this:</p>
  
  <pre><code>stdbuf -o 0 iconv -f L1 -t UTF-8
  </code></pre>
  
  <p>But it looks like iconv is managing the buffering internally itself - it's nothing to do with the Linux pipe buffer.</p>
  
  <p>I'm seeing this with the binary that's packaged with gblic 2.6 and 2.7 in Arch Linux, and I've deplicated it with glibc 2.5 in Debian.</p>
  
  <p>Is there some way around this? I know that streaming character conversions are not simple, but I'd have thought that such a commonly used unix tool would work in streams; it's not at all rare to work with files that won't fit in main memory. Would I have to roll my own binary linked to <code>libiconv</code>?</p>