<p>I'm trying to copy the contents of a large (~350 files, ~40MB total) directory from a Kubernetes pod to my local machine. I'm using the technique described <a href="https://kubernetes.io/docs/reference/kubectl/cheatsheet/#copying-files-and-directories-to-and-from-containers" rel="nofollow noreferrer">here</a>.</p>
<p>Sometimes it succeeds, but very frequently the standard output piped to the <code>tar xf</code> command on my host appears to get truncated. When that happens, I see errors like:
<code>&lt;some file in the archive being transmitted over the pipe&gt;: Truncated tar archive</code></p>
<p>The files in the source directory don't change. The file in the error message is usually different (ie: it appears to be truncated in a different place).</p>
<p>For reference (copied from the document lined to above), this is the analog to what I'm trying to do (I'm using a different pod name and directory names):
<code>kubectl exec -n my-namespace my-pod -- tar cf - /tmp/foo | tar xf - -C /tmp/bar</code></p>
<p>After running it, I expect the contents of my local /tmp/bar to be the same as those in the pod.</p>
<p>However, more often than not, it fails. My current theory (I have a very limited understanding of how kubectl works, so this is all speculation) is that when kubectl determines that the tar command has completed, it terminates -- regardless of whether or not there are remaining bytes in transit (over the network) containing the contents of standard output.</p>
<p>I've tried various combinations of:</p>
<ol>
<li><a href="https://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html" rel="nofollow noreferrer">stdbuf</a></li>
<li>Changing tar's <a href="https://www.gnu.org/software/tar/manual/html_node/Blocking-Factor.html" rel="nofollow noreferrer">blocking factor</a></li>
<li>Making the command take longer to run (by adding <code>&amp;&amp; sleep &lt;x&gt;</code>)</li>
</ol>
<p>I'm not going to list all combinations I've tried, but this is an example that uses everything:
<code>kubectl exec -n my-namespace my-pod -- stdbuf -o 0 tar -b 1 -c -f - -C /tmp/foo . &amp;&amp; sleep 2 | tar xf - -C /tmp/bar</code></p>
<p>There are combinations of that command that I can make work pretty reliably. For example, forgetting about <code>stdbuf</code> and <code>-b 1</code> and just sleeping for 100 seconds, ie:
<code>kubectl exec -n my-namespace my-pod -- tar -c -f - -C /tmp/foo . &amp;&amp; sleep 100 | tar xf - -C /tmp/bar</code></p>
<p>But even more experimentation led me to believe that the block size of tar (512 bytes, I believe?) was still too large (the arguments of <code>-b</code> are a count of blocks, not the size of those blocks). This is the command I'm using for now:
<code>kubectl exec -n my-namespace my-pod -- bash -c 'dd if=&lt;(tar cf - -C /tmp/foo .) bs=16 &amp;&amp; sleep 10' | tar xf - -C /tmp/bar</code></p>
<p>And yes, I HAD to make bs that small and sleep &quot;that big&quot; to make it work. But this at least gives me two variables I can mess with. I did find that if I set bs=1, I didn't have to sleep... but it took a LONG time to move all the data (one byte at a time).</p>
<p>So, I guess my questions are:</p>
<ol>
<li>Is my theory that kubectl truncates standard output after it determines the command given to <code>exec</code> has finished correct?</li>
<li>Is there a better solution to this problem?</li>
</ol>

Ed MacDonald
<p>Maybe you haven't been specific enough for <a href="/questions/tagged/kubectl" class="post-tag" title="show questions tagged &#39;kubectl&#39;" aria-label="show questions tagged &#39;kubectl&#39;" rel="tag" aria-labelledby="kubectl-container">kubectl</a> regarding what the full command that it must contend with really is.  There might be ambiguity as to who should be responsible for the pipe process.  The &quot;--&quot; probably doesn't direct kubectl to include that as part of the command.  That is probably being intercepted by the shell.</p>
<p>Have you tried wrapping all of it in <strong>double-quotes</strong> ?</p>
<pre><code>CMD=&quot;tar cf - /tmp/foo | tar xf - -C /tmp/bar&quot;

kubectl exec -n my-namespace my-pod -- &quot;${CMD}&quot;
</code></pre>
<p>That way it would include the scope of saving at the target as part of the process to monitor for completion.</p>
