<p>I am trying to debug a Java microservice hosted in Kubernetes, in pods which are created dynamically by KEDA feeding off an SQS queue. Pods have dedicated resources of up to 3GB RAM, which to our knowledge is sufficient (although this assumption might need to be revisited).</p>
<p>The service at one point calls ffmpeg, and sometimes randomly this appears to cause catastrophic failure of the process. (Sometimes it succeed on the next attempt of exactly the same job.)</p>
<p>It is so catastrophic that no logs are generated, no exceptions thrown, and even the finally {} block is not even triggered.</p>
<p>This appears to occur in the remote k8 environment only and I have failed to replicate it locally.</p>
<p>What general reasons are there for a Java, executing another program with Execution.executeAndWait(), to fail so catastrophically?</p>
<p>How could I continue investigating the cause?</p>
<pre><code>    public void executeFfmpeg(String[] cmd) {

        log.info(&quot;1&quot;);  // &lt;-- this IS logged!
        int retval = 0;
        try {
            retval = Execution.executeAndWait(cmd, stdout, stderr);
            if ( retval != 0 ) {
                log.error(&quot;FFMPEG broke. stderr: {}&quot;, errorFile);   // &lt;-- this log never occurs
                throw new TranscodingFailedException(&quot;ffmpeg command failed: &quot; + String.valueOf(retval) + &quot; - &quot; + String.join(&quot; &quot;, cmd)); // &lt;-- never thrown
            }
        } catch (InterruptedException e) {
            log.error(&quot;InterruptedException caught in the middle of the execution of ffmpeg. this will now proceed to crash...&quot;, e); // &lt;-- this log never occurs
            throw e;
        } finally {
            log.info(&quot;2&quot;); // &lt;-- THIS IS NOT CALLED!!! even in the finally block!?
            stdout.close();
            stderr.close();
        }
    }


/// Execution class:

    public static int executeAndWait(String[] cmd, Writer stdout, Writer stderr) throws IOException, InterruptedException {
        Process proc = null;
        ExecutorService outExecutor = Executors.newSingleThreadExecutor();
        ExecutorService errExecutor = Executors.newSingleThreadExecutor();

        try {
            proc = exec(cmd, stdout, stderr, outExecutor, errExecutor);
            // block until sub-process exits
            return proc.waitFor();
        } finally {
            shutdownProcess(proc);
            shutdownAndAwaitTermination(outExecutor);
            shutdownAndAwaitTermination(errExecutor);
        }
    }



    private static void shutdownProcess(Process proc) {
        try {
            if (proc != null) {
                proc.destroy();
            }
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }

    private static void shutdownAndAwaitTermination(ExecutorService pool) {
        try {
            pool.shutdown(); // Disable new tasks from being submitted
            try {
                // Wait a while for existing tasks to terminate
                if (!pool.awaitTermination(800, TimeUnit.MILLISECONDS)) {
                    pool.shutdownNow(); // Cancel currently executing tasks
                }
            } catch (InterruptedException ie) {
                // (Re-)Cancel if current thread also interrupted
                pool.shutdownNow();
            }
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }

    private static Process exec(String[] cmd, Writer stdout, Writer stderr, ExecutorService outExecutor, ExecutorService errorExecutor) throws IOException {
        // execute input command in a sub-process
        Process proc = Runtime.getRuntime().exec(cmd);

        StreamConsumer outConsumer = new StreamConsumer(proc.getInputStream(), stdout);
        StreamConsumer errConsumer = new StreamConsumer(proc.getErrorStream(), stderr);

        // execute data read/write in separate threads
        outExecutor.submit(outConsumer);
        errorExecutor.submit(errConsumer);

        return proc;

    }
</code></pre>
<p>An example of the ffmpeg command is:</p>
<pre><code>/usr/local/bin/ffmpeg -i /tmp/transcode-a5ff7706-488a-4e24-9ef8-9657d1254a26626348807122071896/str_CAM_Z2VH_con_H0Zqr2flbT.webm -filter_complex [0:v]setpts=0.8363824*PTS,scale=w=1280:h=720:force_original_aspect_ratio=1,pad=1280:720:(ow-iw)/2:(oh-ih)/2[v0];[0:a]aresample=async=1000[0sync] -map [v0] -map [0sync] -vsync vfr -r 25 -c:v libx264 -c:a mp3 /tmp/transcode-a5ff7706-488a-4e24-9ef8-9657d1254a26626348807122071896/intermediate-str_CAM_Z2VH_con_H0Zqr2flbT.webm.mkv | stdout: /tmp/intermediate-stdout-12310474463787935763.log | stderr: /tmp/intermediate-stderr-12166558954928907997.log
</code></pre>

Brendan Hill
<p>Some suggestions for checking that the sub-process handling is correct:</p>
<p>Carefully check the logs to see if <code>ffmpeg</code> is waiting on input (eg overwrite file confirmation etc). You could eliminate that possibility by closing STDIN explicitly (also see <code>-y</code> flag to force overwrite, or <code>-nostdin</code> to remove use of stdin altogether):</p>
<pre><code>proc.getOutputStream().close();
</code></pre>
<p>Your command looks like a shell command as it contains pipe so isn't valid for direct launch from Java, drop the pipe and following parts, or change command to launch inside a shell which would handle the pipe correctly:</p>
<pre><code>/usr/local/bin/ffmpeg -i ... | stdout: xxx.log | stderr: yyy.log
=&gt; 
/usr/local/bin/ffmpeg -i ...
or
{&quot;bash&quot;,&quot;-c&quot;,&quot;/usr/local/bin/ffmpeg -i ... | stdout: xxx.log | stderr: yyy.log&quot;}
</code></pre>
<p>The STDOUT/ERR streams should be dealt with before calling <code>proc.waitFor()</code>, not afterwards and not after <code>destroy()</code>. You may be better off ensuring that these streams end inside <code>exec()</code>:</p>
<pre><code>// execute data read/write in separate threads
Future&lt;?&gt; so = outExecutor.submit(outConsumer);
Future&lt;?&gt; se = errorExecutor.submit(errConsumer);

// Await STDOUT/ERR termination:
try {
    so.get();
    se.get();
} catch (InterruptedException |ExecutionException e) {
    throw new RuntimeException(&quot;failed&quot;, e);
}
</code></pre>
