<p>I'm trying to verify that shutdown is completing cleanly on Kubernetes, with a .NET Core 2.0 app.</p>

<p>I have an app which can run in two "modes" - one using ASP.NET Core and one as a kind of worker process.  Both use Console and JSON-which-ends-up-in-Elasticsearch-via-Filebeat-sidecar-container logger output which indicate startup and shutdown progress.</p>

<p>Additionally, I have console output which writes directly to stdout when a SIGTERM or Ctrl-C is received and shutdown begins.</p>

<p>Locally, the app works flawlessly - I get the direct console output, then the logger output flowing to stdout on Ctrl+C (on Windows).</p>

<p>My experiment scenario:</p>

<ul>
<li>App deployed to GCS k8s cluster (using <code>helm</code>, though I imagine that doesn't make a difference)</li>
<li>Using <code>kubectl logs -f</code> to stream logs from the specific container</li>
<li>Killing the pod from GCS cloud console site, or deleting the resources via <code>helm delete</code></li>
<li>Dockerfile is <code>FROM microsoft/dotnet:2.1-aspnetcore-runtime</code> and has <code>ENTRYPOINT ["dotnet", "MyAppHere.dll"]</code>, so not wrapped in a <code>bash</code> process or anything</li>
<li>Not specifying a <code>terminationGracePeriodSeconds</code> so guess it defaults to 30 sec</li>
<li>Observing output returned</li>
</ul>

<p>Results:</p>

<ul>
<li>The API pod log streaming showed just the immediate console output, "[SIGTERM] Stop signal received", not the other Console logger output about shutdown process</li>
<li>The worker pod log streaming showed a little more - the same console output and  some Console logger output about shutdown process</li>
<li>The JSON logs didn't seem to pick any of the shutdown log output</li>
</ul>

<p>My conclusions:</p>

<ol>
<li>I don't know if Kubernetes is allowing the process to complete before terminating it, or just issuing SIGTERM then killing things very quick.  I think it should be waiting, but then, why no complete console logger output?</li>
<li>I don't know if console output is cut off when stdout log streaming at some point before processes finally terminates?</li>
<li>I would guess that the JSON stuff doesn't come through to ES because filebeat running in the sidecar terminates even if there's outstanding stuff in files to send</li>
</ol>

<p>I would like to know:</p>

<ul>
<li>Can anyone advise on points 1,2 above?</li>
<li>Any ideas for a way to allow a little extra time or leeway for the sidecar to send stuff up, like a pod container termination order, delay on shutdown for that  container, etc?</li>
</ul>

Kieren Johnstone
<p><code>SIGTERM</code> does indeed signal termination.  The less obvious part is that when the <code>SIGTERM</code> handler returns, everything is considered finished.</p>

<p>The fix is to not return from the <code>SIGTERM</code> handler until the app has finished shutting down.  For example, using a <code>ManualResetEvent</code> and <code>Wait()</code>ing it in the handler.</p>
