<p>I have this spec for Kubernetes CronJob</p>

<pre><code>---
kind: CronJob
apiVersion: batch/v1beta1
metadata:
  name: do-registry-cleanup

spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 4
  jobTemplate:
    spec:
      template:
        spec:
          automountServiceAccountToken: false
          restartPolicy: OnFailure
          containers:
          - name: podtest2
            image: alpine
            args:
            - wget
            - http://some_real_url/test/pod/2
            imagePullPolicy: Always
            lifecycle:
              postStart:
                exec:
                  command:
                  - "sh"
                  - "-c"
                  - "sleep 2s;"
</code></pre>

<p>When i do <code>kubectl describe pod some_pod_name</code> i get this output (truncated) </p>

<pre><code>Normal   Pulling              105s  kubelet, general-rl8c  pulling image "alpine"
Normal   Pulled               105s  kubelet, general-rl8c  Successfully pulled image "alpine"
Normal   Created              105s  kubelet, general-rl8c  Created container
Normal   Started              104s  kubelet, general-rl8c  Started container
Warning  FailedPostStartHook  104s  kubelet, general-rl8c  Exec lifecycle hook ([sh -c sleep 2s;]) for Container "podtest2" in Pod "do-registry-cleanup-1566391980-dvjdn_default(9d87fe8a-c412-11e9-8744-d2e7c0045fbd)" failed - error: command 'sh -c sleep 2s;' exited with 137: , message: ""
Normal   Killing              104s  kubelet, general-rl8c  Killing container with id docker://podtest2:FailedPostStartHook
</code></pre>

<p>As result in this example wget is request url, and i know that sleep command is executed,not broken.
My Question is why are:</p>

<ol>
<li>why is it occured?</li>
<li>what are side effects of this?</li>
</ol>

<hr>

<p>Some additional info. If command is "cmd1; sleep; cmd2" then cmd2 not executed. So by some reason sleep cmd invoke error in container.</p>

<p>Refering to the official documentation:</p>

<h3><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/" rel="nofollow noreferrer">Pod lifecycle</a>:</h3>

<blockquote>
  <p>Once a container enters into <strong>Running</strong> state, <strong>postStart</strong> hook (if any) is executed.  </p>
  
  <p>A container enters into <strong>Terminated</strong> state when it has successfully completed execution or when it has failed for some reason. Regardless, a reason and exit code is displayed, as well as the container’s start and finish time. Before a container enters into Terminated, <strong>preStop</strong> hook (if any) is executed.</p>
</blockquote>

<h3><a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/" rel="nofollow noreferrer">Lifecycle hooks</a>:</h3>

<blockquote>
  <p>There are two hooks that are exposed to Containers:</p>
  
  <p><strong>PostStart</strong></p>
  
  <p>This hook executes immediately after a container is created. However, there is no guarantee that the hook will execute before the container ENTRYPOINT. No parameters are passed to the handler.</p>
  
  <p><strong>PreStop</strong></p>
  
  <p>This hook is called immediately before a container is terminated due to an API request or management event such as liveness probe failure, preemption, resource contention and others. A call to the preStop hook fails if the container is already in terminated or completed state. It is blocking, meaning it is synchronous, so it must complete before the call to delete the container can be sent. No parameters are passed to the handler</p>
</blockquote>

<p>Actually, what is written for PreStop works for PostStart also.</p>

<p>Basically, Kubelet doesn't wait until all hooks are finished. It just terminate everything after main container exits.</p>

<p>For <strong>PreStop</strong> we can only increase grace period, but for <strong>PostStart</strong> we can make the main contaner waiting until the hook is finished.
Here is an example:</p>

<pre><code>kind: CronJob
apiVersion: batch/v1beta1
metadata:
  name: test1
spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 4
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: test1
            image: nginx
            command: ["bash", "-c", "touch file1; while [ ! -f file2 ] ; do ls file*; sleep 1 ; done; ls file*"]
            lifecycle:
              postStart:
                exec:
                  command: ["bash", "-c", "sleep 10; touch file2"]
</code></pre>

<p>If you check the logs of the pod, you'll see that hook has created file before the main container has been terminated. You can see that cycle has run 12 times, instead of 10. That means that PostStart has been started after 2 second after main container starts running. It means, that the container enters into Running state with some delay after start.  </p>

<pre><code>$ kubectl describe cronjob/test1 | grep Created
  Normal  SuccessfulCreate  110s  cronjob-controller  Created job test1-1566402420
$ kubectl describe job/test1-1566402420 | grep Created
  Normal  SuccessfulCreate  2m28s  job-controller  Created pod: test1-1566402420-d5lfr
$ kubectl logs pod/test1-1566402420-d5lfr -c test1
file1
file1
file1
file1
file1
file1
file1
file1
file1
file1
file1
file1
file2
</code></pre>
