<p>I am launching Jobs and I'm trying to use the lifecycle hooks to launch a script at start and another one at shutdown of the container. </p>

<p>I am also specifying resource limits, and they look like this:</p>

<pre><code>resources:
    required:
        memory: 1Gi
        cpu: 1
    limits:
        memory: 1Gi
        cpu: 1
</code></pre>

<p>My cluster currently has <strong>4 nodes with 1  CPU and 4 GB of RAM each</strong>, and is running on EC2 machines.</p>

<p>The <code>postStart</code> script is at the moment very simple, and looks like this:</p>

<pre><code>export SOME_VAR=some_value
node someScript.js
</code></pre>

<p>The only thing the Node script does is update a value on a database, so it's not an especially intensive task.</p>

<p>After launching the job, the following events happen:</p>

<p><a href="https://i.stack.imgur.com/aGXKe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/aGXKe.png" alt="Kubectl Events"></a></p>

<p>As you can see the <code>postStart</code> hook fails with error 137, and gives no error message.</p>

<p>Any help for solving this issue is highly welcome and appreciated.</p>

<h2>Edit 1</h2>

<p>Since the first answer has pointed to the fact that the command executed for the cook might not be correctly built, I think it's important to say that I build the jobs using the API Kubernetes publishes through <code>kubectl proxy</code>.</p>

<p>This is how I specify the <code>lifecycle</code> instructions:</p>

<pre><code>"lifecycle": {
    "postStart": {
        "exec": {
            "command": [
                "/bin/sh",
                "postStart.sh"
             ]
        }
    },
    "preStop": {
        "exec": {
            "command": [
                "/bin/sh",
                "preStop.sh"
            ]
        }
    }
}
</code></pre>

<p>I think this translates to YAML the way it's supposed to; please correct me if I am wrong on this.</p>

<p>You have 2 problems, so you get 2 answers :-)</p>

<h1>Problem 1: too high cpu requirement</h1>

<p>You pod specifies the requirement of <code>cpu: 1</code> - this means 1 cpu core. Your nodes have 1 cpu core in total, but are already running some pods, like kube-proxy. So none of them have a full core available for your application, so the scheduling fails.</p>

<p>The error message <code>No nodes are available that match all of the predicates: Insufficient cpu (4), PodToleratesNodeTaints (1)</code> means:</p>

<ul>
<li>Scheduling is not possible at the moment</li>
<li>Of all nodes, 4 do not have enough cpu to schedule this pod.

<ul>
<li>You can verify this by executing <code>kubectl describe node nameofyournode</code>, and look at the <code>Allocatable:</code> and the <code>Allocated resources:</code> part of the output. In <code>Non-terminated Pods:</code> you will see that is taking up some of your cpu, possibly a kube-proxy pod.</li>
</ul></li>
<li>Of all nodes, 1 has a <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">taint</a> that is not <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">tolerated</a> by the pod (this is the master I imagine)</li>
</ul>

<p>The solution is to lower the requirement for your pod (<code>500mi</code> means 500 millicores, or 0.5 cores):</p>

<pre><code>resources:
  required:
    memory: 1Gi
    cpu: 500mi
  limits:
    memory: 1Gi
    cpu: 500mi
</code></pre>

<p>... or resize your machines so they have 2 cores instead of 1.</p>

<h1>Problem 2: bad postStart command</h1>

<p>Now what is most curious is that somehow in the end the pod did get scheduled, but thereafter killed. Code 126 means <code>Command invoked cannot execute</code>, so the <code>postStart:</code> command is probably invalid. You did not post the full yaml file, but from the error message it looks like you have specified something like:</p>

<pre><code>lifecycle:
  postStart:
    exec:
      command: ["/bin/sh postStart.sh"]
</code></pre>

<p>please check if that is the case. If so, it is incorrect. You need to separate each parameter into a different element in the <code>command</code> array like so:</p>

<pre><code>lifecycle:
  postStart:
    exec:
      command: ["/bin/sh", "postStart.sh"]
</code></pre>

<p>Alternatively, make sure that <code>postStart.sh</code> is marked executable in the container image and specify a shell shebang in the first line (<code>#!/bin/bash</code>). If you do that you can define the postStart hook like this:</p>

<pre><code>lifecycle:
  postStart:
    exec:
      command: ["/path/to/postStart.sh"]
</code></pre>
