<p>We have several applications in a Kubernetes cluster that use Apache Ignite. Ignite creates various thread pools that are sized as follows:</p>

<pre><code>Math.max(8, Runtime.getRuntime().availableProcessors())
</code></pre>

<p>So basically the thread pool will always have at least size 8, but could be more if the system believes there are more processors.</p>

<p>The problem we're having is that some pods are spinning up with pool size 8, and others are using size 36, which is the number of CPUs on the node.</p>

<p>We're using Helm to deploy all apps, but we're <strong>not</strong> setting any CPU limits for any pods. In theory they should all see the same number of available CPUs.</p>

<p>What else could cause pods on the same node to see different views of how many processors are available?</p>

<h2>Update</h2>

<p>We have a health end point in all of our apps that shows the number of CPUS reported by the JVM, using the same <code>Runtime#availableProcessors()</code> method that Ignite uses.</p>

<p>All of our apps, including ones where Ignite thinks there are 36 CPUs, report 2 processors once the process has started.</p>

<p>I found this interesting line in the Java documentation for that method:</p>

<blockquote>
  <p>This value may change during a particular invocation of the virtual machine. Applications that are sensitive to the number of available processors should therefore occasionally poll this property and adjust their resource usage appropriately.</p>
</blockquote>

<p>It seems we're in a race condition where early on in app startup, that value reports 36 but at some point drops to 2. Depending on when the Ignite beans are fired, they're seeing either 36 or 2.</p>

<p><em>tl;dr</em> The underlying issue seems to be when <code>resources.requests.cpu</code> is set exactly to <code>1000m</code>.</p>
<p>I wrote a simple Java app that dumps the available number of processors:</p>
<pre class="lang-java prettyprint-override"><code>public class CpuTest {
    public static void main(String[] args) {
        System.out.println(&quot;Number of CPUs = &quot;
                            + Runtime.getRuntime().availableProcessors());   
    }
}
</code></pre>
<p>I packaged into a Dockerfile and created a simple deployment:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: cputest
  labels:
    app: cputest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cputest
  template:
    metadata:
      labels:
        app: cputest
    spec:
      containers:
      - name: cputest
        image: dev/cputest:latest
        imagePullPolicy: Never
</code></pre>
<p>I ran this on my local RedHat 7 machine which has 24 cores. The output was expected:</p>
<pre><code>Number of CPUs = 24
</code></pre>
<p>I then applied various CPU resource requests to the deployment:</p>
<pre class="lang-yaml prettyprint-override"><code>        resources:
          requests:
            cpu: 1000m
</code></pre>
<p>and re-deployed. The results were interesting:</p>
<ul>
<li>CPU request set to 500m: the app reports 1 CPU</li>
<li><strong>CPU request set to 1000m: the app reports 24 CPU</strong> &lt;==</li>
<li>CPU request set to 1001m: the app reports 2 CPU</li>
<li>CPU request set to 2000m: the app reports 2 CPU</li>
<li>CPU request set to 4000m: the app reports 4 CPU</li>
</ul>
<p>So the issue only arises when the CPU request is set <code>1000m</code> (also tried <code>1</code> and got the same result, where it thinks it has all 24 CPUs).</p>
<p>I went back and looked at all of our apps. Sure enough, the ones where we set the CPU request to exactly <code>1000m</code> are the ones that have the issue. Any other value works as expected.</p>
<p>Of inerest, when I also set CPU limit to <code>1000m</code>, the problem goes away and the JVM reports 1 CPU.</p>
<p>It very well could be this is expected and I don't fully understand how CPU resource and limits are used by Kubernetes, or perhaps an issue with the version we're on (1.12.7).</p>
<p>Either way I at least have an answer as to why some of our pods are seeing different CPUs.</p>
