<p>I would like to make the result of a text classification model (finBERT pytorch model) available through an endpoint that is deployed on Kubernetes.</p>
<p>The whole pipeline is working but it's super slow to process (30 seconds for one sentence) when deployed. If I time the same endpoint in local, I'm getting results in 1 or 2 seconds. Running the docker image in local, the endpoint also takes 2 seconds to return a result.</p>
<p>When I'm checking the CPU usage of my kubernetes instance while the request is running, it doesn't go above 35% so I'm not sure it's related to a lack of computation power?</p>
<p>Did anyone witness such performances issues when making a forward pass to a pytorch model? Any clues on what I should investigate?</p>
<p>Any help is greatly appreciated, thank you!</p>
<p>I am currently using</p>
<p>limits:
cpu: &quot;2&quot;
requests:
cpu: &quot;1&quot;</p>
<p>Python : 3.7
Pytorch : 1.8.1</p>

<p>I had the same issue. Locally my pytorch model would return a prediction in 25 ms and then on Kubernetes it would take 5 seconds. The problem had to do with how many threads torch had available to use. I'm not 100% sure why this works, but reducing the number of threads sped up performance significantly.</p>
<p>Set the following environment variable on your kubernetes pod.
<code>OMP_NUM_THREADS=1</code></p>
<p>After doing that it performed on kubernetes like it did running it locally ~30ms per call.</p>
<p>These are my pod limits:</p>
<ul>
<li>cpu limits <code>1</code></li>
<li>mem limits: <code>1500m</code></li>
</ul>
<p>I was led to discover this from this blog post: <a href="https://www.chunyangwen.com/blog/python/pytorch-slow-inference.html" rel="nofollow noreferrer">https://www.chunyangwen.com/blog/python/pytorch-slow-inference.html</a></p>
