<p><strong>Problem</strong></p>
<p>We are trying to create an inference API that load PyTorch ResNet-101 model on AWS EKS. Apparently, it always killed OOM due to high CPU and Memory usage. Our log shows we need around 900m CPU resources limit. Note that we only tested it using <strong>one</strong> 1.8Mb image. Our DevOps team didn't really like it.</p>
<p><strong>What we have tried</strong></p>
<p>Currently we are using standard PyTorch load model module. We also clean model state dict to clean up the memory usage.</p>
<p>Is there any method to reduce the CPU usage to load PyTorch model?</p>

<p>Have you tried limiting the CPU available to the pods?</p>
<pre class="lang-yaml prettyprint-override"><code>  - name: pytorch-ml-model
    image: pytorch-cpu-hog-model-haha
    resources:
      limits:
        memory: &quot;128Mi&quot;
        cpu: &quot;1000m&quot; # Replace this with CPU amount your devops guys will be happy about
</code></pre>
<p>If your error is OOM, you might want to consider the adding more memory allocated per <code>pod</code>? We as outsiders have no idea how large of memory you would require to execute your models, I would suggest using debugging tools like <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="nofollow noreferrer">PyTorch profiler</a> to understand how much memory you need for your inferencing use-case.</p>
<p>You might also want to consider, using <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html" rel="nofollow noreferrer">memory-optimized</a> worker nodes and applying <a href="https://www.eksworkshop.com/beginner/140_assigning_pods/affinity/" rel="nofollow noreferrer">deployment-node affinity</a> through labels to ensure that inferencing pods are allocated in memory-optimized nodes in your EKS clusters.</p>
