<p>I'm hitting 429 error in Google Container Registry, too many images are pulled simultaneously</p>
<pre><code>Error: Status 429 trying to pull repository [...] &quot;Quota Exceeded.&quot;
</code></pre>
<p>There is a Kubernetes cluster with multiple nodes and pods implement Kubeflow steps. In the Google <a href="https://cloud.google.com/container-registry/docs/troubleshooting" rel="nofollow noreferrer">guide</a> they suggest the following:</p>
<pre><code>To avoid hitting the fixed quota limit, you can:
- Increase the number of IP addresses talking to Container Registry. 
  Quotas are per IP address.
- Add retries that introduce a delay. 
  For example, you could use exponential backoff.
</code></pre>
<p><strong>Questions</strong>:</p>
<ul>
<li>How to increase the number of IP addresses talking to Container Registry?</li>
<li>How and where is it possible to add retries that introduce a delay? Pods are run as steps in the Kubeflow pipelines.</li>
</ul>
<p><strong>Update</strong>:</p>
<ul>
<li>It was Spinnaker which produced many requests to GCR, so it was a different application that exceeded quota and caused issues</li>
</ul>

Konstantin
<p>It seems nothing can be done in terms of the Cloud Registry quota limits because those are <strong>fixed</strong>. According to <a href="https://cloud.google.com/container-registry/quotas" rel="nofollow noreferrer">Container Registry &gt; Doc &gt; Resources &gt; Quotas and limits</a>:</p>
<blockquote>
<p>Any request sent to Container Registry has a 2 hour timeout limit.</p>
<p>The fixed rate limits per client IP address are:</p>
<ul>
<li>50,000 HTTP requests every 10 minutes</li>
<li>1,000,000 HTTP requests per day</li>
</ul>
</blockquote>
<p>Google provides support for GKE, but Kubeflow itself is not supported by Google. This question should be addressed to the Kubeflow support.</p>
<p>A Kubeflow issue with breaching quota limits and a question about how to get container pulls to use more IP addresses can be registered on the project page on GitHub:
<a href="https://github.com/kubeflow/kubeflow/issues" rel="nofollow noreferrer">https://github.com/kubeflow/kubeflow/issues</a></p>
<p>Other support options are available here:
<a href="https://www.kubeflow.org/docs/other-guides/support/" rel="nofollow noreferrer">https://www.kubeflow.org/docs/other-guides/support/</a></p>
<p>If you use CLI, you may try to customize Kubeflow configuration file before deployment or split it into separate deployments to overcome Cloud Registry quota limits. This approach helps for some complicated deployments. An important thing here is to take care of dependencies. First run</p>
<pre><code>kfctl build -v -f ${CONFIG_URI}
</code></pre>
<p>make changes in the file <code>${KF_DIR}/kfctl_gcp_iap.yaml</code>, and then run</p>
<pre><code>kfctl apply -v -f ${CONFIG_URI}
</code></pre>
