<p>I'm trying to install and configure Velero for kubernetes backup. I have followed the <a href="https://github.com/vmware-tanzu/velero-plugin-for-gcp#setup" rel="noreferrer">link</a> to configure it in my GKE cluster. The installation went fine, but velero is not working.</p>
<p>I am using google cloud shell for running all my commands (I have installed and configured velero client in my google cloud shell)</p>
<p>On further inspection on velero deployment and velero pods, I found out that it is not able to pull the image from the docker repository.</p>
<pre><code>kubectl get pods -n velero
NAME                      READY   STATUS              RESTARTS   AGE
velero-5489b955f6-kqb7z   0/1     Init:ErrImagePull   0          20s
</code></pre>
<p>Error from velero pod (kubectl describe pod) (output redacted for readability - only relevant info shown below)</p>
<pre><code>    Events:
  Type     Reason     Age               From                                                  Message
  ----     ------     ----              ----                                                  -------
  Normal   Scheduled  38s               default-scheduler                                     Successfully assigned velero/velero-5489b955f6-kqb7z to gke-gke-cluster1-default-pool-a354fba3-8674
  Warning  Failed     22s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Failed to pull image &quot;velero/velero-plugin-for-gcp:v1.1.0&quot;: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     22s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Error: ErrImagePull
  Normal   BackOff    21s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Back-off pulling image &quot;velero/velero-plugin-for-gcp:v1.1.0&quot;
  Warning  Failed     21s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Error: ImagePullBackOff
  Normal   Pulling    8s (x2 over 37s)  kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Pulling image &quot;velero/velero-plugin-for-gcp:v1.1.0&quot;
</code></pre>
<p>Command used to install velero: (some of the values are given as variables)</p>
<pre><code>velero install \
     --provider gcp \
     --plugins velero/velero-plugin-for-gcp:v1.1.0 \
     --bucket $storagebucket \
     --secret-file ~/velero-backup-storage-sa-key.json
</code></pre>
<p>Velero Version</p>
<pre><code>velero version
Client:
        Version: v1.4.2
        Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
&lt;error getting server version: timed out waiting for server status request to be processed&gt;
</code></pre>
<p>GKE version</p>
<pre><code>v1.15.12-gke.2
</code></pre>

<blockquote>
<p><em>Isn't this a Private Cluster ? – mario 31 mins ago</em></p>
<p><em>@mario this is a private cluster but I can deploy other services without any issues (for eg: I have deployed nginx successfully) –
Sreesan 15 mins ago</em></p>
</blockquote>
<p>Well, this is a <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#docker_hub" rel="noreferrer">know limitation</a> of <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters" rel="noreferrer"><strong>GKE Private Clusters</strong></a>. As you can read in the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#docker_hub" rel="noreferrer">documentation</a>:</p>
<blockquote>
<h3>Can't pull image from public Docker Hub</h3>
<p><strong>Symptoms</strong></p>
<p>A Pod running in your cluster displays a warning in  <code>kubectl describe</code>  such as  <code>Failed to pull image: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)</code></p>
<p><strong>Potential causes</strong></p>
<p>Nodes in a private cluster do not have outbound access to the public
internet. They have limited access to Google APIs and services,
including Container Registry.</p>
<p><strong>Resolution</strong></p>
<p>You cannot fetch images directly from Docker Hub. Instead, use images
hosted on Container Registry. Note that while Container Registry's
<a href="https://cloud.google.com/container-registry/docs/using-dockerhub-mirroring" rel="noreferrer">Docker Hub
mirror</a>
is accessible from a private cluster, it should not be exclusively
relied upon. The mirror is only a cache, so images are periodically
removed, and a private cluster is not able to fall back to Docker Hub.</p>
</blockquote>
<p>You can also compare it with <a href="https://stackoverflow.com/questions/62382321/gke-image-pull-errors-for-specific-public-docker-hub-images/62475193#62475193">this</a> answer.</p>
<p>It can be easily verified on your own by making a simple experiment. Try to run two different nginx deployments. First based on image <code>nginx</code> (which equals to <code>nginx:latest</code>) and the second one based on <code>nginx:1.14.2</code>.</p>
<p>While the first scenario is perfectly feasible because the <code>nginx:latest</code> image can be pulled from <strong>Container Registry's Docker Hub mirror</strong> which is accessible from a private cluster, any attempt of pulling <code>nginx:1.14.2</code> will fail which you'll see in <code>Pod</code> events. It happens because the <strong>kubelet</strong> is not able to find this version of the image in <strong>GCR</strong> and it tries to pull it from public docker registry (<code>https://registry-1.docker.io/v2/</code>), which in <strong>Private Clusters</strong> is not possible. <em>&quot;The mirror is only a cache, so images are periodically removed, and a private cluster is not able to fall back to Docker Hub.&quot;</em> - as you can read in docs.</p>
<p>If you still have doubts, just <code>ssh</code> into your node and try to run following commands:</p>
<pre><code>curl https://cloud.google.com/container-registry/

curl https://registry-1.docker.io/v2/
</code></pre>
<p>While the first one works perfectly, the second one will eventually fail:</p>
<pre><code>curl: (7) Failed to connect to registry-1.docker.io port 443: Connection timed out
</code></pre>
<p>Reason ? - <em>&quot;Nodes in a private cluster do not have outbound access to the public internet.&quot;</em></p>
<h3>Solution ?</h3>
<p>You can search what is currently available in <strong>GCR</strong> <a href="https://console.cloud.google.com/gcr/images/google-containers" rel="noreferrer">here</a>.</p>
<p>In many cases you should be able to get the required image if you don't specify it's exact version (by default <code>latest</code> tag is used). While it can help with <code>nginx</code>, unfortunatelly no version of <a href="https://hub.docker.com/r/velero/velero-plugin-for-gcp" rel="noreferrer">velero/velero-plugin-for-gcp</a> is currently available in Google Container Registry's Docker Hub mirror.</p>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#private-nodes-outbound" rel="noreferrer">Granting private nodes outbound internet access</a> by using <a href="https://cloud.google.com/nat/docs/overview#NATwithGKE" rel="noreferrer">Cloud NAT</a> seems the only reasonable solution that can be applied in your case.</p>
