title: Nginx 504 Gateway Time-out when running in GCP
body: <p>I have a Spring boot application with several components to it that talk to each other. There is an Admin panel that manages accounts and you have the functionality to migrate accounts. If an account has some data to it migrating it could take upwards of 60 seconds as it needs to do a backup, delete the service, and re-create it with the newly selected image.</p>
  <p>If I run the applications locally but still connect them to the GCP environment and I attempt to migrate an account it works fine. But when pushed to master and built on the GCP environment when I try to migrate an account I get the following error:</p>
  <p><a href="https://i.stack.imgur.com/5XQiY.png" rel="nofollow noreferrer">504 Gateway Time-out</a></p>
  <p>So I have ruled out that it's not an issue with my java application and the timeout is potentially coming from the load balancer. I have the following errors in the GCP logs:</p>
  <p>In my ingress-controller I get:</p>
  <pre><code>022-06-17T09:17:48.165983390Z2022/06/17 09:17:48 [error] 1922#1922: *1134692 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 80.192.45.186, server: _, request: &quot;POST /admin/accounts/123456/edit HTTP/1.1&quot;, upstream: &quot;http://XX.XX.XX.XXX:8080/accounts/123456/edit&quot;, host: &quot;somehost.com&quot;, referrer: &quot;https://somehost.com/admin/accounts/123456/edit&quot;
  </code></pre>
  <p>And on in the Logs Explorer, I get the following:</p>
  <pre><code>{
    &quot;textPayload&quot;: &quot;2022/06/17 09:17:48 [error] 1922#1922: *1134692 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 80.192.45.186, server: _, request: \&quot;POST /admin/accounts/123456/edit HTTP/1.1\&quot;, upstream: \&quot;http://XX.XX.XX.XXX:8080/accounts/123456/edit\&quot;, host: \&quot;somehost.com\&quot;, referrer: \&quot;https://somehost.com/admin/accounts/123456/edit\&quot;&quot;,
    &quot;insertId&quot;: &quot;03uw65hlnarbu7v1&quot;,
    &quot;resource&quot;: {
      &quot;type&quot;: &quot;k8s_container&quot;,
      &quot;labels&quot;: {
        &quot;namespace_name&quot;: &quot;ingress-nginx&quot;,
        &quot;container_name&quot;: &quot;controller&quot;,
        &quot;project_id&quot;: &quot;my_project&quot;,
        &quot;cluster_name&quot;: &quot;my_cluster&quot;,
        &quot;pod_name&quot;: &quot;ingress-nginx-controller-cb576846d-6xxvf&quot;,
        &quot;location&quot;: &quot;europe-west1&quot;
      }
    },
    &quot;timestamp&quot;: &quot;2022-06-17T09:17:48.165983390Z&quot;,
    &quot;severity&quot;: &quot;ERROR&quot;,
    &quot;labels&quot;: {
      &quot;k8s-pod/pod-template-hash&quot;: &quot;cb576846d&quot;,
      &quot;k8s-pod/app_kubernetes_io/instance&quot;: &quot;ingress-nginx&quot;,
      &quot;compute.googleapis.com/resource_name&quot;: &quot;my_resource&quot;,
      &quot;k8s-pod/app_kubernetes_io/name&quot;: &quot;ingress-nginx&quot;,
      &quot;k8s-pod/app_kubernetes_io/component&quot;: &quot;controller&quot;
    },
    &quot;logName&quot;: &quot;projects/my_project/logs/stderr&quot;,
    &quot;receiveTimestamp&quot;: &quot;2022-06-17T09:17:50.861580453Z&quot;
  }
  </code></pre>
  <p>I have tried to increase the timeouts in my terraform scripts but to no avail. I have also tried to increase the timeouts from the GCP (as a proof of concept) but they do not seem to take any effect.</p>
  <p>This is my terraform script for the ingress-controller:</p>
  <pre><code>resource &quot;kubernetes_ingress&quot; &quot;nginx-ingress&quot; {
    metadata {
      name      = &quot;nginx-ingress&quot;
      namespace = kubernetes_namespace.nginx.metadata[0].name
      annotations = {
        &quot;kubernetes.io/ingress.class&quot;                 = &quot;gce&quot;
        &quot;kubernetes.io/ingress.global-static-ip-name&quot; = &quot;ext-lb&quot;
        &quot;ingress.gcp.kubernetes.io/pre-shared-cert&quot;   = local.ssl_cert_name
        &quot;networking.gke.io/v1beta1.FrontendConfig&quot;    = kubernetes_manifest.nginx_frontend_config.manifest[&quot;metadata&quot;][&quot;name&quot;]
      }
    }
    spec {
      rule {
        host = local.domain_name
        http {
          path {
            backend {
              service_name = &quot;ingress-nginx-controller&quot;
              service_port = 80
            }
          }
          path {
            path = &quot;/secured/*&quot;
            backend {
              service_name = &quot;ingress-iap-nginx-controller&quot;
              service_port = 80
            }
          }
        }
      }
    }
  }
  </code></pre>
  <p>What might be causing this timeout and how do I increase it?</p>