<p>I am using the ingress-nginx system <a href="https://github.com/kubernetes/ingress-nginx" rel="nofollow noreferrer">https://github.com/kubernetes/ingress-nginx</a>. I'm making extensive use of this project. Jenkins, Consul, Prometheus and more are working just fine using the exact same ingress config as appended to the end.</p>

<p>I am able to access my vault pod directly through port-forwarding with kubectl. But when I attempt to access it over my nginx-ingress, I am returned a 503</p>

<pre><code>kubectl port-forward vault-vault-f9778f86d-srr9n 8200:8200 -n vault

  curl 127.0.0.1:8200/v1/1
    {"errors":["Vault is sealed"]}

➜  vault curl -L vault.me.com/v1/1
&lt;html&gt;
&lt;head&gt;&lt;title&gt;503 Service Temporarily Unavailable&lt;/title&gt;&lt;/head&gt;
&lt;body bgcolor="white"&gt;
&lt;center&gt;&lt;h1&gt;503 Service Temporarily Unavailable&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx/1.13.8&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<p>Looking at the logs I see the following in response to the vault.me.com curl</p>

<pre><code>10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:42:56 +0000] "GET / HTTP/1.1" 308 187 "-" "curl/7.43.0" 77 0.000 [] - - - -
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:42:56 +0000] "GET / HTTP/1.1" 503 213 "-" "curl/7.43.0" 77 0.000 [] - - - -
</code></pre>

<p>Where as if I try to access my consul backend, I see the following.</p>

<pre><code>10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:34 +0000] "GET / HTTP/1.1" 308 187 "-" "curl/7.43.0" 78 0.000 [consul-consul-consul-8500] - - - -
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:39 +0000] "GET / HTTP/1.1" 308 187 "-" "curl/7.43.0" 78 0.000 [consul-consul-consul-8500] - - - -
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:39 +0000] "GET / HTTP/1.1" 301 39 "-" "curl/7.43.0" 78 0.002 [consul-consul-consul-8500] 10.233.114.4:8500 39 0.002 301
10.233.104.128 - [10.233.104.128] - - [19/Apr/2018:20:43:39 +0000] "GET /ui/ HTTP/1.1" 200 30178 "-" "curl/7.43.0" 81 0.001 [consul-consul-consul-8500] 10.233.82.19:8500 30178 0.001 200
</code></pre>

<p>I'm not entire sure whats going on, nor am I clear on how to debug this. I've spent the two hours reading through the source on this tool, but I'm not seeing much of anything.</p>

<p>I read somewhere that any response that isn't 2xx or 3xx  fails nginx's backend health check and will be removed from the pool of backends to be routed to. That would manifest as the 503 that I am seeing. Its true that vault's / does return a non 2xx or 3xx so that would make sense.</p>

<pre><code>vault curl -v 127.0.0.1:8200
* Rebuilt URL to: 127.0.0.1:8200/
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8200 (#0)
&gt; GET / HTTP/1.1
&gt; Host: 127.0.0.1:8200
&gt; User-Agent: curl/7.43.0
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 404 Not Found
&lt; Cache-Control: no-store
&lt; Content-Type: text/plain; charset=utf-8
&lt; X-Content-Type-Options: nosniff
&lt; Date: Thu, 19 Apr 2018 20:46:40 GMT
&lt; Content-Length: 19
&lt;
404 page not found
</code></pre>

<p><strong>Is there a way to change the URL that nginx tries to healthcheck, or disable the healthchecking entirely</strong></p>

<p><a href="https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/" rel="nofollow noreferrer">https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/</a>
From nginx's docs it says if there is a single server in a backend group, it should never be getting marked as unavailable, regardless of return code.</p>

<p>Whats going on?</p>

<blockquote>
  <p>Note that if there is only a single server in a group, the
  fail_timeout and max_fails parameters are ignored and the server is
  never marked unavailable.</p>
</blockquote>

<pre><code>k get svc -n vault
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
vault-vault   ClusterIP   10.233.47.151   &lt;none&gt;        8200/TCP   1h

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx-internal
  name: vault
  namespace: vault
spec:
  rules:
  - host: vault.me.com
    http:
      paths:
      - backend:
          serviceName: vault-vault
          servicePort: 8200
        path: /
  tls:
  - hosts:
    - me.com
    - vault.me.com
    secretName: wildcard-secret
</code></pre>

<p>Unsealing the vault through the port-forward method will change the state of the kubernetes pod level healthchecking, which is what is queried by ingress-nginx to determine if a backend is valid or not.</p>

<p>Once the vault is unsealed you can access the backend as you expect.</p>

<p>At the time of writing, the vault CLI does not appear to respect http 308 redirects. Which gives you a very unhelpful error about golang's parser library. I wasted several hours on this. I hope this information helps someone.</p>

<p><a href="https://github.com/hashicorp/vault/issues/4401" rel="nofollow noreferrer">https://github.com/hashicorp/vault/issues/4401</a></p>
