<p>I have just installed an EFK stack on my Kubernetes cluster using the guide on <a href="https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4" rel="noreferrer">https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4</a></p>

<p>I have it working when accessing it through the proxy as stated in the guide on</p>

<p><a href="http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy" rel="noreferrer">http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy</a></p>

<p>However, I want it to work through my existing ingress controller so I have created a new ingress rule using the yaml below:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  generation: 2
  labels:
    app: kibana
  name: kibana
  namespace: kube-system
spec:
  rules:
  - host: kibana.dev.example1.com
    http:
      paths:
      - backend:
          serviceName: kibana-logging
          servicePort: 5601
        path: /
status:
  loadBalancer:
    ingress:
    - {}
</code></pre>

<p>To my service which runs as:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Kibana
  name: kibana-logging
  namespace: kube-system
spec:
  clusterIP: X.X.195.49
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
</code></pre>

<p>However, when I try and access my URL:
<code>http://kibana.dev.example1.com</code></p>

<p>I get: <code>{"statusCode":404,"error":"Not Found","message":"Not Found"}</code></p>

<p>If I try and access: <code>http://kibana.dev.example1.com/app/kibana#</code></p>

<p>I get: "Kibana did not load properly. Check the server output for more information."</p>

<p>After looking through the logs for both Kibana pod and ingress pod and comparing the results between a successful request through the proxy and an unsuccessful request through the ingress I can see that...</p>

<p>for hitting /</p>

<pre><code>"GET / HTTP/1.1" 200 197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 491 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 197 0.003 200 6101a7003003d34636d2012e53c23ca7
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana HTTP/1.1" 404 85 "http://kibana.dev.example1.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 612 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 5809ac2b33d3e23b200b13c9971d8520
</code></pre>

<p>for hitting /app/kibana#</p>

<pre><code>"GET /app HTTP/1.1" 404 85 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 470 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 54a4abe0cae6d3d4298847a0db0786d6
"GET /app/kibana HTTP/1.1" 200 13301 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 477 0.041 [kube-system-kibana-logging-5601] X.X.22.204:5601 13272 0.040 200 6cb7e7698f5c72e0cd06b3408d8d4673
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/bundles/kibana.style.css?v=16627 HTTP/1.1" 404 85 "https://kibana.dev.example1.com/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 576 0.004 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.004 404 0b825f03c36c2225ab082c2a0bab15f4
</code></pre>

<p>When hitting through the proxy most of these requests return 302s rather than 404s.  Can the ingress not access these URLs in namespace kube-system?</p>

<p>Am I missing something obvious here - maybe hitting the wrong Kibana URL?  I've done a lot of googling and can't find anything similar.</p>

<p>Turns out the problem was with the kibana configuration.</p>

<p>In the kibana deployment yaml there is an environment variable called <code>SERVER_BASEPATH</code> which is set to point at the kibana service proxy.  This was causing the URL to be rewritten each time I tried to access the endpoint externally.</p>

<p>If you comment out this variable and it's value and redeploy kibana then it should work by just hitting the ingress address.</p>

<p>e.g. <a href="http://kibana.dev.example1.com/" rel="noreferrer">http://kibana.dev.example1.com/</a></p>
