<p>This is a similar question as <a href="https://stackoverflow.com/questions/38733801/unable-to-access-kubernetes-dashboard">this</a> but I could not find a resolution from it. I have setup Kubernetes cluster with CoreOS 2 masters and 3 nodes on AWS by following this <a href="https://coreos.com/kubernetes/docs/latest/getting-started.html" rel="nofollow noreferrer">step by step guide</a>. k8s version is 1.4.0 and all servers are in a private subnet, so I build a bastion VPN server on a different VPC and connect to a k8s cluster via the bastion server with VPC peering.</p>

<p>It works basically pretty well but I noticed that I cannot access kubernetes dashboard from a web browser. 
These are my kuberentes dashboard svc and rc yaml files.</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: kubernetes-dashboard
  ports:
  - port: 80
    targetPort: 9090
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: kubernetes-dashboard-v1.4.0
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    version: v1.4.0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
        version: v1.4.0
        kubernetes.io/cluster-service: "true"
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
        scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'
    spec:
      containers:
      - name: kubernetes-dashboard
        image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.4.0
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
          requests:
            cpu: 100m
            memory: 50Mi
        ports:
        - containerPort: 9090
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
</code></pre>

<p>If I just access <code>https://master-host/ui</code>, it returns an authentication error. I understand it and feel no problem because the api server needs an authentication. But when I run <code>kubectl proxy --port=8001</code> then access <code>http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/</code>, a browser returns</p>

<pre><code>Error: 'dial tcp 10.10.93.3:9090: i/o timeout'
Trying to reach: 'http://10.10.93.3:9090/'
</code></pre>

<p>while a request to the api server just works file like <code>http://localhost:8001/static</code> returns:</p>

<pre><code>{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apps",
    "/apis/apps/v1alpha1",
    "/apis/authentication.k8s.io",
    "/apis/authentication.k8s.io/v1beta1",
    "/apis/authorization.k8s.io",
    "/apis/authorization.k8s.io/v1beta1",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/batch/v2alpha1",
    "/apis/certificates.k8s.io",
    "/apis/certificates.k8s.io/v1alpha1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/apis/policy",
    "/apis/policy/v1alpha1",
    "/apis/rbac.authorization.k8s.io",
    "/apis/rbac.authorization.k8s.io/v1alpha1",
    "/apis/storage.k8s.io",
    "/apis/storage.k8s.io/v1beta1",
    "/healthz",
    "/healthz/ping",
    "/logs",
    "/metrics",
    "/swaggerapi/",
    "/ui/",
    "/version"
  ]
}
</code></pre>

<p>It looks pods on master cannot connect to a pod on nodes. From busybox on a node, </p>

<pre><code>kubectl exec busybox -- wget 10.10.93.3:9090
</code></pre>

<p>can fetch an index.html so node-to-node communication should be ok.</p>

<p>A result of service describe:</p>

<pre><code>❯❯❯ kubectl describe svc kubernetes-dashboard --namespace=kube-system                                                                        ⏎ master ⬆ ✭ ✚ ✱ ➜ ◼
Name:           kubernetes-dashboard
Namespace:      kube-system
Labels:         k8s-app=kubernetes-dashboard
            kubernetes.io/cluster-service=true
Selector:       k8s-app=kubernetes-dashboard
Type:           ClusterIP
IP:         10.11.0.82
Port:           &lt;unset&gt; 80/TCP
Endpoints:      10.10.93.9:9090
Session Affinity:   None
No events.
</code></pre>

<p>What else I'm missing? If I use a NodePort I can see the dashboard but I don't want to expose the dashboard. I suspect there is either some missing port I have to open on AWS security group settings, or some flanneld/docker/cni network settings went wrong and it causes the issue.</p>

<p>This is a log of the dashboard pod.</p>

<pre><code>Starting HTTP server on port 9090
Creating API server client for https://10.11.0.1:443
Successful initial request to the apiserver, version: v1.4.0+coreos.1
Creating in-cluster Heapster client
</code></pre>

<p>so it looks nothing actually reached to the dashboard.</p>

<p><em>[Updated]</em>
I found those logs on api-server pod.</p>

<pre><code> proxy.go:186] Error proxying data from backend to client: write tcp [master-ip-address]:443-&gt;[vpn-ip-address]:61980: write: connection timed out
</code></pre>

<p>So obviously something happened when proxing between api server and VPN server.</p>

<p>Ah...finally I noticed that there is a mistake on my AWS security setting. That is, I open TCP 8472 port for flanneld about master => node communication, that should be UDP. I knew that should be UDP so it took very long time until I re-checked it and noticed the mistake.</p>

<p>After I updated the setting, <code>kubectl proxy</code> instantly worked and I can now see kubernetes dashboard.</p>
