title: Kubernetes HA Setup with External Nginx Loadbalancer or F5 Loadbalancer User "system:anonymous" cannot create nodes at the cluster scope
body: <p>I am using external nginx loadbalancer and trying to configure K8s Master but its failing with below error :</p>
  
  <blockquote>
    <p>error uploading configuration: unable to create configmap: configmaps is forbidden: User "system:anonymous" cannot create configmaps in the namespace "kube-system"**</p>
  </blockquote>
  
  <p>For me it more looks like cert issue but i am having hard time to find what i am missing , any help is appreciated in our infrastructure we use F5 loadbalancer in front of apiserver and i am seeing the same issue there this is the env i have created for troubleshooting</p>
  
  <p><strong>kubeadm-config:</strong></p>
  
  <pre><code>    apiVersion: kubeadm.k8s.io/v1alpha2
      kind: MasterConfiguration
      kubernetesVersion: v1.11.0
      apiServerCertSANs:
      - "ec2-23-23-244-63.compute-1.amazonaws.com"
      api:
      controlPlaneEndpoint: "ec2-23-23-244-63.compute-1.amazonaws.com:6443"
      etcd:
      external:
          endpoints:
          - https://172.31.32.160:2379
          caFile: /etc/kubernetes/pki/etcd/ca.crt
          certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
          keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
       networking:
       # This CIDR is a calico default. Substitute or remove for your CNI provider.
       podSubnet: "10.244.0.0/16"
  </code></pre>
  
  <p><strong>Env :</strong>
          Kubelet : 1.11.1 kubeadm 1.11.1 kubectl 1.11.1</p>
  
  <p><strong>Output</strong></p>
  
  <pre><code>    [certificates] Using the existing ca certificate and key.
      [certificates] Using the existing apiserver certificate and key.
      [certificates] Using the existing apiserver-kubelet-client certificate and 
      key.
      [certificates] Using the existing sa key.
      [certificates] Using the existing front-proxy-ca certificate and key.
      [certificates] Using the existing front-proxy-client certificate and key.
      [certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
       [endpoint] WARNING: port specified in api.controlPlaneEndpoint overrides 
    api.bindPort in the controlplane address
       [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
       [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
       [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller- 
    manager.conf"
  
  
  
  
       [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
       [controlplane] wrote Static Pod manifest for component kube-apiserver to 
    "/etc/kubernetes/manifests/kube-apiserver.yaml"
       [controlplane] wrote Static Pod manifest for component kube-controller-manager 
    to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
       [controlplane] wrote Static Pod manifest for component kube-scheduler to 
    "/etc/kubernetes/manifests/kube-scheduler.yaml"
       [init] waiting for the kubelet to boot up the control plane as Static Pods 
    from directory "/etc/kubernetes/manifests"
    [init] this might take a minute or longer if the control plane images have to 
    be pulled
       [apiclient] All control plane components are healthy after 41.036802 seconds
  
  
       [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" 
       in the "kube-system" Namespace
       error uploading configuration: unable to create configmap: configmaps is 
       forbidden: User "system:anonymous" cannot create configmaps in the 
       namespace "kube-system"
  </code></pre>
  
  <p><strong>logs:</strong> </p>
  
  <pre><code>    Unable to register node "ip-172-31-40-157" with API server: nodes is forbidden: User "system:anonymous" cannot create nodes at the cluster scope
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
  on_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "ip-172-31-40-157" not found
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "ip-172-31-40-157" is forbidden: User "system:anonymous" cannot list nodes at t
  tor.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster sco
  :172] Unable to update cni config: No networks found in /etc/cni/net.d
  </code></pre>
  
  <p>Nginx :</p>
  
  <pre><code>   upstream mywebapp1 {
        server 172.31.40.157:6443;
  
     }
  server {
      listen 6443 ssl;
      server_name ec2-23-23-244-63.compute-1.amazonaws.com;
  
      ssl on;
      ssl_certificate         /opt/certificates/server.crt;
      ssl_certificate_key     /opt/certificates/server.key;
      ssl_trusted_certificate /opt/certificates/ca.crt;
  
      location / {
          proxy_pass https://mywebapp1;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
      }
  }
  
  Nginx Server : 172-31-44-203
  Master Server : 172-31-40-157
  </code></pre>
  
  <p><strong>I am using Self Signed Certs and the CA to generate all the certs including the one in nginx are same</strong> </p>
  
  <p>I had same issue in our infrastructure when we use f5 loadbalancer </p>