title: How to implement the load balancing for kube-apiserver using nignx's L7 proxy and SSL authentication ？
body: <p>I'm a beginner in kubernetes and recently tried to implement the load balancing for kube-apiserver using nignx's L7 proxy and SSL authentication.
  Nginx worked fine in L4 proxy mode, but had some problems in L7 proxy mode.
  I chose L7 mode because I wanted to test nginx's retransmission mechanism under a specific status code (e.g. 429).
  I executed the cmd "<strong>kubectl apply -f nginx-deployment.yaml</strong>", and the yaml file is as follows:</p>
  
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    namespace: kube-system
  spec:
    selector:
      matchLabels:
        app: nginx
    replicas: 2
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
          - containerPort: 80
  </code></pre>
  
  <p>But, I found a deployment object was created, but didn't find any pods:</p>
  
  <pre><code>➜  ~ kubectl get deploy --all-namespaces
  NAMESPACE     NAME               READY   UP-TO-DATE   AVAILABLE   AGE
  kube-system   nginx-deployment   2/2     2            2           50m
  ➜  ~ kubectl get pod --all-namespaces
  No resources found
  </code></pre>
  
  <p>The details of the deployment object are described below:</p>
  
  <pre><code>➜  ~ kubectl describe deploy nginx-deployment -n kube-system
  Name:                   nginx-deployment
  Namespace:              kube-system
  CreationTimestamp:      Wed, 04 Mar 2020 12:42:34 +0000
  Labels:                 &lt;none&gt;
  Annotations:            kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-deployment","namespace":"kube-system"},"spec":{"rep...
  Selector:               app=nginx
  Replicas:               2 desired | 0 updated | 0 total | 0 available | 0 unavailable
  StrategyType:           RollingUpdate
  MinReadySeconds:        0
  RollingUpdateStrategy:  25% max unavailable, 25% max surge
  Pod Template:
    Labels:  app=nginx
    Containers:
     nginx:
      Image:        nginx:1.7.9
      Port:         80/TCP
      Host Port:    0/TCP
      Environment:  &lt;none&gt;
      Mounts:       &lt;none&gt;
    Volumes:        &lt;none&gt;
  OldReplicaSets:   &lt;none&gt;
  NewReplicaSet:    &lt;none&gt;
  Events:           &lt;none&gt;
  </code></pre>
  
  <p>When I looked at the events, I found no pod related events:</p>
  
  <pre><code>➜  ~ kubectl get event --all-namespaces       
  NAMESPACE     LAST SEEN   TYPE     REASON                    OBJECT                              MESSAGE
  default       6m12s       Normal   Starting                  node/node1                          Starting kubelet.
  default       6m12s       Normal   NodeHasSufficientMemory   node/node1                          Node node1 status is now: NodeHasSufficientMemory
  default       6m12s       Normal   NodeHasNoDiskPressure     node/node1                          Node node1 status is now: NodeHasNoDiskPressure
  default       6m12s       Normal   NodeHasSufficientPID      node/node1                          Node node1 status is now: NodeHasSufficientPID
  default       6m12s       Normal   NodeAllocatableEnforced   node/node1                          Updated Node Allocatable limit across pods
  default       6m2s        Normal   NodeReady                 node/node1                          Node node1 status is now: NodeReady
  default       5m1s        Normal   Starting                  node/node1                          Starting kube-proxy.
  default       6m5s        Normal   Starting                  node/node2                          Starting kubelet.
  default       6m5s        Normal   NodeHasSufficientMemory   node/node2                          Node node2 status is now: NodeHasSufficientMemory
  default       6m5s        Normal   NodeHasNoDiskPressure     node/node2                          Node node2 status is now: NodeHasNoDiskPressure
  default       6m5s        Normal   NodeHasSufficientPID      node/node2                          Node node2 status is now: NodeHasSufficientPID
  default       6m5s        Normal   NodeAllocatableEnforced   node/node2                          Updated Node Allocatable limit across pods
  default       5m55s       Normal   NodeReady                 node/node2                          Node node2 status is now: NodeReady
  default       4m58s       Normal   Starting                  node/node2                          Starting kube-proxy.
  default       6m30s       Normal   Starting                  node/node3                          Starting kubelet.
  default       6m30s       Normal   NodeHasSufficientMemory   node/node3                          Node node3 status is now: NodeHasSufficientMemory
  default       6m30s       Normal   NodeHasNoDiskPressure     node/node3                          Node node3 status is now: NodeHasNoDiskPressure
  default       6m30s       Normal   NodeHasSufficientPID      node/node3                          Node node3 status is now: NodeHasSufficientPID
  default       6m30s       Normal   NodeAllocatableEnforced   node/node3                          Updated Node Allocatable limit across pods
  default       6m25s       Normal   NodeReady                 node/node3                          Node node3 status is now: NodeReady
  default       6m5s        Normal   Starting                  node/node3                          Starting kubelet.
  default       6m5s        Normal   NodeHasSufficientMemory   node/node3                          Node node3 status is now: NodeHasSufficientMemory
  default       6m5s        Normal   NodeHasNoDiskPressure     node/node3                          Node node3 status is now: NodeHasNoDiskPressure
  default       6m5s        Normal   NodeHasSufficientPID      node/node3                          Node node3 status is now: NodeHasSufficientPID
  default       6m5s        Normal   NodeAllocatableEnforced   node/node3                          Updated Node Allocatable limit across pods
  default       5m55s       Normal   NodeReady                 node/node3                          Node node3 status is now: NodeReady
  default       4m56s       Normal   Starting                  node/node3                          Starting kube-proxy.
  kube-system   6m24s       Normal   LeaderElection            endpoints/kube-controller-manager   master1_ba3e0eac-2c8a-483a-acd6-e467ef013967 became leader
  kube-system   6m24s       Normal   LeaderElection            lease/kube-controller-manager       master1_ba3e0eac-2c8a-483a-acd6-e467ef013967 became leader
  kube-system   5m15s       Normal   LeaderElection            endpoints/kube-controller-manager   master1_1554b52b-29c2-49ca-9166-172b8e22e639 became leader
  kube-system   5m15s       Normal   LeaderElection            lease/kube-controller-manager       master1_1554b52b-29c2-49ca-9166-172b8e22e639 became leader
  kube-system   2m58s       Normal   LeaderElection            endpoints/kube-controller-manager   master3_4719e0df-a3e0-400b-8e76-f00c97e4276e became leader
  kube-system   2m58s       Normal   LeaderElection            lease/kube-controller-manager       master3_4719e0df-a3e0-400b-8e76-f00c97e4276e became leader
  kube-system   101s        Normal   LeaderElection            endpoints/kube-controller-manager   master3_10b341c2-c43f-4a0c-b968-df80bd3c6f5f became leader
  kube-system   101s        Normal   LeaderElection            lease/kube-controller-manager       master3_10b341c2-c43f-4a0c-b968-df80bd3c6f5f became leader
  kube-system   6m19s       Normal   LeaderElection            endpoints/kube-scheduler            master1_de7c551d-c601-41a0-96e7-e3b6c0c91bbb became leader
  kube-system   6m19s       Normal   LeaderElection            lease/kube-scheduler                master1_de7c551d-c601-41a0-96e7-e3b6c0c91bbb became leader
  </code></pre>
  
  <p>I've turned on the nginx SSL proxy feature, as configured below:</p>
  
  <pre><code>pstream kubernetes-api-cluster { 
      server 192.168.1.67:6443 weight=100 max_fails=0 fail_timeout=3s;
      server 192.168.1.68:6443 weight=100 max_fails=0 fail_timeout=3s; 
      server 192.168.1.69:6443 weight=100 max_fails=0 fail_timeout=3s; 
  } 
  
  server { 
      listen 8443 ssl;
      ssl_certificate /etc/nginx/ssl/master/kube-apiserver.pem;           # kube-apiserver cert
      ssl_certificate_key /etc/nginx/ssl/master/kube-apiserver-key.pem;   # kube-apiserver key
      ssl_trusted_certificate /etc/nginx/ssl/ca.pem;                      # ca.pem
      ssl_prefer_server_ciphers on;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
      location / {
          proxy_ssl_certificate /etc/nginx/ssl/admin.pem;                 # kubectl cert
          proxy_ssl_certificate_key /etc/nginx/ssl/admin-key.pem;         # kubectl key
          proxy_ssl_trusted_certificate /etc/nginx/ssl/ca.pem;            # ca.pem
          proxy_pass https://kubernetes-api-cluster;
          proxy_next_upstream error timeout http_500 http_502 http_503 http_504 http_403 http_429 non_idempotent;
          proxy_next_upstream_timeout 3s;
          proxy_next_upstream_tries 5;
          proxy_ignore_client_abort on;
          proxy_set_header Host $host;
          proxy_set_header X-Real-Ip $remote_addr;
          proxy_set_header X-Forwarded-For $remote_addr;
      }
      access_log /var/log/nginx/access.log default;
  }
  </code></pre>
  
  <p>The kube-apiserver configuration is as follows:</p>
  
  <pre><code>KUBE_APISERVER_OPTS="--logtostderr=true \
  --v=7 \
  --anonymous-auth=false \
  --etcd-servers=https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379 \
  --etcd-cafile=/opt/etcd/ssl/ca.pem \
  --etcd-certfile=/opt/etcd/ssl/server.pem \
  --etcd-keyfile=/opt/etcd/ssl/server-key.pem \
  --service-cluster-ip-range=10.254.0.0/16 \
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
  --bind-address=192.168.1.67 \
  --secure-port=6443 \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --token-auth-file=/opt/kubernetes/cfg/token.csv \
  --allow-privileged=true \
  --tls-cert-file=/opt/kubernetes/ssl/master/kube-apiserver.pem  \
  --tls-private-key-file=/opt/kubernetes/ssl/master/kube-apiserver-key.pem \
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --advertise-address=192.168.1.67 \
  --authorization-mode=RBAC,Node \
  --kubelet-https=true \
  --enable-bootstrap-token-auth \
  --kubelet-certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --kubelet-client-key=/opt/kubernetes/ssl/master/kube-apiserver-key.pem \
  --kubelet-client-certificate=/opt/kubernetes/ssl/master/kube-apiserver.pem \
  --service-node-port-range=30000-50000"
  </code></pre>
  
  <p>The kube-controller-manager configuration is as follows:</p>
  
  <pre><code>KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
  --v=7 \
  --bind-address=0.0.0.0 \
  --cluster-name=kubernetes \
  --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \
  --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --authentication-kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \
  --authorization-kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \
  --leader-elect=true \
  --service-cluster-ip-range=10.254.0.0/16 \
  --controllers=*,bootstrapsigner,tokencleaner \
  --tls-cert-file=/opt/kubernetes/ssl/admin.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/admin-key.pem \
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --secure-port=10257 \
  --use-service-account-credentials=true \
  --experimental-cluster-signing-duration=87600h0m0s"
  </code></pre>
  
  <p>The kube-scheduler configuration is shown below:</p>
  
  <pre><code>KUBE_SCHEDULER_OPTS="--logtostderr=true \
  --v=7 \
  --bind-address=0.0.0.0 \
  --port=10251 \
  --secure-port=10259 \
  --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \
  --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --authentication-kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \
  --authorization-kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --tls-cert-file=/opt/kubernetes/ssl/admin.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/admin-key.pem \
  --leader-elect=true"
  </code></pre>
  
  <p><strong>Part of kubelet's log is as follow:</strong></p>
  
  <pre><code>Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790115   25193 factory.go:170] Factory "raw" can handle container "/user.slice/user-1000.slice", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790429   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-update-utmp.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790489   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-modules-load.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790537   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-resolved.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790585   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-tmpfiles-setup.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790638   25193 factory.go:170] Factory "raw" can handle container "/system.slice/etcd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790690   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-tmpfiles-setup-dev.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790738   25193 factory.go:170] Factory "raw" can handle container "/system.slice/lvm2-lvmetad.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790786   25193 factory.go:170] Factory "raw" can handle container "/system.slice/keyboard-setup.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790829   25193 factory.go:170] Factory "raw" can handle container "/user.slice/user-0.slice", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790874   25193 factory.go:170] Factory "raw" can handle container "/user.slice/user-0.slice/user@0.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790924   25193 factory.go:170] Factory "raw" can handle container "/system.slice/kube-scheduler.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.790969   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/sys-fs-fuse-connections.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791017   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-random-seed.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791060   25193 factory.go:170] Factory "raw" can handle container "/system.slice/lxcfs.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791108   25193 factory.go:170] Factory "raw" can handle container "/system.slice/containerd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791154   25193 factory.go:170] Factory "raw" can handle container "/system.slice/cron.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791200   25193 factory.go:170] Factory "raw" can handle container "/system.slice/flanneld.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791250   25193 factory.go:170] Factory "raw" can handle container "/system.slice/snapd.socket", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791297   25193 factory.go:170] Factory "raw" can handle container "/system.slice/apport.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791342   25193 factory.go:170] Factory "raw" can handle container "/system.slice/setvtrgb.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791388   25193 factory.go:170] Factory "raw" can handle container "/user.slice", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791433   25193 factory.go:170] Factory "raw" can handle container "/system.slice/ufw.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791478   25193 factory.go:170] Factory "raw" can handle container "/system.slice/grub-common.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791523   25193 factory.go:170] Factory "raw" can handle container "/system.slice/kube-proxy.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791570   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/snap-core-6350.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791615   25193 factory.go:170] Factory "raw" can handle container "/system.slice", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791660   25193 factory.go:170] Factory "raw" can handle container "/system.slice/accounts-daemon.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791705   25193 factory.go:170] Factory "raw" can handle container "/system.slice/lxd.socket", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791750   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-user-sessions.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791795   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-sysctl.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791837   25193 factory.go:170] Factory "raw" can handle container "/system.slice/lxd-containers.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791889   25193 factory.go:170] Factory "raw" can handle container "/system.slice/polkit.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791938   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-journal-flush.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.791987   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-networkd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792033   25193 factory.go:170] Factory "raw" can handle container "/system.slice/vboxadd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792078   25193 factory.go:170] Factory "raw" can handle container "/system.slice/console-setup.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792124   25193 factory.go:170] Factory "raw" can handle container "/system.slice/rsyslog.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792173   25193 factory.go:170] Factory "raw" can handle container "/system.slice/kmod-static-nodes.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792217   25193 factory.go:170] Factory "raw" can handle container "/system.slice/apparmor.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792294   25193 factory.go:170] Factory "raw" can handle container "/system.slice/irqbalance.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792374   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-networkd-wait-online.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792422   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/sys-kernel-debug.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792467   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/dev-hugepages.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792511   25193 factory.go:170] Factory "raw" can handle container "/system.slice/ssh.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792561   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/dev-mqueue.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792604   25193 factory.go:170] Factory "raw" can handle container "/system.slice/docker.socket", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792645   25193 factory.go:170] Factory "raw" can handle container "/system.slice/atd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792689   25193 factory.go:170] Factory "raw" can handle container "/system.slice/unattended-upgrades.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792734   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/sys-kernel-config.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792779   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-journald.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792853   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-logind.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792918   25193 factory.go:170] Factory "raw" can handle container "/system.slice/snapd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.792963   25193 factory.go:170] Factory "raw" can handle container "/system.slice/kube-apiserver.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793008   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-remount-fs.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793052   25193 factory.go:170] Factory "raw" can handle container "/system.slice/cloud-init-local.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793097   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-udev-trigger.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793139   25193 factory.go:170] Factory "raw" can handle container "/system.slice/system-getty.slice", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793188   25193 factory.go:170] Factory "raw" can handle container "/system.slice/vboxadd-service.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793238   25193 factory.go:170] Factory "raw" can handle container "/system.slice/lvm2-monitor.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793283   25193 factory.go:170] Factory "raw" can handle container "/system.slice/systemd-udevd.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793328   25193 factory.go:170] Factory "raw" can handle container "/system.slice/ebtables.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793401   25193 factory.go:170] Factory "raw" can handle container "/system.slice/dbus.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793574   25193 factory.go:170] Factory "raw" can handle container "/user.slice/user-0.slice/session-615.scope", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793631   25193 factory.go:170] Factory "raw" can handle container "/system.slice/kube-controller-manager.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793685   25193 factory.go:170] Factory "raw" can handle container "/system.slice/cloud-final.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793733   25193 factory.go:170] Factory "raw" can handle container "/system.slice/blk-availability.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793781   25193 factory.go:170] Factory "raw" can handle container "/system.slice/snapd.seeded.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793834   25193 factory.go:170] Factory "systemd" can handle container "/system.slice/snap-core-8689.mount", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793882   25193 factory.go:170] Factory "raw" can handle container "/system.slice/cloud-config.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.793965   25193 factory.go:170] Factory "raw" can handle container "/system.slice/cloud-init.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.794032   25193 factory.go:170] Factory "raw" can handle container "/system.slice/networkd-dispatcher.service", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.794085   25193 factory.go:170] Factory "raw" can handle container "/system.slice/swap.img.swap", but ignoring.
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.971487   25193 eviction_manager.go:229] eviction manager: synchronize housekeeping
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.984592   25193 helpers.go:781] eviction manager: observations: signal=nodefs.inodesFree, available: 2933345, capacity: 3276800, time: 2020-03-04 15:31:02.972779138 +0000 UTC m=+183.120598420
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.984788   25193 helpers.go:781] eviction manager: observations: signal=imagefs.available, available: 28144320Ki, capacity: 51340768Ki, time: 2020-03-04 15:31:02.972779138 +0000 UTC m=+183.120598420
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.984891   25193 helpers.go:781] eviction manager: observations: signal=imagefs.inodesFree, available: 2933345, capacity: 3276800, time: 2020-03-04 15:31:02.972779138 +0000 UTC m=+183.120598420
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.984986   25193 helpers.go:781] eviction manager: observations: signal=pid.available, available: 32388, capacity: 32Ki, time: 2020-03-04 15:31:02.977127121 +0000 UTC m=+183.124946382
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.985193   25193 helpers.go:781] eviction manager: observations: signal=memory.available, available: 779176Ki, capacity: 2040816Ki, time: 2020-03-04 15:31:02.972779138 +0000 UTC m=+183.120598420
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.985310   25193 helpers.go:781] eviction manager: observations: signal=allocatableMemory.available, available: 2040752Ki, capacity: 2040816Ki, time: 2020-03-04 15:31:02.981299125 +0000 UTC m=+183.129118432
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.985451   25193 helpers.go:781] eviction manager: observations: signal=nodefs.available, available: 28144320Ki, capacity: 51340768Ki, time: 2020-03-04 15:31:02.972779138 +0000 UTC m=+183.120598420
  Mar 04 15:31:02 master2 kubelet[25193]: I0304 15:31:02.985520   25193 eviction_manager.go:320] eviction manager: no resources are starved
  Mar 04 15:31:08 master2 kubelet[25193]: I0304 15:31:08.798836   25193 setters.go:77] Using node IP: "192.168.1.68"
  </code></pre>
  
  <p>Where kube-apiserver.pem and kube-apiserver-key.pem are the certificates and private keys configured for nginx listening ports, <strong>admin.pem</strong> is the certificate of kubectl, and <strong>admin-key</strong> is the private key of kubectl. <strong>And they are also applied to the ssl_proxy related configuration of nginx.</strong>
  The kubectl component can be used normally, and the deployment object created can return 200 normally, but the pod is not actually created.
  It will be very grateful that who can help me solve this problem.</p>