<p>I am trying to deploy the 'lighthouse' application in kubernetes cluster with 'v1.18.14' and the pods are up and running but the logs shows that there is a connection refused error on kubernettes service ip 10.233.0.1 on port 443.</p>
<p>Logs of lighthouse pod -</p>
<pre><code>[centos@master elk_stack_6.x]$ kubectl logs lighthouse-webhooks-7f58c9897c-tbvfz -n jx
{&quot;fields.level&quot;:&quot;info&quot;,&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;setting the log level&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;updating the Lighthouse core configuration&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;warning&quot;,&quot;msg&quot;:&quot;unknown plugin&quot;,&quot;plugin&quot;:&quot;blunderbuss&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;warning&quot;,&quot;msg&quot;:&quot;unknown plugin&quot;,&quot;plugin&quot;:&quot;heart&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;updating the Lighthouse plugins configuration&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;warning&quot;,&quot;msg&quot;:&quot;not pushing metrics as there is no push_gateway defined in the config.yaml&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;Lighthouse is now listening on path /hook and port 8080 for WebHooks&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
{&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;Lighthouse is serving prometheus metrics on port 2112&quot;,&quot;time&quot;:&quot;2021-01-21T03:35:52Z&quot;}
E0129 07:23:34.030912       1 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E0129 07:23:35.031560       1 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
</code></pre>
<p>Service under default namespace -</p>
<pre><code>[centos@master elk_stack_6.x]$ kubectl get svc -n default
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.233.0.1   &lt;none&gt;        443/TCP   12d
</code></pre>
<p>Endpoints of default namespace -</p>
<pre><code>[centos@master elk_stack_6.x]$ kubectl get endpoints -n default
NAME         ENDPOINTS             AGE
kubernetes   167.254.204.56:6443   12d
</code></pre>
<p>Trying to connect kubernetes service using curl -</p>
<pre><code>[centos@master ~]$ curl -kv https://10.233.0.1:443
* About to connect() to 10.233.0.1 port 443 (#0)
*   Trying 10.233.0.1...
* Connected to 10.233.0.1 (10.233.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=kube-apiserver
*       start date: Jan 20 13:49:12 2021 GMT
*       expire date: Jan 20 13:49:12 2022 GMT
*       common name: kube-apiserver
*       issuer: CN=kubernetes
&gt; GET / HTTP/1.1
&gt; User-Agent: curl/7.29.0
&gt; Host: 10.233.0.1
&gt; Accept: */*
&gt;
&lt; HTTP/1.1 403 Forbidden
&lt; Cache-Control: no-cache, private
&lt; Content-Type: application/json
&lt; X-Content-Type-Options: nosniff
&lt; Date: Tue, 02 Feb 2021 07:57:21 GMT
&lt; Content-Length: 233
&lt;
{
  &quot;kind&quot;: &quot;Status&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;metadata&quot;: {

  },
  &quot;status&quot;: &quot;Failure&quot;,
  &quot;message&quot;: &quot;forbidden: User \&quot;system:anonymous\&quot; cannot get path \&quot;/\&quot;&quot;,
  &quot;reason&quot;: &quot;Forbidden&quot;,
  &quot;details&quot;: {

  },
  &quot;code&quot;: 403
* Connection #0 to host 10.233.0.1 left intact
</code></pre>
<p>I am facing 403 forbidden error may be because of some missing RBAC policies. Any suggestions would be appreciated.</p>

Bhavani Prasad
<p>Kindly try the HTTPS endpoint as mentioned in the error instead of HTTP.</p>
<p>400 error would mean that the request is indeed reaching the server but not in a format that the server is expecting.</p>
<p>Updated to reflect updated question:-</p>
<p>403 means the user via which the cluster is being accessed is not authorised to use the IP for accessing the resource.</p>
<p>Detailed explanation can be found <a href="https://tufin.medium.com/protecting-your-kubernetes-api-server-5eefeea4cf8a" rel="nofollow noreferrer">protecting your kubernetes api server</a></p>
