<p>I want to write a script to health check our elasticsearch cluster (deploy on kubernetes)</p>
<ol>
<li>I go inside pod which run elasticsearch master container and run below commands:</li>
</ol>
<pre><code>[elasticsearch@elasticsearch-master-0 ~]$ curl localhost:9200/frontend-dev-2021.12.03/_count
{&quot;count&quot;:76,&quot;_shards&quot;:{&quot;total&quot;:1,&quot;successful&quot;:1,&quot;skipped&quot;:0,&quot;failed&quot;:0}}

[elasticsearch@elasticsearch-master-0 ~]$ curl localhost:9200/_cluster/health?pretty
{
  &quot;cluster_name&quot; : &quot;elasticsearch&quot;,
  &quot;status&quot; : &quot;green&quot;,
  &quot;timed_out&quot; : false,
  &quot;number_of_nodes&quot; : 3,
  &quot;number_of_data_nodes&quot; : 3,
  &quot;active_primary_shards&quot; : 617,
  &quot;active_shards&quot; : 1234,
  &quot;relocating_shards&quot; : 0,
  &quot;initializing_shards&quot; : 0,
  &quot;unassigned_shards&quot; : 0,
  &quot;delayed_unassigned_shards&quot; : 0,
  &quot;number_of_pending_tasks&quot; : 0,
  &quot;number_of_in_flight_fetch&quot; : 0,
  &quot;task_max_waiting_in_queue_millis&quot; : 0,
  &quot;active_shards_percent_as_number&quot; : 100.0
}
</code></pre>
<p>As you can see, both index count and health check command are success.
But when I run these command from outside (I give elasticsearch cluster an public endpoint)</p>
<pre><code>root@ip-192-168-1-1:~# curl --user username:password esdev.example.com/frontend-dev-2021.12.03/_count
{&quot;count&quot;:76,&quot;_shards&quot;:{&quot;total&quot;:1,&quot;successful&quot;:1,&quot;skipped&quot;:0,&quot;failed&quot;:0}}

root@ip-192-168-1-1:~# curl --user username:password esdev.example.com/_cluster/health
&lt;html&gt;
&lt;head&gt;&lt;title&gt;403 Forbidden&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;403 Forbidden&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;

</code></pre>
<p>Only the index count command is success, the health check command always produce <code>403 Forbidden</code> error.</p>
<p>I have searched and read through the official <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html" rel="nofollow noreferrer">docs</a> from elasticsearch but event the offcial docs only run command internal elasticsearch cluster or using kibana (http service kubernetes - internal k8s cluster).</p>
<p>How can I health check elasticsearch from outside? Or we can not do this because some mechanism of elasticsearch cluster?</p>
<p>Notes: I create a basic auth nginx (username:password) stand before the elasticsearch and this nginx has an ingressroute from traefik-v2</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    meta.helm.sh/release-name: basic-auth-nginx-dev
    meta.helm.sh/release-namespace: dev
  creationTimestamp: &quot;2021-01-23T08:12:55Z&quot;
  generation: 2
  labels:
    app: basic-auth-nginx-dev
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: traefik.containo.us/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:entryPoints: {}
        f:routes: {}
    manager: Go-http-client
    operation: Update
    time: &quot;2021-01-23T08:12:55Z&quot;
  name: basic-auth-nginx-dev-web
  namespace: dev
  resourceVersion: &quot;103562796&quot;
  selfLink: /apis/traefik.containo.us/v1alpha1/namespaces/dev/ingressroutes/basic-auth-nginx-dev-web
  uid: 5832b501-b2d7-4600-93b6-b3c72c420115
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`esdev.example.com`) &amp;&amp; PathPrefix(`/`)
    priority: 1
    services:
    - kind: Service
      name: basic-auth-nginx-dev
      port: 80
</code></pre>

<p>Could you please show us your nginx config?</p>
<ol>
<li><p>I think the problem come from your nginx because I see the output you show that nginx return 403 for you, not the elasticsearch.</p>
</li>
<li><p>Could you please try another command start with <code>_</code> like <code>_template</code> or something like that, there is a chance your nginx prevent access to path start with <code>_</code> character.</p>
</li>
</ol>
