<p>I have a kubernetes cluster on which I have deployed a opensearch cluster and opensearch dashboard using Helm, I am also able to deploy logstash using helm successfully but I am confused on how to integrate those, I want to feed data to my Opensearch using logstash as my OBJECTIVE as I am not able to find much documentation on it as well. Any help is appreciated....Thanks in advance!</p>
<p>Deployed opensearch using Helm and logstash as well but unable to integrate them</p>
<p><strong>Update here!!!</strong></p>
<p>Have made a few changes to simplify the deployment and more control over the function,</p>
<p>I am testing deployment and service files this time, I will add the files below</p>
<p>Opensearch deployment file</p>
<pre><code>
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: logging
  name: opensearch
  labels:
    component: opensearch
spec:
  selector:
    matchLabels:
      component: opensearch
  replicas: 1
  serviceName: opensearch
  template:
    metadata:
      labels:
        component: opensearch
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          imagePullPolicy: IfNotPresent
          command:
            - sysctl
            - -w
            - vm.max_map_count=262144
          securityContext:
            privileged: true
      containers:
        - name: opensearch
          securityContext:
            capabilities:
              add:
                - IPC_LOCK
          image: opensearchproject/opensearch
          env:
            - name: KUBERNETES_CA_CERTIFICATE_FILE
              value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: &quot;cluster.name&quot;
              value: &quot;opensearch-cluster&quot;
            - name: &quot;network.host&quot;
              value: &quot;0.0.0.0&quot;
            - name: &quot;discovery.seed_hosts&quot;
              value: &quot;[]&quot;
            - name: discovery.type
              value: single-node              
            - name: OPENSEARCH_JAVA_OPTS
              value: -Xmx512M -Xms512M
            - name: &quot;plugins.security.disabled&quot;
              value: &quot;false&quot; 
          ports:
            - containerPort: 9200
              name: http
              protocol: TCP
            - containerPort: 9300
              name: transport
              protocol: TCP
          volumeMounts:
            - name: os-mount
              mountPath: /data
      volumes:
        - name: os-mount
          persistentVolumeClaim:
            claimName: nfs-pvc-os-logging


</code></pre>
<p>Opensearch svc file</p>
<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: opensearch
  namespace: logging
  labels:
    service: opensearch
spec:
  type: ClusterIP
  selector:
    component: opensearch
  ports:
  - port: 9200
    targetPort: 9200

</code></pre>
<p>Opensearch dashboard deployment</p>
<pre><code>---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-dash
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: open-dash
  template:
    metadata:
      labels:
        app: open-dash
    spec:
#      securityContext:
#      runAsUser: 0
            
      containers:
        - name: opensearch-dashboard
          image: opensearchproject/opensearch-dashboards:latest
          ports:
            - containerPort: 80
          env:
#            - name: ELASTICSEARCH_URL
#              value: https://opensearch.logging:9200
#            - name: &quot;SERVER_HOST&quot;
#              value: &quot;localhost&quot;
#            - name: &quot;opensearch.hosts&quot; 
#              value: https://opensearch.logging:9200
            - name: OPENSEARCH_HOSTS
              value:  '[&quot;https://opensearch.logging:9200&quot;]'  


</code></pre>
<p>Opensearch Dashboard svc</p>
<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: opensearch
  namespace: logging
  labels:
    service: opensearch
spec:
  type: ClusterIP
  selector:
    component: opensearch
  ports:
  - port: 9200
    targetPort: 9200


</code></pre>
<p>with the above configuration I am able to get the Dashboard UI open but in Dashboard pod logs I can see a 400 code logs can anyone please try to reproduce this issue, Also I need to integrate the logstash with this stack.</p>
<blockquote>
<p>{&quot;type&quot;:&quot;response&quot;,&quot;@timestamp&quot;:&quot;2023-02-20T05:05:34Z&quot;,&quot;tags&quot;:[],&quot;pid&quot;:1,&quot;method&quot;:&quot;head&quot;,&quot;statusCode&quot;:400,&quot;req&quot;:{&quot;url&quot;:&quot;/app/home&quot;,&quot;method&quot;:&quot;head&quot;,&quot;headers&quot;:{&quot;connection&quot;:&quot;Keep-Alive&quot;,&quot;content-type&quot;:&quot;application/json&quot;,&quot;host&quot;:&quot;3.108.199.0:30406&quot;,&quot;user-agent&quot;:&quot;Manticore 0.9.1&quot;,&quot;accept-encoding&quot;:&quot;gzip,deflate&quot;,&quot;securitytenant&quot;:&quot;<strong>user</strong>&quot;},&quot;remoteAddress&quot;:&quot;10.244.1.1&quot;,&quot;userAgent&quot;:&quot;Manticore 0.9.1&quot;},&quot;res&quot;:{&quot;statusCode&quot;:400,&quot;responseTime&quot;:2,&quot;contentLength&quot;:9},&quot;message&quot;:&quot;HEAD /app/home 400 2ms - 9.0B</p>
</blockquote>
<p>When deploying a <strong>logstash</strong> pod I get an error that</p>
<blockquote>
<p>[WARN ] 2023-02-20 05:13:52.212 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=&gt;&quot;http://logstash:xxxxxx@opensearch.logging:9200/&quot;, :exception=&gt;LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=&gt;&quot;OpenSearch Unreachable: [http://logstash:xxxxxx@opensearch.logging:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond&quot;}</p>
</blockquote>
<p>Can somebody please try to help me with this puzzle</p>
<p>@Benla have made changes as per your recommendation to image now I am getting the following logs in logstash</p>
<blockquote>
<p>[2023-02-20T05:18:43,028][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=&gt;9600, :ssl_enabled=&gt;false}
[2023-02-20T05:18:43,147][INFO ][org.reflections.Reflections] Reflections took 70 ms to scan 1 urls, producing 127 keys and 444 values
[2023-02-20T05:18:43,351][INFO ][logstash.javapipeline    ] Pipeline <code>main</code> is configured with <code>pipeline.ecs_compatibility: v8</code> setting. All plugins in this pipeline will default to <code>ecs_compatibility =&gt; v8</code> unless explicitly configured otherwise.
[2023-02-20T05:18:43,370][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=&gt;&quot;main&quot;, &quot;pipeline.workers&quot;=&gt;16, &quot;pipeline.batch.size&quot;=&gt;125, &quot;pipeline.batch.delay&quot;=&gt;50, &quot;pipeline.max_inflight&quot;=&gt;2000, &quot;pipeline.sources&quot;=&gt;[&quot;/usr/share/logstash/pipeline/logstash.conf&quot;], :thread=&gt;&quot;#&lt;Thread:0x3bf49916@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:131 run&gt;&quot;}
[2023-02-20T05:18:43,811][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {&quot;seconds&quot;=&gt;0.44}
[2023-02-20T05:18:43,816][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=&gt;&quot;0.0.0.0:5044&quot;}
[2023-02-20T05:18:43,821][INFO ][logstash.javapipeline    ][main] Pipeline started {&quot;pipeline.id&quot;=&gt;&quot;main&quot;}
[2023-02-20T05:18:43,835][INFO ][logstash.agent           ] Pipelines running {:count=&gt;1, :running_pipelines=&gt;[:main], :non_running_pipelines=&gt;[]}
[2023-02-20T05:18:43,869][INFO ][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] Starting server on port: 5044</p>
</blockquote>
<p>I started getting these endless loop of logs</p>
<blockquote>
<p>[WARN ] 2023-02-20 05:13:37.191 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=&gt;&quot;http://logstash:xxxxxx@opensearch.logging:9200/&quot;, :exception=&gt;LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=&gt;&quot;OpenSearch Unreachable: [http://logstash:xxxxxx@opensearch.logging:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond&quot;}</p>
</blockquote>

<p>for opensearch sso(keycloak) please use below steps:
opensearch:</p>
<ol>
<li><p>Make a custom iamge for opensearch, for this make 2 files as below.
i.config.yml(for opensearch security plugin)
---
_meta:
type: &quot;config&quot;
config_version: 2</p>
<pre><code> config:
   dynamic:
     http:
       anonymous_auth_enabled: false
     authc:
       internal_auth:
         order: 0
         description: &quot;HTTP basic authentication using the internal user database&quot;
         http_enabled: true
         transport_enabled: true
         http_authenticator:
           type: basic
           challenge: false
         authentication_backend:
           type: internal
       openid_auth_domain:
         http_enabled: true
         transport_enabled: true
         order: 1
         http_authenticator:
           type: openid
           challenge: false
           config:
             subject_key: preferred_username
             roles_key: roles
             openid_connect_url: &quot;https://keycloak-url/realms/realm-name/.well-known/openid-configuration&quot;    
         authentication_backend:
           type: noop 
 ---
</code></pre>
</li>
</ol>
<p>ii.
log4j2.properties(this file will start logs in opensearch so we can see logs which are otherwise turned-off)</p>
<pre><code>    ---
    logger.securityjwt.name = com.amazon.dlic.auth.http.jwt
    logger.securityjwt.level = trace
    ---
</code></pre>
<p>iii. Dockerfile</p>
<pre><code>---
FROM opensearchproject/opensearch:2.5.0
RUN mkdir /usr/share/opensearch/plugins/opensearch-security/securityconfig
COPY config.yaml /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
COPY config.yaml /usr/share/opensearch/config/opensearch-security/config.yml
COPY log4j2.properties /usr/share/opensearch/config/log4j2.properties
---
</code></pre>
<ol start="2">
<li><p>Deploy opensearch with opensearch helm chart(change image with your customimage built using above configs).
opensearch will deploy 3 pods.now go in each pod and fire belo command to start security plugin(do this only once for each pod of opensearch).</p>
<hr />
<p>/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh<br />
-cacert /usr/share/opensearch/config/root-ca.pem <br />
-cert /usr/share/opensearch/config/kirk.pem <br />
-key /usr/share/opensearch/config/kirk-key.pem <br />
-cd /usr/share/opensearch/config/opensearch-security <br />
-h localhost</p>
<hr />
<p>make sure all 3 pods are up and in ready state.
opensearch-dashboard</p>
</li>
</ol>
<p>3.Now we will configure opensearch-dashboard
i. In values.yml of helm chart of opensearch-dashboard search for config</p>
<pre><code>---    
config: 
    opensearch_dashboards.yml: |
        opensearch.hosts: [https://localhost:9200]
        opensearch.ssl.verificationMode: none
        opensearch.username: admin
        opensearch.password: admin
        opensearch.requestHeadersWhitelist: [authorization, securitytenant]
        opensearch_security.multitenancy.enabled: true
        opensearch_security.multitenancy.tenants.preferred: [Private, Global]
        opensearch_security.readonly_mode.roles: [kibana_read_only]
        opensearch_security.cookie.secure: false
        server.host: '0.0.0.0'

        opensearch_security.auth.type: &quot;openid&quot;
        opensearch_security.openid.connect_url: &quot;https://keycloak-url/realms/realm-name/.well-known/openid-configuration&quot;
        opensearch_security.openid.client_id: &quot;admin&quot;
        opensearch_security.openid.client_secret: &quot;asgduasdjsadk&quot;
        opensearch_security.openid.scope: &quot;email openid&quot;
        opensearch_security.openid.base_redirect_url: &quot;https://opensearch_dashboards-url.io&quot;
---
</code></pre>
<p>ii. deploy opensearch_dashboards.</p>
<pre><code>Now once opensearch_dashboards is deployed and pod is in up and ready state you can go to https://opensearch_dashboards-url.io (your opensearch_dashboards url ) and you will see keycloak login form.
</code></pre>
