<p>I'm attempting to use the <a href="https://plugins.jenkins.io/statistics-gatherer/%3E" rel="nofollow noreferrer">Statistics Gathering</a> Jenkins plugin to forward metrics to Logstash. The plugin is configured with the following url: <code>http://logstash.monitoring-observability:9000</code>. Both Jenkins and Logstash are deployed on Kubernetes. When I run a build, which triggers metrics forwarding via this plugin, I see the following error in the logs:</p>
<pre><code>2022-02-19 23:29:20.464+0000 [id=263]   WARNING o.j.p.s.g.util.RestClientUtil$1#failed: The request for url http://logstash.monitoring-observability:9000/ has failed.
java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173
</code></pre>
<p>I get the same behavior when I exec into the jenkins pod and attempt to curl logstash:</p>
<pre><code>jenkins@jenkins-7889fb54b8-d9rvr:/$ curl -vvv logstash.monitoring-observability:9000
*   Trying 10.52.9.143:9000...
* connect to 10.52.9.143 port 9000 failed: Connection refused
* Failed to connect to logstash.monitoring-observability port 9000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to logstash.monitoring-observability port 9000: Connection refused
</code></pre>
<p>I also get the following error in the logstash logs:</p>
<pre><code>[ERROR] 2022-02-20 00:05:43.450 [[main]&lt;tcp] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: &lt;LogStash::Inputs::Tcp port=&gt;9000, codec=&gt;&lt;LogStash::Codecs::JSON id=&gt;&quot;json_f96babad-299c-42ab-98e0-b78c025d9476&quot;, enable_metric=&gt;true, charset=&gt;&quot;UTF-8&quot;&gt;, host=&gt;&quot;jenkins-server.devops-tools&quot;, ssl_verify=&gt;false, id=&gt;&quot;0fddd9afb2fcf12beb75af799a2d771b99af6ac4807f5a67f4ec5e13f008803f&quot;, enable_metric=&gt;true, mode=&gt;&quot;server&quot;, proxy_protocol=&gt;false, ssl_enable=&gt;false, ssl_key_passphrase=&gt;&lt;password&gt;&gt;
  Error: Cannot assign requested address
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
</code></pre>
<p>Here is my jenkins-deployment.yaml:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: devops-tools
  labels:
    app: jenkins-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
            fsGroup: 1000 
            runAsUser: 1000
      serviceAccountName: jenkins-admin
      containers:
        - name: jenkins
          env:
          - name: LOGSTASH_HOST
            value: logstash
          - name: LOGSTASH_PORT
            value: &quot;5044&quot;
          - name: ELASTICSEARCH_HOST
            value: elasticsearch-logging
          - name: ELASTICSEARCH_PORT
            value: &quot;9200&quot;
          - name: ELASTICSEARCH_USERNAME
            value: elastic
          - name: ELASTICSEARCH_PASSWORD
            value: changeme
          image: jenkins/jenkins:lts
          resources:
            limits:
              memory: &quot;2Gi&quot;
              cpu: &quot;1000m&quot;
            requests:
              memory: &quot;500Mi&quot;
              cpu: &quot;500m&quot;
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: &quot;/login&quot;
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: &quot;/login&quot;
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home         
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pv-claim
</code></pre>
<p>Here is my jenkins-service.yaml</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: jenkins-server
  namespace: devops-tools
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector: 
    app: jenkins-server
    k8s-app: jenkins-server
  type: NodePort  
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30000
</code></pre>
<p>Here is my logstash-deployment.yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash-deployment
  namespace: monitoring-observability
  labels:
    app: logstash
spec:
  selector:
    matchLabels:
      app: logstash
  replicas: 1
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        env:
          - name: JENKINS_HOST
            value: jenkins-server
          - name: JENKINS_PORT
            value: &quot;8080&quot;
        image: docker.elastic.co/logstash/logstash:6.3.0
        ports:
        - containerPort: 9000
        volumeMounts:
          - name: config-volume
            mountPath: /usr/share/logstash/config
          - name: logstash-pipeline-volume
            mountPath: /usr/share/logstash/pipeline
      volumes:
      - name: config-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.yml
              path: logstash.yml
      - name: logstash-pipeline-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.conf
              path: logstash.conf
</code></pre>
<p>Here is my logstash-service.yaml</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: logstash
  namespace: monitoring-observability
  labels:
    app: logstash
    kubernetes.io/cluster-service: &quot;true&quot;
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: &quot;logstash&quot;
spec:
  selector:
    app: logstash
  ports:
  - protocol: TCP
    port: 9000
    targetPort: 9000
  type: ClusterIP
</code></pre>
<p>Here is my logstash configmap:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: monitoring-observability
data:
  logstash.yml: |
    http.host: &quot;0.0.0.0&quot;
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    input {
      tcp {
        port =&gt; &quot;9000&quot;
        codec =&gt; &quot;json&quot;
        host =&gt; &quot;jenkins-server.devops-tools&quot;
        ssl_verify =&gt; &quot;false&quot;
      }
    }
    filter {
      if [message] =~ /^\{.*\}$/ {
        json {
          source =&gt; &quot;message&quot;
        }
      }
      if [ClientHost] {
        geoip {
          source =&gt; &quot;ClientHost&quot;
        }
      }
    }
    output {
        elasticsearch {
            hosts =&gt; [ &quot;elasticsearch-logging:9200&quot; ]
        }
    }
</code></pre>
<p>There are no firewalls configured in my cluster that would be blocking traffic on port 9000. I have also tried this same configuration with port <code>5044</code> and get the same results. It seems as though my logstash instance is not actually listening on the <code>containerPort</code>. Why might this be?</p>

<p>I resolved this error by updating the configmap to this:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: monitoring-observability
data:
  logstash.yml: |
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    input {
      tcp {
        port =&gt; &quot;9000&quot;
        codec =&gt; &quot;json&quot;
        ssl_verify =&gt; &quot;false&quot;
      }
    }
    filter {
      if [message] =~ /^\{.*\}$/ {
        json {
          source =&gt; &quot;message&quot;
        }
      }
      if [ClientHost] {
        geoip {
          source =&gt; &quot;ClientHost&quot;
        }
      }
    }
    output {
        elasticsearch {
            hosts =&gt; [ &quot;elasticsearch-logging:9200&quot; ]
        }
    }
</code></pre>
<p>Note that all references to the jenkins host have been removed.</p>
