<p>I have an installer that spins up two pods in my CI flow, let's call them web and activemq. When the web pod starts it tries to communicate with the activemq pod using the k8s assigned amq-deployment-0.activemq pod name. </p>

<p>Randomly, the web will get an unknown host exception when trying to access amq-deployment1.activemq. If I restart the web pod in this situation the web pod will have no problem communicating with the activemq pod.  </p>

<p>I've logged into the web pod when this happens and the /etc/resolv.conf and /etc/hosts files look fine. The host machines /etc/resolve.conf and /etc/hosts are sparse with nothing that looks questionable.</p>

<p>Information:
There is only 1 worker node.</p>

<p>kubectl --version
Kubernetes v1.8.3+icp+ee</p>

<p>Any ideas on how to go about debugging this issue. I can't think of a good reason for it to happen randomly nor resolve itself on a pod restart. </p>

<p>If there is other useful information needed, I can get it. Thank in advance</p>

<p>For activeMQ we do have this service file</p>

<pre><code>apiVersion: v1 kind: Service
metadata:
    name: activemq
    labels:
            app: myapp
            env: dev
spec:
    ports:
        - port: 8161
          protocol: TCP
          targetPort: 8161
          name: http
        - port: 61616
          protocol: TCP
          targetPort: 61616
          name: amq
    selector:
        component: analytics-amq
        app: myapp
        environment: dev
        type: fa-core
    clusterIP: None
</code></pre>

<p>And this ActiveMQ stateful set (this is the template)</p>

<pre><code>kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: pa-amq-deployment
spec:
  replicas: {{ activemqs }}
  updateStrategy:
    type: RollingUpdate
  serviceName: "activemq"
  template:
      metadata:
          labels:
              component: analytics-amq
              app: myapp
              environment: dev
              type: fa-core
      spec:
          containers:
              - name: pa-amq
                image: default/myco/activemq:latest
                imagePullPolicy: Always
                resources:
                      limits:
                          cpu: 150m
                          memory: 1Gi
                livenessProbe:
                    exec:
                        command:
                        - /etc/init.d/activemq
                        - status
                    initialDelaySeconds: 10
                    periodSeconds: 15
                    failureThreshold: 16
                ports:
                    - containerPort: 8161
                      protocol: TCP
                      name: http
                    - containerPort: 61616
                      protocol: TCP
                      name: amq
                envFrom:
                    - configMapRef:
                        name: pa-activemq-conf-all
                    - secretRef:
                        name: pa-activemq-secret
                volumeMounts:
                    - name: timezone
                      mountPath: /etc/localtime
          volumes:
              - name: timezone
                hostPath:
                  path: /usr/share/zoneinfo/UTC
</code></pre>

<p>The Web stateful set:</p>

<pre><code>apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
    name: pa-web-deployment
spec:
    replicas: 1
    updateStrategy:
        type: RollingUpdate
    serviceName: "pa-web"
    template:
        metadata:
            labels:
                component: analytics-web
                app: myapp
                environment: dev
                type: fa-core
        spec:
            affinity:
              podAntiAffinity:
                preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  podAffinityTerm:
                    labelSelector:
                      matchExpressions:
                      - key: component
                        operator: In
                        values:
                        - analytics-web
                    topologyKey: kubernetes.io/hostname
            containers:
                - name: pa-web
                  image: default/myco/web:latest
                  imagePullPolicy: Always
                  resources:
                        limits:
                            cpu: 1
                            memory: 2Gi
                  readinessProbe:
                      httpGet:
                          path: /versions
                          port: 8080
                      initialDelaySeconds: 30
                      periodSeconds: 15
                      failureThreshold: 76
                  livenessProbe:
                      httpGet:
                          path: /versions
                          port: 8080
                      initialDelaySeconds: 30
                      periodSeconds: 15
                      failureThreshold: 80
                  securityContext:
                      privileged: true
                  ports:
                      - containerPort: 8080
                        name: http
                        protocol: TCP
                  envFrom:
                      - configMapRef:
                         name: pa-web-conf-all
                      - secretRef:
                         name: pa-web-secret
                  volumeMounts:
                      - name: shared-volume
                        mountPath: /MySharedPath
                      - name: timezone
                        mountPath: /etc/localtime
            volumes:
                - nfs:
                    server: 10.100.10.23
                    path: /MySharedPath
                  name: shared-volume
                - name: timezone
                  hostPath:
                    path: /usr/share/zoneinfo/UTC
</code></pre>

<p>This web pod also has a similar "unknown host" problem finding an external database we have configured. The issue being resolved similarly by restarting the pod. Here is the configuration of that external service. Maybe it is easier to tackle the problem from this angle? ActiveMQ has no problem using the database service name to find the DB and startup.</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: dbhost
  labels:
    app: myapp
    env: dev
spec:
  type: ExternalName
  externalName: mydb.host.com
</code></pre>

<p>Is it possible that it is a question of which pod, and the app in its container, is started up first and which second?</p>

<p>In any case, connecting using a <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">Service</a> and not the pod name would be recommended as the pod's name assigned by Kubernetes changes between pod restarts.</p>

<p>A way to test connectivity, is to use <code>telnet</code> (or <a href="https://curl.haxx.se/" rel="nofollow noreferrer"><code>curl</code></a> for the protocols it supports), if found in the image:</p>

<pre><code>telnet &lt;host/pod/Service&gt; &lt;port&gt;
</code></pre>
