<p>I have couple of YAML files for mongo-express/mongodb as below:</p>
<pre><code>apiVersion: v1
kind: Secret
metadata:
    name: mongodb-secret
type: Opaque
data:
    mongo-root-username: dXNlcm5hbWU=
    mongo-root-password: cGFzc3dvcmQ=
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
        - name: mongodb
          image: mongo
          ports:
            - containerPort: 27017
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-username
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom: 
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-password

---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
  labels:
    app: mongodb
spec:
  ports:
    - port: 27017
      targetPort: 27017
      protocol: TCP
  selector:
    app: mongodb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-express
  labels:
    app: mongo-express
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo-express
  template:
    metadata:
      labels:
        app: mongo-express
    spec:
      containers:
        - name: mongo-express
          image: mongo-express
          ports:
            - containerPort: 8081
          env:
            - name: ME_CONFIG_MONGODB_ADMINUSERNAME
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-username
            - name: ME_CONFIG_MONGODB_ADMINPASSWORD
              valueFrom: 
                secretKeyRef:
                  name: mongodb-secret
                  key: mongo-root-password
            - name: ME_CONFIG_MONGODB_SERVER
              value: mongodb-service
---
apiVersion: v1
kind: Service
metadata:
  name: mongo-express-service
  labels:
    app: mongo-express
spec:
  ports:
    - port: 8081
      targetPort: 8081
      nodePort: 30000
      protocol: TCP
  type: LoadBalancer
  selector:
    app: mongo-express
</code></pre>
<p>I can apply above YAML file on local <strong>minikube</strong> cluster when I execute minikube service mongo-express-service.
I can also apply it to my 3 nodes <strong>kubernetes</strong> cluster, but the mongo-express <strong>Pod</strong> seems not available to connect to mongodb-service.</p>
<p>This is my initial troubleshoot.</p>
<pre><code>$ kubectl get nodes -o wide
NAME        STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master-20   Ready    master   35d   v1.19.2   192.168.0.20   &lt;none&gt;        Ubuntu 16.04.7 LTS   4.4.0-186-generic   docker://19.3.13
node1       Ready    &lt;none&gt;   35d   v1.19.2   192.168.0.21   &lt;none&gt;        Ubuntu 16.04.7 LTS   4.4.0-186-generic   docker://19.3.13
node2       Ready    &lt;none&gt;   35d   v1.19.2   192.168.0.22   &lt;none&gt;        Ubuntu 16.04.7 LTS   4.4.0-186-generic   docker://19.3.13

$ kubectl get all -o wide
NAME                                     READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
pod/mongo-express-749445c6c9-wlnx8       1/1     Running   0          18s   10.244.2.23   node2   &lt;none&gt;           &lt;none&gt;
pod/mongodb-deployment-8f6675bc5-w9wks   1/1     Running   0          22s   10.244.1.20   node1   &lt;none&gt;           &lt;none&gt;

NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE   SELECTOR
service/kubernetes              ClusterIP      10.96.0.1      &lt;none&gt;        443/TCP          35d   &lt;none&gt;
service/mongo-express-service   LoadBalancer   10.108.20.77   &lt;pending&gt;     8081:30000/TCP   18s   app=mongo-express
service/mongodb-service         ClusterIP      10.98.48.206   &lt;none&gt;        27017/TCP        22s   app=mongodb

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS      IMAGES          SELECTOR
deployment.apps/mongo-express        1/1     1            1           18s   mongo-express   mongo-express   app=mongo-express
deployment.apps/mongodb-deployment   1/1     1            1           22s   mongodb         mongo           app=mongodb

NAME                                           DESIRED   CURRENT   READY   AGE   CONTAINERS      IMAGES          SELECTOR
replicaset.apps/mongo-express-749445c6c9       1         1         1       18s   mongo-express   mongo-express   app=mongo-express,pod-template-hash=749445c6c9
replicaset.apps/mongodb-deployment-8f6675bc5   1         1         1       22s   mongodb         mongo           app=mongodb,pod-template-hash=8f6675bc5


$ kubectl logs mongo-express-749445c6c9-wlnx8
Waiting for mongodb-service:27017...
/docker-entrypoint.sh: line 14: mongodb-service: Try again
/docker-entrypoint.sh: line 14: /dev/tcp/mongodb-service/27017: Invalid argument
Sun Nov  8 05:29:40 UTC 2020 retrying to connect to mongodb-service:27017 (2/5)


$ kubectl logs mongodb-deployment-8f6675bc5-w9wks
about to fork child process, waiting until server is ready for connections.
forked process: 28

...

MongoDB init process complete; ready for start up.

{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.631+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23285,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.634+00:00&quot;},&quot;s&quot;:&quot;W&quot;,  &quot;c&quot;:&quot;ASIO&quot;,     &quot;id&quot;:22601,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;No TransportLayer configured during NetworkInterface startup&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.634+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:4648601, &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.636+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:4615611, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;MongoDB starting&quot;,&quot;attr&quot;:{&quot;pid&quot;:1,&quot;port&quot;:27017,&quot;dbPath&quot;:&quot;/data/db&quot;,&quot;architecture&quot;:&quot;64-bit&quot;,&quot;host&quot;:&quot;mongodb-deployment-8f6675bc5-w9wks&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.636+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23403,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Build Info&quot;,&quot;attr&quot;:{&quot;buildInfo&quot;:{&quot;version&quot;:&quot;4.4.1&quot;,&quot;gitVersion&quot;:&quot;ad91a93a5a31e175f5cbf8c69561e788bbc55ce1&quot;,&quot;openSSLVersion&quot;:&quot;OpenSSL 1.1.1  11 Sep 2018&quot;,&quot;modules&quot;:[],&quot;allocator&quot;:&quot;tcmalloc&quot;,&quot;environment&quot;:{&quot;distmod&quot;:&quot;ubuntu1804&quot;,&quot;distarch&quot;:&quot;x86_64&quot;,&quot;target_arch&quot;:&quot;x86_64&quot;}}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.636+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:51765,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Operating System&quot;,&quot;attr&quot;:{&quot;os&quot;:{&quot;name&quot;:&quot;Ubuntu&quot;,&quot;version&quot;:&quot;18.04&quot;}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.636+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:21951,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Options set by command line&quot;,&quot;attr&quot;:{&quot;options&quot;:{&quot;net&quot;:{&quot;bindIp&quot;:&quot;*&quot;},&quot;security&quot;:{&quot;authorization&quot;:&quot;enabled&quot;}}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.638+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22270,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Storage engine to use detected by data files&quot;,&quot;attr&quot;:{&quot;dbpath&quot;:&quot;/data/db&quot;,&quot;storageEngine&quot;:&quot;wiredTiger&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.639+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22297,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem&quot;,&quot;tags&quot;:[&quot;startupWarnings&quot;]}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:54.639+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22315,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Opening WiredTiger&quot;,&quot;attr&quot;:{&quot;config&quot;:&quot;create,cache_size=479M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:56.498+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813336:498796][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 1 through 2&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:56.889+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813336:889036][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 2 through 2&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.525+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813337:525554][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 1/25728 to 2/256&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.682+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813337:682506][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 1 through 2&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.791+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813337:791351][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 2 through 2&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.880+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813337:880334][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.880+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22430,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;[1604813337:880542][1:0x7f7ffb631a80], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.892+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:4795906, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger opened&quot;,&quot;attr&quot;:{&quot;durationMillis&quot;:3253}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.893+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;RECOVERY&quot;, &quot;id&quot;:23987,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger recoveryTimestamp&quot;,&quot;attr&quot;:{&quot;recoveryTimestamp&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:0,&quot;i&quot;:0}}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.913+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22262,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Timestamp monitor starting&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.940+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:20536,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Flow Control is enabled on this deployment&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.950+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;FTDC&quot;,     &quot;id&quot;:20625,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Initializing full-time diagnostic data capture&quot;,&quot;attr&quot;:{&quot;dataDirectory&quot;:&quot;/data/db/diagnostic.data&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.958+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:23015,   &quot;ctx&quot;:&quot;listener&quot;,&quot;msg&quot;:&quot;Listening on&quot;,&quot;attr&quot;:{&quot;address&quot;:&quot;/tmp/mongodb-27017.sock&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.958+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:23015,   &quot;ctx&quot;:&quot;listener&quot;,&quot;msg&quot;:&quot;Listening on&quot;,&quot;attr&quot;:{&quot;address&quot;:&quot;0.0.0.0&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-11-08T05:28:57.958+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:23016,   &quot;ctx&quot;:&quot;listener&quot;,&quot;msg&quot;:&quot;Waiting for connections&quot;,&quot;attr&quot;:{&quot;port&quot;:27017,&quot;ssl&quot;:&quot;off&quot;}}
</code></pre>

Steven Weng
<p>As suggested in the comments those kind of problems usually indicates issues with coredns and dns resolution. Is it worth to mention that Kubernetes <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="nofollow noreferrer">documentation</a> goes thru couple of good dns troubleshooting steps:</p>
<ul>
<li><p>Check the local DNS configuration first</p>
<pre><code> kubectl exec -ti dnsutils -- cat /etc/resolv.conf
</code></pre>
</li>
<li><p>Check if the DNS pod is running</p>
<pre><code> kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
</code></pre>
</li>
<li><p>Check for errors in the DNS pod</p>
<pre><code> kubectl logs --namespace=kube-system -l k8s-app=kube-dns
</code></pre>
</li>
<li><p>Are DNS endpoints exposed?</p>
<pre><code> kubectl get endpoints kube-dns --namespace=kube-system
</code></pre>
</li>
</ul>
<hr />
<p>To summarize OP confirmed that the issue was related to coredns and changing the nameserver in  /etc/resolve.conf solved the issue.</p>
