<p>While trying to setup Cassandra database in a local Kubernetes cluster on a Mac OS (via Minikube), I am getting connection issues. 
It seems like Node.js is not able to resolve DNS settings correctly, but resolving via command line DOES work.</p>

<p>The setup is as following (simplified):
Cassandra Service</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  type: NodePort
  ports:
    - port: 9042
      targetPort: 9042
      protocol: TCP
      name: http
  selector:
    app: cassandra
</code></pre>

<p>In addition, there's a PersistentVolume and a StatefulSet.</p>

<p>The application itself is very basic</p>

<pre><code>apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: app1
  labels:
    app: app1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
      - name: app1
        image: xxxx.dkr.ecr.us-west-2.amazonaws.com/acme/app1
        imagePullPolicy: "Always"
        ports:
        - containerPort: 3003
</code></pre>

<p>And a service</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: app1
  namespace: default
spec:
  selector:
    app: app1
  type: NodePort
  ports:
    - port: 3003
      targetPort: 3003
      protocol: TCP
      name: http
</code></pre>

<p>there also a simple ingress setup</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: dev.acme.com
    http:
      paths:
      - path: /app1
        backend:
          serviceName: app1
          servicePort: 3003
</code></pre>

<p>And added to <code>/etc/hosts</code> the minikube ip address</p>

<pre><code>192.xxx.xx.xxx dev.acme.com
</code></pre>

<p>So far so good.</p>

<p>When trying to call <code>dev.acme.com/app1</code> via Postman, the node.js app itself is being called correctly (can see in the logs), HOWEVER, the app can not connect to Cassandra and times out with the following error:</p>

<blockquote>
  <p>"All host(s) tried for query failed. First host tried,
  92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."</p>
</blockquote>

<p>The IP <code>92.242.140.2</code> seems to be just a public IP that is related to my ISP, I believe since the app is not able to resolve the service name.</p>

<p>I created a simple node.js script to test dns:</p>

<p><code>var dns = require('dns')
dns.resolve6('cassandra', (err, res) =&gt; console.log('ERR:', err, 'RES:', res))</code></p>

<p>and the response is</p>

<blockquote>
  <p>ERR: { Error: queryAaaa ENOTFOUND cassandra
      at QueryReqWrap.onresolve [as oncomplete] (dns.js:197:19)   errno: 'ENOTFOUND',   code: 'ENOTFOUND',   syscall: 'queryAaaa',   hostname:
  'cassandra' } RES: undefined</p>
</blockquote>

<p>However, and this is where it gets confusing - when I ssh into the pod (app1), I am able to connect to cassandra service using:</p>

<p><code>cqlsh cassandra 9042 --cqlversion=3.4.4</code></p>

<p>So it seems as the pod is "aware" of the service name, but node.js runtime is not.</p>

<p>Any idea what could cause the node.js to not being able to resolve the service name/dns settings?</p>

<p><strong>UPDATE</strong></p>

<p>After re-installing the whole cluster, including re-installing docker, kubectl and minikube I am getting the same issue.</p>

<p>While running <code>ping cassandra</code> from app1 container via ssh, I am getting the following</p>

<blockquote>
  <p>PING cassandra.default.svc.cluster.local (10.96.239.137) 56(84) bytes
  of data. 64 bytes from cassandra.default.svc.cluster.local
  (10.96.239.137): icmp_seq=1 ttl=61 time=27.0 ms</p>
  
  <p>2 packets transmitted, 2 received, 0% packet loss, time 1001ms</p>
</blockquote>

<p>Which seems to be fine.
However, when running from Node.js runtime I am still getting the same error - </p>

<blockquote>
  <p>"All host(s) tried for query failed. First host tried,
  92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."</p>
</blockquote>

<p>These are the services</p>

<pre><code>NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
app1        ClusterIP   None            &lt;none&gt;        3003/TCP         11m
cassandra    NodePort    10.96.239.137   &lt;none&gt;        9042:32564/TCP   38h
kubernetes   ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP          38h
</code></pre>

<p>And these are the pods (all namespaces)</p>

<pre><code>NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
default       app1-85d889db5-m977z                       1/1     Running   0          2m1s
default       cassandra-0                                 1/1     Running   0          38h
kube-system   calico-etcd-ccvs8                           1/1     Running   0          38h
kube-system   calico-node-thzwx                           2/2     Running   0          38h
kube-system   calico-policy-controller-5bb4fc6cdc-cnhrt   1/1     Running   0          38h
kube-system   coredns-86c58d9df4-z8pr4                    1/1     Running   0          38h
kube-system   coredns-86c58d9df4-zcn6p                    1/1     Running   0          38h
kube-system   default-http-backend-5ff9d456ff-84zb5       1/1     Running   0          38h
kube-system   etcd-minikube                               1/1     Running   0          38h
kube-system   kube-addon-manager-minikube                 1/1     Running   0          38h
kube-system   kube-apiserver-minikube                     1/1     Running   0          38h
kube-system   kube-controller-manager-minikube            1/1     Running   0          38h
kube-system   kube-proxy-jj7c4                            1/1     Running   0          38h
kube-system   kube-scheduler-minikube                     1/1     Running   0          38h
kube-system   kubernetes-dashboard-ccc79bfc9-6jtgq        1/1     Running   4          38h
kube-system   nginx-ingress-controller-7c66d668b-rvxpc    1/1     Running   0          38h
kube-system   registry-creds-x5bhl                        1/1     Running   0          38h
kube-system   storage-provisioner                         1/1     Running   0          38h
</code></pre>

<p><strong>UPDATE 2</strong></p>

<p>The code to connect to Cassandra from Node.js:</p>

<pre><code>const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['cassandra:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });

const query = 'SELECT * FROM user';
client.execute(query, [])
  .then(result =&gt; console.log('User with email %s', result.rows[0].email));
</code></pre>

<p>It DOES work when replacing <code>cassandra:9042</code> with <code>10.96.239.137:9042</code> (10.69.239.137 is the ip address received from pinging cassandra via cli).</p>

<p>The Cassandra driver for Node.js uses <code>resolve4</code>/<code>resolve6</code> to do its dns lookup, which bypasses your <code>resolv.conf</code> file. A program like ping uses resolv.conf to resolve 'cassandra' to 'cassandra.default.svc.cluster.local', the actual dns name assigned to your Cassandra service. For a more detailed explanation of name resolution in node.js see <a href="https://stackoverflow.com/questions/40985367/node-js-dns-resolve-vs-dns-lookup">here</a>.</p>

<p>The fix is simple, just pass in the full service name to your client:</p>

<pre><code>const client = new cassandra.Client({ contactPoints: ['cassandra.default.svc.cluster.local:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });
</code></pre>
