<!--

Welcome to ingress-nginx!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-).
More info helps better understanding of the issue (needless to say).

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

-->

<!--

IMPORTANT!!!

Please complete the next sections or the issue will be closed.
This questions are the first thing we need to know to understand the context.

-->

**What happened**:

<!-- (please include exact error messages if you can) -->

We had a unique failure where coredns pods were removed/replaced during an upgrade. ingress-nginx was sending traffic to the old pod IPs which was getting dropped by cilium (because they were stale IPs / not assigned to anything.)

The default dns timeout is 5 retries x 2000ms - so a 10s delay happens, even when global-rate-limit-memcached-connect-timeout is set to 50ms.

https://github.com/kubernetes/ingress-nginx/blob/300f772dd75c9a8d6df7831c97cd5a503235e1dc/rootfs/etc/nginx/lua/util/dns.lua#L99

This is the resulting log message

```
2024/04/26 21:45:14 [error] 1043#1043: *24280775 [lua] global_throttle.lua:105: throttle(): error while processing key: failed to connect: memcached.ingress-nginx.svc.cluster.local could not be resolved (110: Operation timed out), ...
```

**What you expected to happen**:

1- dns should hit the service and not the pod ips so when they're replaced dns doesn't just timeout. (I haven't figured this one out yet.)

2- the 50ms timeout should be followed even if dns is down.

For #2, it's easily reproducible:

Ingress values.yaml

```
controller:
  config:
    global-rate-limit-memcached-host: memcached.ingress-nginx.svc.cluster.local.
```

All the manifests
```
---
apiVersion: v1
kind: Service
metadata:
  name: memcached
  namespace: ingress-nginx
spec:
  selector:
    app: memcached
  ports:
    - protocol: TCP
      port: 11211
      targetPort: 11211
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memcached
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memcached
  template:
    metadata:
      labels:
        app: memcached
    spec:
      containers:
      - name: memcached
        image: docker.io/memcached
        command:
        - memcached
        args:
        - '-m'
        - '64'
        ports:
        - containerPort: 11211
        imagePullPolicy: Always
        readinessProbe:
          tcpSocket:
            port: 11211
          periodSeconds: 10
          failureThreshold: 2
        livenessProbe:
          tcpSocket:
            port: 11211
          periodSeconds: 10
          failureThreshold: 30
        startupProbe:
          tcpSocket:
            port: 11211
          periodSeconds: 1
          failureThreshold: 30
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy
  annotations:
    cert-manager.io/issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/global-rate-limit: "1000000"
    nginx.ingress.kubernetes.io/global-rate-limit-window: "1m"
spec:
  ingressClassName: nginx
  rules:
  - host: "dummy.alphabet5.dev"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy
            port:
              number: 80
  tls:
  - hosts:
    - dummy.alphabet5.dev
    secretName: dummy-alphabet5-dev-tls
---
apiVersion: v1
kind: Service
metadata:
  name: dummy
spec:
  selector:
    app: dummy
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
``` 

block dns for testing

```
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-dns
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: kube-dns
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
    ports:
    - protocol: TCP
      port: 6379

```

This results in the following set of log messages

```
2024/05/02 01:56:41 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:41 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:46 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:51 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:56 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:57:01 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:57:06 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:57:11 [error] 138#138: *7868 [lua] global_throttle.lua:105: throttle(): error while processing key: failed to connect: memcached.ingress-nginx.svc.cluster.local could not be resolved (110: Operation timed out), client: 98.97.11.220, server: dummy.alphabet5.dev, request: "GET / HTTP/2.0", host: "dummy.alphabet5.dev"
```

**NGINX Ingress controller version** (exec into the pod and run nginx-ingress-controller --version.):
<!--
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')

kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version  
-->

NGINX Ingress controller
  Release:       v1.10.1
  Build:         4fb5aac1dd3669daa3a14d9de3e3cdb371b4c518
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

**Kubernetes version** (use `kubectl version`):

```
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7+k3s1
```

(Also on v1.27.8+rke2r1 and v1.28.8+rke2r1)

**Environment**:

- **Cloud provider or hardware configuration**: bare metal, and vms
- **OS** (e.g. from /etc/os-release): `Debian GNU/Linux 12 (bookworm)`, `Ubuntu 22.04`, `Ubuntu 20.04`
- **Kernel** (e.g. `uname -a`): v5.15? (20.04) and `6.1.0-18-amd64`
- **Install tools**: rancher for rke2, and reproduced on as close to stock k3s as possible.



DNS errors cause global-rate-limit-memcached-connect-timeout to not be respected.
This is a GitHub Issue
repo:ingress-nginx
owner:kubernetes
Title : DNS errors cause global-rate-limit-memcached-connect-timeout to not be respected.
Issue date:

--- start body ---
<!--

Welcome to ingress-nginx!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-).
More info helps better understanding of the issue (needless to say).

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

-->

<!--

IMPORTANT!!!

Please complete the next sections or the issue will be closed.
This questions are the first thing we need to know to understand the context.

-->

**What happened**:

<!-- (please include exact error messages if you can) -->

We had a unique failure where coredns pods were removed/replaced during an upgrade. ingress-nginx was sending traffic to the old pod IPs which was getting dropped by cilium (because they were stale IPs / not assigned to anything.)

The default dns timeout is 5 retries x 2000ms - so a 10s delay happens, even when global-rate-limit-memcached-connect-timeout is set to 50ms.

https://github.com/kubernetes/ingress-nginx/blob/300f772dd75c9a8d6df7831c97cd5a503235e1dc/rootfs/etc/nginx/lua/util/dns.lua#L99

This is the resulting log message

```
2024/04/26 21:45:14 [error] 1043#1043: *24280775 [lua] global_throttle.lua:105: throttle(): error while processing key: failed to connect: memcached.ingress-nginx.svc.cluster.local could not be resolved (110: Operation timed out), ...
```

**What you expected to happen**:

1- dns should hit the service and not the pod ips so when they're replaced dns doesn't just timeout. (I haven't figured this one out yet.)

2- the 50ms timeout should be followed even if dns is down.

For #2, it's easily reproducible:

Ingress values.yaml

```
controller:
  config:
    global-rate-limit-memcached-host: memcached.ingress-nginx.svc.cluster.local.
```

All the manifests
```
---
apiVersion: v1
kind: Service
metadata:
  name: memcached
  namespace: ingress-nginx
spec:
  selector:
    app: memcached
  ports:
    - protocol: TCP
      port: 11211
      targetPort: 11211
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memcached
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: memcached
  template:
    metadata:
      labels:
        app: memcached
    spec:
      containers:
      - name: memcached
        image: docker.io/memcached
        command:
        - memcached
        args:
        - '-m'
        - '64'
        ports:
        - containerPort: 11211
        imagePullPolicy: Always
        readinessProbe:
          tcpSocket:
            port: 11211
          periodSeconds: 10
          failureThreshold: 2
        livenessProbe:
          tcpSocket:
            port: 11211
          periodSeconds: 10
          failureThreshold: 30
        startupProbe:
          tcpSocket:
            port: 11211
          periodSeconds: 1
          failureThreshold: 30
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy
  annotations:
    cert-manager.io/issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/global-rate-limit: "1000000"
    nginx.ingress.kubernetes.io/global-rate-limit-window: "1m"
spec:
  ingressClassName: nginx
  rules:
  - host: "dummy.alphabet5.dev"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy
            port:
              number: 80
  tls:
  - hosts:
    - dummy.alphabet5.dev
    secretName: dummy-alphabet5-dev-tls
---
apiVersion: v1
kind: Service
metadata:
  name: dummy
spec:
  selector:
    app: dummy
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
``` 

block dns for testing

```
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-dns
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: kube-dns
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
    ports:
    - protocol: TCP
      port: 6379

```

This results in the following set of log messages

```
2024/05/02 01:56:41 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:41 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:46 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:51 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:56:56 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:57:01 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:57:06 [error] 138#138: send() failed (111: Connection refused) while resolving, resolver: 10.43.0.10:53
2024/05/02 01:57:11 [error] 138#138: *7868 [lua] global_throttle.lua:105: throttle(): error while processing key: failed to connect: memcached.ingress-nginx.svc.cluster.local could not be resolved (110: Operation timed out), client: 98.97.11.220, server: dummy.alphabet5.dev, request: "GET / HTTP/2.0", host: "dummy.alphabet5.dev"
```

**NGINX Ingress controller version** (exec into the pod and run nginx-ingress-controller --version.):
<!--
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')

kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version  
-->

NGINX Ingress controller
  Release:       v1.10.1
  Build:         4fb5aac1dd3669daa3a14d9de3e3cdb371b4c518
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

**Kubernetes version** (use `kubectl version`):

```
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7+k3s1
```

(Also on v1.27.8+rke2r1 and v1.28.8+rke2r1)

**Environment**:

- **Cloud provider or hardware configuration**: bare metal, and vms
- **OS** (e.g. from /etc/os-release): `Debian GNU/Linux 12 (bookworm)`, `Ubuntu 22.04`, `Ubuntu 20.04`
- **Kernel** (e.g. `uname -a`): v5.15? (20.04) and `6.1.0-18-amd64`
- **Install tools**: rancher for rke2, and reproduced on as close to stock k3s as possible.



--- end body ---
