<p>I want to create a custom 403 error page.
Currently I already have an Ingress created and in the annotations I have something like this:</p>

<pre><code>"nginx.ingress.kubernetes.io/whitelist-source-range": "100.01.128.0/20,88.100.01.01"
</code></pre>

<p>So any attempt to access my web app outside that IP range receives a 403 error.</p>

<p>In order to create a custom page I tried adding the following annotations:</p>

<pre><code>"nginx.ingress.kubernetes.io/custom-http-errors": "403",
"nginx.ingress.kubernetes.io/default-backend": "default-http-backend"
</code></pre>

<p>where default-http-backend is the name of an app already deployed.
<a href="https://i.stack.imgur.com/6XixQ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/6XixQ.png" alt="Pod details"></a></p>

<p>the ingress has this:</p>

<pre><code>{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "my-app-ingress",
    "namespace": "my-app-test",
    "selfLink": "/apis/extensions/v1beta1/namespaces/my-app-test/ingresses/my-app-ingress",
    "uid": "8f31f2b4-428d-11ea-b15a-ee0dcf00d5a8",
    "resourceVersion": "129105581",
    "generation": 3,
    "creationTimestamp": "2020-01-29T11:50:34Z",
    "annotations": {
      "kubernetes.io/ingress.class": "nginx",
      "nginx.ingress.kubernetes.io/custom-http-errors": "403",
      "nginx.ingress.kubernetes.io/default-backend": "default-http-backend",
      "nginx.ingress.kubernetes.io/rewrite-target": "/",
      "nginx.ingress.kubernetes.io/whitelist-source-range": "100.01.128.0/20,90.108.01.012"
    }
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "my-app-test.retail-azure.js-devops.co.uk"
        ],
        "secretName": "ssl-secret"
      }
    ],
    "rules": [
      {
        "host": "my-app-test.retail-azure.js-devops.co.uk",
        "http": {
          "paths": [
            {
              "path": "/api",
              "backend": {
                "serviceName": "my-app-backend",
                "servicePort": 80
              }
            },
            {
              "path": "/",
              "backend": {
                "serviceName": "my-app-frontend",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  },
  "status": {
    "loadBalancer": {
      "ingress": [
        {}
      ]
    }
  }
}
</code></pre>

<p>Yet I always get the default 403.
What am I missing?</p>

<p>I've reproduced your scenario and that worked for me.
I will try to guide you in steps I've followed.</p>

<p><strong>Cloud provider</strong>: GKE
<strong>Kubernetes Version</strong>: v1.15.3
<strong>Namespace</strong>: <code>default</code></p>

<p>I'm using 2 deployments of 2 images with a service for each one.</p>

<p><strong>Service 1</strong>: <code>default-http-backend</code> - with nginx image, it will be our default backend.</p>

<p><strong>Service 2</strong>: <code>custom-http-backend</code> - with inanimate/echo-server image, this service will be displayed if the request become from a whitelisted ip.</p>

<p><strong>Ingress</strong>: Nginx ingress with annotations.</p>

<p><strong>Expected behavior:</strong> The ingress will be configured to use <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#default-backend" rel="noreferrer">default-backend</a>, <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-http-errors" rel="noreferrer">custom-http-errors</a> and <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#whitelist-source-range" rel="noreferrer">whitelist-source-range</a> annotations. If the request was made from a whitelisted ip the ingress will redirect to <strong>custom-http-backend</strong>, if not it will be redirect to <strong>default-http-backend</strong>.</p>

<h3>Deployment 1: default-http-backend</h3>

<p>Create a file <code>default-http-backend.yaml</code> with this content:</p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
spec:
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      containers:
      - name: default-http-backend
        image: nginx
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
spec:
  selector:
    app: default-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
</code></pre>

<p>Apply the yaml file: <code>k apply -f default-http-backend.yaml</code></p>

<h3>Deployment 2: custom-http-backend</h3>

<p>Create a file <code>custom-http-backend.yaml</code> with this content:</p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-http-backend
spec:
  selector:
    matchLabels:
      app: custom-http-backend
  template:
    metadata:
      labels:
        app: custom-http-backend
    spec:
      containers:
      - name: custom-http-backend
        image: inanimate/echo-server
        ports:
        - name: http
          containerPort: 8080
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: custom-http-backend
spec:
  selector:
    app: custom-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
</code></pre>

<p>Apply the yaml file: <code>k apply -f custom-http-backend.yaml</code></p>

<h3>Check if services is up and running</h3>

<p><em>I'm using the alias <code>k</code> for <code>kubectl</code></em></p>

<pre><code>➜  ~ k get svc                                 
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
custom-http-backend    ClusterIP   10.125.5.227   &lt;none&gt;        80/TCP    73s
default-http-backend   ClusterIP   10.125.9.218   &lt;none&gt;        80/TCP    5m41s
...
</code></pre>

<pre><code>➜  ~ k get pods
NAME                                    READY   STATUS    RESTARTS   AGE
custom-http-backend-67844fb65d-k2mwl    1/1     Running   0          2m10s
default-http-backend-5485f569bd-fkd6f   1/1     Running   0          6m39s
...
</code></pre>

<p>You could test the service using <a href="https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/" rel="noreferrer">port-forward</a>:</p>

<p><strong>default-http-backend</strong>
<code>k port-forward svc/default-http-backend 8080:80</code>
Try to access <a href="http://localhost:8080" rel="noreferrer">http://localhost:8080</a> in your browse to see the nginx default page.</p>

<p><strong>custom-http-backend</strong>
<code>k port-forward svc/custom-http-backend 8080:80</code>
Try to access <a href="http://localhost:8080" rel="noreferrer">http://localhost:8080</a> in your browse to see the custom page provided by the echo-server image.</p>

<h3>Ingress configuration</h3>

<p>At this point we have both services up and running, we need to install and configure the nginx ingress. You can follow the <a href="https://kubernetes.github.io/ingress-nginx/deploy/" rel="noreferrer">official documentation</a>, this will not covered here.</p>

<p>After installed let's deploy the ingress, based in the code you posted i did some modifications: tls removed, added other domain and removed the path <code>/api</code> for tests purposes only and add my home ip to whitelist.</p>

<p>Create a file <code>my-app-ingress.yaml</code> with the content:</p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/custom-http-errors: '403'
    nginx.ingress.kubernetes.io/default-backend: default-http-backend
    nginx.ingress.kubernetes.io/whitelist-source-range: 207.34.xxx.xx/32
spec:
  rules:
  - host: myapp.rabello.me
    http:
      paths:
      - path: "/"
        backend:
          serviceName: custom-http-backend
          servicePort: 80
</code></pre>

<p>Apply the spec: <code>k apply -f my-app-ingress.yaml</code></p>

<p>Check the ingress with the command:</p>

<pre><code>➜  ~ k get ing
NAME             HOSTS              ADDRESS          PORTS   AGE
my-app-ingress   myapp.rabello.me   146.148.xx.xxx   80      36m
</code></pre>

<p><strong>That's all!</strong></p>

<p>If I test from home with my whitelisted ip, the custom page is showed, but if i try to access using my cellphone in 4G network, the nginx default page is displayed.</p>

<p>Note I'm using ingress and services in the same namespace, if you need work with different namespace you need to use <a href="https://kubernetes.io/docs/concepts/services-networking/service/#externalname" rel="noreferrer">ExternalName</a>.</p>

<p>I hope that helps!</p>

<p><strong>References:</strong>
<a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="noreferrer">kubernetes deployments</a></p>

<p><a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noreferrer">kubernetes service</a></p>

<p><a href="https://kubernetes.github.io/ingress-nginx/" rel="noreferrer">nginx ingress</a></p>

<p><a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/" rel="noreferrer">nginx annotations</a></p>
