<p>I have installed the following two different ingress controllers on my DigitalOcean managed K8S cluster: </p>

<ul>
<li><p>Nginx   </p></li>
<li><p>Istio  </p></li>
</ul>

<p>and they have been assigned to two different IP addresses. My question is, if it is wrong to have two different ingress controllers on the same K8S cluster?  </p>

<p>The reason, why I have done it, because nginx is for tools like harbor, argocd, etc. and istio for microservices.   </p>

<p>I have also figured out, when both are installed alongside each other, sometimes during the deployment, the K8S suddenly goes down.</p>

<p>For example, I have deployed: </p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes-first
  namespace: dev
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: hello-kubernetes-first
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubernetes-first
  namespace: dev
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-kubernetes-first
  template:
    metadata:
      labels:
        app: hello-kubernetes-first
    spec:
      containers:
        - name: hello-kubernetes
          image: paulbouwer/hello-kubernetes:1.7
          ports:
            - containerPort: 8080
          env:
            - name: MESSAGE
              value: Hello from the first deployment!
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
  name: helloworld-ingress
  namespace: dev
spec:
  rules:
    - host: hello.service.databaker.io
      http:
        paths:
          - path: /*
            backend:
              serviceName: hello-kubernetes-first
              servicePort: 80
---
</code></pre>

<p>Then I've got: </p>

<pre><code>Error from server (InternalError): error when creating "istio-app.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.245.107.175:443: i/o timeout  
</code></pre>

<p>You have raised several points - before answering your question, let's take a step back.</p>
<hr />
<h3>K8s Ingress <em>not</em> recommended by Istio</h3>
<p>It is important to note how Istio does not recommend using K8s Ingress:</p>
<blockquote>
<p>Using the <a href="https://istio.io/latest/docs/reference/config/networking/gateway/" rel="nofollow noreferrer">Istio Gateway</a>, rather than Ingress, is recommended to make use of the full feature set that Istio offers, such as rich traffic management and security features.</p>
</blockquote>
<p>Ref: <a href="https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/" rel="nofollow noreferrer">https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/</a></p>
<p>As noted, Istio Gateway (Istio IngressGateway and EgressGateway) acts as the edge, which you can find more in <a href="https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/" rel="nofollow noreferrer">https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/</a>.</p>
<hr />
<h3>Multiple endpoints within Istio</h3>
<p>If you need to assign one public endpoint for business requirement, and another for monitoring (such as Argo CD, Harbor as you mentioned), you can achieve that by using Istio only. There are roughly 2 approaches to this.</p>
<ol>
<li>Create separate Istio IngressGateways - one for main traffic, and another for monitoring</li>
<li>Create one Istio IngressGateway, and use <a href="https://istio.io/latest/docs/reference/config/networking/gateway/" rel="nofollow noreferrer">Gateway</a> definition to handle multiple access patterns</li>
</ol>
<p>Both are valid, and depending on requirements, you may need to choose one way or the other.</p>
<p>As to the Approach #2., it is where Istio's traffic management system shines. It is a great example of Istio's power, but the setup is slightly complex if you are new to it. So here goes an example.</p>
<p><strong>Example of Approach #2</strong></p>
<p>When you create Istio IngressGateway by following the <a href="https://istio.io/latest/docs/setup/install/istioctl/" rel="nofollow noreferrer">default installation</a>, it would create <code>istio-ingressgateway</code> like below (I overly simplified YAML definition):</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
  name: istio-ingressgateway
  namespace: istio-system
  # ... other attributes ...
spec:
  type: LoadBalancer
  # ... other attributes ...
</code></pre>
<p>This LB Service would then be your endpoint. (I'm not familiar with DigitalOcean K8s env, but I suppose they would handle LB creation.)</p>
<p>Then, you can create Gateway definition like below:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: your-gateway
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
    - port:
        number: 3000
        name: https-your-system
        protocol: HTTPS
      hosts:
        - &quot;your-business-domain.com&quot;
        - &quot;*.monitoring-domain.com&quot;
      # ... other attributes ...
</code></pre>
<p>You can then create 2 or more <a href="https://istio.io/latest/docs/reference/config/networking/virtual-service/" rel="nofollow noreferrer">VirtualService</a> definitions.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: business-virtsvc
spec:
  gateways:
    - istio-ingressgateway.istio-system.svc.cluster.local
  hosts:
    - &quot;your-business-domain.com&quot;
  http:
    - match:
        - port: 3000
      route:
        - destination:
            host: some-business-pod
            port:
              number: 3000
    # ... other attributes ...
</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: monitoring-virtsvc
spec:
  gateways:
    - istio-ingressgateway.istio-system.svc.cluster.local
  hosts:
    - &quot;harbor.monitoring-domain.com&quot;
  http:
    - match:
        - port: 3000
      route:
        - destination:
            host: harbor-pod
            port:
              number: 3000
    # ... other attributes ...
</code></pre>
<p>NOTE: The above is assuming a lot of things, such as port mapping, traffic handling, etc.. Please check out the official doc for details.</p>
<hr />
<p>So, back to the question after long detour:</p>
<h2>Question: [Is it] wrong to have two different ingress controllers on the same K8S cluster[?]</h2>
<p>I believe it is OK, though this can cause an error like you are seeing, as two ingress controller fight for the K8s Ingress resource.</p>
<p>As mentioned above, if you are using Istio, it's better to stick with Istio IngressGateway instead of K8s Ingress. If you need K8s Ingress for some specific reason, you could use other Ingress controller for K8s Ingress, like Nginx.</p>
<p>As to the error you saw, it's coming from Nginx deployed webhook, that <code>ingress-nginx-controller-admission.nginx.svc</code> is not available. This means you have created a K8s Ingress <code>helloworld-ingress</code> with <code>kubernetes.io/ingress.class: istio</code> annotation, but Nginx webhook is interfering with K8s Ingress handling. The webhook is then failing to handle the resource, as the Pod / Svc responsible for webhook traffic is not found.</p>
<p>The error itself just says something is unhealthy in K8s - potentially not enough Node allocated to the cluster, and thus Pod allocation not happening. It's also good to note that Istio does require some CPU and memory footprint, which may be putting more pressure to the cluster.</p>
