<p>I am using deploying an outward facing service, that is exposed behind a nodeport and then an istio ingress. The deployment is using manual sidecar injection. Once the deployment, nodeport and ingress are running, I can make a request to the istio ingress.</p>

<p>For some unkown reason, the request does not route through to my deployment and instead displays the text "no healthy upstream". Why is this, and what is causing it?</p>

<p>I can see in the http response that the status code is 503 (Service Unavailable) and the server is "envoy". The deployment is functioning as I can map a port forward to it and everything works as expected.</p>

<p>Just in case, like me, you get curious... Even though in my scenario it was clear the case of the error...</p>

<p><strong>Error cause:</strong> <em>I had two versions of the same service (v1 and v2), and an Istio VirtualService configured with traffic route destination using weights. Then, 95% goes to v1 and 5% goes to v2. As I didn't have the v1 deployed (yet), of course, the error "503 - no healthy upstream" shows up 95% of the requests</em>.</p>

<p>Ok, even so, I knew the problem and how to fix it (just deploy v1), I was wondering...  But, how can I have more information about this error?  How could I get a deeper analysis of this error to find out what was happening? </p>

<p>This is a way of investigating using the configuration command line utility of Istio, the istioctl:</p>

<pre><code># 1) Check the proxies status --&gt;
  $ istioctl proxy-status
# Result --&gt;
  NAME                                                   CDS        LDS        EDS        RDS          PILOT                       VERSION
  ...
  teachstore-course-v1-74f965bd84-8lmnf.development      SYNCED     SYNCED     SYNCED     SYNCED       istiod-86798869b8-bqw7c     1.5.0
  ...
  ...

# 2) Get the name outbound from JSON result using the proxy (service with the problem) --&gt;
  $ istioctl proxy-config cluster teachstore-course-v1-74f965bd84-8lmnf.development --fqdn teachstore-student.development.svc.cluster.local -o json
# 2) If you have jq install locally (only what we need, already extracted) --&gt;
  $ istioctl proxy-config cluster teachstore-course-v1-74f965bd84-8lmnf.development --fqdn teachstore-course.development.svc.cluster.local -o json | jq -r .[].name
# Result --&gt;
  outbound|80||teachstore-course.development.svc.cluster.local
  inbound|80|9180-tcp|teachstore-course.development.svc.cluster.local
  outbound|80|v1|teachstore-course.development.svc.cluster.local
  outbound|80|v2|teachstore-course.development.svc.cluster.local

# 3) Check the endpoints of "outbound|80|v2|teachstore-course..." using v1 proxy --&gt;
  $ istioctl proxy-config endpoints teachstore-course-v1-74f965bd84-8lmnf.development --cluster "outbound|80|v2|teachstore-course.development.svc.cluster.local"
# Result (the v2, 5% of the traffic route is ok, there are healthy targets) --&gt;
  ENDPOINT             STATUS      OUTLIER CHECK     CLUSTER
  172.17.0.28:9180     HEALTHY     OK                outbound|80|v2|teachstore-course.development.svc.cluster.local
  172.17.0.29:9180     HEALTHY     OK                outbound|80|v2|teachstore-course.development.svc.cluster.local

# 4) However, for the v1 version "outbound|80|v1|teachstore-course..." --&gt;
$ istioctl proxy-config endpoints teachstore-course-v1-74f965bd84-8lmnf.development --cluster "outbound|80|v1|teachstore-course.development.svc.cluster.local"
  ENDPOINT             STATUS      OUTLIER CHECK     CLUSTER
# Nothing! Emtpy, no Pods, that's explain the "no healthy upstream" 95% of time.
</code></pre>
