<p>I am using Istio in AWS EKS cluster. I am using the pre-installed prometheus and grafana to monitor pods, Istio mesh, Istio service workloads. </p>

<p>I have three services running in three different workspace,</p>

<pre><code>Service 1:- service1.namespace1.svc.cluster.local
Service 2 :- service2.namespace2.svc.cluster.local
Service 3:- service3.namespace3.svc.cluster.local
</code></pre>

<p>I can find the latency of each service end points from <code>Istio Service Dashboard</code> in Grafana . But, it just shows the latency for service end points, not the end point prefix. Though the overall service end point latency is fine but I want to check which path is taking time in a service end point.</p>

<p>Let's say  <code>P50 Latency</code> of <code>service1.namespace1.svc.cluster.local</code> is 2.91ms , but I also want to check latency of each path. It has four paths,</p>

<pre><code>service1.namespace1.svc.cluster.local/login =&gt; Loging Path , P50 Latency = ?
service1.namespace1.svc.cluster.local/signup =&gt; Singup Path , P50 Latency = ?
service1.namespace1.svc.cluster.local/auth =&gt; Auth path , P50 Latency = ?
service1.namespace1.svc.cluster.local/list =&gt; List path , P50 Latency = ?
</code></pre>

<p>I am not sure if it is possible in Prometheus and Grafana stack. What is the recommended way to achieve it ?</p>

<pre><code>Istioctl version --remote 

client version: 1.5.1
internal-popcraftio-ingressgateway version: 
citadel version: 1.4.3
galley version: 1.4.3
ingressgateway version: 1.5.1
pilot version: 1.4.3
policy version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
pilot version: 1.5.1
office-popcraftio-ingressgateway version: 
data plane version: 1.4.3 (83 proxies), 1.5.1 (4 proxies)
</code></pre>

codekube
<p>To my knowledge this is not something that the Istio metrics can provide. However, you should take a look at the available metrics that your server framework provides, if any. So, this is application (framework)-dependent. See for instance for SpringBoot ( <a href="https://docs.spring.io/spring-metrics/docs/current/public/prometheus" rel="nofollow noreferrer">https://docs.spring.io/spring-metrics/docs/current/public/prometheus</a> ) or Vert.x ( <a href="https://vertx.io/docs/vertx-micrometer-metrics/java/#_http_server" rel="nofollow noreferrer">https://vertx.io/docs/vertx-micrometer-metrics/java/#_http_server</a> )</p>

<p>One thing to be aware of, with HTTP path-based metrics, is that it is likely to make the metrics cardinality explode, if not used with care. Imagine some of your paths contain unbounded dynamic values (e.g. <code>/object/123465</code> , with <code>123456</code> being an ID), if that path is stored as a Prometheus label, that would mean under the hood that Prometheus will create one metric per ID, which is likely to cause performance issues on Prometheus and risk out-of-memory on your app.</p>

<p>This is I think a good reason to NOT have Istio providing path-based metrics. While on the other end, frameworks can have the sufficient knowledge to provide metrics based on path template instead of actual path (e.g. <code>/object/$ID</code> instead of <code>/object/123465</code>), which solves the cardinality problem.</p>

<p>PS: Kiali has some documentation about runtimes monitoring, that may help: <a href="https://kiali.io/documentation/runtimes-monitoring/" rel="nofollow noreferrer">https://kiali.io/documentation/runtimes-monitoring/</a></p>
