<p>We have a service running behind istio virtual service. Occasionally the request timeouts at the client side while everything is normal inside the service.</p>
<p>To solve this problem, we want to record time cost for all queries, and use these data to check time cost for specific query (by specifying filter fields included in the query) when timeouts happen again.</p>
<p>Best way to do that is using something like plugin with istio virtual service. But seems istio not provide such thing. How to do that?</p>
<p><strong>software info</strong><br />
istio v1.14<br />
kubernetes v1.20</p>

<p>You can check the istio standard metrics and istio with prometheus integration.</p>
<p><a href="https://istio.io/latest/docs/reference/config/metrics/#:%7E:text=For%20HTTP%2C%20HTTP%2F2%2C,measures%20the%20duration%20of%20requests" rel="nofollow noreferrer">https://istio.io/latest/docs/reference/config/metrics/#:~:text=For%20HTTP%2C%20HTTP%2F2%2C,measures%20the%20duration%20of%20requests</a></p>
<p>Standard metrics will be helpful to found the timeout issue, latency and other details between service to service communication and ingress to service also if exposed.</p>
<p>You can get P50, P90 and other metrics.
<a href="https://i.stack.imgur.com/3lbcT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3lbcT.png" alt="enter image description here" /></a></p>
<p>I would recommend checking out and installing istio and it's <strong>service dashboard</strong> : <a href="https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengistio-intro-topic.htm" rel="nofollow noreferrer">https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengistio-intro-topic.htm</a></p>
<p>Also tracing might be helpful to check the request tracing and time it's taken to process.</p>
<p><a href="https://istio.io/latest/docs/tasks/observability/distributed-tracing/jaeger/" rel="nofollow noreferrer">https://istio.io/latest/docs/tasks/observability/distributed-tracing/jaeger/</a></p>
