<p>My team is using istio  (version 1.2.8) on our k8s (v 1.15.6)  landscape,
and we want to provide Prometheus with <strong>external IP</strong>, when applying the <code>vs</code> 
We were able to access Prometheus in the browser but without and <code>css/js</code> files. we got <strong>404</strong> error for the <code>.js</code> (see logs below) files (see envoy logs below) as the <code>/static</code> files are not served.</p>

<p><strong>This is the UI we got</strong> (no css and js files are served) </p>

<p><img src="https://user-images.githubusercontent.com/34491236/70865211-8877d700-1f63-11ea-92fd-d388103d6d9f.png" alt="image"></p>

<p>instead of the following Prometheus default UI (when using loadbalancer or port forwarding...) </p>

<p><img src="https://user-images.githubusercontent.com/34491236/70865296-87937500-1f64-11ea-98e4-2f80586c51d5.png" alt="image"></p>

<p>This is the minimal steps to see the issue:</p>

<p>Install <a href="https://github.com/helm/charts/tree/master/stable/prometheus" rel="nofollow noreferrer">Prometheus</a> via helm <strong>as-is</strong>(latest- <strong>we didn't change any default</strong> config of Prometheus from the chart  ) </p>

<p>Take the name of the <code>service</code> (with <code>kubectl get svc</code> on the <code>ns</code> which the service deployed)   put it on the <code>destination-&gt;host</code> section in the VS (update the <code>gw</code> host etc) and apply the <code>VS</code> file</p>

<p><strong>vs.yaml</strong></p>

<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
    - match:
        - uri:
            prefix: /prometheus
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80


</code></pre>

<p>BTW,</p>

<p>If I just change the type of Prometheus to use <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L851" rel="nofollow noreferrer"><code>LoadBalancer</code></a> it work, I was able to get <code>external-ip</code> and see istio UI as expected.</p>

<p>another info, if I remove the following </p>

<pre><code>      rewrite:
        uri: /graph
</code></pre>

<p>I got <code>404 error</code> in the browser without any data from prom</p>

<p>in the browser without js/css files , the network in the browser is like following:</p>

<p><img src="https://user-images.githubusercontent.com/34491236/70866298-8b2cf900-1f70-11ea-9962-bfd6ba8c5f78.png" alt="image"></p>

<p>I even try the following which doesn't work either</p>

<pre><code>        - uri:
            prefix: /prometheus
      rewrite:
        uri: /static

</code></pre>

<p>or </p>

<pre><code>        - uri:
            prefix: /prometheus/static

</code></pre>

<p>our <code>gateway</code> spec looks like following</p>

<pre><code>...
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - lzs.dev10.int.str.cloud.rpn
    port:
      name: https-manager
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/de-tls/tls.key
      serverCertificate: /etc/istio/de-tls/tls.crt
</code></pre>

<p>using port forwarding (local) or <code>loadbalancer</code> for Prometheus it works. 
How can we make it work with istio ? </p>

<p><strong>update</strong></p>

<p>I've tried also to add the static and got the same results:</p>

<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
    - match:
        - uri:
            prefix: /prometheus
        - uri:
            prefix: /static
        - uri:
            regex: '^.*\.(ico|png|jpg)$'
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80


</code></pre>

<p><strong>update 2</strong></p>

<p>after using the yaml which provided as answer, now I see the ui with the css etc however it's not functional , I got error: <code>Error loading available metrics!</code> 
in the browser debug mode network tab I can see that the following is not working</p>

<p>This is the logs for envoy for the error</p>

<pre><code>[2019-12-17T09:04:18.670Z] "GET /api/v1/query?query=time()&amp;_=1576573457737 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "57592874-27f5-4b57-9dea-1bcf13365f60" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.13:443 100.96.0.1:24664 lzs.dev10.int.str.cloud.rpn
[2019-12-17T09:04:18.670Z] "GET /api/v1/label/__name__/values?_=1576573457738 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "edad441d-58fe-4214-aae0-a0aec9012030" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.13:443 100.96.0.1:24664 lzs.dev10.int.str.cloud.rpn
</code></pre>

<p><a href="https://i.stack.imgur.com/VN5c4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VN5c4.png" alt="enter image description here"></a></p>

<p>(we are not talking about Prometheus which comes with istio, we need to install diff Prometheus on diff namespace...) </p>

<p><strong>This is the logs from envoy</strong> </p>

<p>2019-12-15T13:57:16.977357Z info    Envoy proxy is ready</p>

<p>[2019-12-15 14:29:51.226][14][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, 
[2019-12-15 15:00:50.980][14][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, 
<strong>[2019-12-15T15:11:02.572Z] "GET /prometheus HTTP/2" 200 - "-" "-" 0 5785 2 1 "100.96.3.1"</strong> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "531e2f39-0c9f-44d3-b11b-e336126ea836" "lzs.dev10.int.str.cloud.rpn" "100.96.0.16:9090" outbound|80||prom-prometheus-server.mon.svc.cluster.local - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn
<strong>[2019-12-15T15:11:02.705Z] "GET /static/vendor/js/jquery-3.3.1.min.js?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 -</strong> "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "40119d8d-2103-4453-b589-e1561d44d363" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn
<strong>[2019-12-15T15:11:02.705Z] "GET /static/vendor/js/popper.min.js?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 -</strong> "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "dbdf2a2a-cfd3-422a-82f4-e6e466407671" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn
[2019-12-15T15:11:02.706Z] "GET /static/vendor/bootstrap-4.3.1/js/bootstrap.min.js?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "efd95571-03e9-492d-98ff-b4910d1646d6" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn
[2019-12-15T15:11:02.706Z] "GET /static/vendor/bootstrap-4.3.1/css/bootstrap.min.css?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "8ae04546-06cb-4ba0-8430-f04388811460" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn
[2019-12-15T15:11:02.706Z] "GET /static/css/prometheus.css?v=6f92ce56053866194ae5937012c1bec40f1dd1d9 HTTP/2" 404 NR "-" "-" 0 0 0 - "100.96.3.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0" "70c88f5f-b582-4dd9-a2e2-47605c812344" "lzs.dev10.int.str.cloud.rpn" "-" - - 100.96.2.10:443 100.96.3.1:32972 lzs.dev10.int.str.cloud.rpn</p>

<p>For start I found some information about <a href="https://archive.istio.io/v1.2/docs/setup/kubernetes/platform-setup/" rel="nofollow noreferrer">istio 1.2</a> that tells </p>

<blockquote>
  <p>Istio 1.2 has been tested with these Kubernetes releases: 1.12, 1.13, 1.14.</p>
</blockquote>

<p>So if You use kubernetes 1.15 I would recommend to upgrade your istio to latest version.</p>

<hr>

<p>About prometheus, I see You want to use prometheus helm chart, why won't You use built-in prometheus? As provided in <a href="https://istio.io/docs/setup/additional-setup/config-profiles/" rel="nofollow noreferrer">istio documentation</a> prometheus is enabled in versions default,demo and sds.</p>

<hr>

<p>Based on istio <a href="https://istio.io/docs/tasks/observability/gateways/" rel="nofollow noreferrer">remotely accessign telemetry addons</a> You can use either secure(<a href="https://istio.io/docs/tasks/observability/gateways/#option-1-secure-access-https" rel="nofollow noreferrer">https</a>) or insecure(<a href="https://istio.io/docs/tasks/observability/gateways/#option-2-insecure-access-http" rel="nofollow noreferrer">http</a>) option to expose prometheus.</p>

<hr>

<p>Personally i did an insecure reproduction by following above tutorial and everything is working. </p>

<p><strong>Kubernetes Version</strong>: 1.13.11-gke.14</p>

<p><strong>Istio Version</strong>: 1.4.2</p>

<p>Steps to follow</p>

<p>1.Install </p>

<ul>
<li><a href="https://istio.io/docs/setup/getting-started/#download" rel="nofollow noreferrer">Istioctl</a></li>
<li><a href="https://istio.io/docs/setup/install/istioctl/#install-istio-using-the-default-profile" rel="nofollow noreferrer">Istio default</a></li>
</ul>

<p>2.Expose prometheus</p>

<pre><code>cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: prometheus-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15030
      name: http-prom
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prometheus-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - prometheus-gateway
  http:
  - match:
    - port: 15030
    route:
    - destination:
        host: prometheus
        port:
          number: 9090
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: prometheus
  namespace: istio-system
spec:
  host: prometheus
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF 
</code></pre>

<p>3.Result</p>

<p><a href="https://i.stack.imgur.com/44e9K.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/44e9K.png" alt="enter image description here"></a></p>

<hr>

<p><strong>EDIT</strong></p>

<p>Could You try use this yaml?</p>

<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
  - match:
        - uri:
            prefix: /prometheus
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80
  - match:
        - uri:
            prefix: /static
        - uri:
            regex: '^.*\.(ico|png|jpg)$'
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80
</code></pre>

<p><strong>EDIT2</strong> Please add /api prefix to your second match like below</p>

<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prom-virtualservice
  namespace: mon
spec:
  gateways:
    - de-system-gateway.ws-system.svc.cluster.local
  hosts:
    - lzs.dev10.int.str.cloud.rpn
  http:
  - match:
        - uri:
            prefix: /prometheus
      rewrite:
        uri: /graph
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80
  - match:
        - uri:
            prefix: /static
        - uri:
            regex: '^.*\.(ico|png|jpg)$'
        - uri:
            prefix: /api
      route:
        - destination:
            host: prom-prometheus-server
            port:
              number: 80
</code></pre>

<p><strong>EDIT3</strong> </p>

<blockquote>
  <p>In your answer you separate it to two matches , why?</p>
</blockquote>

<p>This <a href="https://istio.io/docs/reference/config/networking/virtual-service/" rel="nofollow noreferrer">link</a> is the answer here I think, You rewrite /prometheus to /graph since it's main Prometheus url, and that's okay. But you can't rewrite /static, /api to /graph because you need those paths to grab files and metrics, if it won't match then error 404 appears.</p>
