<p><strong>Summary</strong></p>

<p>I'm trying to figure out how to properly use the OR <code>|</code> operator in a Prometheus query because my imported Grafana dashboard is not working.</p>

<p><strong>Long version</strong></p>

<p>I'm trying to debug a Grafana dashboard based on some data scraped from my Kubernetes pods running <a href="https://github.com/AppMetrics/Prometheus" rel="nofollow noreferrer">AppMetrics/Prometheus</a>; the dashboard is <a href="https://grafana.com/dashboards/2204" rel="nofollow noreferrer">here</a>. Basically what happens is that when the value "All" for the <code>server</code> is selected on the Grafana dashboard (<code>server</code> is an individual pod in this case), no data appears. However, when I select an individual pod, then data does appear.</p>

<p>Here's an example of the same metric scraped from the two pods:</p>

<pre><code># HELP application_httprequests_transactions 
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 5.006965628
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 1367
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.5"} 0.000202825
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.75"} 0.000279318
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.95"} 0.000329862
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.99"} 0.055584233

# HELP application_httprequests_transactions 
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 6.10214788
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 1363
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.5"} 0.000218548
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.75"} 0.000277483
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.95"} 0.033821094
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.99"} 0.097113234
</code></pre>

<p>I ran the Query inspector in Grafana to find out which query it is calling, and then ran the PromQL query in Prometheus itself. Basically, when I execute the following PromQL queries individually, they return data:</p>

<pre><code>rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv"}[15m])*60

rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-jdq78"}[15m])*60
</code></pre>

<p>However, when I try to use PromQL's <code>|</code> operator to combine them, I don't get data back:</p>

<pre><code>rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60
</code></pre>

<p>Here's the raw output from Grafana's query inspector:</p>

<pre><code>xhrStatus:"complete"

request:Object

method:"GET"
url:"api/datasources/proxy/56/api/v1/query_range?query=rate(application_httprequests_transactions_count%7Benv%3D%22test%22%2Capp%3D%22MyApp%22%2Cserver%3D%22myapp-test-58d94bf78d-jdq78%7Cmyapp-test-58d94bf78d-l9tdv%7Cmyapp-test-5b8c9845fb-7lklm%7Cmyapp-test-5b8c9845fb-8jf7n%7Cmyapp-test-5b8c9845fb-d9x5c%7Cmyapp-test-5b8c9845fb-fw4gj%7Cmyapp-test-5b8c9845fb-vtl9z%7Cmyapp-test-5b8c9845fb-vv7xv%7Cmyapp-test-5b8c9845fb-wq9bs%7Cmyapp-test-5b8c9845fb-xqfrt%7Cmyapp-test-69999d58b5-549vd%7Cmyapp-test-69999d58b5-lmp8x%7Cmyapp-test-69999d58b5-nbvt9%7Cmyapp-test-69999d58b5-qphj2%7Cmyapp-test-6b8dcc5ffb-gjjvj%7Cmyapp-test-6b8dcc5ffb-rxfk2%7Cmyapp-test-7fdf446767-bzhm2%7Cmyapp-test-7fdf446767-hp46w%7Cmyapp-test-7fdf446767-rhqhq%7Cmyapp-test-7fdf446767-wxmm2%22%7D%5B1m%5D)*60&amp;start=1540574190&amp;end=1540574505&amp;step=15"

response:Object

status:"success"

data:Object

resultType:"matrix"

result:Array[0] =&gt; []
</code></pre>

<p>I opened a GitHub issue for this as well; it has a quick GIF screen recording showing what I mean: <a href="https://github.com/AppMetrics/Prometheus/issues/43" rel="nofollow noreferrer">AppMetrics/Prometheus#43</a></p>

Scott Crooks
<p><code>|</code> is for regular expressions, PromQL doesn't have a <code>|</code> operator (but it does have an <code>or</code> operator). You need to specify that the matcher is a regex rather than an exact match with <code>=~</code>:</p>

<pre><code>rate(application_httprequest_transactions_count{env="test",app="MyApp",server=~"myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60
</code></pre>
