<p>I am new to prometheus/alertmanager.</p>

<p>I have created a cron job which executes shell script every minute.  This shell script generates "test.prom" file (with a gauge metric in it) in the same directory which is assigned to <code>--textfile.collector.directory</code> argument (to node-exporter).  I verified (using curl <a href="http://localhost:9100/metrics" rel="nofollow noreferrer">http://localhost:9100/metrics</a>)  that the node-exporter exposes that custom metric correctly.</p>

<p>When I tried to run a query against that custom metric in prometheus dashboard, it does not show up any results (it says no data found). </p>

<p>I could not figure out why the query against the metric exposed via node-exporter textfile collector fails. <strong>Any clues what I missed ?  Also please let me know how to check and ensure that prometheus scraped my custom metric 'test_metric` ?</strong></p>

<p>My query in prometheus dashboard is <code>test_metric != 0</code> (in prometheus dashboard) which did not give any results.  But I exposed <code>test_metric</code> via node-exporter textfile.  </p>

<p>Any help is appreciated !!</p>

<p>BTW, the node-exporter is running as docker container in Kubernetes environment.</p>

<p>I had a similar situation, but it was not a configuration problem.</p>

<p>Instead, my data included timestamps:</p>

<pre><code># HELP network_connectivity_rtt Round Trip Time to each node
# TYPE network_connectivity_rtt gauge
network_connectivity_rtt{host="home"} 53.87 1541426242
network_connectivity_rtt{host="hop_1"} 58.8 1541426242
network_connectivity_rtt{host="hop_2"} 21.93 1541426242
network_connectivity_rtt{host="hop_3"} 71.69 1541426242
</code></pre>

<p>PNE was picking them up without any problem <em>once I reloaded it</em>. As prometheus is running under systemd, I had to check the logs like this:</p>

<pre><code>journalctl --system -u prometheus.service --follow
</code></pre>

<p>There I read this line:</p>

<pre><code>msg="Error on ingesting samples that are too old or are too far into the future"
</code></pre>

<p>Once I removed the timestamps, values started appearing. This lead me to read more in detail about the timestamps, and I found out they have to be in <em>miliseconds</em>. So this format now is ok:</p>

<pre><code># HELP network_connectivity_rtt Round Trip Time to each node
# TYPE network_connectivity_rtt gauge
network_connectivity_rtt{host="home"} 50.47 1541429581376
network_connectivity_rtt{host="hop_1"} 3.38 1541429581376
network_connectivity_rtt{host="hop_2"} 11.2 1541429581376
network_connectivity_rtt{host="hop_3"} 20.72 1541429581376
</code></pre>

<p>I hope it helps someone else.</p>
