{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/node_exporter/issues/245","repository_url":"https://api.github.com/repos/prometheus/node_exporter","labels_url":"https://api.github.com/repos/prometheus/node_exporter/issues/245/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/node_exporter/issues/245/comments","events_url":"https://api.github.com/repos/prometheus/node_exporter/issues/245/events","html_url":"https://github.com/prometheus/node_exporter/issues/245","id":157055385,"number":245,"title":"textfile collector reporting 'sample with repeated timestamp but different value'","user":{"login":"mrwacky42","id":355590,"avatar_url":"https://avatars.githubusercontent.com/u/355590?v=3","gravatar_id":"","url":"https://api.github.com/users/mrwacky42","html_url":"https://github.com/mrwacky42","followers_url":"https://api.github.com/users/mrwacky42/followers","following_url":"https://api.github.com/users/mrwacky42/following{/other_user}","gists_url":"https://api.github.com/users/mrwacky42/gists{/gist_id}","starred_url":"https://api.github.com/users/mrwacky42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrwacky42/subscriptions","organizations_url":"https://api.github.com/users/mrwacky42/orgs","repos_url":"https://api.github.com/users/mrwacky42/repos","events_url":"https://api.github.com/users/mrwacky42/events{/privacy}","received_events_url":"https://api.github.com/users/mrwacky42/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-05-26T19:04:07Z","updated_at":"2016-05-27T00:55:50Z","closed_at":"2016-05-27T00:55:50Z","body":"With the update to Prometheus 0.19.1, we can now start finding out why it is constantly logging about out-of-order samples.\r\n\r\nWe are running `node_exporter` 0.12.0 with a command line like so:\r\n\r\n     /usr/local/bin/node_exporter $A_BUNCH_OF_ARGS --collector.textfile.directory=/var/spool/prometheus\r\n\r\nCurrently, we have a single file in that directory that cron writes once a minute. Our scrape interval is 60s.\r\nPrometheus [sees everything twice](https://www.youtube.com/watch?v=wPg21R_9Io8). Well, the mtime of the file is seen twice anyway.\r\n\r\n```json\r\n{\r\n  \"time\": \"2016-05-26T18:46:33Z\",\r\n  \"source\": \"scrape.go:460\",\r\n  \"sample\": {\r\n    \"value\": [\r\n      1464288393.089,\r\n      \"1464288361.424913\"\r\n    ],\r\n    \"metric\": {\r\n      \"job\": \"node\",\r\n      \"instance\": \"example.com\",\r\n      \"file\": \"sysstat.prom\",\r\n      \"__name__\": \"node_textfile_mtime\"\r\n    }\r\n  },\r\n  \"msg\": \"Sample discarded\",\r\n  \"level\": \"debug\",\r\n  \"error\": \"sample with repeated timestamp but different value\"\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/prometheus/node_exporter/issues/comments/222034632","html_url":"https://github.com/prometheus/node_exporter/issues/245#issuecomment-222034632","issue_url":"https://api.github.com/repos/prometheus/node_exporter/issues/245","id":222034632,"user":{"login":"mrwacky42","id":355590,"avatar_url":"https://avatars.githubusercontent.com/u/355590?v=3","gravatar_id":"","url":"https://api.github.com/users/mrwacky42","html_url":"https://github.com/mrwacky42","followers_url":"https://api.github.com/users/mrwacky42/followers","following_url":"https://api.github.com/users/mrwacky42/following{/other_user}","gists_url":"https://api.github.com/users/mrwacky42/gists{/gist_id}","starred_url":"https://api.github.com/users/mrwacky42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrwacky42/subscriptions","organizations_url":"https://api.github.com/users/mrwacky42/orgs","repos_url":"https://api.github.com/users/mrwacky42/repos","events_url":"https://api.github.com/users/mrwacky42/events{/privacy}","received_events_url":"https://api.github.com/users/mrwacky42/received_events","type":"User","site_admin":false},"created_at":"2016-05-27T00:55:50Z","updated_at":"2016-05-27T00:55:50Z","body":"AAAAAAND lastly..\r\n\r\nAfter restarting for real:\r\n```\r\n$ curl -s 0:10000/metrics |grep build_info|grep -v ^#\r\nnode_exporter_build_info{branch=\"master\",goversion=\"go1.6.2\",revision=\"df8dcd2\",version=\"0.12.0\"} 1\r\n$ curl -s 0:10000/metrics |grep -v ^# |grep textfile_mtime\r\nnode_textfile_mtime{file=\"elasticsearch_cluster_stats.prom\"} 1.464310441823659e+09\r\nnode_textfile_mtime{file=\"sysstat.prom\"} 1.4643104417836597e+09"}}
{"actor":{"display_login":"mrwacky42"}}