{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/node_exporter/issues/868","repository_url":"https://api.github.com/repos/prometheus/node_exporter","labels_url":"https://api.github.com/repos/prometheus/node_exporter/issues/868/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/node_exporter/issues/868/comments","events_url":"https://api.github.com/repos/prometheus/node_exporter/issues/868/events","html_url":"https://github.com/prometheus/node_exporter/issues/868","id":309033637,"number":868,"title":"node_exporter hangs in a statfs syscall","user":{"login":"MarkusTeufelberger","id":950561,"avatar_url":"https://avatars0.githubusercontent.com/u/950561?v=4","gravatar_id":"","url":"https://api.github.com/users/MarkusTeufelberger","html_url":"https://github.com/MarkusTeufelberger","followers_url":"https://api.github.com/users/MarkusTeufelberger/followers","following_url":"https://api.github.com/users/MarkusTeufelberger/following{/other_user}","gists_url":"https://api.github.com/users/MarkusTeufelberger/gists{/gist_id}","starred_url":"https://api.github.com/users/MarkusTeufelberger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarkusTeufelberger/subscriptions","organizations_url":"https://api.github.com/users/MarkusTeufelberger/orgs","repos_url":"https://api.github.com/users/MarkusTeufelberger/repos","events_url":"https://api.github.com/users/MarkusTeufelberger/events{/privacy}","received_events_url":"https://api.github.com/users/MarkusTeufelberger/received_events","type":"User","site_admin":false},"labels":[{"id":36419757,"url":"https://api.github.com/repos/prometheus/node_exporter/labels/duplicate","name":"duplicate","color":"cccccc","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-27T16:03:36Z","updated_at":"2018-03-28T11:07:14Z","closed_at":null,"author_association":"NONE","body":"<!--\r\n\tPlease note: GitHub issues should only be used for feature requests and\r\n\tbug reports. For general usage/help/discussions, please refer to one of:\r\n\r\n\t- #prometheus on freenode\r\n\t- the Prometheus Users list: https://groups.google.com/forum/#!forum/prometheus-users\r\n\r\n\tBefore filing a bug report, note that running node_exporter in Docker is\r\n\tnot recommended, for the reasons detailed in the README:\r\n\r\n\thttps://github.com/prometheus/node_exporter#using-docker\r\n\r\n\tFinally, also note that node_exporter is focused on *NIX kernels, and the\r\n\tWMI exporter should be used instead on Windows.\r\n\r\n\tFor bug reports, please fill out the below fields and provide as much detail\r\n\tas possible about your issue.  For feature requests, you may omit the\r\n\tfollowing template.\r\n-->\r\n### Host operating system: output of `uname -a`\r\n`Linux xxxXXXxxx 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux`\r\n\r\n### node_exporter version: output of `node_exporter --version`\r\n````\r\nnode_exporter, version 0.15.2 (branch: HEAD, revision: 98bc64930d34878b84a0f87dfe6e1a6da61e532d)\r\n  build user:       root@d5c4792c921f\r\n  build date:       20171205-14:50:53\r\n  go version:       go1.9.2\r\n````\r\n\r\n### node_exporter command line flags\r\n<!-- Please list all of the command line flags -->\r\n`--web.listen-address=127.0.0.1:9100 '--collector.diskstats.ignored-devices=^(ram|loop|fd|rpc_pipefs)\\d*$' --collector.diskstats --collector.filesystem --collector.loadavg --collector.meminfo --collector.stat --collector.textfile --collector.time --collector.netdev --collector.netstat --collector.filefd --collector.vmstat --collector.sockstat --collector.tcpstat --collector.entropy --collector.systemd --collector.ntp --collector.systemd.private --collector.textfile.directory=/etc/prometheus-node-exporter/textfile-collector`\r\n\r\n### Are you running node_exporter in Docker?\r\n<!-- Please note the warning above. -->\r\nNo.\r\n\r\n### What did you do that produced an error?\r\nApparently there were issues with NFS causing some mounts to become unresponsive\r\n\r\n### What did you expect to see?\r\nAll metrics except the NFS ones are still exported.\r\n\r\n### What did you see instead?\r\nnode_exporter hangs in a statfs syscall.\r\n\r\n(2414 is the PID of node-exporter)\r\n````\r\nroot@xxxXXXxxx ~ # cat /proc/2414/stack                                                                                                                                                                           \r\n[<ffffffffc048bb54>] rpc_wait_bit_killable+0x24/0xa0 [sunrpc]                                                                                                                                                       \r\n[<ffffffffc048ca0e>] __rpc_execute+0x15e/0x420 [sunrpc]                                                                                                                                                             \r\n[<ffffffffc048ef01>] rpc_execute+0x61/0xc0 [sunrpc]                                                                                                                                                                 \r\n[<ffffffffc047fe48>] rpc_run_task+0xf8/0x140 [sunrpc]                                                                                                                                                               \r\n[<ffffffffc0482020>] rpc_call_sync+0x50/0x90 [sunrpc]                                                                                                                                                               \r\n[<ffffffffc08c5729>] nfs3_rpc_wrapper.constprop.12+0x39/0xa0 [nfsv3]                                                                                                                                                \r\n[<ffffffffc08c63d6>] nfs3_proc_getattr+0x56/0xb0 [nfsv3]                                                                                                                                                            \r\n[<ffffffffc0893270>] __nfs_revalidate_inode+0xd0/0x360 [nfs]                                                                                                                                                        \r\n[<ffffffffc088a785>] nfs_weak_revalidate+0xb5/0x110 [nfs]                                                                                                                                                           \r\n[<ffffffffa746271c>] complete_walk+0x6c/0x80                                                                                                                                                                        \r\n[<ffffffffa7464650>] path_lookupat+0x110/0x230                                                                                                                                                                      \r\n[<ffffffffa7467948>] filename_lookup+0xb8/0x1a0                                                                                                                                                                     \r\n[<ffffffffa7467b06>] user_path_at_empty+0x36/0x40                                                                                                                                                                   \r\n[<ffffffffa748fce6>] user_statfs+0x46/0xa0                                                                                                                                                                          \r\n[<ffffffffa748fd67>] SYSC_statfs+0x27/0x60                                                                                                                                                                          \r\n[<ffffffffa74900ee>] SyS_statfs+0xe/0x10                                                                                                                                                                            \r\n[<ffffffffa7c001a1>] entry_SYSCALL_64_fastpath+0x24/0xab                                                                                                                                                            \r\n[<ffffffffffffffff>] 0xffffffffffffffff  \r\n````\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/node_exporter/issues/comments/376848626","html_url":"https://github.com/prometheus/node_exporter/issues/868#issuecomment-376848626","issue_url":"https://api.github.com/repos/prometheus/node_exporter/issues/868","id":376848626,"user":{"login":"MarkusTeufelberger","id":950561,"avatar_url":"https://avatars0.githubusercontent.com/u/950561?v=4","gravatar_id":"","url":"https://api.github.com/users/MarkusTeufelberger","html_url":"https://github.com/MarkusTeufelberger","followers_url":"https://api.github.com/users/MarkusTeufelberger/followers","following_url":"https://api.github.com/users/MarkusTeufelberger/following{/other_user}","gists_url":"https://api.github.com/users/MarkusTeufelberger/gists{/gist_id}","starred_url":"https://api.github.com/users/MarkusTeufelberger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarkusTeufelberger/subscriptions","organizations_url":"https://api.github.com/users/MarkusTeufelberger/orgs","repos_url":"https://api.github.com/users/MarkusTeufelberger/repos","events_url":"https://api.github.com/users/MarkusTeufelberger/events{/privacy}","received_events_url":"https://api.github.com/users/MarkusTeufelberger/received_events","type":"User","site_admin":false},"created_at":"2018-03-28T11:07:14Z","updated_at":"2018-03-28T11:07:14Z","author_association":"NONE","body":"No to take the analogy too far, but I don't want the car windows to turn blank if the steering wheel is stuck so I don't have any way of knowing which state the system is in... Especially in this situation I want my GPS to tell me I'm still in my garage and not on the highway instead of just telling me nothing at all (not even logging which metric is stuck!).\r\n\r\nIf I understand you correctly, you are suggesting that I automatically run `df` and collect the output of that as a text file because node_exporter can't do this for me?!\r\n\r\nI agree that having syscalls that put the calling thread in uninterruptible sleep potentially indefinitely with no way to cancel or timeout sucks. There are solutions to this problem though, I'm sure. Also, if this is not recommended and discussed since 2 years in a bug report and you don't want to introduce state to the metrics collection, why is the broken/problematic nfs module still enabled by default instead of having a documented alternative with systemd timers, text exporters and nonblocking RPC metrics?"}}
{"actor":{"display_login":"MarkusTeufelberger"}}