{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5036","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5036/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5036/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5036/events","html_url":"https://github.com/zfsonlinux/zfs/issues/5036","id":173669753,"number":5036,"title":"Thousands of kernel worker threads -> `htop` slow","user":{"login":"haasn","id":1149047,"avatar_url":"https://avatars.githubusercontent.com/u/1149047?v=3","gravatar_id":"","url":"https://api.github.com/users/haasn","html_url":"https://github.com/haasn","followers_url":"https://api.github.com/users/haasn/followers","following_url":"https://api.github.com/users/haasn/following{/other_user}","gists_url":"https://api.github.com/users/haasn/gists{/gist_id}","starred_url":"https://api.github.com/users/haasn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haasn/subscriptions","organizations_url":"https://api.github.com/users/haasn/orgs","repos_url":"https://api.github.com/users/haasn/repos","events_url":"https://api.github.com/users/haasn/events{/privacy}","received_events_url":"https://api.github.com/users/haasn/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-08-28T21:45:55Z","updated_at":"2016-08-28T23:30:44Z","closed_at":"2016-08-28T23:30:44Z","body":"I've been observing this issue since switching to ZFS. Basically, I have a few thousand kworker threads on my machine, way, way more than on other machines (including ones also using ZFS).\r\n\r\nI'm not entirely sure where the issue originates from. Any help in tracking it down would be appreciated.\r\n\r\nHere's a big example of what my kernel threads look like: https://0x0.st/MF7.txt\r\n\r\nAbout 6000-7000 of them (the vast majority) are `kworker` threads. The number seems to be slowly changing (i.e. it's not entirely static), both growing and decreasing over time. (But the change happens in batches - it will be static for many seconds, and then later it will have an entirely different number)\r\n\r\nI've inspect a bunch of them via `/proc/NNNN/stack` and they all seem to be doing the same thing:\r\n\r\n```\r\n[<ffffffff81132964>] worker_thread+0xc4/0x480\r\n[<ffffffff811328a0>] worker_thread+0x0/0x480\r\n[<ffffffff81137d48>] kthread+0xb8/0xd0\r\n[<ffffffff81a49a4e>] ret_from_fork+0x1e/0x40\r\n[<ffffffff81137c90>] kthread+0x0/0xd0\r\n[<ffffffffffffffff>] 0xffffffffffffffff\r\n```\r\n\r\n`ret_from_fork` in particular seems rather curious. Given the high number, and the fact that they're every so slowly growing, it sort of seems like some component is suffering from overzealous or recursive forking.\r\n\r\nUnfortunately, I can't easily test what would happen if I disable ZFS, because I can currently only boot off of ZFS. But I will try reproducing the issue with a non-hardened kernel to at least rule that out as a cause.\r\n\r\nGentoo Linux 4.7.2-hardened-gnu, spl/zfs are git master."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/243006420","html_url":"https://github.com/zfsonlinux/zfs/issues/5036#issuecomment-243006420","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/5036","id":243006420,"user":{"login":"haasn","id":1149047,"avatar_url":"https://avatars.githubusercontent.com/u/1149047?v=3","gravatar_id":"","url":"https://api.github.com/users/haasn","html_url":"https://github.com/haasn","followers_url":"https://api.github.com/users/haasn/followers","following_url":"https://api.github.com/users/haasn/following{/other_user}","gists_url":"https://api.github.com/users/haasn/gists{/gist_id}","starred_url":"https://api.github.com/users/haasn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haasn/subscriptions","organizations_url":"https://api.github.com/users/haasn/orgs","repos_url":"https://api.github.com/users/haasn/repos","events_url":"https://api.github.com/users/haasn/events{/privacy}","received_events_url":"https://api.github.com/users/haasn/received_events","type":"User","site_admin":false},"created_at":"2016-08-28T23:30:44Z","updated_at":"2016-08-28T23:30:44Z","body":"Update: I managed to track down a the triggering condition for this bug, and I no longer believe ZFS is directly involved.\r\n\r\nThe issue seems to be at least some very complicated interaction betwen icinga2, systemd and check commands requiring root (to which the check_zfs script belongs). Basically I noticed that every time the process count increases (happens in spikes of a few thousand processes at once), it's *precisely* when one of the icinga check commands triggers, and this also throws out a bunch of systemd messages (it seems to create and tear down an entire session for the user root multiple times, not sure why), and apparently this leaves behind some (lots) of kworker ghosts (also not sure why).\r\n\r\nAt any rate, disabling these check commands in icinga2 reduces the kernel worker thread count of a running system to about 100. The fact that I was always sitting at around 6000 seems to be because the kernel threads eventually start expiring after a while, and 6000 was the sweet spot where the number of threads spawned per check invocation (i.e. per minute) equaled the number of threads that died per minute.\r\n\r\nAnyway, since I don't suspect ZFS at this moment I'll close this issue until further notice (e.g. in case I establish some link between ZFS and the issue again). Sorry for the noise."}}
{"actor":{"display_login":"haasn"}}