{"action":"created","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/3350","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/3350/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/3350/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/3350/events","html_url":"https://github.com/flynn/flynn/issues/3350","id":174632265,"number":3350,"title":"Load average spike on node, probably due to zil_clean processes","user":{"login":"temujin9","id":627414,"avatar_url":"https://avatars.githubusercontent.com/u/627414?v=3","gravatar_id":"","url":"https://api.github.com/users/temujin9","html_url":"https://github.com/temujin9","followers_url":"https://api.github.com/users/temujin9/followers","following_url":"https://api.github.com/users/temujin9/following{/other_user}","gists_url":"https://api.github.com/users/temujin9/gists{/gist_id}","starred_url":"https://api.github.com/users/temujin9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/temujin9/subscriptions","organizations_url":"https://api.github.com/users/temujin9/orgs","repos_url":"https://api.github.com/users/temujin9/repos","events_url":"https://api.github.com/users/temujin9/events{/privacy}","received_events_url":"https://api.github.com/users/temujin9/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/flynn/flynn/labels/kind/bug","name":"kind/bug","color":"d93f0b"},{"url":"https://api.github.com/repos/flynn/flynn/labels/upstream/zfs","name":"upstream/zfs","color":"d4c5f9"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-09-01T21:38:14Z","updated_at":"2016-09-07T18:54:16Z","closed_at":"2016-09-07T18:54:16Z","body":"I've got a cluster with a node which has had a sustained load of 6-8 since the 29th.\r\n\r\n```\r\nWarning: Permanently added '10.100.55.213' (ECDSA) to the list of known hosts.\r\n 21:06:39 up 4 days,  3:01,  0 users,  load average: 2.67, 2.60, 2.79\r\n\r\nWarning: Permanently added '10.100.55.216' (ECDSA) to the list of known hosts.\r\n 21:06:44 up 4 days,  3:01,  0 users,  load average: 5.79, 6.00, 6.23\r\n\r\nWarning: Permanently added '10.100.55.201' (ECDSA) to the list of known hosts.\r\n 21:06:51 up 4 days,  3:01,  0 users,  load average: 3.27, 2.92, 3.17\r\n\r\nWarning: Permanently added '10.100.55.217' (ECDSA) to the list of known hosts.\r\n 21:06:53 up 4 days,  3:01,  0 users,  load average: 2.32, 2.41, 2.58\r\n\r\nWarning: Permanently added '10.100.55.218' (ECDSA) to the list of known hosts.\r\n 21:07:10 up 4 days,  3:02,  0 users,  load average: 2.61, 2.59, 2.70\r\n```\r\n\r\nProcess list on the affected node shows a lot of zil_clean processes:\r\n```\r\n$ ps aux | grep -c zil_clean\r\n8976\r\n```\r\n\r\nhttps://bazaarvoice.box.com/s/bwl6lb43hb0y9r2mipjhj8onojlhx73p has tarballs of /var/log/flynn, as collect-debug-info kept hanging. http://zfs-discuss.opensolaris.narkive.com/z6yLSOEf/lots-of-zil-clean-threads seems to describe the same error."},"comment":{"url":"https://api.github.com/repos/flynn/flynn/issues/comments/245380679","html_url":"https://github.com/flynn/flynn/issues/3350#issuecomment-245380679","issue_url":"https://api.github.com/repos/flynn/flynn/issues/3350","id":245380679,"user":{"login":"temujin9","id":627414,"avatar_url":"https://avatars.githubusercontent.com/u/627414?v=3","gravatar_id":"","url":"https://api.github.com/users/temujin9","html_url":"https://github.com/temujin9","followers_url":"https://api.github.com/users/temujin9/followers","following_url":"https://api.github.com/users/temujin9/following{/other_user}","gists_url":"https://api.github.com/users/temujin9/gists{/gist_id}","starred_url":"https://api.github.com/users/temujin9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/temujin9/subscriptions","organizations_url":"https://api.github.com/users/temujin9/orgs","repos_url":"https://api.github.com/users/temujin9/repos","events_url":"https://api.github.com/users/temujin9/events{/privacy}","received_events_url":"https://api.github.com/users/temujin9/received_events","type":"User","site_admin":false},"created_at":"2016-09-07T18:54:16Z","updated_at":"2016-09-07T18:54:16Z","body":"This seems to be resolved without code changes. Closing it for now."}}
{"actor":{"display_login":"temujin9"}}