{"action":"created","issue":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/475","repository_url":"https://api.github.com/repos/lizardfs/lizardfs","labels_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/475/labels{/name}","comments_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/475/comments","events_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/475/events","html_url":"https://github.com/lizardfs/lizardfs/issues/475","id":177320596,"number":475,"title":"3.10.2: serious performance regression","user":{"login":"onlyjob","id":1061236,"avatar_url":"https://avatars.githubusercontent.com/u/1061236?v=3","gravatar_id":"","url":"https://api.github.com/users/onlyjob","html_url":"https://github.com/onlyjob","followers_url":"https://api.github.com/users/onlyjob/followers","following_url":"https://api.github.com/users/onlyjob/following{/other_user}","gists_url":"https://api.github.com/users/onlyjob/gists{/gist_id}","starred_url":"https://api.github.com/users/onlyjob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/onlyjob/subscriptions","organizations_url":"https://api.github.com/users/onlyjob/orgs","repos_url":"https://api.github.com/users/onlyjob/repos","events_url":"https://api.github.com/users/onlyjob/events{/privacy}","received_events_url":"https://api.github.com/users/onlyjob/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lizardfs/lizardfs/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"blink69","id":3777450,"avatar_url":"https://avatars.githubusercontent.com/u/3777450?v=3","gravatar_id":"","url":"https://api.github.com/users/blink69","html_url":"https://github.com/blink69","followers_url":"https://api.github.com/users/blink69/followers","following_url":"https://api.github.com/users/blink69/following{/other_user}","gists_url":"https://api.github.com/users/blink69/gists{/gist_id}","starred_url":"https://api.github.com/users/blink69/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blink69/subscriptions","organizations_url":"https://api.github.com/users/blink69/orgs","repos_url":"https://api.github.com/users/blink69/repos","events_url":"https://api.github.com/users/blink69/events{/privacy}","received_events_url":"https://api.github.com/users/blink69/received_events","type":"User","site_admin":false},"assignees":[{"login":"blink69","id":3777450,"avatar_url":"https://avatars.githubusercontent.com/u/3777450?v=3","gravatar_id":"","url":"https://api.github.com/users/blink69","html_url":"https://github.com/blink69","followers_url":"https://api.github.com/users/blink69/followers","following_url":"https://api.github.com/users/blink69/following{/other_user}","gists_url":"https://api.github.com/users/blink69/gists{/gist_id}","starred_url":"https://api.github.com/users/blink69/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blink69/subscriptions","organizations_url":"https://api.github.com/users/blink69/orgs","repos_url":"https://api.github.com/users/blink69/repos","events_url":"https://api.github.com/users/blink69/events{/privacy}","received_events_url":"https://api.github.com/users/blink69/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2016-09-16T00:15:54Z","updated_at":"2016-09-20T00:24:41Z","closed_at":null,"body":"Shortly after upgrade to 3.10.2 _master_ exhibited  serious performance regression with 100% CPU usage (one core) and _very slow_ response to clients, etc. Performance of the entire cluster is crippled. \r\n\r\nI couldn't find anything relevant in logs but my suspicion is that regression is somehow related to trash which is unusually big (8.1+ TiB; 1400000+ files). It looks like master struggles to remove files from trash and that it takes much longer than before so removing nightly snapshot ( `settrashtime -r 2`) created too much stress for _mfsmaster_.\r\n\r\nBesides it appears that master uses main thread to process trash and that's one of the reasons why performance is affected so much..."},"comment":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/comments/248167521","html_url":"https://github.com/lizardfs/lizardfs/issues/475#issuecomment-248167521","issue_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/475","id":248167521,"user":{"login":"onlyjob","id":1061236,"avatar_url":"https://avatars.githubusercontent.com/u/1061236?v=3","gravatar_id":"","url":"https://api.github.com/users/onlyjob","html_url":"https://github.com/onlyjob","followers_url":"https://api.github.com/users/onlyjob/followers","following_url":"https://api.github.com/users/onlyjob/following{/other_user}","gists_url":"https://api.github.com/users/onlyjob/gists{/gist_id}","starred_url":"https://api.github.com/users/onlyjob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/onlyjob/subscriptions","organizations_url":"https://api.github.com/users/onlyjob/orgs","repos_url":"https://api.github.com/users/onlyjob/repos","events_url":"https://api.github.com/users/onlyjob/events{/privacy}","received_events_url":"https://api.github.com/users/onlyjob/received_events","type":"User","site_admin":false},"created_at":"2016-09-20T00:24:41Z","updated_at":"2016-09-20T00:24:41Z","body":"2710 is a big relief but `Can't read data from socket: timeout` still happens on user's home folders and 100% CPU utilisation and slowness are still there... I'll test further..."}}
{"actor":{"display_login":"onlyjob"}}