{"action":"created","issue":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/377","labels_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/377/labels{/name}","comments_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/377/comments","events_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/377/events","html_url":"https://github.com/lizardfs/lizardfs/issues/377","id":129147903,"number":377,"title":"Chunkserver communication timed out","user":{"login":"FATruden","id":1313540,"avatar_url":"https://avatars.githubusercontent.com/u/1313540?v=3","gravatar_id":"","url":"https://api.github.com/users/FATruden","html_url":"https://github.com/FATruden","followers_url":"https://api.github.com/users/FATruden/followers","following_url":"https://api.github.com/users/FATruden/following{/other_user}","gists_url":"https://api.github.com/users/FATruden/gists{/gist_id}","starred_url":"https://api.github.com/users/FATruden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FATruden/subscriptions","organizations_url":"https://api.github.com/users/FATruden/orgs","repos_url":"https://api.github.com/users/FATruden/repos","events_url":"https://api.github.com/users/FATruden/events{/privacy}","received_events_url":"https://api.github.com/users/FATruden/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-27T13:43:53Z","updated_at":"2016-01-28T10:43:54Z","closed_at":null,"body":"Hi friends. \r\n\r\nI have a chunkserver'hangs with the following message in the logs:\r\n\r\n`Jan 26 16:41:43 ak34 mfsmount[20791]: read file error, inode: 4, index: 0, chunk: 97, version: 1 - Chunkserver communication timed out: 172.24.32.134:9422 (try counter: 1)`\r\n`Jan 26 16:41:45 ak34 mfsmount[20791]: read file error, inode: 4, index: 0, chunk: 97, version: 1 - Chunkserver communication timed out: 172.24.32.134:9422 (try counter: 2)`\r\n`Jan 26 16:41:47 ak34 mfsmount[20791]: read file error, inode: 4, index: 0, chunk: 97, version: 1 - Chunkserver communication timed out: 172.24.32.134:9422 (try counter: 3)`\r\n\r\nThis error occurs only when the read operations. \r\nI have a master, chunkserver(with one disk) and a client on a single server(for testing only).\r\n\r\nI have the following setting in chunkserver conf:\r\n\r\nNR_OF_NETWORK_WORKERS = 2\r\nNR_OF_HDD_WORKERS_PER_NETWORK_WORKER = 8\r\n\r\nI'm using rhel 6.6 and lizardfs 3.9.4.\r\nI tried few different disks but it didn't help.\r\n\r\nUnfortunately I can't reproduce this behavior on my other test stands, but to leave this problem without attention I can't either.\r\n\r\nMaybe someone has ideas why this is happening?"},"comment":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/comments/176118278","html_url":"https://github.com/lizardfs/lizardfs/issues/377#issuecomment-176118278","issue_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/377","id":176118278,"user":{"login":"FATruden","id":1313540,"avatar_url":"https://avatars.githubusercontent.com/u/1313540?v=3","gravatar_id":"","url":"https://api.github.com/users/FATruden","html_url":"https://github.com/FATruden","followers_url":"https://api.github.com/users/FATruden/followers","following_url":"https://api.github.com/users/FATruden/following{/other_user}","gists_url":"https://api.github.com/users/FATruden/gists{/gist_id}","starred_url":"https://api.github.com/users/FATruden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FATruden/subscriptions","organizations_url":"https://api.github.com/users/FATruden/orgs","repos_url":"https://api.github.com/users/FATruden/repos","events_url":"https://api.github.com/users/FATruden/events{/privacy}","received_events_url":"https://api.github.com/users/FATruden/received_events","type":"User","site_admin":false},"created_at":"2016-01-28T10:43:54Z","updated_at":"2016-01-28T10:43:54Z","body":"@psarna, thanks for the quick reply.\r\nThe server has no load, and the disks are idle. \r\nI run a simple `dd if=/mnt/filename of/dev/null iflag=direct` and it hangs and in the log appear these error.\r\nThis occurs up until I kill the dd-process hard.\r\nAfter that, the chunkserver process obeys the commands stop and restart. I have to kill him.\r\nI have tried to use default values of \r\n\r\nNR_OF_NETWORK_WORKERS\r\nNR_OF_HDD_WORKERS_PER_NETWORK_WORKER\r\n\r\nand nothing helped.\r\n\r\nI understand that without clear steps to reproduce, it is difficult to say. But perhaps you have some thoughts which will help me to find the cause."}}