{"action":"created","issue":{"url":"https://api.github.com/repos/minio/minio/issues/6612","repository_url":"https://api.github.com/repos/minio/minio","labels_url":"https://api.github.com/repos/minio/minio/issues/6612/labels{/name}","comments_url":"https://api.github.com/repos/minio/minio/issues/6612/comments","events_url":"https://api.github.com/repos/minio/minio/issues/6612/events","html_url":"https://github.com/minio/minio/issues/6612","id":368995057,"node_id":"MDU6SXNzdWUzNjg5OTUwNTc=","number":6612,"title":"Too many file descriptors","user":{"login":"pepelisu","id":4609725,"node_id":"MDQ6VXNlcjQ2MDk3MjU=","avatar_url":"https://avatars0.githubusercontent.com/u/4609725?v=4","gravatar_id":"","url":"https://api.github.com/users/pepelisu","html_url":"https://github.com/pepelisu","followers_url":"https://api.github.com/users/pepelisu/followers","following_url":"https://api.github.com/users/pepelisu/following{/other_user}","gists_url":"https://api.github.com/users/pepelisu/gists{/gist_id}","starred_url":"https://api.github.com/users/pepelisu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pepelisu/subscriptions","organizations_url":"https://api.github.com/users/pepelisu/orgs","repos_url":"https://api.github.com/users/pepelisu/repos","events_url":"https://api.github.com/users/pepelisu/events{/privacy}","received_events_url":"https://api.github.com/users/pepelisu/received_events","type":"User","site_admin":false},"labels":[{"id":399254082,"node_id":"MDU6TGFiZWwzOTkyNTQwODI=","url":"https://api.github.com/repos/minio/minio/labels/community","name":"community","color":"006b75","default":false},{"id":574010253,"node_id":"MDU6TGFiZWw1NzQwMTAyNTM=","url":"https://api.github.com/repos/minio/minio/labels/priority:%20medium","name":"priority: medium","color":"fbca04","default":false},{"id":643968487,"node_id":"MDU6TGFiZWw2NDM5Njg0ODc=","url":"https://api.github.com/repos/minio/minio/labels/triage","name":"triage","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-11T07:52:21Z","updated_at":"2018-10-11T11:43:47Z","closed_at":null,"author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\nAfter stop and restart the daemon in distributed mode (2  nodes/ 4 disk each) the terminal is full of errors like this one:\r\n```\r\n2018-10-11 07:20:52.010224 I | http: Accept error: accept tcp 0.0.0.0:9000: accept4: too many open files; retrying in 10ms\r\n```\r\nWhen login in Minio Browser only one bucket is listed and when try to access it, it says ```server not initialized please try again.```\r\n\r\n## Expected Behavior\r\nMinio should restart without errors. \r\n\r\n## Current Behavior\r\nAs mentioned above when stopped and started again, the output in terminal is a list of errors (too many open files) that seems endless, it has not stopped after 12h. \r\nIf minio server is stopped in one of the nodes (server2) a big list with errors is shown all in the form of: \r\n```\r\ngoroutine 115801674 [IO wait]:\r\ninternal/poll.runtime_pollWait(0x7f8ee8d739f8, 0x72, 0xc4d9e77960)\r\n\t/opt/go/src/runtime/netpoll.go:173 +0x57\r\ninternal/poll.(*pollDesc).wait(0xc44cc26018, 0x72, 0xffffffffffffff00, 0x15dca60, 0x1d7a618)\r\n\t/opt/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b\r\ninternal/poll.(*pollDesc).waitRead(0xc44cc26018, 0xc42dd07000, 0x1000, 0x1000)\r\n\t/opt/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r\ninternal/poll.(*FD).Read(0xc44cc26000, 0xc42dd07000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/opt/go/src/internal/poll/fd_unix.go:157 +0x17d\r\nnet.(*netFD).Read(0xc44cc26000, 0xc42dd07000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/opt/go/src/net/fd_unix.go:202 +0x4f\r\nnet.(*conn).Read(0xc43ce0b878, 0xc42dd07000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/opt/go/src/net/net.go:176 +0x6a\r\ngithub.com/minio/minio/cmd/http.(*TimeoutConn).Read(0xc445a5dbf0, 0xc42dd07000, 0x1000, 0x1000, 0xc47010c840, 0xc4b576d0e0, 0xc4d9e77bfc)\r\n\t/q/.q/sources/gopath/src/github.com/minio/minio/cmd/http/timeoutconn.go:46 +0x5f\r\nnet/http.(*persistConn).Read(0xc42e03ab40, 0xc42dd07000, 0x1000, 0x1000, 0xc4d9e77b98, 0x403355, 0xc47010c840)\r\n\t/opt/go/src/net/http/transport.go:1453 +0x136\r\nbufio.(*Reader).fill(0xc4b4e93860)\r\n\t/opt/go/src/bufio/bufio.go:100 +0x11e\r\nbufio.(*Reader).Peek(0xc4b4e93860, 0x1, 0x0, 0x0, 0x0, 0xc47010c6c0, 0x0)\r\n\t/opt/go/src/bufio/bufio.go:132 +0x3a\r\nnet/http.(*persistConn).readLoop(0xc42e03ab40)\r\n\t/opt/go/src/net/http/transport.go:1601 +0x185\r\ncreated by net/http.(*Transport).dialConn\r\n\t/opt/go/src/net/http/transport.go:1237 +0x95a\r\n\r\nrax    0xca\r\nrbx    0x20c6520\r\nrcx    0x45b6a3\r\nrdx    0x0\r\nrdi    0x20c6668\r\nrsi    0x0\r\nrbp    0x7ffe7ee711d0\r\nrsp    0x7ffe7ee71188\r\nr8     0x0\r\nr9     0x0\r\nr10    0x0\r\nr11    0x286\r\nr12    0x433ffce91dce9\r\nr13    0x45290a\r\nr14    0x200\r\nr15    0x4\r\nrip    0x45b6a1\r\nrflags 0x286\r\ncs     0x33\r\nfs     0x0\r\ngs     0x0\r\n\r\n```\r\nConsequently the other node (server1) stops to throw errors. Stopping minio in server1, after stopping server2, throws no error. \r\n\r\n## Possible Solution\r\nWriting this I realize that server1 and server2 where not running the same minio version. They where working before the issue with server1=```RELEASE.2018-10-06T00-15-16Z```, while server2 was running: ```RELEASE.2018-09-01T00-38-25Z```\r\nI stopped both servers, server2 with errors first, then server1 without. After update server2 to ```RELEASE.2018-10-06T00-15-16Z``` and restart both servers they started without errors and now minio web interface is showing all the buckets. \r\nSo possible solution, always run the same server release in all the distributed systems. \r\n\r\n## Steps to Reproduce (for bugs)\r\nI am not 100% sure that this will work as steps to reproduce, but lets say that:\r\n1. At least two servers in distributed mode, one running updated and the other running old version of the server\r\n2. Copy some data into a bucket.\r\n3. Start and stop the servers.\r\n\r\n## Context\r\nI was changing from manual start to a daemon installation with SysVinit. \r\n\r\n## Regression\r\nMore than a regression this is a incompatibility. \r\n\r\n## Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\n* Version used (`minio version`): RELEASE.2018-10-06T00-15-16Z and RELEASE.2018-09-01T00-38-25Z\r\n* Environment name and version (e.g. nginx 1.9.1): binary minio executable\r\n* Server type and version: NA\r\n* Operating System and version (`uname -a`): Linux 4.14.69-0-vanilla #1-Alpine SMP Mon Sep 10 19:33:23 UTC 2018 x86_64 Linux\r\n"},"comment":{"url":"https://api.github.com/repos/minio/minio/issues/comments/428924324","html_url":"https://github.com/minio/minio/issues/6612#issuecomment-428924324","issue_url":"https://api.github.com/repos/minio/minio/issues/6612","id":428924324,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODkyNDMyNA==","user":{"login":"nitisht","id":5156139,"node_id":"MDQ6VXNlcjUxNTYxMzk=","avatar_url":"https://avatars0.githubusercontent.com/u/5156139?v=4","gravatar_id":"","url":"https://api.github.com/users/nitisht","html_url":"https://github.com/nitisht","followers_url":"https://api.github.com/users/nitisht/followers","following_url":"https://api.github.com/users/nitisht/following{/other_user}","gists_url":"https://api.github.com/users/nitisht/gists{/gist_id}","starred_url":"https://api.github.com/users/nitisht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nitisht/subscriptions","organizations_url":"https://api.github.com/users/nitisht/orgs","repos_url":"https://api.github.com/users/nitisht/repos","events_url":"https://api.github.com/users/nitisht/events{/privacy}","received_events_url":"https://api.github.com/users/nitisht/received_events","type":"User","site_admin":false},"created_at":"2018-10-11T11:43:47Z","updated_at":"2018-10-11T11:43:47Z","author_association":"MEMBER","body":"@pepelisu what is the client and the file system you're using. \r\n\r\nWe made some changes in the way servers interact with each other (RPC calls) in release `RELEASE.2018-10-06T00-15-16Z` "}}
{"actor":{"display_login":"nitisht"}}