{"action":"created","issue":{"url":"https://api.github.com/repos/gluster/glusterd2/issues/1166","repository_url":"https://api.github.com/repos/gluster/glusterd2","labels_url":"https://api.github.com/repos/gluster/glusterd2/issues/1166/labels{/name}","comments_url":"https://api.github.com/repos/gluster/glusterd2/issues/1166/comments","events_url":"https://api.github.com/repos/gluster/glusterd2/issues/1166/events","html_url":"https://github.com/gluster/glusterd2/issues/1166","id":350031365,"node_id":"MDU6SXNzdWUzNTAwMzEzNjU=","number":1166,"title":"System idle state causes to peers offline","user":{"login":"rmadaka","id":34678351,"node_id":"MDQ6VXNlcjM0Njc4MzUx","avatar_url":"https://avatars3.githubusercontent.com/u/34678351?v=4","gravatar_id":"","url":"https://api.github.com/users/rmadaka","html_url":"https://github.com/rmadaka","followers_url":"https://api.github.com/users/rmadaka/followers","following_url":"https://api.github.com/users/rmadaka/following{/other_user}","gists_url":"https://api.github.com/users/rmadaka/gists{/gist_id}","starred_url":"https://api.github.com/users/rmadaka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rmadaka/subscriptions","organizations_url":"https://api.github.com/users/rmadaka/orgs","repos_url":"https://api.github.com/users/rmadaka/repos","events_url":"https://api.github.com/users/rmadaka/events{/privacy}","received_events_url":"https://api.github.com/users/rmadaka/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-13T13:32:36Z","updated_at":"2018-08-13T13:41:40Z","closed_at":null,"author_association":"NONE","body":"-> Having 3 node cluster (n1, n2, n3) with external etcd.\r\n-> kept idle for 40 mints.\r\n-> I have  checked the peer status n2 and n3 went to offline like below.\r\n\r\nperfromed from node n1:\r\n\r\n```glustercli peer status\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+--t-----+\r\n|                  ID                  |               NAME                |  CLIENT ADDRESSES  |   PEER ADDRESSES   | ONLINE |  PID  |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n| 46b846fd-32e8-4046-9080-fb827413ccff | dhcp35-194.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.194:24008 | no     |       |\r\n|                                      |                                   | 10.70.35.194:24007 |                    |        |       |\r\n| 5d9f6dbc-3231-4e78-93ba-148ed8fd6249 | dhcp35-33.lab.eng.blr.redhat.com  | 127.0.0.1:24007    | 10.70.35.33:24008  | no     |       |\r\n|                                      |                                   | 10.70.35.33:24007  |                    |        |       |\r\n| 91b6edad-6246-4b77-9a0d-6bf92a6a1514 | dhcp35-219.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.219:24008 | yes    | 30476 |\r\n|                                      |                                   | 10.70.35.219:24007 |                    |        |       |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n```\r\n-> I have performed 'systemctl restart glusterd2' on node n2, then node n2 peer status showing online, afterwards i have performed 'systemctl restart glusterd2 ' on node n3, now node n3 peer status showing online, node n2 peer status went to offline.\r\n-> I have tried couple of times restarting glusterd2 on node n2 and n3, recently restarted node only will be in online previously restarted node will go to offline.\r\n-> Out 3 nodes only 2 nodes will be in online one node will be in offline state.\r\noutput like below:\r\n----------------------\r\n```[root@dhcp35-219 ~]# glustercli peer status\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n|                  ID                  |               NAME                |  CLIENT ADDRESSES  |   PEER ADDRESSES   | ONLINE |  PID  |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n| 46b846fd-32e8-4046-9080-fb827413ccff | dhcp35-194.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.194:24008 | yes    |  7946 |\r\n|                                      |                                   | 10.70.35.194:24007 |                    |        |       |\r\n| 5d9f6dbc-3231-4e78-93ba-148ed8fd6249 | dhcp35-33.lab.eng.blr.redhat.com  | 127.0.0.1:24007    | 10.70.35.33:24008  | no     |       |\r\n|                                      |                                   | 10.70.35.33:24007  |                    |        |       |\r\n| 91b6edad-6246-4b77-9a0d-6bf92a6a1514 | dhcp35-219.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.219:24008 | yes    | 30476 |\r\n|                                      |                                   | 10.70.35.219:24007 |                    |        |       |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n[root@dhcp35-219 ~]# glustercli peer status\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n|                  ID                  |               NAME                |  CLIENT ADDRESSES  |   PEER ADDRESSES   | ONLINE |  PID  |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n| 46b846fd-32e8-4046-9080-fb827413ccff | dhcp35-194.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.194:24008 | no     |       |\r\n|                                      |                                   | 10.70.35.194:24007 |                    |        |       |\r\n| 5d9f6dbc-3231-4e78-93ba-148ed8fd6249 | dhcp35-33.lab.eng.blr.redhat.com  | 127.0.0.1:24007    | 10.70.35.33:24008  | yes    | 11368 |\r\n|                                      |                                   | 10.70.35.33:24007  |                    |        |       |\r\n| 91b6edad-6246-4b77-9a0d-6bf92a6a1514 | dhcp35-219.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.219:24008 | yes    | 30476 |\r\n|                                      |                                   | 10.70.35.219:24007 |                    |        |       |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+-------+\r\n```\r\n\r\n-> Out of n2 and n3, even if any node is coming to online none of the glustercli commands are working..\r\n\r\ncommand:\r\nglustercli volume list\r\nError getting Volumes list\r\n\r\nFailed to connect to glusterd. Please check if\r\n- Glusterd is running(http://127.0.0.1:24007) and reachable from this node.\r\n- Make sure Endpoints specified in the command is valid\r\n\r\n-> After 5 mints node n1 also went to offline.\r\n\r\ncommand:\r\n[root@dhcp35-219 ~]# glustercli peer status\r\nFailed to get Peers list\r\n\r\nFailed to connect to glusterd. Please check if\r\n- Glusterd is running(http://127.0.0.1:24007) and reachable from this node.\r\n- Make sure Endpoints specified in the command is valid\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/gluster/glusterd2/issues/comments/412521899","html_url":"https://github.com/gluster/glusterd2/issues/1166#issuecomment-412521899","issue_url":"https://api.github.com/repos/gluster/glusterd2/issues/1166","id":412521899,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjUyMTg5OQ==","user":{"login":"rmadaka","id":34678351,"node_id":"MDQ6VXNlcjM0Njc4MzUx","avatar_url":"https://avatars3.githubusercontent.com/u/34678351?v=4","gravatar_id":"","url":"https://api.github.com/users/rmadaka","html_url":"https://github.com/rmadaka","followers_url":"https://api.github.com/users/rmadaka/followers","following_url":"https://api.github.com/users/rmadaka/following{/other_user}","gists_url":"https://api.github.com/users/rmadaka/gists{/gist_id}","starred_url":"https://api.github.com/users/rmadaka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rmadaka/subscriptions","organizations_url":"https://api.github.com/users/rmadaka/orgs","repos_url":"https://api.github.com/users/rmadaka/repos","events_url":"https://api.github.com/users/rmadaka/events{/privacy}","received_events_url":"https://api.github.com/users/rmadaka/received_events","type":"User","site_admin":false},"created_at":"2018-08-13T13:41:40Z","updated_at":"2018-08-13T13:41:40Z","author_association":"NONE","body":"After reboot of nodes, all 3 nodes came back to online..\r\n```[root@dhcp35-219 ~]# glustercli peer status\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+------+\r\n|                  ID                  |               NAME                |  CLIENT ADDRESSES  |   PEER ADDRESSES   | ONLINE | PID  |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+------+\r\n| 46b846fd-32e8-4046-9080-fb827413ccff | dhcp35-194.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.194:24008 | yes    | 1461 |\r\n|                                      |                                   | 10.70.35.194:24007 |                    |        |      |\r\n| 5d9f6dbc-3231-4e78-93ba-148ed8fd6249 | dhcp35-33.lab.eng.blr.redhat.com  | 127.0.0.1:24007    | 10.70.35.33:24008  | yes    | 1470 |\r\n|                                      |                                   | 10.70.35.33:24007  |                    |        |      |\r\n| 91b6edad-6246-4b77-9a0d-6bf92a6a1514 | dhcp35-219.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.219:24008 | yes    | 1463 |\r\n|                                      |                                   | 10.70.35.219:24007 |                    |        |      |\r\n+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+------+\r\n```"}}
{"actor":{"display_login":"rmadaka"}}