{"action":"created","issue":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/5373","repository_url":"https://api.github.com/repos/arangodb/arangodb","labels_url":"https://api.github.com/repos/arangodb/arangodb/issues/5373/labels{/name}","comments_url":"https://api.github.com/repos/arangodb/arangodb/issues/5373/comments","events_url":"https://api.github.com/repos/arangodb/arangodb/issues/5373/events","html_url":"https://github.com/arangodb/arangodb/issues/5373","id":323893558,"node_id":"MDU6SXNzdWUzMjM4OTM1NTg=","number":5373,"title":"Cluster with arangodb starter - Failed to get master URL","user":{"login":"p30arena","id":1261332,"node_id":"MDQ6VXNlcjEyNjEzMzI=","avatar_url":"https://avatars1.githubusercontent.com/u/1261332?v=4","gravatar_id":"","url":"https://api.github.com/users/p30arena","html_url":"https://github.com/p30arena","followers_url":"https://api.github.com/users/p30arena/followers","following_url":"https://api.github.com/users/p30arena/following{/other_user}","gists_url":"https://api.github.com/users/p30arena/gists{/gist_id}","starred_url":"https://api.github.com/users/p30arena/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/p30arena/subscriptions","organizations_url":"https://api.github.com/users/p30arena/orgs","repos_url":"https://api.github.com/users/p30arena/repos","events_url":"https://api.github.com/users/p30arena/events{/privacy}","received_events_url":"https://api.github.com/users/p30arena/received_events","type":"User","site_admin":false},"labels":[{"id":699898031,"node_id":"MDU6TGFiZWw2OTk4OTgwMzE=","url":"https://api.github.com/repos/arangodb/arangodb/labels/1%20Analyzing","name":"1 Analyzing","color":"1d76db","default":false},{"id":6585198,"node_id":"MDU6TGFiZWw2NTg1MTk4","url":"https://api.github.com/repos/arangodb/arangodb/labels/1%20Question","name":"1 Question","color":"fbca04","default":false},{"id":271864709,"node_id":"MDU6TGFiZWwyNzE4NjQ3MDk=","url":"https://api.github.com/repos/arangodb/arangodb/labels/3%20Cluster","name":"3 Cluster","color":"f9d0c4","default":false},{"id":696951598,"node_id":"MDU6TGFiZWw2OTY5NTE1OTg=","url":"https://api.github.com/repos/arangodb/arangodb/labels/Waiting%20User%20Reply","name":"Waiting User Reply","color":"d93f0b","default":false}],"state":"open","locked":false,"assignee":{"login":"OmarAyo","id":32968582,"node_id":"MDQ6VXNlcjMyOTY4NTgy","avatar_url":"https://avatars2.githubusercontent.com/u/32968582?v=4","gravatar_id":"","url":"https://api.github.com/users/OmarAyo","html_url":"https://github.com/OmarAyo","followers_url":"https://api.github.com/users/OmarAyo/followers","following_url":"https://api.github.com/users/OmarAyo/following{/other_user}","gists_url":"https://api.github.com/users/OmarAyo/gists{/gist_id}","starred_url":"https://api.github.com/users/OmarAyo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OmarAyo/subscriptions","organizations_url":"https://api.github.com/users/OmarAyo/orgs","repos_url":"https://api.github.com/users/OmarAyo/repos","events_url":"https://api.github.com/users/OmarAyo/events{/privacy}","received_events_url":"https://api.github.com/users/OmarAyo/received_events","type":"User","site_admin":false},"assignees":[{"login":"OmarAyo","id":32968582,"node_id":"MDQ6VXNlcjMyOTY4NTgy","avatar_url":"https://avatars2.githubusercontent.com/u/32968582?v=4","gravatar_id":"","url":"https://api.github.com/users/OmarAyo","html_url":"https://github.com/OmarAyo","followers_url":"https://api.github.com/users/OmarAyo/followers","following_url":"https://api.github.com/users/OmarAyo/following{/other_user}","gists_url":"https://api.github.com/users/OmarAyo/gists{/gist_id}","starred_url":"https://api.github.com/users/OmarAyo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OmarAyo/subscriptions","organizations_url":"https://api.github.com/users/OmarAyo/orgs","repos_url":"https://api.github.com/users/OmarAyo/repos","events_url":"https://api.github.com/users/OmarAyo/events{/privacy}","received_events_url":"https://api.github.com/users/OmarAyo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":31,"created_at":"2018-05-17T06:53:25Z","updated_at":"2018-08-10T07:52:44Z","closed_at":null,"author_association":"NONE","body":"## my environment running ArangoDB\r\nI'm using the latest ArangoDB of the respective release series:\r\n- [x] 3.3.8\r\n\r\nMode:\r\n- [x] Cluster\r\n\r\nStorage-Engine:\r\n- [x] mmfiles  - Found out from Logs\r\n\r\nOn this operating system:\r\n- [x] Linux \r\n  - [x] CentOS 7 .rpm\r\n\r\n## Using Hypervisor\r\nI'm using **VMware Workstation** to create 3 nodes:\r\n\r\n192.168.136.130\r\n192.168.136.131\r\n192.168.136.132\r\n\r\n## OS Config\r\n\r\n```\r\nfirewall-cmd --zone=public --add-port=8528/tcp --permanent\r\nfirewall-cmd --zone=public --add-port=8529/tcp --permanent\r\nfirewall-cmd --zone=public --add-port=8530/tcp --permanent\r\nfirewall-cmd --zone=public --add-port=8531/tcp --permanent\r\nfirewall-cmd --reload\r\n```\r\n\r\n```\r\n[root@localhost ~]# df -Th\r\nFilesystem          Type      Size  Used Avail Use% Mounted on\r\n/dev/mapper/cl-root xfs        17G  4.9G   13G  29% /\r\ndevtmpfs            devtmpfs  478M     0  478M   0% /dev\r\ntmpfs               tmpfs     489M     0  489M   0% /dev/shm\r\ntmpfs               tmpfs     489M  6.7M  482M   2% /run\r\ntmpfs               tmpfs     489M     0  489M   0% /sys/fs/cgroup\r\n/dev/sda1           xfs      1014M  139M  876M  14% /boot\r\ntmpfs               tmpfs      98M     0   98M   0% /run/user/0\r\n```\r\n\r\n```\r\n[root@localhost ~]# ip addr\r\n1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1\r\n    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r\n    inet 127.0.0.1/8 scope host lo\r\n       valid_lft forever preferred_lft forever\r\n    inet6 ::1/128 scope host\r\n       valid_lft forever preferred_lft forever\r\n2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000\r\n    link/ether 00:0c:29:da:f8:e7 brd ff:ff:ff:ff:ff:ff\r\n    inet 192.168.136.130/24 brd 192.168.136.255 scope global ens33\r\n       valid_lft forever preferred_lft forever\r\n    inet6 fe80::9cbe:9eaa:5a81:2c83/64 scope link\r\n       valid_lft forever preferred_lft forever\r\n```\r\n\r\n```\r\n\r\n[root@localhost ~]# cat /etc/selinux/config\r\n\r\n# This file controls the state of SELinux on the system.\r\n# SELINUX= can take one of these three values:\r\n#     enforcing - SELinux security policy is enforced.\r\n#     permissive - SELinux prints warnings instead of enforcing.\r\n#     disabled - No SELinux policy is loaded.\r\nSELINUX=disabled\r\n# SELINUXTYPE= can take one of three two values:\r\n#     targeted - Targeted processes are protected,\r\n#     minimum - Modification of targeted policy. Only selected processes are protected.\r\n#     mls - Multi Level Security protection.\r\nSELINUXTYPE=targeted\r\n\r\n```\r\n\r\n## Issue\r\n\r\nfirst I ran the **arangodb command**, the **cluster was up and runnnig**,\r\nthen we encountered a **power failure**,\r\n**after restart** I saw that arangodb was listening to port 8529,\r\nso I **killed** the process and **ran the arangodb command again**,\r\nthis is the output I'm receiving:\r\n\r\n```\r\n[root@localhost ~]# arangodb\r\n2018/05/16 09:54:13 Starting arangodb version 0.11.2, build 4d6e075\r\n2018/05/16 09:54:13 Relaunching service with id 'ed10f41b' on :8528...\r\n2018/05/16 09:54:13 ArangoDB Starter listening on 0.0.0.0:8528 (:8528)\r\n2018/05/16 09:54:13 Looking for a running instance of agent on port 8531\r\n2018/05/16 09:54:13 Starting agent on port 8531\r\n2018/05/16 09:54:14 Looking for a running instance of dbserver on port 8530\r\n2018/05/16 09:54:14 Starting dbserver on port 8530\r\n2018/05/16 09:54:15 Looking for a running instance of coordinator on port 8529\r\n2018/05/16 09:54:15 Starting coordinator on port 8529\r\n2018/05/16 09:54:19 agent up and running (version 3.3.8).\r\n2018/05/16 09:55:23 Failed to get master URL, retrying in 5sec (context deadline exceeded)\r\n2018/05/16 09:55:38 Failed to get master URL, retrying in 5sec (context deadline exceeded)\r\n2018/05/16 09:55:53 Failed to get master URL, retrying in 5sec (context deadline exceeded)\r\n2018/05/16 09:56:08 Failed to get master URL, retrying in 5sec (context deadline exceeded)\r\n2018/05/16 09:56:23 Failed to get master URL, retrying in 5sec (context deadline exceeded)\r\n2018/05/16 09:56:38 Failed to get master URL, retrying in 5sec (context deadline exceeded)\r\n2018/05/16 09:56:46 dbserver not ready after 5min!: Status trail: []int{}\r\n2018/05/16 09:56:47 coordinator not ready after 5min!: Status trail: []int{}\r\n```\r\n\r\n**then I specified the ip address explicitly,** \r\n\r\n`arangodb --starter.address=192.168.136.130`\r\n\r\nbut got the same output\r\n\r\nI noticed a time difference between servers,\r\nso first I synced the timezone\r\nthen installed and started the NTP service\r\nstill no luck\r\n\r\nI also changed the ip address from\r\n`tcp://192.168.136.130:port`\r\nfor agent and dbserver and coordinator, but no difference\r\n\r\n## Logs\r\n\r\n```\r\ncoordinator8529/arangod.log\r\n\r\n2018-05-16T16:55:16Z [4208] INFO {agencycomm} Flaky agency communication to http+tcp://192.168.136.130:8531. Unsuccessful consecutive tries: 46 (76.64s). Network checks advised.\r\n\r\n```\r\n```\r\ndbserver8530/arangod.log\r\n\r\n2018-05-16T16:55:16Z [4209] INFO {agencycomm} Flaky agency communication to http+tcp://192.168.136.130:8531. Unsuccessful consecutive tries: 46 (76.64s). Network checks advised.\r\n```\r\n\r\n```\r\nagent8531/arangod.log\r\n\r\n2018-05-16T16:55:16Z [4832] WARNING {agency} Got bad callback from AppendEntriesRPC: comm_status(8), last(0), follower(AGNT-c7e9ca21-3bd8-44ec-b4e0-81f6c7afd589), time(0.00341606)\r\n2018-05-16T16:55:16Z [4832] WARNING {agency} Last confirmation from peer AGNT-c7e9ca21-3bd8-44ec-b4e0-81f6c7afd589 was received more than minPing ago: 4.06152\r\n```\r\n\r\n```\r\n[root@localhost ~]# lsof -i :8528\r\nCOMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME\r\narangodb 15234 root    5u  IPv6 1048961      0t0  TCP *:8528 (LISTEN)\r\n[root@localhost ~]# lsof -i :8529\r\n[root@localhost ~]# lsof -i :8530\r\n[root@localhost ~]# lsof -i :8531\r\n```\r\n\r\n**after a couple of minutes**\r\n\r\n```\r\n[root@localhost ~]# lsof -i :8528\r\nCOMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME\r\narangodb 15234 root    5u  IPv6 1048961      0t0  TCP *:8528 (LISTEN)\r\n[root@localhost ~]# lsof -i :8529\r\n[root@localhost ~]# lsof -i :8530\r\n[root@localhost ~]# lsof -i :8531\r\nCOMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME\r\narangodb 15234 root    7u  IPv4 1049997      0t0  TCP localhost.localdomain:43790->localhost.localdomain:8531 (ESTABLISHED)\r\narangodb 15234 root    8u  IPv4 1049964      0t0  TCP localhost.localdomain:43752->localhost.localdomain:8531 (ESTABLISHED)\r\narangodb 15234 root    9u  IPv4 1050080      0t0  TCP localhost.localdomain:43928->localhost.localdomain:8531 (ESTABLISHED)\r\narangodb 15234 root   10u  IPv4 1050282      0t0  TCP localhost.localdomain:44260->localhost.localdomain:8531 (ESTABLISHED)\r\narangodb 15234 root   11u  IPv4 1050438      0t0  TCP localhost.localdomain:44530->localhost.localdomain:8531 (ESTABLISHED)\r\narangodb 15234 root   12u  IPv4 1050639      0t0  TCP localhost.localdomain:44786->localhost.localdomain:8531 (ESTABLISHED)\r\narangodb 15234 root   13u  IPv4 1050807      0t0  TCP localhost.localdomain:45026->localhost.localdomain:8531 (ESTABLISHED)\r\narangod  15240 root   35u  IPv6 1049963      0t0  TCP *:8531 (LISTEN)\r\narangod  15240 root   36u  IPv6 1050082      0t0  TCP localhost.localdomain:8531->localhost.localdomain:43928 (ESTABLISHED)\r\narangod  15240 root   37u  IPv6 1049967      0t0  TCP localhost.localdomain:8531->localhost.localdomain:43752 (ESTABLISHED)\r\narangod  15240 root   38u  IPv6 1049999      0t0  TCP localhost.localdomain:8531->localhost.localdomain:43790 (ESTABLISHED)\r\narangod  15240 root   39u  IPv6 1050284      0t0  TCP localhost.localdomain:8531->localhost.localdomain:44260 (ESTABLISHED)\r\narangod  15240 root   40u  IPv6 1050440      0t0  TCP localhost.localdomain:8531->localhost.localdomain:44530 (ESTABLISHED)\r\narangod  15240 root   41u  IPv6 1050641      0t0  TCP localhost.localdomain:8531->localhost.localdomain:44786 (ESTABLISHED)\r\narangod  15240 root   42u  IPv6 1050809      0t0  TCP localhost.localdomain:8531->localhost.localdomain:45026 (ESTABLISHED)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/comments/412005628","html_url":"https://github.com/arangodb/arangodb/issues/5373#issuecomment-412005628","issue_url":"https://api.github.com/repos/arangodb/arangodb/issues/5373","id":412005628,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjAwNTYyOA==","user":{"login":"Napsty","id":3702203,"node_id":"MDQ6VXNlcjM3MDIyMDM=","avatar_url":"https://avatars3.githubusercontent.com/u/3702203?v=4","gravatar_id":"","url":"https://api.github.com/users/Napsty","html_url":"https://github.com/Napsty","followers_url":"https://api.github.com/users/Napsty/followers","following_url":"https://api.github.com/users/Napsty/following{/other_user}","gists_url":"https://api.github.com/users/Napsty/gists{/gist_id}","starred_url":"https://api.github.com/users/Napsty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Napsty/subscriptions","organizations_url":"https://api.github.com/users/Napsty/orgs","repos_url":"https://api.github.com/users/Napsty/repos","events_url":"https://api.github.com/users/Napsty/events{/privacy}","received_events_url":"https://api.github.com/users/Napsty/received_events","type":"User","site_admin":false},"created_at":"2018-08-10T07:52:44Z","updated_at":"2018-08-10T07:52:44Z","author_association":"NONE","body":"> Was your standalone instance using mmFiles or RocksDB as storage-engine?\r\nIt was using RocksDB\r\n\r\n> all your Starter command used for deploying the Cluster\r\n\r\n```\r\nroot@arango1:~# arangodb --starter.address=192.168.15.161 --starter.data-dir=/var/lib/arangodb3 --server.storage-engine=rocksdb --log.verbose\r\n\r\nroot@arango2:~# arangodb --starter.join=192.168.15.161 --starter.data-dir=/var/lib/arangodb3 --server.storage-engine=rocksdb --log.verbose\r\n\r\nroot@arango3:~# arangodb --starter.join=192.168.15.161 --starter.data-dir=/var/lib/arangodb3 --server.storage-engine=rocksdb --log.verbose\r\n```\r\n\r\n> was the Standalone instance stopped before trying to deploy the Cluster\r\nYes\r\n\r\n> deploy your Cluster on an empty directory\r\nSo does that mean, that it's not possible to create an ArangoDB cluster out of a standalone ArangoDB server? When I use an empty directory as data dir, the existing databases and collections from the standalone ArangoDB are gone of course. A clarification on this topic would help a lot.\r\n"}}
{"actor":{"display_login":"Napsty"}}