{"action":"created","issue":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6666","repository_url":"https://api.github.com/repos/orientechnologies/orientdb","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6666/labels{/name}","comments_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6666/comments","events_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6666/events","html_url":"https://github.com/orientechnologies/orientdb/issues/6666","id":175476165,"number":6666,"title":"adding 2nd node in cluster: dbase copy hang with \"node not online yet\"","user":{"login":"rdelangh","id":15123823,"avatar_url":"https://avatars.githubusercontent.com/u/15123823?v=3","gravatar_id":"","url":"https://api.github.com/users/rdelangh","html_url":"https://github.com/rdelangh","followers_url":"https://api.github.com/users/rdelangh/followers","following_url":"https://api.github.com/users/rdelangh/following{/other_user}","gists_url":"https://api.github.com/users/rdelangh/gists{/gist_id}","starred_url":"https://api.github.com/users/rdelangh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdelangh/subscriptions","organizations_url":"https://api.github.com/users/rdelangh/orgs","repos_url":"https://api.github.com/users/rdelangh/repos","events_url":"https://api.github.com/users/rdelangh/events{/privacy}","received_events_url":"https://api.github.com/users/rdelangh/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/waiting%20reply","name":"waiting reply","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"lvca","id":312606,"avatar_url":"https://avatars.githubusercontent.com/u/312606?v=3","gravatar_id":"","url":"https://api.github.com/users/lvca","html_url":"https://github.com/lvca","followers_url":"https://api.github.com/users/lvca/followers","following_url":"https://api.github.com/users/lvca/following{/other_user}","gists_url":"https://api.github.com/users/lvca/gists{/gist_id}","starred_url":"https://api.github.com/users/lvca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvca/subscriptions","organizations_url":"https://api.github.com/users/lvca/orgs","repos_url":"https://api.github.com/users/lvca/repos","events_url":"https://api.github.com/users/lvca/events{/privacy}","received_events_url":"https://api.github.com/users/lvca/received_events","type":"User","site_admin":false},"assignees":[{"login":"lvca","id":312606,"avatar_url":"https://avatars.githubusercontent.com/u/312606?v=3","gravatar_id":"","url":"https://api.github.com/users/lvca","html_url":"https://github.com/lvca","followers_url":"https://api.github.com/users/lvca/followers","following_url":"https://api.github.com/users/lvca/following{/other_user}","gists_url":"https://api.github.com/users/lvca/gists{/gist_id}","starred_url":"https://api.github.com/users/lvca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvca/subscriptions","organizations_url":"https://api.github.com/users/lvca/orgs","repos_url":"https://api.github.com/users/lvca/repos","events_url":"https://api.github.com/users/lvca/events{/privacy}","received_events_url":"https://api.github.com/users/lvca/received_events","type":"User","site_admin":false}],"milestone":null,"comments":58,"created_at":"2016-09-07T11:13:01Z","updated_at":"2016-10-18T21:43:25Z","closed_at":null,"body":"## OrientDB Version, operating system, or hardware.\n- v2.2.7\n\n## Operating System\n- [x] Linux\n\nI have my database \"cdrarch\" running on a first server node, which has been started in distributed mode, and is ONLINE:\norientdb {db=cdrarch}> list servers\n\n```\nCONFIGURED SERVERS\n+----+-----------------+------+-----------+-----------------------+-----------------+-----------------+---------------+---------------+---------+\n|#   |Name             |Status|Connections|StartedOn              |Binary           |HTTP             |UsedMemory     |FreeMemory     |MaxMemory|\n+----+-----------------+------+-----------+-----------------------+-----------------+-----------------+---------------+---------------+---------+\n|0   |node1473107743544|ONLINE|1          |2016-09-07 10:27:24.808|10.49.69.190:2424|10.49.69.190:2480|5.51GB (38.74%)|1.71GB (12.05%)|14.22GB  |\n+----+-----------------+------+-----------+-----------------------+-----------------+-----------------+---------------+---------------+---------+\n```\n\nI have copied the \"hazelcast.xml\" file to a second server, and started that server as well.\n\nThe logs on both nodes say that they get a socket connection: extract from the logs on the second node:\n\n```\n2016-09-07 12:18:51:293 WARNI Assigning distributed node name: node1473243531293 [OHazelcastPlugin]\n\n2016-09-07 12:18:52:399 INFO  [10.49.69.189]:2434 [cdrcontrols] [3.6.3] Creating MulticastJoiner [Node]\n2016-09-07 12:18:52:405 INFO  [10.49.69.189]:2434 [cdrcontrols] [3.6.3] Address[10.49.69.189]:2434 is STARTING [LifecycleService]\n...\n2016-09-07 12:18:52:668 INFO  [10.49.69.189]:2434 [cdrcontrols] [3.6.3] Connecting to /10.49.69.190:2434, timeout: 0, bind-any: true [InitConnectionTask]\n2016-09-07 12:18:52:675 INFO  [10.49.69.189]:2434 [cdrcontrols] [3.6.3] Established socket connection between /10.49.69.189:42394 and /10.49.69.190:2434 [TcpIpConnectionManager]\n2016-09-07 12:18:58:692 INFO  [10.49.69.189]:2434 [cdrcontrols] [3.6.3]\n\nMembers [2] {\n        Member [10.49.69.190]:2434\n        Member [10.49.69.189]:2434 this\n}\n```\n\nThen the database 'GratefulDeadConcerts' gets replicated:\n\n```\n2016-09-07 12:19:03:593 INFO  [node1473243531293] Current node started as MASTER for database 'GratefulDeadConcerts' [OHazelcastPlugin]\n2016-09-07 12:19:03:596 INFO  [node1473243531293] New distributed configuration for database: GratefulDeadConcerts (version=1)\n\nCLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)\n+--------+-----------+----------+-----------------+\n|        |           |          |     MASTER      |\n|        |           |          |     ONLINE      |\n+--------+-----------+----------+-----------------+\n|CLUSTER |writeQuorum|readQuorum|node1473107743544|\n+--------+-----------+----------+-----------------+\n|*       |     1     |    1     |        X        |\n|internal|     1     |    1     |                 |\n+--------+-----------+----------+-----------------+\n\n [OHazelcastPlugin]\n2016-09-07 12:19:03:596 INFO  [node1473243531293] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/databases/GratefulDeadConcerts/distribut\ned-config.json [OHazelcastPlugin]\n2016-09-07 12:19:03:600 INFO  [node1473243531293] Distributed servers status:\n\n+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+\n|Name              |Status  |Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory               |\n+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+\n|node1473243531293*|STARTING|                                    |0    |12:18:50 |10.49.69.189:2424|10.49.69.189:2480|75.77MB/491.00MB (15.43%)|\n|node1473107743544 |ONLINE  |GratefulDeadConcerts=ONLINE (MASTER)|3    |10:27:24 |10.49.69.190:2424|10.49.69.190:2480|5.35GB/14.22GB (37.63%)  |\n|                  |        |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                         |\n+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+\n...\n2016-09-07 12:19:05:102 INFO  [node1473243531293] Publishing ONLINE status for database node1473243531293.GratefulDeadConcerts... [ODistributedDatabaseImpl]\n2016-09-07 12:19:05:109 INFO  [node1473243531293] New distributed configuration for database: GratefulDeadConcerts (version=31)\n\nCLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)\n+-------------+----+-----------+----------+-----------------+-----------------+\n|             |    |           |          |     MASTER      |     MASTER      |\n|             |    |           |          |     ONLINE      |     ONLINE      |\n+-------------+----+-----------+----------+-----------------+-----------------+\n|CLUSTER      |  id|writeQuorum|readQuorum|node1473107743544|node1473243531293|\n+-------------+----+-----------+----------+-----------------+-----------------+\n|*            |    |     2     |    1     |        X        |        o        |\n|_studio_1    |  50|     2     |    1     |        o        |        X        |\n|_studio_5    |  54|     2     |    1     |        o        |        X        |\n|_studio_6    |  55|     2     |    1     |        o        |        X        |\n|_studio_7    |  56|     2     |    1     |        o        |        X        |\n|e_4          |  21|     2     |    1     |        o        |        X        |\n|e_5          |  22|     2     |    1     |        o        |        X        |\n...\n|written_by_7 |  40|     2     |    1     |        o        |        X        |\n+-------------+----+-----------+----------+-----------------+-----------------+\n\n [OHazelcastPlugin]\n2016-09-07 12:19:05:109 INFO  [node1473243531293] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/databases/GratefulDeadConcerts/distribut\ned-config.json [OHazelcastPlugin]\n```\n\nSo far so good, it seems.\n\nThen it starts with my big (300GB) database \"cdrarch\", but here we get problems:\n\n```\n2016-09-07 12:19:05:119 INFO  [node1473243531293] Current node started as MASTER for database 'cdrarch' [OHazelcastPlugin]\n2016-09-07 12:19:05:120 INFO  [node1473243531293] New distributed configuration for database: cdrarch (version=1)\n\nCLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)\n+--------+-----------+----------+-----------------+\n|        |           |          |     MASTER      |\n|        |           |          |     ONLINE      |\n+--------+-----------+----------+-----------------+\n|CLUSTER |writeQuorum|readQuorum|node1473107743544|\n+--------+-----------+----------+-----------------+\n|*       |     1     |    1     |        X        |\n|internal|     1     |    1     |                 |\n+--------+-----------+----------+-----------------+\n2016-09-07 12:19:05:121 INFO  [node1473243531293] Saving distributed configuration file for database 'cdrarch' to: /opt/orientdb/databases/cdrarch/distributed-config.json [OHazelcast\nPlugin]\n2016-09-07 12:19:05:124 INFO  [node1473243531293] Distributed servers status:\n\n+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+\n|Name              |Status  |Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory               |\n+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+\n|node1473243531293*|STARTING|                                    |0    |12:18:50 |10.49.69.189:2424|10.49.69.189:2480|75.77MB/491.00MB (15.43%)|\n|node1473107743544 |ONLINE  |GratefulDeadConcerts=ONLINE (MASTER)|3    |10:27:24 |10.49.69.190:2424|10.49.69.190:2480|5.35GB/14.22GB (37.63%)  |\n|                  |        |cdrarch=ONLINE (MASTER)             |     |         |                 |                 |                         |\n|                  |        |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                         |\n+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+\n [OHazelcastPlugin]\n2016-09-07 12:19:05:131 WARNI [node1473243531293]->[[node1473107743544]] Requesting deploy of database 'cdrarch' on local server... [OHazelcastPlugin]\n2016-09-07 12:19:05:444 WARNI [node1473243531293] Moving existent database 'cdrarch' in '/opt/orientdb/databases/cdrarch' to '/opt/orientdb/databases//../backup/databases/cdrarch' an\nd get a fresh copy from a remote node... [OHazelcastPlugin]\n2016-09-07 12:19:05:445 SEVER [node1473243531293] Error on moving existent database 'cdrarch' located in '/opt/orientdb/databases/cdrarch' to '/opt/orientdb/databases/../backup/datab\nases/cdrarch'. Deleting old database... [OHazelcastPlugin]\n2016-09-07 12:19:05:445 INFO  [node1473243531293]<-[node1473107743544] Copying remote database 'cdrarch' to: /tmp/orientdb/install_cdrarch.zip [OHazelcastPlugin]\n2016-09-07 12:19:05:446 INFO  [node1473243531293]<-[node1473107743544] Installing database 'cdrarch' to: /opt/orientdb/databases/cdrarch... [OHazelcastPlugin]\n2016-09-07 12:19:05:447 INFO  [node1473243531293] - writing chunk #1 offset=0 size=0b [OHazelcastPlugin]\n2016-09-07 12:19:05:448 INFO  [node1473243531293] Database copied correctly, size=0b [ODistributedAbstractPlugin$3]\n2016-09-07 12:19:07:476 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]\n2016-09-07 12:19:09:476 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=2, queue=0 worker=0) [ODistributedWorker]\n...\n```\n\n-> what's happening?\n\n"},"comment":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/comments/254648316","html_url":"https://github.com/orientechnologies/orientdb/issues/6666#issuecomment-254648316","issue_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6666","id":254648316,"user":{"login":"rdelangh","id":15123823,"avatar_url":"https://avatars.githubusercontent.com/u/15123823?v=3","gravatar_id":"","url":"https://api.github.com/users/rdelangh","html_url":"https://github.com/rdelangh","followers_url":"https://api.github.com/users/rdelangh/followers","following_url":"https://api.github.com/users/rdelangh/following{/other_user}","gists_url":"https://api.github.com/users/rdelangh/gists{/gist_id}","starred_url":"https://api.github.com/users/rdelangh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdelangh/subscriptions","organizations_url":"https://api.github.com/users/rdelangh/orgs","repos_url":"https://api.github.com/users/rdelangh/repos","events_url":"https://api.github.com/users/rdelangh/events{/privacy}","received_events_url":"https://api.github.com/users/rdelangh/received_events","type":"User","site_admin":false},"created_at":"2016-10-18T21:43:25Z","updated_at":"2016-10-18T21:43:25Z","body":"@lvca \r\n\"Stop one of the server\" ;-) this whole issue is about not being able to launch a 2nd server...\r\nSo the only running server has been stopped, and a backup has been taken, the server has been restarted afterwards. The resulting backup zip file is nearly 20GB size, the original database subdir contains approx 31GB.\r\nCan you please explain what the point was of this action (the backup)? What does this provide you as added information to help troubleshoot the fact that a 2nd node cannot join the 'group' (where 'group' equals 1 ONLINE server) ?"}}
{"actor":{"display_login":"rdelangh"}}