{"action":"created","issue":{"url":"https://api.github.com/repos/codership/galera/issues/143","repository_url":"https://api.github.com/repos/codership/galera","labels_url":"https://api.github.com/repos/codership/galera/issues/143/labels{/name}","comments_url":"https://api.github.com/repos/codership/galera/issues/143/comments","events_url":"https://api.github.com/repos/codership/galera/issues/143/events","html_url":"https://github.com/codership/galera/issues/143","id":43794170,"number":143,"title":"Log says 'SYNCED' but wsrep_cluster_state = 1 (JOINING)","user":{"login":"freddielunchbird","id":168628,"avatar_url":"https://avatars.githubusercontent.com/u/168628?v=3","gravatar_id":"","url":"https://api.github.com/users/freddielunchbird","html_url":"https://github.com/freddielunchbird","followers_url":"https://api.github.com/users/freddielunchbird/followers","following_url":"https://api.github.com/users/freddielunchbird/following{/other_user}","gists_url":"https://api.github.com/users/freddielunchbird/gists{/gist_id}","starred_url":"https://api.github.com/users/freddielunchbird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freddielunchbird/subscriptions","organizations_url":"https://api.github.com/users/freddielunchbird/orgs","repos_url":"https://api.github.com/users/freddielunchbird/repos","events_url":"https://api.github.com/users/freddielunchbird/events{/privacy}","received_events_url":"https://api.github.com/users/freddielunchbird/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2014-09-24T17:47:58Z","updated_at":"2016-03-22T13:54:58Z","closed_at":null,"body":"Hi,\r\n\r\nI have a two node cluster:\r\nUbuntu 14.04\r\nPXC 5.6.20-68.0   , galera 3.7(r7f44a18)   \r\nxtrabackup 2.2.4\r\n\r\nI have a problem to add a third node (same software as for the other two nodes). The node gets stuck in 'joining', but the error log reports 'SYNCED'.\r\n\r\nJoiner error log:\r\n\r\n140924 17:31:51 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql\r\n140924 17:31:51 mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql\r\n140924 17:31:51 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position\r\n2014-09-24 17:31:51 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Read nil XID from storage engines, skipping position init\r\n2014-09-24 17:31:51 15952 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'\r\n2014-09-24 17:31:51 15952 [Note] WSREP: wsrep_load(): Galera 3.7(r7f44a18) by Codership Oy <info@codership.com> loaded successfully.\r\n2014-09-24 17:31:51 15952 [Note] WSREP: CRC-32C: using \"slicing-by-8\" algorithm.\r\n2014-09-24 17:31:51 15952 [Warning] WSREP: Could not open saved state file for reading: /var/lib/mysql//grastate.dat\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Passing config to GCS: base_host = 10.177.194.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.w\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Service thread queue flushed.\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1\r\n2014-09-24 17:31:51 15952 [Note] WSREP: wsrep_sst_grab()\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Start replication\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1\r\n2014-09-24 17:31:51 15952 [Note] WSREP: protonet asio version 0\r\n2014-09-24 17:31:51 15952 [Note] WSREP: Using CRC-32C for message checksums.\r\n2014-09-24 17:31:51 15952 [Note] WSREP: backend: asio\r\n2014-09-24 17:31:51 15952 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)\r\n2014-09-24 17:31:51 15952 [Note] WSREP: restore pc from disk failed\r\n2014-09-24 17:31:51 15952 [Note] WSREP: GMCast version 0\r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567\r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') multicast: , ttl: 1\r\n2014-09-24 17:31:51 15952 [Note] WSREP: EVS version 0\r\n2014-09-24 17:31:51 15952 [Note] WSREP: PC version 0\r\n2014-09-24 17:31:51 15952 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.177.192.227:4567,10.179.2.66:4567,10.177.194.12:4567'\r\n2014-09-24 17:31:51 15952 [Warning] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' points to own listening address, blacklisting\r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid ab9c8371 is blacklisted, skipping\r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: \r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid ab9c8371 is blacklisted, skipping\r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid ab9c8371 is blacklisted, skipping\r\n2014-09-24 17:31:51 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid ab9c8371 is blacklisted, skipping\r\n2014-09-24 17:31:52 15952 [Note] WSREP: declaring 5fea8f01 at tcp://10.177.192.227:4567 stable\r\n2014-09-24 17:31:52 15952 [Note] WSREP: declaring ce5494a7 at tcp://10.179.2.66:4567 stable\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Node 5fea8f01 state prim\r\n2014-09-24 17:31:52 15952 [Note] WSREP: save pc into disk\r\n2014-09-24 17:31:52 15952 [Note] WSREP: gcomm: connected\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Opened channel 'my_wsrep_cluster'\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Waiting for SST to complete.\r\n2014-09-24 17:31:52 15952 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3\r\n2014-09-24 17:31:52 15952 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.\r\n2014-09-24 17:31:52 15952 [Note] WSREP: STATE EXCHANGE: sent state msg: b0a379a9-4410-11e4-be8e-76386af2d3c6\r\n2014-09-24 17:31:52 15952 [Note] WSREP: STATE EXCHANGE: got state msg: b0a379a9-4410-11e4-be8e-76386af2d3c6 from 0 (server03)\r\n2014-09-24 17:31:52 15952 [Note] WSREP: STATE EXCHANGE: got state msg: b0a379a9-4410-11e4-be8e-76386af2d3c6 from 2 (server06)\r\n2014-09-24 17:31:52 15952 [Note] WSREP: STATE EXCHANGE: got state msg: b0a379a9-4410-11e4-be8e-76386af2d3c6 from 1 (server01)\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Quorum results:\r\n\tversion    = 3,\r\n\tcomponent  = PRIMARY,\r\n\tconf_id    = 159,\r\n\tmembers    = 2/3 (joined/total),\r\n\tact_id     = 102139,\r\n\tlast_appl. = -1,\r\n\tprotocols  = 0/6/3 (gcs/repl/appl),\r\n\tgroup UUID = 01ec4764-3fea-11e4-887f-da10e62749e4\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Flow-control interval: [28, 28]\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 102139)\r\n2014-09-24 17:31:52 15952 [Note] WSREP: State transfer required: \r\n\tGroup state: 01ec4764-3fea-11e4-887f-da10e62749e4:102139\r\n\tLocal state: 00000000-0000-0000-0000-000000000000:-1\r\n2014-09-24 17:31:52 15952 [Note] WSREP: New cluster view: global state: 01ec4764-3fea-11e4-887f-da10e62749e4:102139, view# 160: Primary, number of nodes: 3, my index: 1, protocol version 3\r\n2014-09-24 17:31:52 15952 [Warning] WSREP: Gap in state sequence. Need state transfer.\r\n2014-09-24 17:31:52 15952 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.177.194.12' --auth 'root:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15952'  '' '\r\nWSREP_SST: [INFO] Streaming with xbstream (20140924 17:31:52.963)\r\nWSREP_SST: [INFO] Using socat as streamer (20140924 17:31:52.965)\r\nWSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140924 17:31:53.301)\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.177.194.12:4444/xtrabackup_sst\r\n2014-09-24 17:31:53 15952 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.\r\n2014-09-24 17:31:53 15952 [Note] WSREP: REPL Protocols: 6 (3, 2)\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Service thread queue flushed.\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Assign initial position for certification: 102139, protocol version: 3\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Service thread queue flushed.\r\n2014-09-24 17:31:53 15952 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (01ec4764-3fea-11e4-887f-da10e62749e4): 1 (Operation not permitted)\r\n\t at galera/src/replicator_str.cpp:prepare_for_IST():455. IST will be unavailable.\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Member 1.0 (server01) requested state transfer from '*any*'. Selected 0.0 (server03)(SYNCED) as donor.\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 102139)\r\n2014-09-24 17:31:53 15952 [Note] WSREP: Requesting state transfer: success, donor: 0\r\nWSREP_SST: [INFO] Proceeding with SST (20140924 17:31:54.298)\r\nWSREP_SST: [INFO] Cleaning the existing datadir (20140924 17:31:54.302)\r\nremoved ‘/var/lib/mysql/gvwstate.dat’\r\nWSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140924 17:31:54.319)\r\n2014-09-24 17:31:55 15952 [Note] WSREP: (ab9c8371, 'tcp://0.0.0.0:4567') turning message relay requesting off\r\n2014-09-24 17:33:42 15952 [Note] WSREP: 0.0 (server03): State transfer to 1.0 (server01) complete.\r\n2014-09-24 17:33:42 15952 [Note] WSREP: Member 0.0 (server03) synced with group.\r\nWSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/ (20140924 17:33:42.257)\r\nWSREP_SST: [INFO] Evaluating innobackupex --no-version-check  --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20140924 17:33:42.264)\r\nWSREP_SST: [INFO] Total time on joiner: 0 seconds (20140924 17:33:50.769)\r\nWSREP_SST: [INFO] Removing the sst_in_progress file (20140924 17:33:50.775)\r\n2014-09-24 17:33:50 15952 [Note] WSREP: SST complete, seqno: 102137\r\n2014-09-24 17:33:50 15952 [Note] Plugin 'FEDERATED' is disabled.\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Using atomics to ref count buffer pool pages\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: The InnoDB memory heap is disabled\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Memory barrier is not used\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Compressed tables use zlib 1.2.3.4\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Using Linux native AIO\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Not using CPU crc32 instructions\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Initializing buffer pool, size = 908.0M\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Completed initialization of buffer pool\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18311 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18320 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18312 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18319 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18318 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18317 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18315 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18314 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18316 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Warning] InnoDB: Setting thread 18313 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:50 15952 [Note] InnoDB: Highest supported file format is Barracuda.\r\n2014-09-24 17:33:51 15952 [Note] InnoDB: 128 rollback segment(s) are active.\r\n2014-09-24 17:33:51 15952 [Warning] InnoDB: Setting thread 18325 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:51 15952 [Note] InnoDB: Waiting for purge to start\r\n2014-09-24 17:33:51 15952 [Warning] InnoDB: Setting thread 18326 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:51 15952 [Warning] InnoDB: Setting thread 18327 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:51 15952 [Warning] InnoDB: Setting thread 18328 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:33:51 15952 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 767569480\r\n2014-09-24 17:33:51 15952 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: f306c24e-4410-11e4-a863-bc764e08d443.\r\n2014-09-24 17:33:51 15952 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.\r\n2014-09-24 17:33:51 15952 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.\r\n2014-09-24 17:33:51 15952 [Note] Server hostname (bind-address): '*'; port: 3306\r\n2014-09-24 17:33:51 15952 [Note] IPv6 is available.\r\n2014-09-24 17:33:51 15952 [Note]   - '::' resolves to '::';\r\n2014-09-24 17:33:51 15952 [Note] Server socket created on IP: '::'.\r\n2014-09-24 17:33:51 15952 [Warning] 'proxies_priv' entry '@ root@server04a' ignored in --skip-name-resolve mode.\r\n2014-09-24 17:33:51 15952 [Note] Event Scheduler: Loaded 0 events\r\n2014-09-24 17:33:51 15952 [Note] WSREP: Signalling provider to continue.\r\n2014-09-24 17:33:51 15952 [Note] WSREP: inited wsrep sidno 1\r\n2014-09-24 17:33:51 15952 [Note] WSREP: SST received: 01ec4764-3fea-11e4-887f-da10e62749e4:102137\r\n2014-09-24 17:33:51 15952 [Note] /usr/sbin/mysqld: ready for connections.\r\nVersion: '5.6.20-68.0-56'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona XtraDB Cluster (GPL), Release 25.7, wsrep_25.7.r4126\r\n2014-09-24 17:33:51 15952 [Note] WSREP: 1.0 (server01): State transfer from 0.0 (server03) complete.\r\n2014-09-24 17:33:51 15952 [Note] WSREP: Shifting JOINER -> JOINED (TO: 102139)\r\n2014-09-24 17:33:51 15952 [Note] WSREP: Member 1.0 (server01) synced with group.\r\n2014-09-24 17:33:51 15952 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 102139)\r\n\r\n\r\nmysql> show global status like '%wsrep%';\r\n+------------------------------+---------------------------------------------------------+\r\n| Variable_name                | Value                                                   |\r\n+------------------------------+---------------------------------------------------------+\r\n| wsrep_local_state_uuid       | 01ec4764-3fea-11e4-887f-da10e62749e4                    |\r\n| wsrep_protocol_version       | 6                                                       |\r\n| wsrep_last_committed         | 102137                                                  |\r\n| wsrep_replicated             | 0                                                       |\r\n| wsrep_replicated_bytes       | 0                                                       |\r\n| wsrep_repl_keys              | 0                                                       |\r\n| wsrep_repl_keys_bytes        | 0                                                       |\r\n| wsrep_repl_data_bytes        | 0                                                       |\r\n| wsrep_repl_other_bytes       | 0                                                       |\r\n| wsrep_received               | 3                                                       |\r\n| wsrep_received_bytes         | 298                                                     |\r\n| wsrep_local_commits          | 0                                                       |\r\n| wsrep_local_cert_failures    | 0                                                       |\r\n| wsrep_local_replays          | 0                                                       |\r\n| wsrep_local_send_queue       | 0                                                       |\r\n| wsrep_local_send_queue_max   | 2                                                       |\r\n| wsrep_local_send_queue_min   | 0                                                       |\r\n| wsrep_local_send_queue_avg   | 0.333333                                                |\r\n| wsrep_local_recv_queue       | 0                                                       |\r\n| wsrep_local_recv_queue_max   | 1                                                       |\r\n| wsrep_local_recv_queue_min   | 0                                                       |\r\n| wsrep_local_recv_queue_avg   | 0.000000                                                |\r\n| wsrep_local_cached_downto    | 18446744073709551615                                    |\r\n| wsrep_flow_control_paused_ns | 0                                                       |\r\n| wsrep_flow_control_paused    | 0.000000                                                |\r\n| wsrep_flow_control_sent      | 0                                                       |\r\n| wsrep_flow_control_recv      | 0                                                       |\r\n| wsrep_cert_deps_distance     | 0.000000                                                |\r\n| wsrep_apply_oooe             | 0.000000                                                |\r\n| wsrep_apply_oool             | 0.000000                                                |\r\n| wsrep_apply_window           | 0.000000                                                |\r\n| wsrep_commit_oooe            | 0.000000                                                |\r\n| wsrep_commit_oool            | 0.000000                                                |\r\n| wsrep_commit_window          | 0.000000                                                |\r\n| wsrep_local_state            | 1                                                       |\r\n| wsrep_local_state_comment    | Joining                                                 |\r\n| wsrep_cert_index_size        | 0                                                       |\r\n| wsrep_causal_reads           | 0                                                       |\r\n| wsrep_cert_interval          | 0.000000                                                |\r\n| wsrep_incoming_addresses     | 10.177.192.227:3306,10.177.194.12:3306,10.179.2.66:3306 |\r\n| wsrep_evs_repl_latency       | 0/0/0/0/0                                               |\r\n| wsrep_cluster_conf_id        | 160                                                     |\r\n| wsrep_cluster_size           | 3                                                       |\r\n| wsrep_cluster_state_uuid     | 01ec4764-3fea-11e4-887f-da10e62749e4                    |\r\n| wsrep_cluster_status         | Primary                                                 |\r\n| wsrep_connected              | ON                                                      |\r\n| wsrep_local_bf_aborts        | 0                                                       |\r\n| wsrep_local_index            | 1                                                       |\r\n| wsrep_provider_name          | Galera                                                  |\r\n| wsrep_provider_vendor        | Codership Oy <info@codership.com>                       |\r\n| wsrep_provider_version       | 3.7(r7f44a18)                                           |\r\n| wsrep_ready                  | OFF                                                     |\r\n+------------------------------+---------------------------------------------------------+\r\n52 rows in set (0.00 sec)\r\n\r\nRunning any query on the \"joining\" node gives \"Unknown command\", so it is not Synced (and wsrep isn't ready).\r\n\r\nBut if i have writes on the Cluster (writing to the node that is not the Donor nor the joiner):\r\n\r\n140924 17:43:51 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql\r\n140924 17:43:51 mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql\r\n140924 17:43:51 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position\r\n2014-09-24 17:43:51 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Read nil XID from storage engines, skipping position init\r\n2014-09-24 17:43:51 27304 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'\r\n2014-09-24 17:43:51 27304 [Note] WSREP: wsrep_load(): Galera 3.7(r7f44a18) by Codership Oy <info@codership.com> loaded successfully.\r\n2014-09-24 17:43:51 27304 [Note] WSREP: CRC-32C: using \"slicing-by-8\" algorithm.\r\n2014-09-24 17:43:51 27304 [Warning] WSREP: Could not open saved state file for reading: /var/lib/mysql//grastate.dat\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Passing config to GCS: base_host = 10.177.194.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.w\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Service thread queue flushed.\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1\r\n2014-09-24 17:43:51 27304 [Note] WSREP: wsrep_sst_grab()\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Start replication\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1\r\n2014-09-24 17:43:51 27304 [Note] WSREP: protonet asio version 0\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Using CRC-32C for message checksums.\r\n2014-09-24 17:43:51 27304 [Note] WSREP: backend: asio\r\n2014-09-24 17:43:51 27304 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)\r\n2014-09-24 17:43:51 27304 [Note] WSREP: restore pc from disk failed\r\n2014-09-24 17:43:51 27304 [Note] WSREP: GMCast version 0\r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567\r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1\r\n2014-09-24 17:43:51 27304 [Note] WSREP: EVS version 0\r\n2014-09-24 17:43:51 27304 [Note] WSREP: PC version 0\r\n2014-09-24 17:43:51 27304 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.177.192.227:4567,10.179.2.66:4567,10.177.194.12:4567'\r\n2014-09-24 17:43:51 27304 [Warning] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' points to own listening address, blacklisting\r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid 58aa13d7 is blacklisted, skipping\r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: \r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid 58aa13d7 is blacklisted, skipping\r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid 58aa13d7 is blacklisted, skipping\r\n2014-09-24 17:43:51 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') address 'tcp://10.177.194.12:4567' pointing to uuid 58aa13d7 is blacklisted, skipping\r\n2014-09-24 17:43:51 27304 [Note] WSREP: declaring 5fea8f01 at tcp://10.177.192.227:4567 stable\r\n2014-09-24 17:43:51 27304 [Note] WSREP: declaring ce5494a7 at tcp://10.179.2.66:4567 stable\r\n2014-09-24 17:43:51 27304 [Note] WSREP: Node 5fea8f01 state prim\r\n2014-09-24 17:43:51 27304 [Note] WSREP: save pc into disk\r\n2014-09-24 17:43:52 27304 [Note] WSREP: gcomm: connected\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Opened channel 'my_wsrep_cluster'\r\n2014-09-24 17:43:52 27304 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Waiting for SST to complete.\r\n2014-09-24 17:43:52 27304 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 58f731ba-4412-11e4-970e-0ff58de80f93\r\n2014-09-24 17:43:52 27304 [Note] WSREP: STATE EXCHANGE: sent state msg: 58f731ba-4412-11e4-970e-0ff58de80f93\r\n2014-09-24 17:43:52 27304 [Note] WSREP: STATE EXCHANGE: got state msg: 58f731ba-4412-11e4-970e-0ff58de80f93 from 0 (server01)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: STATE EXCHANGE: got state msg: 58f731ba-4412-11e4-970e-0ff58de80f93 from 1 (server03)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: STATE EXCHANGE: got state msg: 58f731ba-4412-11e4-970e-0ff58de80f93 from 2 (server06)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Quorum results:\r\n\tversion    = 3,\r\n\tcomponent  = PRIMARY,\r\n\tconf_id    = 161,\r\n\tmembers    = 2/3 (joined/total),\r\n\tact_id     = 104692,\r\n\tlast_appl. = -1,\r\n\tprotocols  = 0/6/3 (gcs/repl/appl),\r\n\tgroup UUID = 01ec4764-3fea-11e4-887f-da10e62749e4\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Flow-control interval: [28, 28]\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 104692)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: State transfer required: \r\n\tGroup state: 01ec4764-3fea-11e4-887f-da10e62749e4:104692\r\n\tLocal state: 00000000-0000-0000-0000-000000000000:-1\r\n2014-09-24 17:43:52 27304 [Note] WSREP: New cluster view: global state: 01ec4764-3fea-11e4-887f-da10e62749e4:104692, view# 162: Primary, number of nodes: 3, my index: 0, protocol version 3\r\n2014-09-24 17:43:52 27304 [Warning] WSREP: Gap in state sequence. Need state transfer.\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.177.194.12' --auth 'root:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '27304'  '' '\r\nWSREP_SST: [INFO] Streaming with xbstream (20140924 17:43:52.285)\r\nWSREP_SST: [INFO] Using socat as streamer (20140924 17:43:52.288)\r\nWSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140924 17:43:52.648)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.177.194.12:4444/xtrabackup_sst\r\n2014-09-24 17:43:52 27304 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.\r\n2014-09-24 17:43:52 27304 [Note] WSREP: REPL Protocols: 6 (3, 2)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Service thread queue flushed.\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Assign initial position for certification: 104692, protocol version: 3\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Service thread queue flushed.\r\n2014-09-24 17:43:52 27304 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (01ec4764-3fea-11e4-887f-da10e62749e4): 1 (Operation not permitted)\r\n\t at galera/src/replicator_str.cpp:prepare_for_IST():455. IST will be unavailable.\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Member 0.0 (server01) requested state transfer from '*any*'. Selected 1.0 (server03)(SYNCED) as donor.\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 104781)\r\n2014-09-24 17:43:52 27304 [Note] WSREP: Requesting state transfer: success, donor: 1\r\nWSREP_SST: [INFO] Proceeding with SST (20140924 17:43:53.812)\r\nWSREP_SST: [INFO] Cleaning the existing datadir (20140924 17:43:53.818)\r\nremoved ‘/var/lib/mysql/gvwstate.dat’\r\nWSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140924 17:43:53.844)\r\n2014-09-24 17:43:55 27304 [Note] WSREP: (58aa13d7, 'tcp://0.0.0.0:4567') turning message relay requesting off\r\n2014-09-24 17:45:43 27304 [Note] WSREP: 1.0 (server03): State transfer to 0.0 (server01) complete.\r\n2014-09-24 17:45:43 27304 [Note] WSREP: Member 1.0 (server03) synced with group.\r\nWSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/ (20140924 17:45:43.511)\r\nWSREP_SST: [INFO] Evaluating innobackupex --no-version-check  --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20140924 17:45:43.520)\r\nWSREP_SST: [INFO] Total time on joiner: 0 seconds (20140924 17:45:55.201)\r\nWSREP_SST: [INFO] Removing the sst_in_progress file (20140924 17:45:55.209)\r\n2014-09-24 17:45:55 27304 [Note] WSREP: SST complete, seqno: 113877\r\n2014-09-24 17:45:55 27304 [Note] Plugin 'FEDERATED' is disabled.\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Using atomics to ref count buffer pool pages\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: The InnoDB memory heap is disabled\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Memory barrier is not used\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Compressed tables use zlib 1.2.3.4\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Using Linux native AIO\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Not using CPU crc32 instructions\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Initializing buffer pool, size = 908.0M\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Completed initialization of buffer pool\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29744 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29746 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29752 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29745 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29753 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29751 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29750 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29748 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29747 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29749 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Highest supported file format is Barracuda.\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: 128 rollback segment(s) are active.\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29759 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Note] InnoDB: Waiting for purge to start\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29760 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29761 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Warning] InnoDB: Setting thread 29762 nice to 0 failed, current nice 10, errno 13\r\n2014-09-24 17:45:55 27304 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 784149207\r\n2014-09-24 17:45:55 27304 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: a2d4c7d5-4412-11e4-a92d-bc764e08d443.\r\n2014-09-24 17:45:55 27304 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.\r\n2014-09-24 17:45:55 27304 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.\r\n2014-09-24 17:45:55 27304 [Note] Server hostname (bind-address): '*'; port: 3306\r\n2014-09-24 17:45:55 27304 [Note] IPv6 is available.\r\n2014-09-24 17:45:55 27304 [Note]   - '::' resolves to '::';\r\n2014-09-24 17:45:55 27304 [Note] Server socket created on IP: '::'.\r\n2014-09-24 17:45:55 27304 [Warning] 'proxies_priv' entry '@ root@server04a' ignored in --skip-name-resolve mode.\r\n2014-09-24 17:45:56 27304 [Note] Event Scheduler: Loaded 0 events\r\n2014-09-24 17:45:56 27304 [Note] WSREP: Signalling provider to continue.\r\n2014-09-24 17:45:56 27304 [Note] WSREP: inited wsrep sidno 1\r\n2014-09-24 17:45:56 27304 [Note] WSREP: SST received: 01ec4764-3fea-11e4-887f-da10e62749e4:113877\r\n2014-09-24 17:45:56 27304 [Note] /usr/sbin/mysqld: ready for connections.\r\nVersion: '5.6.20-68.0-56'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona XtraDB Cluster (GPL), Release 25.7, wsrep_25.7.r4126\r\n2014-09-24 17:45:56 27304 [Note] WSREP: 0.0 (server01): State transfer from 1.0 (server03) complete.\r\n2014-09-24 17:45:56 27304 [Note] WSREP: Shifting JOINER -> JOINED (TO: 115797)\r\n2014-09-24 17:45:57 27304 [Note] WSREP: Member 0.0 (server01) synced with group.\r\n2014-09-24 17:45:57 27304 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 115972)\r\n2014-09-24 17:45:57 27304 [Note] WSREP: Synchronized with group, ready for connections\r\n2014-09-24 17:45:57 27304 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.\r\n\r\n\r\nmysql> show global status like '%wsrep%';\r\n+------------------------------+---------------------------------------------------------+\r\n| Variable_name                | Value                                                   |\r\n+------------------------------+---------------------------------------------------------+\r\n| wsrep_local_state_uuid       | 01ec4764-3fea-11e4-887f-da10e62749e4                    |\r\n| wsrep_protocol_version       | 6                                                       |\r\n| wsrep_last_committed         | 117348                                                  |\r\n| wsrep_replicated             | 14                                                      |\r\n| wsrep_replicated_bytes       | 3814                                                    |\r\n| wsrep_repl_keys              | 14                                                      |\r\n| wsrep_repl_keys_bytes        | 434                                                     |\r\n| wsrep_repl_data_bytes        | 2484                                                    |\r\n| wsrep_repl_other_bytes       | 0                                                       |\r\n| wsrep_received               | 12674                                                   |\r\n| wsrep_received_bytes         | 16708102                                                |\r\n| wsrep_local_commits          | 0                                                       |\r\n| wsrep_local_cert_failures    | 0                                                       |\r\n| wsrep_local_replays          | 0                                                       |\r\n| wsrep_local_send_queue       | 0                                                       |\r\n| wsrep_local_send_queue_max   | 2                                                       |\r\n| wsrep_local_send_queue_min   | 0                                                       |\r\n| wsrep_local_send_queue_avg   | 0.008696                                                |\r\n| wsrep_local_recv_queue       | 0                                                       |\r\n| wsrep_local_recv_queue_max   | 11118                                                   |\r\n| wsrep_local_recv_queue_min   | 0                                                       |\r\n| wsrep_local_recv_queue_avg   | 4917.729604                                             |\r\n| wsrep_local_cached_downto    | 104693                                                  |\r\n| wsrep_flow_control_paused_ns | 541185                                                  |\r\n| wsrep_flow_control_paused    | 0.000004                                                |\r\n| wsrep_flow_control_sent      | 1                                                       |\r\n| wsrep_flow_control_recv      | 1                                                       |\r\n| wsrep_cert_deps_distance     | 1301.121760                                             |\r\n| wsrep_apply_oooe             | 0.651109                                                |\r\n| wsrep_apply_oool             | 0.005186                                                |\r\n| wsrep_apply_window           | 2.845866                                                |\r\n| wsrep_commit_oooe            | 0.000000                                                |\r\n| wsrep_commit_oool            | 0.000000                                                |\r\n| wsrep_commit_window          | 2.267070                                                |\r\n| wsrep_local_state            | 4                                                       |\r\n| wsrep_local_state_comment    | Synced                                                  |\r\n| wsrep_cert_index_size        | 143                                                     |\r\n| wsrep_causal_reads           | 0                                                       |\r\n| wsrep_cert_interval          | 0.394991                                                |\r\n| wsrep_incoming_addresses     | 10.177.194.12:3306,10.177.192.227:3306,10.179.2.66:3306 |\r\n| wsrep_evs_repl_latency       | 0.000593955/0.00157661/0.00837575/0.00108117/116        |\r\n| wsrep_cluster_conf_id        | 162                                                     |\r\n| wsrep_cluster_size           | 3                                                       |\r\n| wsrep_cluster_state_uuid     | 01ec4764-3fea-11e4-887f-da10e62749e4                    |\r\n| wsrep_cluster_status         | Primary                                                 |\r\n| wsrep_connected              | ON                                                      |\r\n| wsrep_local_bf_aborts        | 0                                                       |\r\n| wsrep_local_index            | 0                                                       |\r\n| wsrep_provider_name          | Galera                                                  |\r\n| wsrep_provider_vendor        | Codership Oy <info@codership.com>                       |\r\n| wsrep_provider_version       | 3.7(r7f44a18)                                           |\r\n| wsrep_ready                  | ON                                                      |\r\n+------------------------------+---------------------------------------------------------+\r\n52 rows in set (0.00 sec)\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/codership/galera/issues/comments/199826382","html_url":"https://github.com/codership/galera/issues/143#issuecomment-199826382","issue_url":"https://api.github.com/repos/codership/galera/issues/143","id":199826382,"user":{"login":"SylvainA77","id":18008540,"avatar_url":"https://avatars.githubusercontent.com/u/18008540?v=3","gravatar_id":"","url":"https://api.github.com/users/SylvainA77","html_url":"https://github.com/SylvainA77","followers_url":"https://api.github.com/users/SylvainA77/followers","following_url":"https://api.github.com/users/SylvainA77/following{/other_user}","gists_url":"https://api.github.com/users/SylvainA77/gists{/gist_id}","starred_url":"https://api.github.com/users/SylvainA77/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SylvainA77/subscriptions","organizations_url":"https://api.github.com/users/SylvainA77/orgs","repos_url":"https://api.github.com/users/SylvainA77/repos","events_url":"https://api.github.com/users/SylvainA77/events{/privacy}","received_events_url":"https://api.github.com/users/SylvainA77/received_events","type":"User","site_admin":false},"created_at":"2016-03-22T13:54:58Z","updated_at":"2016-03-22T13:54:58Z","body":"i am also experiencing the very same problem at a client's using rsync as an sst method. so it does not seem to be xtrabackup related."}}