title: Percona XtraDB Cluster SST not working using rsync: wsrep_sst_rsync
body: <p>I'm sure there is a simple fix for this, but forgive me I'm new at PXC. I'm using rsync to transfer state of the bootstrapping node to node2. node2 is the node I am joining to the cluster. I originally tried Xtrabackup, but ran into problems which I will explore at another time. For now I'm using rsync for what I thought would be simplicity. If you scroll down to the [ERROR] you will see where the problems are causing the State Transfer to be interrupted. What could be causing this?</p>
  
  <pre><code>*WSREP_SST: **[ERROR]** find/rsync returned code 123: (20141228 02:24:40.505)
  2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Failed to read from: **wsrep_sst_rsync** --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket*    
  
  2014-12-28 02:24:24 9446 [Note] WSREP: (2861d1d7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
  2014-12-28 02:24:24 9446 [Note] WSREP: declaring 41f79045 at tcp://192.168.70.2:4567 stable
  2014-12-28 02:24:24 9446 [Note] WSREP: Node 2861d1d7 state prim
  2014-12-28 02:24:24 9446 [Note] WSREP: save pc into disk
  2014-12-28 02:24:24 9446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
  2014-12-28 02:24:24 9446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 42495aba-8e30-11e4-9596-c702faf22ad0
  2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: sent state msg: 42495aba-8e30-11e4-9596-c702faf22ad0
  2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 42495aba-8e30-11e4-9596-c702faf22ad0 from 0 (node3)
  2014-12-28 02:24:24 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 42495aba-8e30-11e4-9596-c702faf22ad0 from 1 (node2)
  2014-12-28 02:24:24 9446 [Note] WSREP: Quorum results:
      version    = 3,
      component  = PRIMARY,
      conf_id    = 1,
      members    = 1/2 (joined/total),
      act_id     = 0,
      last_appl. = 0,
      protocols  = 0/6/3 (gcs/repl/appl),
      group UUID = 48ec9889-8ddc-11e4-9efd-da6610fd24da
  2014-12-28 02:24:24 9446 [Note] WSREP: Flow-control interval: [23, 23]
  2014-12-28 02:24:24 9446 [Note] WSREP: New cluster view: global state: 48ec9889-8ddc-11e4-9efd-da6610fd24da:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
  2014-12-28 02:24:24 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  2014-12-28 02:24:24 9446 [Note] WSREP: REPL Protocols: 6 (3, 2)
  2014-12-28 02:24:24 9446 [Note] WSREP: Service thread queue flushed.
  2014-12-28 02:24:24 9446 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
  2014-12-28 02:24:24 9446 [Note] WSREP: Service thread queue flushed.
  2014-12-28 02:24:25 9446 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node3)(SYNCED) as donor.
  2014-12-28 02:24:25 9446 [Note] WSREP: Shifting SYNCED -&gt; DONOR/DESYNCED (TO: 0)
  2014-12-28 02:24:25 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  2014-12-28 02:24:25 9446 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0''
  2014-12-28 02:24:25 9446 [Note] WSREP: sst_donor_thread signaled with 0
  2014-12-28 02:24:25 9446 [Note] WSREP: Flushing tables for SST...
  2014-12-28 02:24:25 9446 [Note] WSREP: Provider paused at 48ec9889-8ddc-11e4-9efd-da6610fd24da:0 (5)
  2014-12-28 02:24:25 9446 [Note] WSREP: Tables flushed.
  WSREP_SST: [INFO] Preparing binlog files for transfer: (20141228 02:24:26.201)
  mysql-bin.000015
  2014-12-28 02:24:27 9446 [Note] WSREP: (2861d1d7, 'tcp://0.0.0.0:4567') turning message relay requesting off
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest1.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest2.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest3.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest4.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest5.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest6.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest7.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/sbtest/sbtest8.ibd": Permission denied (13)
  rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
  rsync: send_files failed to open "/var/lib/mysql/mysql/innodb_index_stats.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/mysql/innodb_table_stats.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/mysql/slave_master_info.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/mysql/slave_relay_log_info.ibd": Permission denied (13)
  rsync: send_files failed to open "/var/lib/mysql/mysql/slave_worker_info.ibd": Permission denied (13)
  rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
  WSREP_SST: **[ERROR]** find/rsync returned code 123: (20141228 02:24:40.505)
  2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Failed to read from: **wsrep_sst_rsync** --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0'
  2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0': 255 (Unknown error 255)
  2014-12-28 02:24:40 9446 [Note] WSREP: resuming provider at 5
  2014-12-28 02:24:40 9446 [Note] WSREP: Provider resumed.
  2014-12-28 02:24:40 9446 **[ERROR]** WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '192.168.70.2:4444/rsync_sst' --auth 'sst:sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '48ec9889-8ddc-11e4-9efd-da6610fd24da:0'
  2014-12-28 02:24:40 9446 [Warning] WSREP: 0.0 (node3): State transfer to 1.0 (node2) failed: -255 (Unknown error 255)
  2014-12-28 02:24:40 9446 [Note] WSREP: Shifting DONOR/DESYNCED -&gt; JOINED (TO: 0)
  2014-12-28 02:24:40 9446 [Note] WSREP: Member 0.0 (node3) synced with group.
  2014-12-28 02:24:40 9446 [Note] WSREP: Shifting JOINED -&gt; SYNCED (TO: 0)
  2014-12-28 02:24:40 9446 [Note] WSREP: Synchronized with group, ready for connections
  2014-12-28 02:24:40 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  2014-12-28 02:24:41 9446 [Note] WSREP: forgetting 41f79045 (tcp://192.168.70.2:4567)
  2014-12-28 02:24:41 9446 [Note] WSREP: Node 2861d1d7 state prim
  2014-12-28 02:24:41 9446 [Note] WSREP: save pc into disk
  2014-12-28 02:24:41 9446 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
  2014-12-28 02:24:41 9446 [Note] WSREP: forgetting 41f79045 (tcp://192.168.70.2:4567)
  2014-12-28 02:24:41 9446 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2
  2014-12-28 02:24:41 9446 [Note] WSREP: STATE EXCHANGE: sent state msg: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2
  2014-12-28 02:24:41 9446 [Note] WSREP: STATE EXCHANGE: got state msg: 4c2e3544-8e30-11e4-a0dc-d280b597b8d2 from 0 (node3)
  2014-12-28 02:24:41 9446 [Note] WSREP: Quorum results:
      version    = 3,
      component  = PRIMARY,
      conf_id    = 2,
      members    = 1/1 (joined/total),
      act_id     = 0,
      last_appl. = 0,
      protocols  = 0/6/3 (gcs/repl/appl),
      group UUID = 48ec9889-8ddc-11e4-9efd-da6610fd24da
  2014-12-28 02:24:41 9446 [Note] WSREP: Flow-control interval: [16, 16]
  2014-12-28 02:24:41 9446 [Note] WSREP: New cluster view: global state: 48ec9889-8ddc-11e4-9efd-da6610fd24da:0, view# 3: Primary, number of nodes: 1, my index: 0, protocol version 3
  2014-12-28 02:24:41 9446 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  2014-12-28 02:24:41 9446 [Note] WSREP: REPL Protocols: 6 (3, 2)
  2014-12-28 02:24:41 9446 [Note] WSREP: Service thread queue flushed.
  2014-12-28 02:24:41 9446 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
  2014-12-28 02:24:41 9446 [Note] WSREP: Service thread queue flushed.
  2014-12-28 02:24:46 9446 [Note] WSREP:  cleaning up 41f79045 (tcp://192.168.70.2:4567)
  </code></pre>