title: How to copy files as fast as possible?
body: <p>I am running my shell script on <code>machineA</code> which copies the files from <code>machineB</code> and <code>machineC</code> to <code>machineA</code>.</p>
  
  <p>If the file is not there in <code>machineB</code>, then it should be there in <code>machineC</code> for sure. So I will try to copy from <code>machineB</code> first, if it is not there in <code>machineB</code> then I will go to <code>machineC</code> to copy the same files. </p>
  
  <p>In <code>machineB</code> and <code>machineC</code> there will be a folder like this <code>YYYYMMDD</code> inside this folder - </p>
  
  <pre><code>/data/pe_t1_snapshot
  </code></pre>
  
  <p>So whatever date is the latest date in this format <code>YYYYMMDD</code> inside the above folder - I will pick that folder as the full path from where I need to start copying the files - </p>
  
  <p>so suppose if this is the latest date folder <code>20140317</code> inside <code>/data/pe_t1_snapshot</code> then this will be the full path for me - </p>
  
  <pre><code>/data/pe_t1_snapshot/20140317
  </code></pre>
  
  <p>from where I need to start copying the files in <code>machineB</code> and <code>machineC</code>. I need to copy around <code>400</code> files in <code>machineA</code> from <code>machineB</code> and <code>machineC</code> and each file size is <code>1.5 GB</code>.</p>
  
  <p>Currently I have my below shell script which works fine as I am using <code>scp</code> but somehow it takes ~<code>2 hours</code> to copy the <code>400</code> files in machineA which is too long for me I guess. :(</p>
  
  <p>Below is my shell script -</p>
  
  <pre><code>#!/bin/bash
  
  readonly PRIMARY=/export/home/david/dist/primary
  readonly SECONDARY=/export/home/david/dist/secondary
  readonly FILERS_LOCATION=(machineB machineC)
  readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot
  PRIMARY_PARTITION=(0 3 5 7 9) # this will have more file numbers around 200
  SECONDARY_PARTITION=(1 2 4 6 8) # this will have more file numbers around 200
  
  dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
  dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
  
  echo $dir1
  echo $dir2
  
  if [ "$dir1" = "$dir2" ]
  then
      # delete all the files first
      find "$PRIMARY" -mindepth 1 -delete
      for el in "${PRIMARY_PARTITION[@]}"
      do
          scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
      done
  
      # delete all the files first
      find "$SECONDARY" -mindepth 1 -delete
      for sl in "${SECONDARY_PARTITION[@]}"
      do
          scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.
      done
  fi
  </code></pre>
  
  <p>I am copying <code>PRIMARY_PARTITION</code> files in <code>PRIMARY</code> folder and <code>SECONDARY_PARTITION</code> files in <code>SECONDARY</code> folder in <code>machineA</code>.</p>
  
  <p>Is there any way to move the  files faster in <code>machineA</code>. Can I copy 10 files at a time or 5 files at a time in parallel to speed up this process or any other approach?</p>
  
  <p>NOTE: <code>machineA</code> is running on <code>SSD</code></p>
  
  <p><strong>UPDATE:-</strong></p>
  
  <p>Parallel Shell Script which I tried, top portion of shell script is same as shown above.</p>
  
  <pre><code>if [ "$dir1" = "$dir2" ] &amp;&amp; [ "$length1" -gt 0 ] &amp;&amp; [ "$length2" -gt 0 ]
  then
      find "$PRIMARY" -mindepth 1 -delete
      for el in "${PRIMARY_PARTITION[@]}"
      do
          (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.) &amp;
            WAITPID="$WAITPID $!"        
      done
  
      find "$SECONDARY" -mindepth 1 -delete
      for sl in "${SECONDARY_PARTITION[@]}"
      do
          (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.) &amp;
            WAITPID="$WAITPID $!"        
      done
       wait $WAITPID
       echo "All files done copying."
  fi
  </code></pre>
  
  <p><strong>Errors I got with parallel shell script-</strong> </p>
  
  <pre><code>channel 24: open failed: administratively prohibited: open failed
  channel 25: open failed: administratively prohibited: open failed
  channel 26: open failed: administratively prohibited: open failed
  channel 28: open failed: administratively prohibited: open failed
  channel 30: open failed: administratively prohibited: open failed
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  channel 32: open failed: administratively prohibited: open failed
  channel 36: open failed: administratively prohibited: open failed
  channel 37: open failed: administratively prohibited: open failed
  channel 38: open failed: administratively prohibited: open failed
  channel 40: open failed: administratively prohibited: open failed
  channel 46: open failed: administratively prohibited: open failed
  channel 47: open failed: administratively prohibited: open failed
  channel 49: open failed: administratively prohibited: open failed
  channel 52: open failed: administratively prohibited: open failed
  channel 54: open failed: administratively prohibited: open failed
  channel 55: open failed: administratively prohibited: open failed
  channel 56: open failed: administratively prohibited: open failed
  channel 57: open failed: administratively prohibited: open failed
  channel 59: open failed: administratively prohibited: open failed
  mux_client_request_session: session request failed: Session open refused by peer
  channel 61: open failed: administratively prohibited: open failed
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  mux_client_request_session: session request failed: Session open refused by peer
  channel 64: open failed: administratively prohibited: open failed
  mux_client_request_session: session request failed: Session open refused by peer
  channel 68: open failed: administratively prohibited: open failed
  channel 72: open failed: administratively prohibited: open failed
  channel 74: open failed: administratively prohibited: open failed
  channel 76: open failed: administratively prohibited: open failed
  channel 78: open failed: administratively prohibited: open failed
  </code></pre>