title: Multi-server file storage and sharing causes website to hang - other websites on same server remain accessible
body: <p>I'm trying to make a system where files are stored on Main Servers, and if a user chooses, these files can be copied over to Backup Servers to prevent data loss at a HDD crash but it's not going as planned. When files are copied from their Main Server to a Backup Server the entire website hangs. No one can visit the website anymore until all transfers are complete.</p>
  
  <p>What's weird is that other websites, running on the same lighttpd server, remain accessible and even phpmyadmin keeps on trucking.</p>
  
  <p>STEP 1:
  -First, a "job" gets created in mysql table "redundfilesjobs" for a file that needs more backup servers than it currently has.
  -Then, an available backup server gets assigned to said job</p>
  
  <p>STEP 2:
  -After a user approves all the jobs that have a server assigned at once, each job gets copied to "redundfiles" with the field "processed" set to 0 after which the "sendfilestoservers" function is immediatly called
  -The sendfilestoservers function is built so it can accept both a pre-defined job for sending, as well as send all files with the field "processed" set to 0
  -In this case, it gets called with a pre-defined job ID that needs to be processed</p>
  
  <p>sendfilestoservers($lastid);</p>
  
  <p>The function:</p>
  
  <pre><code>&lt;?php
      function sendfilestoservers($specificjob){
  
      if($specificjob != ''){ $result = mysql_query("SELECT * FROM redundfiles WHERE processed=0 and id=$specificjob"); }
      else{ $result = mysql_query("SELECT * FROM redundfiles WHERE processed=0"); }
  
      while($row = mysql_fetch_array($result))
      {
          $fileid=$row['fileID'];
          $result2 = mysql_query("SELECT files.name, files.subfolder, mainservers.path, mainservers.ftpaddress FROM files LEFT JOIN mainservers ON files.serverID = mainservers.id WHERE files.id=$fileid");
          while($row2 = mysql_fetch_array($result2))
          {
              $serverid = $row['serverID'];
              $result3 = mysql_query("SELECT * FROM redundservers WHERE id=$serverid");
              while($row3 = mysql_fetch_array($result3))
              {
                  file_get_contents($row3['sendfileaddress'].'?f='.$row2['name']."&amp;fo=$row2[subfolder]&amp;ftpaddress=$row2[ftpaddress]&amp;mspath=$row2[path]");
  
                  $sql = "UPDATE redundfiles SET processed=1 WHERE id='".$row['id']."'";
                  mysql_query($sql) or die(mysql_error());
              }
          }
      }
      }
  ?&gt;
  </code></pre>
  
  <p>//As you can see it calls the assigned backup servers' address (example: <a href="http://user:pass@serverX.XXX.XX/get.php" rel="nofollow">http://user:pass@serverX.XXX.XX/get.php</a> ) for receiving files, and sends with it directives: file name, the folder it's in on the main server, the ftp address for the main server, and also the path on the main server where all files are stored (inside their respective subfolders)
  //The backup server is now going to processed all those GET values, and ends up retrieving the file from the main server over FTP using linux' wget</p>
  
  <p>Config on backup server X</p>
  
  <pre><code>&lt;?php
  $filename = $_GET['f']; //Filename
  $subfolder = $_GET['fo']; //Folder file is stored in
  $mainserverID = $_GET['ms']; //Main server the file resides on
  $mainserverpath = $_GET['mspath']; //Path on the main server where all the files are stored
  $mainserverftpaddress = $_GET['ftpaddress']; //Main server's FTP address or IP, as it may differ from the web address.
  $dlto = "/var/www/serverX.XXX.XXX/"; //This backup server's main path. This needs to be changed on every backup server.
  $dltodir = "files"; //Folder files are stored in (follows $dlto)
  $serverID = $_GET['s']; //Backup-servers ID as it's stored in the main servers database, some other scripts use this.
  
  function execOutput($command) {
  $output = array($command);
  exec($command.' 2&gt;&amp;1', $output);
  return implode("\n", $output);
  }
  ?&gt;
  </code></pre>
  
  <p>"get.php" script on backup server X</p>
  
  <pre><code>&lt;?php
  include("config.php");
  
      if (file_exists($dltodir.'/'.$subfolder.'/'.$filename))
      {
      }
      else
      {
          exec('mkdir '.$dlto.$dltodir.'/'.$subfolder);
          exec('chmod 777 '.$dlto.$dltodir.'/'.$subfolder);
          exec('wget ftp://user:pass@'.$mainserverftpaddress.'/'.$mainserverpath.'/'.$subfolder.'/'.$filename.' -P '.$dlto.$dltodir.'/'.$subfolder);
  
      }
  ?&gt;
  </code></pre>
  
  <p>//And then we go back to Step 2 for the next approved file.</p>
  
  <p>The first steps (creating the job in "redundfilesjobs" and copying the job to "redundfiles") go fast and I don't notice them causing hang ups. But for some reason the actual sending (or rather, having the backup server retrieve it) just makes the website freeze up.</p>
  
  <p>I hope I provided all the info required for everyone to understand how the system is supposed to work. Can anyone tell me why the entire website hangs and becomes unaccessible for everyone until all files are copied?</p>