title: Terminating spawn sessions in expect
body: <p>I'm trying to address an issue with an Expect script that logs into a very large number of devices (thousands). The script is about 1500 lines and fairly involved; its job is to audit managed equipment on a network with many thousands of nodes. As a result, it logs into the devices via telnet, runs commands to check on the health of the equipment, logs this information to a file, and then logs out to proceed to the next device.</p>
  
  <p>This is where I'm running into my problem; every <code>expect</code> in my script includes a timeout and an eof like so:</p>
  
  <pre><code>timeout {
      lappend logmsg "$rtrname timed out while &lt;description of expect statement&gt;"
      logmessage
      close
      wait
      set session 0
      continue
  }
  eof {
      lappend logmsg "$rtrname disconnected while &lt;description of expect statement&gt;"
      logmessage
      set session 0
      continue
  }
  </code></pre>
  
  <p>My final <code>expect</code> closes each spawn session manually:</p>
  
  <pre><code>-re "OK.*#" {
      close
      send_user "Closing session... "
      wait
      set session 0
      send_user "closed.\n\n"
      continue
  }
  </code></pre>
  
  <p>The continues bring the script back to the while loop that initiates the next spawn session, assuming session = 0.</p>
  
  <p>The set session 0 tracks when a spawn session closes either manually by the timeout or via EOF before a new spawn session is opened, and everything seems to indicate that the spawn sessions are being closed, yet after a thousand or so spawned sessions, I get the following error:</p>
  
  <pre><code>spawn telnet &lt;IP removed&gt;
  too many programs spawned?  could not create pipe: too many open files
  </code></pre>
  
  <p>Now, I'm a network engineer, not a UNIX admin or professional programmer, so can someone help steer me towards my mistake? Am I closing telnet spawn sessions but not properly closing a channel? I wrote a second, test script, that literally just connects to devices one by one and disconnects immediately after a connection is formed. It doesn't log in or run any commands as my main script does, and it works flawlessly through thousands of connections. That script is below:</p>
  
  <pre><code>#!/usr/bin/expect -f
  
  #SPAWN TELNET LIMIT TEST
  
  set ifile [open iad.list]
  set rtrname ""
  set sessions 0
  
  while {[gets $ifile rtrname] != -1} {
  set timeout 2
  spawn telnet $rtrname
  incr sessions
  send_user "Session# $sessions\n"
  expect  {
      "Connected" {
                  close
                  wait
                  continue
                  }
      timeout     {
                  close
                  wait
                  continue
                  }
      eof         {
                  continue
                  }
  }
  </code></pre>
  
  <p>In my main script I'm logging every single connection and why they may EOF or timeout (via the logmessage process which writes a specific reason to a file), and even when I see nothing but successful spawned connections and closed connections, I get the same problem with my main script but not the test script. </p>
  
  <p>I've been doing some reading on killing process IDs, but as I understand it, <em>close</em> should be killing the process ID of the current spawn session, and <em>wait</em> should be halting the script until the process is dead. I've also tried using a simple "exit" command from the devices to close the telnet connection, but this doesn't produce any better results.</p>
  
  <p>I may simply need a suggestion on how to better track the opening and closing of my sessions and ensure that, between devices, no spawn sessions remain open. Any help that can be offered will be much appreciated.</p>
  
  <p>Thank you!</p>