title: handle a non-persistent eof in a tcl readhandler when non-blocking
body: <p>I am running tcl8.5/tk8.5 application on Windows (issue happens on XP, 7 and 8) with an FTDI D2xx driver in BitBash mode.  Also using ftd2xx c extension to access the FTDI dll.</p>
  
  <p>I have a report from a user the app works fine initially, but after a day of non-use, living in the background, it suddenly grows from 5M to 100M and begins to consume 99% cpu.  (This is not good!)</p>
  
  <p>I have had issues with the USB before this, particularly if the USB is "hot unplugged".  It can cause the app to block and Windows cannot kill it.  My app never needs to read the USB, it just writes to it (controls it), but I discovered that in BitBash mode, the FTDI chip sends a continuous stream of data back. Handling the reads clears the readbuffer so if the USB is unplugged, there isn't a pending read to block and I can exit gracefully.</p>
  
  <p>But now, I think the readhandler is giving me issues.  I have written a small test chunk of code that I mimics the actual app that  I think gets to the root of the issue.  In any case, I don't understand tcl's behavior.</p>
  
  <p>Here is the code:</p>
  
  <pre><code>    package require -exact ftd2xx 1.2.1
  
      # define a read handler
      # gets called by a fileevent readable
      #
      proc readchan {} {
          variable cnt
          set len [gets $::handle buf]
  
          # buffer is non zero
  
          if {$len &gt; 0} {
              incr cnt
              set end [eof $::handle]
              puts "$len ($cnt) eof $end"; 
              # if it wasn't an eof output the buffer
              if {!$end} {
                      puts $buf
              }
          } elseif {$len == 0} {
              # was a zero length read
              puts -nonewline "0"
              set end [eof $::handle]
              if { !$end } {
                      # eof makes the $len invalid?
                      puts "-0"
                      #puts "\nlen is 0 and eof is $end - exit!";
                      #exit
              }
          } else {
              # len was negative (-1) so data in buffer but no end of line (in binary mode)
              if { [eof $::handle] } {
                      puts "EOF w/len 0"
              } else {
                      puts -nonewline "."
              }
          }
  }
  
  #  main code 
  #
  # find the usb device and open it
  set usb [ftd2xx list]
  lassign $usb d
  lassign $d d id e loc f serial
  puts "found USB $serial"
  set handle [ftd2xx open -serial $serial]
  puts "Opened USB $handle"
  #
  # configure it for bitbash mode and for binary, non blocking
  #
  set bitmode 0xFF01
  chan configure $handle -tranlation binary -bitmode $bitmode -blocking 0
  fileevent $handle readable readchan
  
  # output something to the usb every 100ms or so required to cause failure
  #
  while {1} {
      set continue 0
      after 100 {set continue 1}
  
      # putting:
      # "a" made it crash after 245 "timeouts" - no flush
      # null made it past 255 - no flush
      # "aa" made it crash after 164 "timeouts - no flush
  
      puts $handle "aa"
  
      #flushing $handle makes it crash after first flush
      #flush $handle
      puts -nonewline "w"
      vwait continue
  }
  </code></pre>
  
  <p>Here is the output from the above code:</p>
  
  <pre><code>$ tclsh85 usbfailtest.tcl
  found USB AH009L40
  Opened USB ftd2xx0
  w...w....w....w....w....w....w...w....w.126976 (1) eof 1  &lt;&lt;&lt; 126K length with eof true
  ..w....w....w....w...w....w....w....w..126976 (2) eof 1   &lt;&lt;&lt; "w" output for usb write
  .w....w....w...w....w....w....w....w...126976 (3) eof 1   &lt;&lt;&lt; this is 3rd non-zero read
  w...w....w....w....w....w...w....w....w.126976 (4) eof 1  &lt;&lt;&lt; "." output when len -1
  ..w....w...w....w....w....w....w...w...126976 (5) eof 1   &lt;&lt;&lt; line takes about 1 sec 
  w....w....w....w....w...w....w....w....w126976 (6) eof 1
  .
  . (output skipped)
  .
  .w....w....w....w...w....w....w....w...126976 (156) eof 1
  w....w....w...w....w....w....w....w....w126976 (157) eof 1
  ..w....w....w....w....w....w...w....w..126976 (158) eof 1
  .w....w....w...w....w....w....w....w...126976 (159) eof 1
  w....w...w....w....w....w....w....w...w.126976 (160) eof 1
  ..w....w....w....w....w...w....w....w..126976 (161) eof 1
  .w....w....w....w...w....w....w....w...126976 (162) eof 1
  w....w...w....w....w....w....w....w...w.126976 (163) eof 1
  ..w....w....w....w....w...w....w102695 (164) eof 0  &lt;&lt; a 102K buffer with EOF false
  aa  &lt;&lt; all but two of the 102K buffer were null?
  0-0 &lt;&lt; zero length buffer with no eof
  2 (165) eof 0
  aa  &lt;&lt; two length buffer, no eof, and we got two chars
  3 (166) eof 0
  aaa &lt;&lt; three length bufffer, no eof, and we got three chars
  2 (167) eof 0
  aa  &lt;&lt; etc.
  0-0  &lt;&lt; another zero zero  these scroll out *very* fast.
  0-0
  0-0
  0-0
  2 (168) eof 0
  aa  &lt;&lt; an so forth
  0-0
  0-0
  .
  . (output skipped)
  .
  43 (268) eof 0
  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  ..w...22192 (269) eof 1
  w...w....w....w....w....w...w....w....w.126976 (270) eof 1  &lt;&lt; revert to "slow" mode
  ..w....w....w....w...w....w....w....w..126976 (271) eof 1
  .w....w....w...w....w....w....w....w...126976 (272) eof 1
  w....w...w....w....w....w....w....w...w.126976 (273) eof 1
  ..w....w....w....w....w...w....w....w..126976 (274) eof 1
  </code></pre>
  
  <p>An explanation of the output:</p>
  
  <p>To begin with, the read handler is invoked multiple times (each ".") with an incomplete buffer (length -1) and no eof condition. Eventually something either times out or an internal buffer limit is hit and the read is forced to complete. At the same time an EOF raised. Each line of "...126987" takes about a second to write.</p>
  
  <p>After some number of EOF's (ie, lines of ....126987 - in this case 164 - very repeatable), dependent on the amount of data WRITTEN to the channel and whether the channel is flushed a read completed without an EOF (the 164th line).</p>
  
  <p>Up to this point, the read event interrupt rate was tolerable, but then, it skyrockets and consumes lots of cycles handling it.  On  slower machines, there isn't any time left to do anything useful.</p>
  
  <p>I have many questions about this that are difficult to articulate.  But to start with:</p>
  
  <p>I don't understand why the read handler would get called with 0 bytes pending and a non-eof condition.  Doesn't there have to be at least one byte in the buffer to be "readable"?</p>
  
  <p>I don't understand why the EOF is transient.  I would expect that an EOF on a USB port means something like it is unplugged, but it isn't.</p>
  
  <p>If I flush the channel after writing to the USB, ALL I get are 0 length reads, and this persists even if I kill the app and restart it.  The only way to clear this (to the slow mode) is to unplug the device from the USB and start over (without flushes).  I am not sure what to make of this.  I have to flush the channel to get the write to actually go to the FTDI chip.</p>
  
  <p>Should I be using a read with a fixed length instead of a gets for this?</p>