title: How to implement breakpoints in Lua using the standard hooks?
body: <p>This question is motivated by Exercise 25.7 on p. 264 of <em>Programming in Lua</em> (4th ed.), and more specifically, the optimization proposed in the hint (I've emphasized it in the quote below):</p>
  
  <blockquote>
    <p><strong>Exercise 25.7:</strong> Write a library for breakpoints.  It should offer at least two functions</p>
    
    <p><code>setbreakpoint(function, line)      --&gt; returns handle</code></p>
    
    <p><code>removebreakpoint(handle)</code></p>
    
    <p>We specify a breakpoint by a function and a line inside that function.  When the program hits a breakpoint, the library should call <code>debug.debug</code>.  (Hint: for a basic implementation, use a line hook that checks whether it is in a breakpoint; to improve performance, <strong>use a call hook to trace program execution and only turn on the line hook when the program is running the target function</strong>.)</p>
  </blockquote>
  
  <p>I can't figure out how to implement the optimization described in the hint.</p>
  
  <p>Consider the following code (this is, of course, an artificial example concocted only for the sake of this question):</p>
  
  <pre><code> function tweedledum ()
     while true do
       local ticket = math.random(1000)
       if ticket % 5  == 0 then tweedledee() end
       if ticket % 17 == 0 then break end
     end
   end
  
   function tweedledee ()
     while true do
       local ticket = math.random(1000)
       if ticket % 5  == 0 then tweedledum() end
       if ticket % 17 == 0 then break end
     end
   end
  
   function main ()
     tweedledum()
   end
  </code></pre>
  
  <p>Function <code>main</code> is supposed to represent the program's entrypoint.  Functions <code>tweedledum</code> and <code>tweedledee</code> are almost identical to each other, and do little more than invoke each other repeatedly.</p>
  
  <p>Suppose I set a breakpoint on <code>tweedledum</code>'s assignment line.  I can implement a call hook can check whether <code>tweedledum</code> has been invoked, and then sets a line hook that will check when the desired line is being invoked<sup>1</sup>.</p>
  
  <p>More likely than not, <code>tweedledum</code> will invoke <code>tweedledee</code> before it breaks out of its loop.  Suppose that this happens.  The currently enabled line hook can detect that it is no longer in <code>tweedledum</code>, and re-install the call hook.</p>
  
  <p>At this point the execution can switch from <code>tweedledee</code> to <code>tweedledum</code> in one of two ways:</p>
  
  <ol>
  <li><code>tweedledee</code> can invoke <code>tweedledum</code> (yet again);</li>
  <li><code>tweedledee</code> can <em>return</em> to its invoker, which happens to be <code>tweedledum</code>.</li>
  </ol>
  
  <p><em>And here's the problem:</em> the call hook can detect the event in (1) but it cannot detect the event in (2).</p>
  
  <p>Granted, this example is very artificial, but it's the simplest way I could come up with to illustrate the problem.</p>
  
  <p>The best approach I can think of (and it's very weak!) is to keep track of the stack depth <code>N</code> at the <em>first</em> invokation of <code>tweedledum</code>, have the line hook reinstall the call hook only when the stack depth sinks below <code>N</code>.  Thus, the line hook will be in force as long as <code>tweedledee</code> is in the stack, whether it is being executed or not.</p>
  
  <p>Is it possible to implement the optimization described in the hint using only the standard hooks available in Lua?<sup>2</sup></p>
  
  <hr>
  
  <p><sup><sup>1</sup> My understanding is that, by installing the line hook, the call hook essentially uninstalls itself.  AFAICT, only one hook can be active per coroutine.  <strong><em>Please do correct me if I am wrong.</em></strong></sup></p>
  
  <p><sup><sup>2</sup> Namely: call, line, return, and count hooks.</sup></p>