title: Python coroutine that consumes and produces
body: <p>In a web-crawler of mine, I have a class that keeps track of urls to crawl, removing duplicates, etc:</p>
  
  <pre><code>class VisitOnlyOnceClerk(object):
      def __init__(self):
          self.visited = set()
          self.to_visit = set()
  
      def enqueue(self, url, referer):
          if not url in self.visited:
              self.to_visit.add((url, referer))
  
      def __bool__(self):
          return bool(self.to_visit)
  
      def __iter__(self):
          while self.to_visit:
              (url, referer) = self.to_visit.pop()
              self.visited.add(url)
              yield (url, referer)
  </code></pre>
  
  <p>This provides an iterable that can also have values added to it:</p>
  
  <pre><code>clerk = VisitOnlyOnceClerk()
  clerk.enqueue(starting_point, starting_point)
  for (url, referer) in clerk:
      # get the url, and clerk.enqueue() all the links from it ...
  </code></pre>
  
  <p>This seems like a task that could be performed by a coroutine. I have seen examples of coroutines that solely produce values, and examples that solely consume values, but none that do both. Something like:</p>
  
  <pre><code>def visit_once_clerk():
      visited = set()
      to_visit = set([(yield)])
  
      for i in to_visit:
          visited.add(i[1])
          extras = (yield i)
          if extras:
              to_visit.union(i for i in extras if i[1] not in visited)
  </code></pre>
  
  <p>This, of course, does not work like I think it should. Are coroutines even the right tool here? What's the right way to use them in this case?</p>