title: Lazy evaluation / Stream / FRP code for node.js fs.readdir Async recursive directory search
body: <p>I try to implement a list structure reading FileSystem Directory tree using node.js</p>
  
  <p>DIR/file structure:</p>
  
  <pre><code>DIR1
       DIR2R
       file1
       file2
       file3
       …
  </code></pre>
  
  <p>-> list structure:    </p>
  
  <pre><code>("DIR1" ("DIR2" "file1" "file2" "file3" …))
  
   DIR()   //looks like a C function
   file    //looks like an Atom value
  </code></pre>
  
  <p>so, for instance, a given UNIX root DIR: </p>
  
  <pre><code>root bin  cd
            ls
  
       lib  file1
            file2
  
       opt
  
       usr  file3
  
       file4
  </code></pre>
  
  <p>is equivalent to a list:</p>
  
  <pre><code>("root" ("bin" ("cd" "ls")
           "lib" ("file1" "file2") 
           "opt" ()
           "usr" ("file3")
           "file4"
          )
  )
  </code></pre>
  
  <hr>
  
  <p>Having said that, I want to implement this code with some <strong>lazy async sequence (stream/ Infinite list?)</strong>.</p>
  
  <p><strong><a href="https://stackoverflow.com/questions/5827612/node-js-fs-readdir-recursive-directory-search">node.js fs.readdir recursive directory search</a>
  is a good reference.</strong></p>
  
  <p>node.js has many excellent libraries such as <a href="https://github.com/mikeal/node-utils/tree/master/file" rel="nofollow noreferrer">file.walk</a> etc. , but this is a very interesting subject, in fact, many code examples exsist.</p>
  
  <p>In other words, a good subject to learn a basic concept of lazy async sequence (stream/ Infinite list?), and the code can be a good library, I want to implement from a scratch.</p>
  
  <hr>
  
  <hr>
  
  <p><strong><a href="https://github.com/substack/stream-handbook" rel="nofollow noreferrer">substack/stream-handbook</a> covers the basics of how to write node.js
  programs with <a href="http://nodejs.org/docs/latest/api/stream.html" rel="nofollow noreferrer">streams</a> .
  Truly Excellent article.</strong></p>
  
  <hr>
  
  <p>so, here is my thought on this subject - Recursive Dir tree processing:</p>
  
  <ul>
  <li><p>This subject deal with a recursive search, and there are 2 ways: <strong>async</strong> and <strong>sync</strong> , for  various reasons, I chose <strong>async</strong> method.</p></li>
  <li><p>The search result is equivalent to a <strong>list structure</strong>, where <strong>Functional programming</strong>  paradigm does matter (Lisp/Scheme).</p></li>
  <li><p>For JavaScript Functional programming, there are libraries such as <strong><a href="http://underscorejs.org/" rel="nofollow noreferrer">underscore.js</a></strong>, and recently, <strong><a href="http://danieltao.com/lazy.js/" rel="nofollow noreferrer">lazy.js</a></strong> with one important difference: <strong>lazy evaluation</strong> (also known as deferred execution) where appears to <strong>outperform underscore.js</strong>.</p></li>
  <li><p>lazy.js wraps Stream processing in Node.js as well. I also found <a href="http://streamjs.org/" rel="nofollow noreferrer">stream.js</a> says: <em>an infinite number of elements. Their power comes from being lazily evaluated, which in simple terms means that they can contain infinite items.</em> .</p></li>
  <li><p>Stream data(infinite list) realizes <strong>Reactive Functional Programming(FRP)</strong> paradigm. lazy.js actually demonstrates a mouse event FRP implementation. <a href="https://github.com/Reactive-Extensions/RxJS" rel="nofollow noreferrer">RxJS</a> is a FRP library: <strong><em>RxJS or Reactive Extensions</strong> for JavaScript is a library for transforming, composing, and querying <strong>streams of data</strong>.</em></p></li>
  <li><p><strong>async</strong> method of this subject <em>is</em> a <strong>FRP</strong> subject.</p></li>
  <li><p>Therefore, <strong>lazy evaluation</strong> is a core factor of faster List processing Functional Programming, and expands list to infinite list(stream) which integrates async/events to Stream data source to process in Functional Programming paradigm (FunctionalReactiveProgramming).</p></li>
  <li><p><strong><a href="https://github.com/pkrumins/node-lazy" rel="nofollow noreferrer">node-lazy</a> Introduction</strong> states clearly: <em>Lazy comes really handy when you need to treat a stream of events like a list. The best use case currently is returning a lazy list from an asynchronous function, and having data pumped into it via events. In asynchronous programming you can't just return a regular list because you don't yet have data for it. The usual solution so far has been to provide a callback that gets called when the data is available. But doing it this way you lose the power of chaining functions and creating pipes, which leads to not that nice interfaces. (See the 2nd example below to see how it improved the interface in one of my modules.)</em></p></li>
  </ul>
  
  <hr>
  
  <p>At last, here is my Question on this subject - Recursive Dir tree processing:</p>
  
  <p><strong>Are there any sample code to implement this subject exclusively in lazy evaluation or FRP manner?</strong></p>
  
  <p>I know lazy.js, stream.js, or linq.js is based on lazy evaluation, but any of them can not <strong>define node.js fs.readdir as a stream data for async processing (FRP )</strong>. According to <a href="http://danieltao.com/lazy.js/docs/AsyncSequence.html" rel="nofollow noreferrer">http://danieltao.com/lazy.js/docs/AsyncSequence.html</a> , it's not yet implement, correct?</p>
  
  <p><a href="https://github.com/Reactive-Extensions/RxJS" rel="nofollow noreferrer">RxJS</a> probably can, but I don't know how. </p>
  
  <p><a href="https://www.google.co.jp/search?q=rx.Observable&amp;oq=rx.Observable&amp;aqs=chrome.0.57j59&amp;sourceid=chrome&amp;ie=UTF-8#gs_rn=17&amp;gs_ri=psy-ab&amp;tok=lt8hdO2Fy31TF6PSUrTROA&amp;pq=rx.observable&amp;cp=24&amp;gs_id=lx&amp;xhr=t&amp;q=rx.Observable%20fs.readdir&amp;es_nrs=true&amp;pf=p&amp;safe=off&amp;sclient=psy-ab&amp;oq=rx.Observable%20fs.readdir&amp;gs_l=&amp;pbx=1&amp;bav=on.2,or.r_cp.r_qf.&amp;bvm=bv.48705608,d.aGc&amp;fp=aa4be614a32a08d0&amp;biw=1155&amp;bih=607" rel="nofollow noreferrer">Google <strong>rx.Observable fs.readdir</strong> hits nothing.</a></p>
  
  <p>Any ideas?</p>