title: Building an infinite scrolling list in typescript with rxjs5
body: <p>I am trying to build an infinitely scrolling list using TypeScript and rxjs.  That is, I want the application to fetch few pages of results from the backend, then fetch more results whenever the user scrolls near the bottom.</p>
  
  <p>I have an Observable, built with <code>Observable.prototype.expand()</code>, which will give me all results, eventually fetching all pages from the server.  However due to the nature of Observable, I cannot pause this process. Once I subscribe it will inevitably fetch all results as fast as possible.  I need a different solution, where I can pull from the stream of results at the speed I need.</p>
  
  <p>Things are made more complicated by the fact, that I cannot fetch the second page from the API directly, every page contains the info I need to fetch the next one. A reply looks like this:</p>
  
  <pre><code>interface GraphApiResponse {
      data?: any[];
      paging?: {
          cursors: {
              before: string,
              after: string
          },
          next?: string,
          previous?: string
      };
  }
  </code></pre>
  
  <p>The presence of the <code>paging.next</code> indicates there is another page and <code>paging.cursors.after</code> is used to actually retrieve it.</p>
  
  <p>I cannot seem to figure out how to implement this, without making it messy.  However, an infinite list seems like such a common problem, that it is unlikely there would not be a good solution for it.  How should I go about implementing this, without turning the thing into a mess?</p>
  
  <h1>Things I have tried</h1>
  
  <h2>Iterable of Promises</h2>
  
  <p>My first thought was using an Iterable of Promises, however I do not know how many results I will get, forcing me to build an infinite <code>Iterable&lt;Promise&lt;Response?&gt;&gt;</code> whose Promises will all resolve to <code>undefined</code> after a certain point.  However since it is infinite I can't iterate over that normally (it would fill the entire available memory with Promises), actually using the results when they are in that form means getting every Promise in the resolve function of the previous one.</p>
  
  <p>This solution seems like it might work, but with every line I write it gets less readable and more complex.</p>
  
  <h2>Merging it in with a behavior subject</h2>
  
  <p>While googling on the issue I found <a href="https://stackoverflow.com/questions/35343183/rxjs-control-observable-invocation">a related SO question</a> as well as <a href="https://github.com/ReactiveX/rxjs/issues/71" rel="noreferrer">a GitHub issue on rxjs backpressure</a>, both contain code snippets by Ben Lesh, that can apparently be used to add backpressure to an Observable, sadly, no matter what I try, I cannot get the source Observable to emit it's values slower than it generates them, they always just get buffered somewhere, which means, that the network requests will happen no matter what.</p>
  
  <p><strong>From GitHub:</strong></p>
  
  <pre><code>// this behavior subject is basically your "give me the next batch" mechanism.
  // in this example, we're going to make 5 async requests back to back before requesting more.
  const BATCH_SIZE = 5;
  const requests = new BehaviorSubject(BATCH_SIZE); // start by requesting five items
  
  // for every request, pump out a stream of events that represent how many you have left to fulfill
  requests.flatMap((count) =&gt; Observable.range(0, count).map(n =&gt; count - n - 1))
    // then concat map that into an observable of what you want to control with backpressure
    // you might have some parameterization here you need to handle, this example is simplified
    // handle side effects with a `do` block
    .concatMap(() =&gt; getSomeObservableOfDataHere().do(stuffWithIt), (remaining) =&gt; remaining)
    // narrow it down to when there are no more left to request,
    // and pump another batch request into the BehaviorSubject
    .filter(remaining =&gt; remaining === 0)
    .mapTo(BATCH_SIZE)
    .subscribe(requests);
  </code></pre>
  
  <p><strong>From StackOverflow:</strong></p>
  
  <pre><code>// start with 5 values
  const controller = new Rx.BehaviorSubject(5);
  
  // some observable source, in this case, an interval.
  const source = Rx.Observable.interval(100)
  
  const controlled = controller.flatMap(
        // map your count into a set of values
        (count) =&gt; source.take(count), 
        // additional mapping for metadata about when the block is done
        (count, value, _, index) =&gt; {
          return { value: value, done: count - index === 1 }; 
        })
        // when the block is done, request 5 more.
        .do(({done}) =&gt; done &amp;&amp; controller.next(5))
        // we only care about the value for output
        .map(({value}) =&gt; value);
  
  
  // start our subscription
  controlled.subscribe(x =&gt; {
    console.log(x)
  });
  </code></pre>
  
  <p>I might be wrong about this, but it seems to me, that once I subscribe to an Observable it produces its values as fast as possible, with no way to slow it down, so this is probably not a solution.</p>
  
  <h2>Using ixjs</h2>
  
  <p>It seems that <a href="https://github.com/Reactive-Extensions/IxJS" rel="noreferrer">ixjs</a> is meant to be a solution to my problem, however that repository has not been updated in a long time.  There apparently is <a href="https://github.com/ReactiveX/IxJS" rel="noreferrer">a reimplementation in TypeScript</a>, however that seems to be early in development and not well documented jet.</p>
  
  <p>I would rather not depend on a framework used by so few people for what is actually a very simple problem.</p>
  
  <h2>Refactoring the app</h2>
  
  <p>I have searched online for implementations of an infinite scrolling list in TypeScript (with Angular).  My current approach is having a Service, that provides an Object, that can be used to get all the results. Then I have a Component showing them.  Alternatives seem to be <a href="http://blog.brecht.io/infinite-scroll-with-rxjs-and-angular2/" rel="noreferrer">doing the checking for scroll position right in the service that queries the backend</a>, or <a href="https://codeburst.io/angular-2-simple-infinite-scroller-directive-with-rxjs-observables-a989b12d4fb1" rel="noreferrer">having the component fetch a new Observable from the backend service when the user scrolls</a>.</p>
  
  <p>Both solutions would force me to mix code, that is currently neatly separated.  I would prefer having the Service return something, that I can just feed into the Component, without the Component having to know about network requests, or the Service having to know about scroll position.</p>