title: Reactor lazy pagination with expand
body: <p>Based on <a href="https://stackoverflow.com/questions/53274568/how-to-collect-paginated-api-responses-using-spring-boot-webclient">How to collect paginated API responses using spring boot WebClient?</a></p>
  
  <p>I created the following crawler class</p>
  
  <pre><code>class GitlabCrawler(private val client: WebClient, private val token: String) {
  
      fun fetchCommits(project: URI): Flux&lt;Commit&gt; {
          return fetchCommitsInternal(project).expand { cr: ClientResponse? -&gt;
                  val nextUrl = getNextUrl(cr)
  
                  nextUrl?.let { fetchCommitsInternal(URI.create(it)) }
                          ?: Mono.empty&lt;ClientResponse&gt;()
          }.limitRate(1)
                  .flatMap { cr: ClientResponse? -&gt; cr?.bodyToFlux(Commit::class.java) ?: Flux.empty() }
  
  
      }
  
      private fun getNextUrl(cr: ClientResponse?):String? {
          // TODO replace with proper link parsing
          return cr?.headers()?.header(HttpHeaders.LINK)?.firstOrNull()
                  ?.splitToSequence(",")
                  ?.find { it.endsWith("rel=\"next\"") }
                  ?.let { it.substring(it.indexOf('&lt;') + 1, it.lastIndexOf('&gt;')) }
      }
  
      private fun fetchCommitsInternal(url: URI): Mono&lt;ClientResponse&gt; {
          return client.get()
                  .uri(url)
                  .accept(MediaType.APPLICATION_JSON_UTF8)
                  .header("Private-Token", token)
                  .exchange()
      }
  }
  
  
  data class Commit(
          val id: String,
          val message: String,
          @JsonProperty("parent_ids") val parentIds: List&lt;String&gt;,
          @JsonProperty("created_at") val createdAt: String)
  </code></pre>
  
  <p>I'd like to avoid unnecessary request, but it performs more request than needed to fulfill the request.</p>
  
  <pre><code>gitlabCrawler.fetchCommits(URI.create("https://...")).take(15).collectList().block()
  </code></pre>
  
  <p>Would only need one request as each page contains 20 entries, but it starts the second page request. It seems to always request one more page than necessary. I tried using <code>limitRate</code> but that doesn't seem to have an effect.</p>
  
  <p>Is there a way to make it lazy, i.e., only request the next page when the current is exhausted?</p>