title: RecyclerView Jumps in middle of the list when paginatining
body: <p>I have this page which I've setUp with MVVM and pagination using <code>RecyclerView</code> and Pull to refresh.
  Clicking on some items in this <code>RecyclerView</code> will navigate to another fragment.</p>
  
  <p>My problem is whenever I load the page for the first time and scroll all the way down in works perfectly. Pull to refresh will work as well.</p>
  
  <p>But when I navigate to the other fragment and get back, scroll all the way top, swipe to refresh: The <code>RecyclerView</code> will jump to the middle of the page (Right on the item where I clicked to navigate to the other fragment)</p>
  
  <p><strong>My Fragment</strong></p>
  
  <pre><code>override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
          super.onViewCreated(view, savedInstanceState)
          val adapter = GroupAdapter&lt;ViewHolder&gt;().apply { add(mainSection) }
          val gridLayoutManager = GridLayoutManager(activity, adapter.spanCount)
          gridLayoutManager.spanSizeLookup = adapter.spanSizeLookup
          with(recyclerView) {
              layoutManager = gridLayoutManager
              setHasFixedSize(true)
              addOnScrollListener(PaginationScrollListener {
                  viewModel.onEndOfList()
              })
          }
          recyclerView.adapter = adapter
  
  
          pullToRefresh.apply {
              setProgressBackgroundColorSchemeColor(
                  ContextCompat.getColor(context, R.color.window_level_2)
              )
              setColorSchemeColors(ContextCompat.getColor(context, R.color.brand_primary))
              setOnRefreshListener { viewModel.onRefresh() }
          }
      }
  </code></pre>
  
  <p><strong>My ViewModel</strong></p>
  
  <pre><code>    fun onRefresh() {
          page = 0
          widgetItems.clear()
          _widgetListObservable.postValue(widgetItems)
          finishedLoading = false
          isFirstFetch = true
          getItems()
      }
  
  private fun getItems() {
          isLoading = true
          dataSource.getPage(page)
              .subscribeOn(backgroundThread.getScheduler())
              .observeOn(mainThread.getScheduler())
              .flatMap {
                  Flowable.fromIterable(it)
                      .toList()
                      .toFlowable()
              }
              .doAfterTerminate {
                  isLoading = false
              }
              .subscribe(Consumer {
                  finishedLoading = it.isEmpty()
  
                  if (isFirstFetch &amp;&amp; finishedLoading) {
                      _isMyPaymentsEmptyObservable.postValue(true)
                  }
                  widgetItems.addAll(it)
                  _widgetListObservable.postValue(widgetItems)
                  page++
                  isFirstFetch = false
              }, {
                println(it)
              })
  </code></pre>
  
  <p><strong>EDIT</strong>
  when I remove the <code>onRefreshListener</code> on the <code>swipeToRefresh</code>in the first fragment it works. I have no idea why this happens?</p>
  
  <p>Thanks for reading.</p>