title: How to click 'Next' for pagination using Requests-HTML library
body: <p>I'm new to webscraping and so I'm looking into a bunch of different approaches. One of them (the one I'm most excited about) is to use Python library 'requests-html', which supports rendering of Javascript content.  </p>
  
  <p>Basically I'm wondering how I can click the 'Next' button to get on a given page to get more JS rendered content. Using the example webpage "<a href="https://us-proxy.org/" rel="nofollow noreferrer">https://us-proxy.org/</a>".  I can pretty easily get the xpath to the 'Next' button, but I don't know how to actuate it.</p>
  
  <p>My understanding of 'requests-html' is that it works basically by controlling a headless chromium instance and using it to render pages with Javascript. So far so good, I've done a little experimenting with it so far and it works pretty well, but the documentation mentions that it includes pagination functionality.  Frankly I can't seem to get it to work and aside from saying it's there the creators docs don't elaborate much.</p>
  
  <p>I'd love it if someone could explain how people are accomplishing this with this library or even if they could just point me to some more fleshed out docs if there are any online.  I've spent some time trolling around looking to see if I could find any, but there's surprisingly little that I was able to turn up about 'requests-html' at all given how powerful it seems to be. I checked ReadTheDocs as well and it had basically no information.  I suppose I can go out and to the side of the problem a little and dig into pyppeteer but that seems like potentially a whole other rabbit hole...</p>
  
  <h1>---------------------------- My sample script ----------------------</h1>
  
  <pre><code>targetURL = "https://us-proxy.org/"
  
  print("script running")
  
  # create an HTML Session Object
  session = HTMLSession()
  
  # Use the object to needed webpage
  responseObject =  session.get(targetURL)
  responseObject.html.next()
  
  option_tagsNoRender = responseObject.html.xpath("//td")
  print("\n\nNo Rend: ", len(option_tagsNoRender) )
  print("\n\n", option_tagsNoRender[0].full_text)
  
  
  # Run Javascript Code on target webpage
  responseObject.html.render(sleep=10)
  
  
  option_tags = responseObject.html.xpath("//td")
  print("\n\nPost Render: ", len(option_tags) )
  #print(dir(option_tags[0]))
  
  print("\n\n", option_tags[0].full_text)
  
  for tag in option_tags:
      #pass
      print(tag.full_text)
  </code></pre>
  
  <h1>---------------- End ------------------------------------------</h1>
  
  <p>Since the creators documentation did in fact mention a '.next()' method I tried executing it (as seen in the sample script).  Based on the output it doesn't seem to be doing anything (though I have no idea if I'm implementing it correctly). The results of the tag search come out the same whether I comment out that line or leave it in.  It does NOT appear to be advancing the JS rendered page. </p>
  
  <p>Appreciate any guidance or insights.  Thanks!</p>
  
  <p>Update:</p>
  
  <p>Ok, so I took a closer look at the docs and they do mention you can interact with the page if you use the 'keep_page' option during the render call.  That led me down a discovery path to pyppeteers docs (which is of course employed within requests-html) which seem to reveal a pretty straight forward '.click()' method that didn't seem to choke on the xpath selector of the next button that I fed it.  Unfortunately, I still don't see any evidence that it successfully clicks the link and I keep getting an error saying "RuntimeWarning: coroutine 'clickNext' was never awaited".  </p>
  
  <p>Here is the simple function I wrote to click the next button that is now giving me that message:</p>
  
  <h1>-------------------------------------------------------</h1>
  
  <p>async def clickNext():</p>
  
  <pre><code>await asyncio.wait([ responseObject.html.page.click('//a[@aria-controls][@data-dt-idx="9"]'), 
                       responseObject.html.page.waitFor(5000),])
  </code></pre>
  
  <h1>-------------------------------------------------------</h1>
  
  <p>I have zero experience thus far with async programming and coroutines, so int he mean time I'm digging into that, but if anyone has any insights into what I'm doing wrong with this I'd appreciate any info you cared to share.  Thanks!</p>