title: How to scroll down with Phantomjs to load dynamic content
body: <p>I am trying to scrape links from a page that generates content dynamically as the user scroll down to the bottom (infinite scrolling). I have tried doing different things with Phantomjs but not able to gather links beyond first page. Let say the element at the bottom which loads content has class <code>.has-more-items</code>. It is available until final content is loaded while scrolling and then becomes unavailable in DOM (display:none). Here are the things I have tried-</p>
  
  <ul>
  <li>Setting viewportSize to a large height right after <code>var page = require('webpage').create();</code></li>
  </ul>
  
  <blockquote>
    <p>page.viewportSize = {             width: 1600,            height: 10000,
            };</p>
  </blockquote>
  
  <ul>
  <li>Using <code>page.scrollPosition = { top: 10000, left: 0 }</code> inside <code>page.open</code> but have no effect like-</li>
  </ul>
  
  <blockquote>
  <pre><code>page.open('http://example.com/?q=houston', function(status) {
     if (status == "success") {
        page.scrollPosition = { top: 10000, left: 0 };  
     }
  });
  </code></pre>
  </blockquote>
  
  <ul>
  <li>Also tried putting it inside <code>page.evaluate</code> function but that gives </li>
  </ul>
  
  <blockquote>
    <p>Reference error: Can't find variable page</p>
  </blockquote>
  
  <ul>
  <li>Tried using jQuery and JS code inside <code>page.evaluate</code> and <code>page.open</code> but to no avail-</li>
  </ul>
  
  <blockquote>
    <p>$("html, body").animate({ scrollTop: $(document).height() }, 10,
    function() {
            //console.log('check for execution');
        });</p>
  </blockquote>
  
  <p>as it is and also inside <code>document.ready</code>. Similarly for JS code-</p>
  
  <pre><code>window.scrollBy(0,10000)
  </code></pre>
  
  <p>as it is and also inside <code>window.onload</code></p>
  
  <p>I am really struck on it for 2 days now and not able to find a way. Any help or hint would be appreciated.</p>
  
  <p><strong>Update</strong></p>
  
  <p>I have found a helpful piece of code at <a href="https://groups.google.com/forum/?fromgroups=#!topic/phantomjs/8LrWRW8ZrA0" rel="noreferrer">https://groups.google.com/forum/?fromgroups=#!topic/phantomjs/8LrWRW8ZrA0</a></p>
  
  <pre><code>var hitRockBottom = false; while (!hitRockBottom) {
      // Scroll the page (not sure if this is the best way to do so...)
      page.scrollPosition = { top: page.scrollPosition + 1000, left: 0 };
  
      // Check if we've hit the bottom
      hitRockBottom = page.evaluate(function() {
          return document.querySelector(".has-more-items") === null;
      }); }
  </code></pre>
  
  <p>Where <code>.has-more-items</code> is the element class I want to access which is available at the bottom of the page initially and as we scroll down, it moves further down until all data is loaded and then becomes unavailable.</p>
  
  <p>However, when I tested it is clear that it is running into infinite loops without scrolling down (I render pictures to check). I have tried to replace <code>page.scrollPosition = { top: page.scrollPosition + 1000, left: 0 };</code> with codes from below as well (one at a time)</p>
  
  <pre><code>window.document.body.scrollTop = '1000';
  location.href = ".has-more-items";
  page.scrollPosition = { top: page.scrollPosition + 1000, left: 0 };
  document.location.href=".has-more-items";
  </code></pre>
  
  <p>But nothing seems to work.</p>