title: Puppeteer, looping through links selected by xpath
body: <p>I'm new to puppeteer (and not so great at javascript in general) and am trying to write some basic functionality to:</p>
  
  <ul>
  <li>Get all the links from an XPath</li>
  <li>Loop through and click those links</li>
  <li>Screenshot and save the HTML of the page</li>
  <li>Go back, screenshot and save the HTML of the records page to be save in the same directory of the others and start the process over</li>
  </ul>
  
  <p>The error I get is: </p>
  
  <blockquote>
    <p>Evaluation failed: DOMException: Failed to execute 'querySelector' on 'Document': '0' is not a valid selector</p>
  </blockquote>
  
  <p>Here's the code I have:</p>
  
  <p><a href="https://i.stack.imgur.com/sILpz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sILpz.png" alt="enter image description here"></a></p>
  
  <p>I'm fairly confident all the code works, except for the issues i'm having getting the right things to click on with my XPath. The website I'm getting these from is: </p>
  
  <p><a href="https://hrlb.oregon.gov/bspa/licenseelookup/searchdir.asp?searchby=lastname&amp;searchfor=a&amp;stateselect=none&amp;Submit=Search" rel="nofollow noreferrer">https://hrlb.oregon.gov/bspa/licenseelookup/searchdir.asp?searchby=lastname&amp;searchfor=a&amp;stateselect=none&amp;Submit=Search</a></p>
  
  <p>code:</p>
  
  <pre><code>const records = await page.$x('//table[2]//tr[td[a]]//td[1]/a');
  let int = 0;
  for (let record in records) {
      await Promise.all([
          page.waitForNavigation(),
          page.click(record)
      ]);
  
      await Promise.all([makeDirectory('screenshots/item'+int), makeDirectory('screenshots/item'+int+'/base'), makeDirectory('screenshots/item'+int+'/record')]);
      let recordPath = "screenshots/item"+int+"/record/record.html";
      let basePath = "screenshots/item"+int+"/base/base.html";
  
      page.screenshot({path: "screenshots/item"+int+"/record/record.png", fullPage: true});
      let recordBody = await page.evaluate(() =&gt; document.body.innerHTML);
      await saveHtml(recordPath, recordBody);
  
      await Promise.all([
          page.waitForNavigation(),
          page.goBack()
      ]);
  
      await page.screenshot({path: "screenshots/item"+int+"/base/base.png", fullPage: true});
      let baseBody = await page.evaluate(() =&gt; document.body.innerHTML);
      await saveHtml(basePath, baseBody);
  
      int++;
      console.log(record);
  }
  
  async function makeDirectory(path) {
      mkdirp(path, function(err) {
          if (err) throw err;
      });
  };
  
  async function saveHtml(path, html) {
      await fs.writeFile(path, html, (err) =&gt; {
          if (err) throw err;
      });
  };
  </code></pre>
  
  <p>Note: I'm required to use XPath :(</p>
  
  <p>Updated 6/25/18
  This now gives me all the links from the xpath selector. Which i then iterate over and just use a page.goto, to go to the correct site.</p>
  
  <pre><code>const linksXPath = '//table[2]//tr[td[a]]//td[1]/a';
  const links = await page.evaluate((selector) =&gt; {
      let results = [];
  let query = document.evaluate(selector,
    document,
    null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
    for (let i=0, length=query.snapshotLength; i&lt;length; ++i) {
      results.push(query.snapshotItem(i).href);
    }
      return results;
  }, linksXPath);
  </code></pre>