title: Web Automation to go to the next page - Invoking method that returns Promise within await block) - await is only valid in async
body: <p><strong>Background:</strong></p>
  
  <p>I am writing a Nodejs script with puppeteer to web scrape data from a web page. I'm not familiar with Nodejs, promises, or puppeteer. I've tried many things and done research for a few days.</p>
  
  <p><strong>Application Flow:</strong></p>
  
  <ol>
  <li>With automation, go to a website</li>
  <li>Scrape data from the page, push to an array</li>
  <li>If there is a "next page" click the next page button</li>
  <li>Scrape data from the page, push to same array</li>
  <li>Repeat</li>
  </ol>
  
  <p><strong>Problem:</strong></p>
  
  <p>My problem is with #3. With web automation, clicking the next page button. 
  All I want, is to use the .click() method in puppeteer, to click on the button selector. However, .click() returns a Promise. Since it's a promise, I need keyword <code>await</code>, but you can't have <code>await</code> in the for loop (or any block other than <code>async</code>).</p>
  
  <p><strong>What Have I Tried:</strong></p>
  
  <p>I've tried creating another <code>async</code> function, with statements for <code>await page.click();</code>and calling that function in the problem area. I've tried creating a regular function with page.click() and calling that in the problem area. Refactoring everything to have it not work as well. I'm not really understanding Promises and Async/Await even after reading about it for a few days.</p>
  
  <p><strong>What I Want Help With:</strong></p>
  
  <p>Help with invoking the .click() method inside the problem area or any help with selecting the 'Next Page' using web automation.</p>
  
  <p><strong>Pseudo Code:</strong></p>
  
  <pre><code>let scrape = async () =&gt; {
      await //do.some.automation;
  
      const result = await page.evaluate(() =&gt; {
          for (looping each page) {
              if (there is a next page) {
                  for (loop through data) {
                      array.push(data);
                      //----PROBLEM----
                      //use automation to click the selector of the next page button
                      //--------------   
                  }
              }
          }
          return data;
      });
      //close browser
      return result;
  };
  scrape().then((value) =&gt; {
      //output data here;
      });
  });
  </code></pre>
  
  <p><strong>All Code:</strong></p>
  
  <pre><code>let scrape = async () =&gt; {
      const browser = await puppeteer.launch({
          headless: false 
      });
      const page = await browser.newPage();
      await page.goto("GO TO A WEBSITE");
      await page.click("CLICK A BUTTON");
      await page.waitFor(2000);
  
      //Scraping
      const result = await page.evaluate(() =&gt; {
          let pages = document.getElementsByClassName("results-paging")[2];
          let allPages = pages.getElementsByClassName("pagerLink");
          let allJobs = [];
          //Loop through each page
          for (var j = 0; j &lt; allPages.length; j++) {
              let eachPage = pages.getElementsByClassName("pagerLink")[j].innerHTML;
              if (eachPage) {
                  //Scrape jobs on single page
                  let listSection = document.getElementsByTagName("ul")[2];
                  let allList = listSection.getElementsByTagName("li");
                  for (var i = 0; i &lt; allList.length; i++) {
                      let eachList = listSection.getElementsByTagName("li")[i].innerText;
                      allJobs.push(eachList);
  
                      //--------PROBLEM-------------
                      await page.click('#selector_of_next_page');
                      //----------------------------  
                  }
              }
              else {
                  window.alert("Fail");
              }
          }
          return allJobs;
      });
      browser.close();
      return result;
  };
  
  scrape().then((value) =&gt; {
      let data = value.join("\r\n");
      console.log(data);
      fs.writeFile("RESULTS.txt", data, function (err) {
          console.log("SUCCESS MESSAGE");
      });
  });
  </code></pre>
  
  <p><strong>Error Message:</strong></p>
  
  <blockquote>
    <p>SyntaxError: await is only valid in async function</p>
  </blockquote>