title: Kimurai scraping inconsistent returning results
body: <p>I'm attempting to use Kimurai to get product information off of a local grocery website. The problem I have is it will only actually grab the data maybe 1 out of 10 times. I have already built the scraper for the individual product pages and found that unless I used sleep for 5 seconds that the results were inconsistent. I've been playing around with this and even setting it up to 30 seconds (much longer than the page takes to load for me) it still doesn't work consistently and anything over 30 seconds seems to time out the search.</p>
  <p>My goal is to have Kimurai load the search the page (<a href="https://www.loblaws.ca/search?search-bar=%5Bsearchterm%5D" rel="nofollow noreferrer">https://www.loblaws.ca/search?search-bar=[searchterm]</a>) and gather the Name, and price of the products (and eventually the URL of the product page, but I haven't got that far yet)</p>
  <p>Controller Code:</p>
  <pre><code>def search
       searchterm = params[:lookup]
       url = 'https://www.loblaws.ca/search?search-bar=' + searchterm     
      response = FoodsearchSpider.process(url)
      if response[:status] == :completed &amp;&amp; response[:error].nil?
        flash.now[:notice] = &quot;Successfully scraped url&quot;
      else
        flash.now[:alert] = response[:error]
      end
        rescue StandardError =&gt; e
      flash.now[:alert] = &quot;Error: #{e}&quot;
      end    
  </code></pre>
  <p>Model Code: <a href="https://pastebin.com/Aq2G9BFL" rel="nofollow noreferrer">https://pastebin.com/Aq2G9BFL</a></p>
  <pre><code>class FoodsearchSpider &lt; Kimurai::Base 
   @name = 'foodsearch_spider'
    @engine = :selenium_chrome
    @config = {
        user_agent: &quot;Mozilla/5.0 Firefox/61.0&quot;,
    }
   
    def self.process(url)
      @start_urls = [url]
      self.crawl!
    end
   
    def parse(response, url:, data: {})
        sleep 30
        response.xpath(&quot;//li[@class='product-tile-group__list__item']&quot;).each do |result|
          puts &quot;in the loop&quot;
          item = {}
          item[:name]      = result.xpath(&quot;//span[@class='product-name product-name--product-tile']&quot;)&amp;.text&amp;.squish
          item[:price]    = result.xpath(&quot;//*[@id='site-content']/div/div/div/div[2]/div/div[2]/div[2]/div[2]/div/ul/li[1]/div/div/div[3]/div[3]/ul[1]/li[1]/span/span[1]&quot;)&amp;.text&amp;.squish
        puts &quot;**********&quot;
        puts item
        puts &quot;***********&quot;
        end
    end
  end
  
  </code></pre>
  <p>everything consistently works fine up until</p>
  <p><code>response.xpath(&quot;//li[@class='product-tile-group__list__item']&quot;).each do |result|</code></p>
  <p>If it goes into the loop it will work fine but that only happens occasionally.</p>
  <p>When it doesn't go into the loop it still reports that it completed successfully,  but it never enters the loop or sets the array with the values.</p>
  <p>Console log from failure:</p>
  <pre><code>Processing by IngredientsController#search as HTML
    Parameters: {&quot;utf8&quot;=&gt;&quot;✓&quot;, &quot;authenticity_token&quot;=&gt;&quot;7fcXSWF4IAajptTR5cCfLu+cswbWniSbiBvAcCQvVvwaa66Ce2dOfmevx38c7ApXDMBHGaJFQEyhJUg5sjLitg==&quot;, &quot;lookup&quot;=&gt;&quot;peppers&quot;, &quot;commit&quot;=&gt;&quot;Submit&quot;}
    User Load (0.2ms)  SELECT  &quot;users&quot;.* FROM &quot;users&quot; WHERE &quot;users&quot;.&quot;id&quot; = ? ORDER BY &quot;users&quot;.&quot;id&quot; ASC LIMIT ?  [[&quot;id&quot;, 1], [&quot;LIMIT&quot;, 1]]
  searching URL
  ****************
  https://www.loblaws.ca/search?search-bar=peppers
  ****************
  I, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Spider: started: foodsearch_spider
  D, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: BrowserBuilder (selenium_chrome): created browser instance
  D, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: BrowserBuilder (selenium_chrome): enabled custom user_agent
  D, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: BrowserBuilder (selenium_chrome): enabled native headless_mode
  I, [2021-03-20 19:40:18 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Browser: started get request to: https://www.loblaws.ca/search?search-bar=peppers
  2021-03-20 19:40:18 WARN Selenium [DEPRECATION] :driver_path is deprecated. Use :service with an instance of Selenium::WebDriver::Service instead.
  I, [2021-03-20 19:40:20 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Browser: finished get request to: https://www.loblaws.ca/search?search-bar=peppers
  I, [2021-03-20 19:40:20 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Info: visits: requests: 1, responses: 1
  D, [2021-03-20 19:40:22 +0000#27612] [C: 47268811491340] DEBUG -- foodsearch_spider: Browser: driver.current_memory: 255830
  I, [2021-03-20 19:40:52 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Browser: driver selenium_chrome has been destroyed
  I, [2021-03-20 19:40:52 +0000#27612] [C: 47268811491340]  INFO -- foodsearch_spider: Spider: stopped: {:spider_name=&gt;&quot;foodsearch_spider&quot;, :status=&gt;:completed, :error=&gt;nil, :environment=&gt;&quot;development&quot;, :start_time=&gt;2021-03-20 19:40:18 +0000, :stop_time=&gt;2021-03-20 19:40:52 +0000, :running_time=&gt;&quot;34s&quot;, :visits=&gt;{:requests=&gt;1, :responses=&gt;1}, :items=&gt;{:sent=&gt;0, :processed=&gt;0}, :events=&gt;{:requests_errors=&gt;{}, :drop_items_errors=&gt;{}, :custom=&gt;{}}}
    Rendering ingredients/search.html.erb within layouts/application
    Rendered ingredients/search.html.erb within layouts/application (0.9ms)
  Completed 200 OK in 34363ms (Views: 63.6ms | ActiveRecord: 0.2ms)
  </code></pre>
  <p>Occasionally this actually works and it starts running the code inside the loop (the xpaths inside the code aren't set correctly yet) but it's like maybe 10 percent of the time.</p>