title: Why is Selenium getting chrome error when running celery tasks?
body: <p>I am getting the following error when Celery tasks try to use Selenium;</p>
  
  <pre><code>[2020-06-12 23:23:04,228: WARNING/ForkPoolWorker-1] Message: unknown error: Chrome failed to start: exited abnormally.
    (chrome not reachable)
    (The process started from chrome location /opt/python/run/venv/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  [2020-06-12 23:23:04,230: ERROR/ForkPoolWorker-1] Task Update All Results[73583ec6-474a-4f20-8d88-d7be0a150642] raised unexpected: NameError("name 'sleep' is not defined",)
  Traceback (most recent call last):
    File "/opt/python/bundle/42/app/fixtures/models.py", line 1035, in get_flash_result
      driver = webdriver.Chrome(options=opts)
    File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
      desired_capabilities=desired_capabilities)
    File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
      self.start_session(capabilities, browser_profile)
    File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
      response = self.execute(Command.NEW_SESSION, parameters)
    File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
      self.error_handler.check_response(response)
    File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
      raise exception_class(message, screen, stacktrace)
  selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.
    (chrome not reachable)
    (The process started from chrome location /opt/python/run/venv/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  </code></pre>
  
  <p>I have a Django app on an EC2 server via Elastic Beanstalk which runs periodic tasks using Celery however when webdriver is set I get the error.</p>
  
  <p>The Celery task calls a class method on my model which contains the selenium code.  The error doesn't occur if I run the same code via Django shell or call the method from the admin dashboard - just when the Celery task is running.</p>
  
  <p>the error occurs after the following code;</p>
  
  <pre class="lang-py prettyprint-override"><code>from selenium import webdriver
  from selenium.common.exceptions import TimeoutException, NoSuchElementException, ElementClickInterceptedException, WebDriverException
  from selenium.webdriver.common.by import By
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.chrome.options import Options
  ...
  opts = Options()
  opts.headless = True
  opts.add_argument("--no-sandbox")
  opts.add_argument("--disable-dev-shm-usage")
  opts.add_argument("--remote-debugging-port=9222")
  assert opts.headless  # Operating in headless mode
  
  driver = webdriver.Chrome(options=opts)
  </code></pre>
  
  <p>My Django app runs on a virtual environment - /opt/python/run/venv - and chromedriver and chrome are available to it;
  /opt/python/run/venv/bin/chromedriver &amp; 
  /opt/python/run/venv/bin/google-chrome --> /opt/google/chrome/google-chrome</p>
  
  <p>versions;</p>
  
  <pre><code>(venv) [ec2-user@ip-172-31-12-230 ~]$ google-chrome --version &amp;&amp; which google-chrome
  Google Chrome 83.0.4103.97 
  /opt/python/run/venv/bin/google-chrome
  (venv) [ec2-user@ip-172-31-12-230 ~]$ chromedriver --version &amp;&amp; which chromedriver
  ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416})
  /opt/python/run/venv/bin/chromedriver
  
  (venv) [ec2-user@ip-172-31-12-230 ~]$ pip list
  Package               Version
  --------------------- ----------
  celery                4.4.2
  Django                3.0.4
  django-celery-beat    2.0.0
  selenium              3.141.0
  
  </code></pre>
  
  <p><strong>EDIT</strong>
  This was linked to another question <a href="https://stackoverflow.com/questions/53073411/selenium-webdriverexceptionchrome-failed-to-start-crashed-as-google-chrome-is">here</a> and closed, however the link doesn't answer my question as I'm still getting the error.  I tried making the following adjustments and validations;</p>
  
  <ol>
  <li>Google Chrome and Chromedriver are up to date</li>
  <li>executable path (/opt/python/run/venv/bin/chromedriver) to chromedriver was added to webdriver()</li>
  <li>.add_argument('--headless') was moved to be the first option set</li>
  <li>google-chrome in venv and /usr/bin are symlinks</li>
  <li>binary location option added</li>
  <li>tried deleting chrome and chromedriver and reinstalling (a few times)</li>
  <li>tried changing options to just '--no-sandbox' and '--disable-dev-shm-usage'</li>
  <li>I've tried putting it in a loop and adding a sleep</li>
  <li>Celery is not using root user or sudo</li>
  </ol>
  
  <p>I <strong>only</strong> encounter this error when Celery worker executes tasks, I can run the same code being run by the worker in the django shell or via the admin dashboard and don't receive the error so it feels like this is a celery error and not my selenium options.</p>