title: Celery/django duplicate key violations
body: <p>I have a single celery worker with 5 threads. It's scraping websites and saving domains to DB via django's ORM.</p>
  
  <p>Here is roughly how it looks like:</p>
  
  <pre><code>domain_all = list(Domain.objects.all())
  needs_domain = set()
  
  for x in dup_free_scrape:
      domain = x['domain']
      if any(domain.lower() == s.name.lower() for s in domain_all):
          x['domainn'] = [o for o in domain_all if domain.lower() == o.name.lower()][0]
      else:
          print('adding: {}'.format(domain))
          needs_domain.add(domain)
  
  create_domains = [Domain(name=b.lower()) for b in needs_domain]
  create_domains_ids = Domain.objects.bulk_create(create_domains)
  </code></pre>
  
  <p>Probably not the best way, but it checks domains in one dict(dup_free_scrape) against all domains already in database.</p>
  
  <p>It can go over hundreds or even thousands before encountering the error, but sometimes it does:</p>
  
  <blockquote>
    <p>Task keywords.domains.model_work[285c3e74-8e47-4925-9ab6-a99540a24665]
    raised unexpected: IntegrityError('duplicate key value violates unique
    constraint "keywords_domain_name_key"\nDETAIL:  Key
    (name)=(domain.com) already exists.\n',)
    django.db.utils.IntegrityError: duplicate key value violates unique
    constraint "keywords_domain_name_key"</p>
  </blockquote>
  
  <p>The only reason for this issue I can think of would be: One thread saved domain to DB while another was in the middle of code above?</p>
  
  <p>I can't find any good solutions, but here is and idea(not sure if any good): Wrap whole thing in transaction and if databaise raises error simplty retry(query database for "Domain.objects.all()" again).</p>