title: How to efficiently search list elements in a string in python
body: <p>I have a list of concepts (<code>myconcepts</code>) and a list of sentences (<code>sentences</code>) as follows.</p>
  
  <pre><code>concepts = [['natural language processing', 'text mining', 'texts', 'nlp'], ['advanced data mining', 'data mining', 'data'], ['discourse analysis', 'learning analytics', 'mooc']]
  
  
  sentences = ['data mining and text mining', 'nlp is mainly used by discourse analysis community', 'data mining in python is fun', 'mooc data analysis involves texts', 'data and data mining are both very interesting']
  </code></pre>
  
  <p>In a nutshell, I want to find the <code>concepts</code> in <code>sentences</code>. More specifically, given a list in <code>concepts</code> (e.g., <code>['natural language processing', 'text mining', 'texts', 'nlp']</code>), I want to identify these concepts in the sentence and replace them by its first element (i.e. <code>natural language processing</code>). </p>
  
  <p><strong>Example:</strong>
  So, if we consider the sentence <code>data mining and text mining</code>; the results should be <code>advanced data mining and natural language processing</code>. (because the first elements of <code>data mining</code> and <code>text mining</code> are <code>advanced data mining</code> and <code>natural language processing</code> respectively).</p>
  
  <p>The results of the above dummy data should be:</p>
  
  <pre><code>['advanced data mining and natural language processing', 'natural language processing is mainly used by discourse analysis community', 'advanced data mining in python is fun', 'discourse analysis advanced data mining analysis involves natural language processing', 'advanced data mining and advanced data mining are both very interesting']
  </code></pre>
  
  <p>I am currently doing this using regex as follows:</p>
  
  <pre><code>concepts_re = []
  
  for item in sorted_wikipedia_redirects:
          item_re = "|".join(re.escape(item) for item in item)
          concepts_re.append(item_re)
  
  sentences_mapping = []
  
  for sentence in sentences:
      for terms in concepts:
          if len(terms) &gt; 1:
              for item in terms:
                  if item in sentence:
                      sentence = re.sub(concepts_re[concepts.index(terms)], item[0], sentence)
      sentences_mapping.append(sentence)
  </code></pre>
  
  <p>In my real data set I have about 8 million <code>concepts</code>. Therefore, my approach is very inefficient and takes like 5 minutes to process one sentence. I would like to know if there is any efficient way of doing this in python.</p>
  
  <p>For those who would like to process a long list of <code>concepts</code> to measure the time, I have attached a bit longer list herewith: <a href="https://drive.google.com/file/d/1OsggJTDZx67PGH4LupXIkCTObla0gDnX/view?usp=sharing" rel="noreferrer">https://drive.google.com/file/d/1OsggJTDZx67PGH4LupXIkCTObla0gDnX/view?usp=sharing</a> </p>
  
  <p>I am happy to provide more details if needed.</p>