<p>You can rewrite your regex to have the structure of a trie (instead of <code>word|worse|wild</code> use <code>w(or(d|se)|ild)</code>), or even better, ditch the regex and use the <a href="https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm" rel="nofollow noreferrer">Aho–Corasick</a> algorithm. Of course you can use a library for that, for instance <a href="https://github.com/vi3k6i5/flashtext" rel="nofollow noreferrer">FlashText</a> (which is a <a href="https://arxiv.org/abs/1711.00046" rel="nofollow noreferrer">slimmed down version</a> of Aho-Corasick, specialized for searching and replacing whole words as in your case).</p>
<p>The author of FlashText claims <em>»<a href="https://www.freecodecamp.org/news/regex-was-taking-5-days-flashtext-does-it-in-15-minutes-55f04411025f/" rel="nofollow noreferrer">Regex was taking 5 days to run. So I built a tool that did it in 15 minutes.</a>«</em></p>
Fastest way to remove/substitute substrings from a long text
<p>I have a large corpus that I want to remove certain words from. Similar to removing stopwords from the text, but rather I now want to remove bigrams from the corpus. I have my list of bigrams, but obviously the simple list comprehension way to remove stopwords isn't going to cut it.  I was thinking to use regex and compile a pattern from a list of words and then substituting the words. Here is some sample code:</p>
<pre><code>txt = 'He was the type of guy who liked Christmas lights on his house in the middle of July. He picked up trash in his spare time to dump in his neighbors yard. If eating three-egg omelets causes weight-gain, budgie eggs are a good substitute. We should play with legos at camp. She cried diamonds. She had some amazing news to share but nobody to share it with. He decided water-skiing on a frozen lake wasn’t a good idea. His eyes met mine on the street. When he asked her favorite number, she answered without hesitation that it was diamonds. She is never happy until she finds something to be unhappy about; then, she is overjoyed.'
</code></pre>
<p>--</p>
<pre><code>import re
words_to_remove = ['this is', 'We should', 'Christmas lights']
pattrn = re.compile(r' | '.join(words_to_remove))
pattrn.sub(' ',txt)

%timeit pattrn.sub(' ',txt)
</code></pre>
<p>--</p>
<pre><code>timeit 1: 9.18 µs ± 11.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>
<p>Is there a faster way for me to remove these bigrams? The len of the actual corpus is 556,694,135 characters and the number of bigrams is 3,205,182 this is really slow when doing it on the actual dataset.</p>