title: Removing ngrams containing stopwords using tidytext
body: <p><strong>UPDATE:</strong> Thanks for the input so far. I rewritten the question and added a better example to highlight the implicit requirements that were not covered in my first example.</p>
  
  <p><strong>Question</strong>
  I am looking for a general <code>tidy</code> solution to removing ngrams containing stopwords. In short, ngrams are strings of words separated by a space. A unigram contains 1 word, a bigram 2 words, and so on. My goal would be to apply this on a data frame after using <code>unnest_tokens()</code>. The solution should work with a data frame containing a mix of ngrams of any length (uni, bi, tri..), or at least bi &amp; tri and above.</p>
  
  <ul>
  <li>For more information on ngrams, see wiki: <a href="https://en.wikipedia.org/wiki/N-gram" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/N-gram</a></li>
  <li>I am aware of this question: <a href="https://stackoverflow.com/questions/46685498/remove-ngrams-with-leading-and-trailing-stopwords">Remove ngrams with leading and trailing stopwords</a>. However, I am looking for a general solution, that does not require the stopword to be leading or trailing and that would also scale somewhat nicely.</li>
  <li>as pointed out in the comments, there is a solution for bigrams documented here: <a href="https://www.tidytextmining.com/ngrams.html#counting-and-filtering-n-grams" rel="nofollow noreferrer">https://www.tidytextmining.com/ngrams.html#counting-and-filtering-n-grams</a></li>
  </ul>
  
  <p><strong>New example data</strong></p>
  
  <pre class="lang-r prettyprint-override"><code>ngram_df &lt;- tibble::tribble(
    ~Document,                   ~ngram,
            1,                    "the",
            1,              "the basis",
            1,                  "basis",
            1,       "basis of culture",
            1,                "culture",
            1,        "is ground water",
            1,           "ground water",
            1, "ground water treatment"
    )
  stopword_df &lt;- tibble::tribble(
    ~word, ~lexicon,
    "the", "custom",
     "of", "custom",
     "is", "custom"
    )
  desired_output &lt;- tibble::tribble(
    ~Document,                   ~ngram,
            1,                  "basis",
            1,                "culture",
            1,           "ground water",
            1, "ground water treatment"
    )
  </code></pre>
  
  <p><sup>Created on 2019-03-21 by the <a href="https://reprex.tidyverse.org" rel="nofollow noreferrer">reprex package</a> (v0.2.1)</sup></p>
  
  <p><strong>Desired behaviour</strong></p>
  
  <ul>
  <li>the <code>ngram_df</code> should be transformed into the <code>desired_output</code>, using the stopwords from the <code>word</code> column in the <code>stopword_df</code>.</li>
  <li>every row containing a stopword should be removed</li>
  <li>word boundaries should be respected (i.e. looking for <code>is</code> should not remove <code>basis</code>)</li>
  </ul>
  
  <p><hr>
  <sup>my first attempt for a reprex below:</p>
  
  <p>example data</p>
  
  <pre class="lang-r prettyprint-override"><code>library(tidyverse)
  library(tidytext)
  df &lt;- "Groundwater remediation is the process that is used to treat polluted groundwater by removing the pollutants or converting them into harmless products." %&gt;% 
    enframe() %&gt;% 
    unnest_tokens(ngrams, value, "ngrams", n = 2)
  #apply magic here
  
  df
  #&gt; # A tibble: 21 x 2
  #&gt;     name ngrams                 
  #&gt;    &lt;int&gt; &lt;chr&gt;                  
  #&gt;  1     1 groundwater remediation
  #&gt;  2     1 remediation is         
  #&gt;  3     1 is the                 
  #&gt;  4     1 the process            
  #&gt;  5     1 process that           
  #&gt;  6     1 that is                
  #&gt;  7     1 is used                
  #&gt;  8     1 used to                
  #&gt;  9     1 to treat               
  #&gt; 10     1 treat polluted         
  #&gt; # ... with 11 more rows
  </code></pre>
  
  <p>example list of stopwords</p>
  
  <pre><code>stopwords &lt;- c("is", "the", "that", "to")
  </code></pre>
  
  <p>desired output</p>
  
  <pre class="lang-r prettyprint-override"><code>#&gt; Source: local data frame [9 x 2]
  #&gt; Groups: &lt;by row&gt;
  #&gt; 
  #&gt; # A tibble: 9 x 2
  #&gt;    name ngrams                 
  #&gt;   &lt;int&gt; &lt;chr&gt;                  
  #&gt; 1     1 groundwater remediation
  #&gt; 2     1 treat polluted         
  #&gt; 3     1 polluted groundwater   
  #&gt; 4     1 groundwater by         
  #&gt; 5     1 by removing            
  #&gt; 6     1 pollutants or          
  #&gt; 7     1 or converting          
  #&gt; 8     1 them into              
  #&gt; 9     1 harmless products
  </code></pre>
  
  <p><sup>Created on 2019-03-20 by the <a href="https://reprex.tidyverse.org" rel="nofollow noreferrer">reprex package</a> (v0.2.1)</sup></p>
  
  <p>(example sentence from: <a href="https://en.wikipedia.org/wiki/Groundwater_remediation" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Groundwater_remediation</a>)
  </sup></p>