title: Fastest way to determine if an ordered sublist is in a large lists of lists?
body: <p>Suppose I have a my_huge_list_of_lists with 2,000,000 lists in it, each list about 50 in length.</p>
  
  <p>I want to shorten the 2,000,000 <code>my_huge_list_of_lists</code> by discarding sublists that do not contain two elements in the sequence.</p>
  
  <p>So far I have:</p>
  
  <pre><code># https://stackoverflow.com/questions/3313590/check-for-presence-of-a-sliced-list-in-python
  def check_if_list_is_sublist(lst, sublst):
      #checks if a list appears in order in another larger list.
      n = len(sublst)
      return any((sublst == lst[i:i + n]) for i in xrange(len(lst) - n + 1))
  
  my_huge_list_of_lists = [x for x in my_huge_list_of_lists
                              if not check_if_list_is_sublist(x, [a,b])]
  my_huge_list_of_lists = [x for x in my_huge_list_of_lists
                              if not check_if_list_is_sublist(x, [b,a])]
  </code></pre>
  
  <p>The consecutiveness of the search term [a,b] or [b,a] is important so I can't use a <code>set.issubset()</code>.</p>
  
  <p>I find this slow.  I'd like to speed it up.  I've considered a few options like using an 'early exit' and statement:</p>
  
  <pre><code>my_huge_list_of_lists = [x for x in my_huge_list_of_lists
                              if (a in x and not check_if_list_is_sublist(x, [a,b]))]
  </code></pre>
  
  <p>and less times in the <code>for</code> loop with an <code>or</code> statement:</p>
  
  <pre><code>my_huge_list_of_lists = [x for x in my_huge_list_of_lists
                              if not (check_if_list_is_sublist(x, [a,b])
                                      or check_if_list_is_sublist(x, [b,a]))]
  </code></pre>
  
  <p>and also working on speeding up the function (WIP)</p>
  
  <pre><code># https://stackoverflow.com/questions/48232080/the-fastest-way-to-check-if-the-sub-list-exists-on-the-large-list
  def check_if_list_is_sublist(lst, sublst):
          checks if a list appears in order in another larger list.
          set_of_sublists = {tuple(sublst) for sublist in lst}
  </code></pre>
  
  <p>and done some searching on Stack Overflow; but can't think of a way because the number of times <code>check_if_list_is_sublist()</code> is called is <code>len(my_huge_list) * 2</code>.</p>
  
  <p>edit: add some user data as requested</p>
  
  <pre><code>from random import randint
  from string import ascii_lowercase
  my_huge_list_of_lists = [[ascii_lowercase[randint(0, 25)] for x in range(50)] for y in range(2000000)]
  my_neighbor_search_fwd = [i,c]
  my_neighbor_search_rev = my_neighbor_search_fwd.reverse()
  </code></pre>