title: What is the most efficient way to query multiple values from a single column in SQLite?
body: <p>I have a database in SQLite I use to store hashes. Often times I will need query upwards of a hundred thousand hashes against the dataset of approximately 20 million hashes. There are several tables in the database containing these 20 some ought millions of hashes. Each table has the same structure.</p>
  
  <p>For example,</p>
  
  <pre><code>ID,md5,sha1,source
  
  0,abcd1234...,abcd4321...,bills_hashes
  
  1,cdef5678...,cdef8765...,bobs_hashes
  </code></pre>
  
  <p>Currently, my code (Python) reads in the input text file and creates a list of comparison hashes. The hashes in the list are then created into a set/list (forgot what it's technically called in SQL). Then for each table in the database a query is created akin to the following and added to a list.</p>
  
  <pre><code>SELECT * FROM zacks_table WHERE md5 COLLATE NOCASE in ('hash1', 'hash2', ... 'hash156921');
  
  SELECT * FROM gregs_table WHERE md5 COLLATE NOCASE in ('hash1', 'hash2', ... 'hash156921');
  </code></pre>
  
  <p>There is usually about 12 tables so about 12 queries are added to a list. Then all 12 queries are executed, and all the resulting return rows are collated and then processed.</p>
  
  <p>The issue I'm having is that the querying / result gathering is taking a long time with the larger data sets. Once I get over 50,000 hashes it seems to take hours sometimes. The hash columns in the database are indexed.</p>
  
  <p>How can I improve my queries to speed up my returns so I can process this data quickly (as can be)? I am using the sqlite3 module and Python 2.7 to perform these operations.</p>