 <p><strong>Disclaimer:</strong> <em>Do not down vote without explaining why, because OP didn't include his entire code base or hardware/infrastructure design. But if i've made critical errors in my code or logic, explain them and down vote accordingly.</em></p> <p>Lets start off with defining the bottle-necks you'll encounter (some obvious, some not).</p> <ul> <li>Harddrives - They're slow and doesn't cache large quantities</li> <li>Re-reading the same data multiple times</li> <li>Memory, you can't store a 13GB file, <em>or maybe you do and that's an option?</em></li> </ul> <h2>To solve these, you could go multiple routes.<br></h2> <ol> <li>One clear benefitial way would be to read the large data into a database (postgresql or mariadb for instance). But I'll assume this isn't at all possible for now.</li> </ol> <p>In regards to a CSV reader, they're fine but maybe not so efficient.<br> Since you're going to read through both files anyway I would consider the following:</p> <ol start="2"> <li><p>Read the 13GB file line by line and save the <code>ID</code> in a dictionary without checking if the key/value exists. <em>(why? Because checking if the value exists is slower than just overwriting it and dictionaries also has a added bonus that keys are unique so duplicates will be weeded out)</em> <strong>or</strong> add it to a <code>set()</code> as described by many others. </p> <p>followed by reading the smaller file line by line and checking your <code>dict</code> or <code>set</code> if it contains the <code>ID</code>.</p></li> </ol> <h2><code>dict()</code> vs <code>set()</code> vs <code>list()</code></h2> <p>Here's a comparison between the three data types <code>set()</code>, <code>list()</code> and <code>dict()</code>:<br> <em>code used: <a href="https://gist.github.com/Torxed/b3872299c5659ea0ea09" rel="nofollow">test.py</a></em></p> <pre><code>(11.719028949737549, 999950, 'Using dict() for inserts') (1.8462610244750977, 'dict() time spent looking through the data') (11.793760061264038, 999961, 'Using set() for inserts') (7.019757986068726, 'set() time spent looking through the data') (15min+, 'Using list()') # Honestly, I never let it run to it's finish.. It's to slow. </code></pre> <p>As you can see <code>dict</code> is marginally quicker than <code>set</code>, while <code>list()</code> just falls behind completely (see description of why by <a href="http://stackoverflow.com/a/35944150/929999">Antti</a>). I should point out my data is a bit skewed since it's a quick and dirty demo of the speed difference, but the general idea should still come across. </p> <h2>Using dict() as a solution</h2> <p>So if you do not have access to a database version of the source-data, and you need to use Python, id go with something along the lines of:</p> <pre><code>delimiter = b'\x01' big_source = {} with open('13_gig_source.log', 'rb') as fh: for line in fh: csv = line.split(delimiter) big_source[csv[4]] = None # 5:th column, change to match CUSTOMER_ID output = open('result.log', 'wb') with open('smaller_file.log', 'rb') as fh: for line in fh: csv = line.split(delimiter) if csv[4] in big_source: output.write(csv[4] + b'\n') output.close() </code></pre> <p>Since I didn't know which column the data exists on i didn't optimize the <code>split()</code>.<br> For instance, if it's the last column you're trying to fetch, do <code>line.rsplit(delimiter, 1)[-1]</code> instead. Or if it's the 3:d column do <code>line.split(delimiter, 3)[2]</code> because it will abort the process of looking for the next position of the <code>delimiter</code> in the <code>split()</code> functions.</p> <h2>Using linux tools</h2> <p>Yes, some tools might be better off suited for this such as <code>awk</code> because of the fact that it's a specific tool written in C to perform a very specific task. Even tho Python is based on C it still has a lot of abstraction layers on-top of that C code and will for the most part be slower than the counterpart C tools written for specific tasks.</p> <p>Now I have no data to test this with, nor am i a <em>PRO With Nix-Commands</em> or <em>PWN-C</em> for short. So I'll leave someone else to give you examples of this, but i found this:</p> <ul> <li><a href="http://stackoverflow.com/questions/27253776/merge-two-csv-files-according-to-matching-rows-and-add-new-columns-in-linux">merge two csv files according to matching rows and add new columns in linux</a></li> </ul> <p>And it might be helpful.</p>