Take the 2-minute tour ×

Let's say I have a file DATA with 10,000,000 lines. I have another file IDS with 100,000 strings. I want to extract all lines from DATA that contain one of the strings from IDS. An additional condition is that there is a 1:1 relationship between the files, so every ID has one line of DATA and every DATA has one ID.

What is the most efficient, least complicated way to do this using standard linux command-line utilities?

My ideas so far:

  1. Build a huge regex and use grep (easy, may exceeed some limit within grep)
  2. Go through IDS line by line and grep DATA for each string separately, merge results. (easy, probably very inefficient)
  3. Build a hashmap of IDS in python, loop through DATA, extract ID and check against hash map (a bit harder)
share|improve this question
4. man join 5. Use a real database –  n.m. Feb 27 '13 at 18:55
I am trying to use join now (which should be exactly what I need) but I am running into some trouble with sorting my data: stackoverflow.com/questions/15133894/… –  Nils Feb 28 '13 at 11:06
add comment

2 Answers

up vote 3 down vote accepted
grep -F -f IDS DATA

Don't miss -F: it prevents from interpreting IDS as regular expressions, and enables a much more efficient Aho-Korasick algorithm.

share|improve this answer
add comment

If IDS contains the exact strings you need to find in DATA, one string per line, try using

grep --file=IDS DATA > results
share|improve this answer
add comment

Your Answer


