Take the 2-minute tour ×

Hey! Working in linux/shell env, how can I accomplish the following:

text file 1 contains:


text file 2 contains:


I need to extract the entries in file 2 which are unique to file 1. So '6' and '7' in this example.

How do I do this from the command line?

many thanks!

share|improve this question
Is it homework? If positive, please tag it as so. –  karlphillip Jan 17 '11 at 19:57
what is the separator of values? –  Ass3mbler Jan 17 '11 at 19:59
good catch! each value is on its own line; so newline sep. –  mark Jan 17 '11 at 20:20
add comment

6 Answers

up vote 6 down vote accepted
$ awk 'FNR==NR{a[$0]++}FNR!=NR && !a[$0]{print}' file1 file2
share|improve this answer
+1: Wow. awk to the rescue! Nice! –  Davidann Jan 17 '11 at 20:15
@David thanks, I just optimized it to be much more concise –  SiegeX Jan 17 '11 at 20:20
sweet! this works great but what if the values are each on a separate line, not separated by space as in my example (i actually had them on a new line but SO formatted them on the same line)? –  mark Jan 17 '11 at 20:22
@mark my code will work for both cases, but if each number is on a separate line, you can completely remove the RS="[ \n]" to make the code shorter. Also, welcome to SO. –  SiegeX Jan 17 '11 at 20:26
@mark by the way, to prevent SO from formatting your code, highlight it then either press CTRL+K or click the { } icon. I edited your question already with this change. –  SiegeX Jan 17 '11 at 20:28
show 1 more comment

Using some lesser-known utilities:

sort file1 > file1.sorted
sort file2 > file2.sorted
comm -1 -3 file1.sorted file2.sorted

This will output duplicates, so if there is 1 3 in file1, but 2 in file2, this will still output 1 3. If this is not what you want, pipe the output from sort through uniq before writing it to a file:

sort file1 | uniq > file1.sorted
sort file2 | uniq > file2.sorted
comm -1 -3 file1.sorted file2.sorted

There are lots of utilities in the GNU coreutils package that allow for all sorts of text manipulations.

share|improve this answer
Good call on these utilities. You can combine this to a much simpler form and remove the need for temp files: comm -13 <(sort file1) <(sort file2) I still prefer awk only because it runs a single process instead of 3 as it doesn't require sorted files. This can make a big difference on large files. –  SiegeX Jan 17 '11 at 20:45
join can also be used for this. –  Dennis Williamson Jan 17 '11 at 22:48
@SiegeX - I personally prefer the version with 3 commands - that way if I need to tweak on of the commands (or for example, get updated file1) I don't need to re-run the WHOLE thing; which can be a benefit for really large files. Also, the syntax you provided sounds like bash, it may not work on other shells (/bin/sh or csh derivatives) –  DVK Jan 18 '11 at 2:43
classic Unix solution. +1 –  DVK Jan 18 '11 at 2:43
add comment

with grep:

grep -F -x -v -f file_1 file_2 
share|improve this answer
This leads to wrong results, as can be shown if a . (dot) is added to file_1. grep -F -x -v -f file_1 file_2 is indeed correct. –  xebeche Sep 29 '13 at 12:45
@xebeche: Thx! Corrected the code line according to your suggestion. –  sid_com Dec 11 '13 at 10:50
add comment

If you are really set on doing this from the command line, this site (search for "no duplicates found") has an awk example that searches for duplicates. It may be a good starting point to look at that.

However, I'd encourage you to use Perl or Python for this. Basically, the flow of the program would be:

findUniqueValues(file1, file2){
    contents1 = array of values from file1
    contents2 = array of values from file2
    foreach(value2 in contents2){
        foreach(value1 in contents1){
            if (value2 == value1) found=true
        if(!found) print value2

This isn't the most elegant way of doing this, since it has a O(n^2) time complexity, but it will do the job.

share|improve this answer
thank a lot David, i'll have a look! –  mark Jan 17 '11 at 20:28
add comment

here's another awk solution

$ awk 'FNR==NR{a[$0]++;next}(!($0 in a))' file1 file2
share|improve this answer
What are the rules where you can use () in lieu of {}? I'm assuming this isn't a gawk'ism because you tend to use gawk when that's the case. –  SiegeX Jan 19 '11 at 18:43
as you know, awk syntax consists of /pattern/{action}. ((!$0 in a)) is the "pattern" part. {action} is printing by default. Its like you can do NR==1 (for example). –  ghostdog74 Jan 20 '11 at 0:19
I guess I'm more curious about the double set of parens, why isn't (!$0 in a) sufficient? Btw, if you prefix your comment with @username then username actually gets a notification that there is a comment to them pending, otherwise they won't. The @username prefix isn't necessary only if you are the person who wrote the question and/or answer people are commenting on. So technically I didn't need to do it for this comment to you. –  SiegeX Jan 20 '11 at 18:26
@SiegeX, no the double parenthesis doesn't matter in this case. Its a habit of mine. the double parenthesis is needed though, if there are more conditions. –  ghostdog74 Jan 21 '11 at 0:19
Thanks for the tip, sounds like a good habit to have. I have a similar one in that I quote variables inside bash's [[ ]]. I'd rather just always do it and not have to worry when I don't need to (even though I already know, heh). –  SiegeX Jan 21 '11 at 0:23
add comment

I was wondering which of the following solutions was the "fastest" for "larger" files:

awk 'FNR==NR{a[$0]++;next}!($0 in a)' file1 file2          # awk2 by ghostdog74
join -v 2 <(sort file1) <(sort file2)
grep -v -F -x -f file1 file2

Results of my benchmarks in short:

  • Do not use grep -Fxf, it's much slower (2-4 times in my tests).
  • comm is slightly faster than join.
  • If file1 and file2 are already sorted, comm and join are much faster than awk1 + awk2. (Of course, they do not assume sorted files.)
  • awk1 + awk2, supposedly, use more RAM and less CPU. Real run times are lower for comm probably due to the fact that it uses more threads. CPU times are lower for awk1 + awk2.

For the sake of brevity I omit full details. However, I assume that anyone interested can contact me or just repeat the tests. Roughly, the setup was

# Debian Squeeze, Bash 4.1.5, LC_ALL=C, slow 4 core CPU
$ wc file1 file2
  321599   321599  8098710 file1
  321603   321603  8098794 file2

Typical results of fastest runs

awk2: real 0m1.145s  user 0m1.088s  sys 0m0.056s  user+sys 1.144
awk1: real 0m1.369s  user 0m1.324s  sys 0m0.044s  user+sys 1.368
comm: real 0m0.980s  user 0m1.608s  sys 0m0.184s  user+sys 1.792
join: real 0m1.080s  user 0m1.756s  sys 0m0.140s  user+sys 1.896
grep: real 0m4.005s  user 0m3.844s  sys 0m0.160s  user+sys 4.004

BTW, for the awkies: It seems that a[$0]=1 is faster than a[$0]++, and (!($0 in a)) is faster than (!a[$0]). So, for an awk solution I suggest:

awk 'FNR==NR{a[$0]=1;next}!($0 in a)' file1 file2
share|improve this answer
add comment

Your Answer


