title: Remove duplicate rows in big file
body: <p>How do I remove duplicate rows of string while reading a .fastq(text) file using Java? Currently work with next-generation sequencing (ngs) and need to remove duplicate readings from file sequenced (big data). I'm trying with <a href="https://en.wikipedia.org/wiki/Bloom_filter" rel="nofollow noreferrer">Counting Bloom Filter</a> and <a href="https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf" rel="nofollow noreferrer">Cuckoo filter</a> in java but still can not succeed.</p>