Across from each single strand of DNA, we typically see adenine pair with thymine, and cytosine pair with guanine to form a parallel complementary strand as described below.  Two nucleotide sequences which correspond to each other in this manner are referred to as complementary:
A frayed end refers to a region of a double stranded (or other multi-stranded) DNA molecule near the end with a significant proportion of non-complementary sequences; that is, a sequence where nucleotides on the adjacent strands do not match up correctly:
The term "frayed" is used because the incorrectly matched nucleotides tend to avoid bonding, thus appearing similar to the strands in a fraying piece of rope.
Although non-complementary sequences are also possible in the middle of double stranded DNA, mismatched regions away from the ends are not referred to as "frayed".