Gaylordsville is located in the northwest corner of New Milford. It is part of the valley known to the Native Americans as the Wheniuck or Red Plumb Plain. On the east the boundary is Quanuctnic or Long Mountain, but it has never been decided whether it should be at the foot of the mountain or somewhere up on top. The southern boundary is also vague, usually considered to be an imaginary line leaving the Housatonic River somewhere south of the Tory Cave and extending across Squash Hollow. The Sherman town line forms the western boundary, although several homes in Sherman are usually considered to be part of the Gaylordsville community.