The first building of the First Presbyterian Church of Sag Harbor was erected in 1766. Known as the "Old Barn Church", it was a "simple building of uncouth shape," a wood-frame building with walls and a roof, but neither a ceiling nor interior plastered walls. According to the Rev. Nathaniel S. Prime, pastor from 1806-09, "If a shower of rain occurred during public worship, the minister was obliged to retreat to the corner of the ample pulpit to escape the falling drops."