Noah Phelps was the fourth of nine children born to David Phelps and Abigail Pettibone. David Phelps was the son of Joseph Phelps and his third wife, Mary Case. David married Abigail Pettibone on April 25, 1731 and died of smallpox on December 9, 1760. Noah married Lydia Griswold of Windsor, Connecticut on June 10, 1761 in Simsbury, Connecticut. They had six children: Noah Amherst (May 3, 1762), Lydia Griswold (February 25, 1764), Chandler Conway (October 22, 1766), George P. (August 18, 1773), Elisha Phelps (November 16, 1779), and a son (November 18, 1783, died at birth).