The Gray House was designed in 1810 by architect Ithiel Town, whose earliest known work it is. It was built for the zoologist William Dandridge Peck, and originally stood at the corner of Garden and Linnaean Streets in Cambridge, Massachusetts, on the grounds of the Harvard College Botanical Garden. Subsequent occupants included botanist Thomas Nuttall and Harvard presidents James Walker and Jared Sparks.
Asa Gray purchased the house in 1842 and moved in during the summer of 1844, after receiving an appointment to a professorship at Harvard that he would hold for 45 years. Already a rising star in the world of botany, Gray in 1848 published The General of the Plants of the United States, which was not only groundbreaking for the content, but also in its presentation. His discovery of relationships between plants of North America and East Asia was influential in the growth of the field of plant geography.