Gibbs was born in New Haven, Connecticut. He belonged to an old Yankee family that had produced distinguished American clergymen and academics since the 17th century. He was the fourth of five children and the only son of Josiah Willard Gibbs Sr., and his wife Mary Anna, "née" Van Cleve. On his father's side, he was descended from Samuel Willard, who served as acting President of Harvard College from 1701 to 1707. On his mother's side, one of his ancestors was the Rev. Jonathan Dickinson, the first president of the College of New Jersey (later Princeton University). Gibbs's given name, which he shared with his father and several other members of his extended family, derived from his ancestor Josiah Willard, who had been Secretary of the Province of Massachusetts Bay in the 18th century. His paternal grandmother, Mercy (Prescott) Gibbs, was the sister of Rebecca Minot Prescott Sherman, the wife of American founding father Roger Sherman; and he was the second cousin of Roger Sherman Baldwin, see the Amistad case below.