His paternal grandparents were Francis Sarason Gibbs and Eliza Gay (née Hosmer) Gibbs and his maternal grandparents were Edward Alexander Duer and Sarah Anna (née Vanderpoel) Duer. He was a descendant of mineralogist George Gibbs and the great-nephew of the chemist Oliver Wolcott Gibbs with whom he shared all three names. The younger Gibbs, however, disdained the "Oliver" and never used it. Through his maternal grandfather, he was a descendant of William Duer, a member of the Continental Congress and signer of the United States Articles of Confederation, and through his maternal grandmother, he was a direct descendant of U.S. President Martin Van Buren. He was also a direct descendant of Oliver Wolcott Sr., signer of the Declaration of Independence, and Secretary of the Treasury under George Washington and John Adams. He also descended from the Livingston family and the Schuyler family.