One of his ancestors was John Lothropp, also an ancestor of Benedict Arnold and Joseph Smith, Jr. One of his distant relatives from his mother's side is the author Laura Ingalls Wilder. His maternal grandfather Warren Delano II, a descendant of "Mayflower" passengers Richard Warren, Isaac Allerton, Degory Priest, and Francis Cooke, during a period of twelve years in China made more than a million dollars in the tea trade in Macau, Canton, and Hong Kong, but upon coming back to the United States, he lost it all in the Panic of 1857.