 He was the son of Philip Livingston (1686–1749), the 2nd Lord of Livingston Manor.His grandfather's older brother was Robert Livingston and his younger brothers included Philip Livingston, a signer of the United States Declaration of Independence, and William Livingston, a signer of the United States Constitution and the 1st Governor of New Jersey.His maternal grandparents were David Van Horne and Anne (née French) Van Horne, herself the daughter of Philip French III and Susanna Brockholst.