 He was the son of Loyalist Philip Peter Livingston (1740–1810), who was known as "Gentleman Phil," and Cornelia (née Van Horne) Livingston (b.1759).His siblings included Charles Ludlow Livingston (1800–1873), Philip Livingston and Richard Livingston.