"Callendar House" was built on a bluff overlooking the river in 1794 by Henry Gilbert Livingston (1758–1817), whose country seat was next door. He was the great-grandson of Robert Livingston the Elder through his grandfather Gilbert. In 1795 he sold it to his cousin Philip Henry Livingston (1769–1831), great-grandson of Gilbert's older brother Philip Livingston (1686–1749), the second Lord of Livingston Manor. Philip and his wife called it "Sunning Hill" and occupied it until 1828, when it was purchased by Robert L. Tillotson. In 1835, Tillotson sold the property to architect Richard T. Auchmuty.