 Smith lived at Tresco Abbey and developed the gardens on Tresco.He never married, but is believed to have had illegitimate children by several of his female servants.He had a close relationship and a lifelong correspondence with Lady Sophia Tower, a married woman (and only daughter of the 1st Earl Brownlow) who often visited him at Tresco.