He was the elder son of Sir John Patteson the judge, by his second wife, Frances Duke Coleridge who was a niece of the poet Samuel Taylor Coleridge. Patteson was brought up in Devon at Feniton Court, where his family resided, so as to be near the home of his mother's relatives at Ottery St Mary. After three years at The King's School, Ottery St Mary, Patteson was placed in 1838 at Eton College, under his uncle, the Reverend Edward Coleridge, son-in-law of John Keate, once headmaster there.