Anstey was the third child and only son of the Rev. Dr. Christopher Anstey, the rector of Brinkley in Cambridgeshire, and his wife Mary Thompson, born on 31 October 1724 in Trumpington. He was educated at Eton College and King's College, Cambridge, where he distinguished himself for his Latin verses. He became a fellow of his college in 1745 but the degree of M.A. was withheld from him in 1749 owing to his defiance of the university authorities and the offence caused by an address that is said to have begun "Doctors without doctrine, artless masters of arts, and bachelors more worthy of the rod than the laurel..." He joined the Middle Temple in 1746, but was not called to the bar.