The word "toff" is thought to come from the word "tuft", which was a gold tassel worn by titled undergraduates at the University of Oxford or the University of Cambridge.
The use of the term "toff" suggests a connection to the tradition of gold tassels worn by distinguished students at prestigious universities like Oxford and Cambridge.