He was educated at Harrow where, notwithstanding the fears of his family that a childhood head injury caused by a dangerous fall from a horse had seriously damaged his intellect, he showed aptitude in Greek and Latin. He made good academic progress, but after he had settled a quarrel with another pupil by an organised fist-fight, his father removed him from the school. (Fist fights were tolerated at Harrow: it was the fact of Brudenell's receiving punishment for unauthorised absence while having a broken bone in his hand attended to by a London surgeon that had annoyed the earl). He was subsequently educated at home. Here, as the only son among seven sisters, he developed into something of a spoilt child, accustomed to getting his own way. This has been seen as a cause of his arrogance and stubbornness in later life.