Brumby went to the University of Melbourne in 1971 and completed a Bachelor of Commerce degree. He then got his Diploma of Education from the State College of Victoria, Rusden, in 1975. He worked at Eaglehawk, Victoria, as a teacher for three years and then became a union official.