In 1958 he formed the South African Sports Association, and as Secretary was strongly opposed to a proposed cricket tour by Frank Worrell’s West Indies to South Africa in 1959, leading a successful campaign to have it cancelled. In 1962 Brutus was a co-founder of the South African Non-Racial Olympic Committee (SANROC), an organisation that would be heavily influential in the banning of apartheid-era South Africa from the Olympics in 1964. In 1961, Brutus was banned for his political activities as part of SANROC. As South Africa attempted, in 1968, to get back into the Olympics by arguing that they would field multi-racial teams, SANROC successfully pointed out that those teams were chosen on a segregated basis, leading to South Africa's continued ban from 1968 until 1992.