Brown also played college football as a safety and punt returner for the University of Michigan from 1972 to 1974. While playing for Michigan, compiled 526 punt return yards (11.7 yards per return), three punt returns for touchdowns, 174 tackles, nine interceptions, 202 interception return yards, and 15 pass breakups. He was selected as a consensus first-team defensive back on the 1973 College Football All-America Team and a unanimous first-team pick on the 1974 College Football All-America Team. He was posthumously inducted into the College Football Hall of Fame in 2007.