Milbury was a three-year letterman at Colgate University from 1972 to 1974. A defenseman who wore uniform number 7, he was the team's co-leader in assists with 19 in his junior year. As senior captain, he had his best season with the Red Raiders with 30 points (4 goals, 26 assists). He also led the squad in penalty minutes in both campaigns with 68 in 1973 and 85 in 1974. His totals in 76 games played were 6 goals, 55 assists, 61 points and 203 penalty minutes.