Walker led the nation in scoring in his senior year of 1966–67, averaging 30.4 points a game. He caught and passed UCLA's Lew Alcindor in the final weeks of the season. Walker's 2,000-plus career points led Providence for four decades, until his all-time scoring record was broken in 2005 by Ryan Gomes. Walker was able to accomplish this in only three seasons; at the time, freshmen were not allowed to play varsity basketball. Jimmy Walker ended his college career in the quarterfinals of the 1967 NIT in the last basketball tournament held at the third Madison Square Garden, when he scored 36 points but missed a jump shot at the end of the game as Providence bowed to Marquette 81-80 in overtime. Walker later said that the missed shot was the only thing that went wrong for him in Madison Square Garden.