Nelson led the Saint Joseph's Hawks to a 27–0 regular season record in 2003–04. The Hawks' first loss came in the Atlantic 10 Tournament to Xavier. Nelson and junior guard Delonte West formed what was largely considered the nation's best backcourt, helping the Hawks earn a #1 seed in the NCAA Tournament. They advanced to the Elite Eight and were within seconds of the Final Four before Oklahoma State Cowboys' John Lucas III hit a three-pointer with only a few seconds remaining (after the make, Nelson dribbled up the court and had a chance to tie the game, but his 15-foot shot fell short). Saint Joseph's finished with a 30–2 record, the best in the university's history. Nelson averaged 20.6 points, 5.3 assists, and 2.9 steals per game. He received the Lowe's Senior CLASS Award his final year, recognizing him as the nation's top senior men's basketball player. He left the Hawks as the best player in the program's history, as its all-time leader in scoring (2094 points), assists (714), and steals (256). Nelson's number was retired by the university on April 23, 2004.