In their first three seasons under Colson, the Lobos struggled to a 39–44 combined record. Four-year starting PG Phil Smith led those early teams, averaging 14 points, six assists, and two steals per game over his career, setting then-records for highest scoring freshman, most steals in a season, and most career assists. A highlight of Colson's first season was his fourth game as head coach, hosting an Arizona State team that would reach #3 in the nation led by future NBA players Byron Scott, Fat Lever, Alton Lister, and Sam Williams. The callow Lobo squad led for much of the game, encouraged by a delirious crowd at The Pit. The Sun Devils battled back to tie the game at the end of regulation before escaping with a win in OT, 86–83. Despite the struggles of the team, attendance at The Pit actually improved in Colson's early years, remaining in the top four nationally.