Lee showed talent for basketball since an early age, and he impressed many college scouts with his game style. He accepted an offer to play for Marquette University, starring there from 1974 to 1978. In 1974 Lee asked his coach Al McGuire to allow him to play for the United States Olympic basketball team. However, his coach had sent someone else and Lee went to Puerto Rico where he qualified for the Puerto Rican national basketball team. When Puerto Rico played against the U.S. in the 1976 Summer Olympics, Lee made 15 out of 18 field goals and scored 35 points. The U.S. still avoided an upset, defeating Puerto Rico by one point, 95-94.