Gavitt has also been involved in Olympic basketball. In 1980, he was selected as the head coach of the Olympic basketball team, only to lose out on the opportunity due to the boycott of the Moscow games by the United States. He would also go on to serve on the Olympic governing body, including a presidency from 1988 to 1992. It was during his tenure that he developed the concept of the "Dream Team," an Olympic basketball team composed of the NBA's best.