Olyphant attended Modesto's Fred C. Beyer High School. Growing up, he was "embarrassed" by the idea of acting, but enjoyed art and drawing. He swam competitively throughout his childhood and was a finalist at the 1986 Nationals, in the 200m Individual Medley. He was then recruited to the University of Southern California by USC Trojans swimming coach Peter Daland. When Olyphant first visited the campus as part of a recruitment trip, he hoped to study architecture but was told it would be unmanageable with his training schedule. Instead, he opted to earn a Bachelor of Fine Arts.