Baylor was born September 16, 1934 in Washington, DC. He attended high school at Washington, D.C.'s Springarn High. He was an All-American at Seattle University. In 1958 he led his team to the NCAA finals.