Olds played on the varsity college football team in both 1941 and 1942. At 6 feet 2 inches in height (1.88m) and weighing 205 pounds (92 kg), he played tackle on both offense and defense, lettering both seasons. Army's record in 1941 was 5–3–1, with wins over The Citadel, VMI, Yale, Columbia, and West Virginia, a scoreless tie with Notre Dame, and losses to Harvard, Penn and Navy. The loss to the midshipmen was followed eight days later by the attack on Pearl Harbor.