Wofford was born in Junction City, Kansas to an equestrian family. His father Col. John W. Wofford, who died when James was 10, rode for the United States Show Jumping Team in the 1932 Los Angeles Olympic Games. Col. Wofford went on to coach eventers and show jumpers at the 1952 Helsinki Olympic Games. This was the first Olympics where civilians were permitted to participate in eventing, and nineteen-year-old Jeb Wofford—Col. Wofford’s son and James’ brother—won a bronze medal. Col. Wofford was also the founder and first president of the United States Equestrian Team.