Elder did not play a full round of 18 holes until he was 16. He took jobs in pro shops and locker rooms, in addition to caddying where he developed his game by watching his clients, and playing when he had the opportunity. Elder's game developed sufficiently for him to start hustling. His career took a big step after playing a match with heavyweight boxer Joe Louis, which led to Louis’s golf instructor, Ted Rhodes, taking Elder under his wing for three years. Under the tutelage of Rhodes, Elder was able to polish his game and he began playing in tournaments.