Wilhem was born on July 26, 1922, in Huntersville, North Carolina. His parents, John and Ethel (née Stanley) Wilhelm, were tenant farmers and had eleven children. He played baseball at Cornelius High School in Cornelius, North Carolina. There he started trying to throw the knuckleball. Because he could not throw very fast, he thought that being able to throw a knuckleball would make him successful. He used a tennis ball to practice.