Haslam was born in 1930 in Detroit, Michigan, the son of Melitta (Fry) and James Arthur Haslam. His father was a World War I veteran who was, at the time, working for the Studebaker Corporation. The family lived for a time in Pennsylvania before moving to St. Petersburg, Florida. Jim was a star football player at St. Petersburg High School, and had offers to play at several colleges. He eventually chose the University of Tennessee, following a recruiting visit from legendary U.T. coach Robert Neyland.