Jim played baseball for the Beverly Hills Yankees, where he pitched and also hit home runs as an outfielder. The family eventually moved to Scottsdale, Arizona, where Jim played baseball, basketball, and football at Scottsdale High School. He earned All-State honors in each of these sports, also graduating with a 3.4 grade-point average in 1963. Palmer also showed his prowess at American Legion Baseball. The University of Southern California, UCLA, and Arizona State University each offered him full scholarships; Stanford University offered a partial scholarship as well.