Palmer played high school football at Immaculate High School a small Catholic high school located in Danbury, Connecticut. After high school, he attended and played college football at Southern Connecticut State University (SCSU). Palmer played quarterback for Southern Connecticut State from 1968 to 1971. He earned both bachelor's and master's degrees at SCSU, where he was inducted into the school's athletic Hall of Fame. He is also a member of the Hall of Fame at New Haven and Immaculate High School.