Evers was the third of the five children born to Jesse (Wright) and James Evers. The family included his father Jesse's two children from a previous marriage. The Evers family owned a small farm, and his father also worked at a sawmill. Evers walked twelve miles to attend segregated schools, and earned his high school diploma. Evers served in the United States Army during World War II from 1943 to 1945. He fought in the Battle of Normandy in June 1944. After the end of the war, Evers was honorably discharged as a sergeant.