Gaylord was the son of Evan and Ruby Perry, who were farmers. Evan Perry had been a noted athlete. Gaylord grew up with his older brother James "Jim" Perry and younger sister Carolyn in Williamston and the small area of Farmlife, a populated place located within the Township of Griffins, a minor division of Martin County. Gaylord assisted his father with farming on their family's land in this area. Jim and Gaylord both began playing baseball with their father during their lunch break on the farm as youths, and later all three would play on the same local semi-pro team.