Davidson was born in Springfield, Massachusetts on March 6, 1917, to Clarence ("Davie") Herbert Davidson and Grace Cordelia Anthony. The family lived in the Philippines from shortly after Davidson's birth until he was about 4. Then, having lived in Amherst and Philadelphia, the family finally settled on Staten Island when Davidson was 9 or 10. He then began to attend public school and had to begin in first grade with much younger children. He then attended the Staten Island Academy, starting in fourth grade.