Drew was born in 1904 into an African-American middle-class family in Washington, D.C. His father, Richard, was a carpet layer and his mother, Nora Burrell, trained as a teacher. Drew and three of his four younger siblings grew up in Washington's largely middle-class and interracial Foggy Bottom neighborhood. From 1920 until his marriage in 1939, Drew's permanent address was in Arlington County, Virginia, although he graduated from Washington's Dunbar High School in 1922 and usually resided elsewhere during that period of time.