Dillard's first principal was Joseph A. Ely. When the building, now the Old Dillard Museum, was built in 1924 it was an elementary school; by the time he left in 1937, when he moved to Crispus Attacks High School (today a middle school), he had succeeded in expanding Dillard to include high school classes. He was responsible for getting the school name changed in 1930 to honor James H. Dillard, a white philanthropist, educator, and promoter of education for black children. He was replaced by Clarence C. Walker, Sr., who was principal until his death in 1942.