John Dillard settled in the area around what is now Dillard with his son, James Dillard, "circa" 1823. James took title to of land by purchasing the lottery rights of land holders in Rabun County. John Dillard served as a lieutenant in the Virginia Militia in the American Revolution, during which he was in the Battle of Guilford Court House and other battles. Both James and John were residents of Buncombe County, North Carolina prior to their settlement on lands now comprising Dillard.