Freeman married Inez Virginia Goddin on February 5, 1914. They had three children: Mary Tyler, Anne Ballard, and James Douglas. Mary Tyler Freeman married Leslie Cheek, Jr., longtime director of the Virginia Museum of Fine Arts, and became a founder or influential officer of several important community organizations, as well as president of the Robert E. Lee Memorial Foundation. The family lived (and Freeman died) in a mansion he named "Westbourne" in Richmond's west end, a house listed (in 2000) in the National Register of Historic Places.