In 1915, James married Marion Templeton. Their only child, Marion James, became a historian of ancient Greece and Rome, and an art collector.
In addition to his work, Charles James became an avid gardener, specializing in delphinium propagation. He also became a beekeeper, and an expert on the social life of bees.