In 1949 Callaway married Laura Elizabeth "Beth" Walton, a native of Hamilton in Harris County, Georgia. She was a daughter of Ralph Outlaw Walton and the former Octavia Perry. She met Howard Callaway in the sixth grade. She graduated from Agnes Scott College in Atlanta in 1947, where she majored in mathematics and chemistry. In 1979, she received a Master of Science in Botany from Western State College in Gunnison, Colorado. Mrs. Callaway was a founding member of the Lady Bird Johnson Wildflower Center in Austin, Texas, and served as well on the board of visitors for the Ida Cason Callaway Foundation and the board of the United States National Arboretum in Washington, D.C. Lynda Johnson Robb, older daughter of Lady Bird Johnson called Mrs. Callaway "a take-charge woman, and when she gave her heart to Mother Nature, you knew that native plants and the environment were better off." The couple were together for more than sixty years until her death in 2009.