Noe's daughter-in-law, the former Betty Schlesinger (1926–2014), a native of Abbeville in Vermilion Parish, was active in the New Orleans Opera Association, the Women's Opera Guild, the New Orleans Ballet Association, the Louisiana Arts Council, and the Louisiana State University Foundation. U.S. President Ronald W. Reagan appointed her to serve on the advisory committee of the Kennedy Center in Washington, D.C. She was a Knight of the Equestrian Order of the Holy Sepulchre of Jerusalem and a recipient of the Order of St. Louis. Jimmy Jr. and Betty Noe, who married in 1955, had five children, Lisa Deane (Tenny), James Noe, III, Erin May (Chet), George Noe (Elizabeth), and Claire Koch (Josh).