Gardner was born at the Manor House Uttoxeter, which is commemorated by a plaque. He married Susannah Hyde Gale (c. 1760 – 20 April 1823) on 20 May 1769. She was a Jamaican heiress and the daughter of Francis Gale, a plantation owner, and Susanna Hall. He married Susanna Hyde Gale, daughter and heiress of Francis Gale of Liguania, Jamaica, and widow of Sabine Turner, in Kingston Jamaica on 20 May 1769. They had nine sons and a daughter, and his eldest son, Vice-Admiral Alan Hyde Gardner, succeeded to his titles. The first son, Alan Gardner, 2nd Baron Gardner, became an admiral in the Royal Navy; the second son, Francis, also became an admiral; and the third son William, became a major general.