Pamela Digby was born in Farnborough, Hampshire, England, the daughter of Edward Digby, 11th Baron Digby, and his wife, Constance Pamela Alice, the daughter of Henry Campbell Bruce, 2nd Baron Aberdare. She was educated by governesses in the ancestral home at Minterne Magna in Dorset, along with her three younger siblings. Her great-great aunt was the nineteenth-century adventurer and courtesan Jane Digby (1807–1881), notorious for her exotic travels and scandalous personal life. Pamela was to follow in her ancestor's footsteps, and has been called "the 20th-century's most influential courtesan".