Arthur Bryant was the son of Sir Francis Morgan Bryant, who was the chief clerk to the Prince of Wales, and wife Margaret (May) née Edmunds. His father would later hold a number of offices in the royal secretariat, eventually becoming registrar of the Royal Victorian Order. Arthur grew up in a house bordering the Buckingham Palace gardens near the Royal Mews. There he developed a feel for the trappings of traditional British protocol and a strong attachment to the history of England.