Thomas Patrick John Anson was born 25 April 1939. He was the only son of Lieutenant-Colonel Thomas William Arnold Anson, Viscount Anson (1913–1958), the eldest son and heir apparent of Thomas Edward Anson, 4th Earl of Lichfield (1883–1960). His mother was born Anne Bowes-Lyon (1917–1980), a niece of Queen Elizabeth The Queen Mother. His parents divorced in 1948, and his mother subsequently became Princess Anne of Denmark after her remarriage to Prince George Valdemar of Denmark in 1950. He had one sister, Elizabeth Georgiana (1941–2020), who married Sir Geoffrey Adam Shakerley, 6th Baronet Shakerley.