Simpson was born in Bayview, near Cavendish, Prince Edward Island, Canada, as the third son and fourth child of James Simpson, Jr. and Janet Clark. Author Harold H. Simpson has gathered an extensive genealogy of Cavendish families in "Cavendish: Its History, Its People". His research establishes the Clark family (A. B. Simpson's mother's side) as one of the founding families of Cavendish in 1790, along with the Simpson family, and he traces common ancestors between Albert B. Simpson and Lucy Maud Montgomery, the author of "Anne of Green Gables".