Alfred Newton was born near Geneva in Switzerland, the fifth son of William Newton of Elveden Hall in Suffolk, MP for ; his mother Elizabeth (1789–1843) was the daughter of Richard Slater Milnes, MP for . The family wealth was founded on sugar plantations in the Caribbean, where Alfred's grandfather Samuel Newton had a plantation in St Kitts, and a property in St Croix. William Newton returned to England in 1813, purchasing the property of Elveden, near Thetford from the Earl of Albermarle. Elveden (pronounced and sometimes spelt 'Eldon') was built in 1770 by Admiral Augustus Keppel. After the Newtons left, Elveden Hall and its estate were bought by Prince Duleep Singh in 1863, and later by the Guinness family (Earl of Iveagh).