Manning was born on 15 July 1808 at his grandfather's home, Copped Hall, Totteridge, Hertfordshire. He was the third and youngest son of William Manning, a West India merchant, who served as a director and (1812–1813) as a governor of the Bank of England and also sat in Parliament for 30 years, representing in the Tory interest Plympton Earle, Lymington, Evesham and Penryn consecutively. Manning's mother, Mary (died 1847), daughter of Henry Leroy Hunter, of Beech Hill, and sister of Sir Claudius Stephen Hunter, 1st Baronet, came of a family said to be of French extraction.