He was the second son of Thomas Metcalfe and Susannah Selina Sophia. His father first went to India in 1767 as a cadet in the British Army, and at the time of Metcalfe's birth was serving as a major in the Bengal Army. He later became a Member of Parliament, director of the British East India Company and was created a baronet on 21 December 1802. Thomas Metcalfe married Susannah in Calcutta in 1782. She was the daughter of merchant John Debonnaire, a trader at Fort St. George, Madras, who subsequently settled at the Cape of Good Hope. It was her second marriage, her first being with Major John Smith in Madras from 1776 until his death. His parents returned to England shortly after the birth of their son.