Gouger was the fifth son of nine children of George Gouger (1763–1802), who was a prosperous city merchant, and his wife Anne, née Sibley.
Gouger was the fifth child out of nine siblings born to George Gouger, a successful city trader, and his wife Anne Sibley.