Gouger was the fifth son of nine children of George Gouger (1763–1802), who was a prosperous city merchant, and his wife Anne, née Sibley.
Gouger was the fifth son of nine children of George Gouger (1763–1802), who was a wealthy urban trader, and his wife Anne, née Sibley.