Crabbe was born in Aldeburgh, Suffolk, the eldest child of George Crabbe Sr. The elder George Crabbe had been a teacher at a village school in Orford, Suffolk, and later in Norton, near Loddon, Norfolk; he later became a tax collector for salt duties, a position that his own father had held. As a young man he married an older widow named Craddock, who became the mother of his six children: George, his brothers Robert, John, and William, his sister Mary, and another sister who died as an infant.