Cox was born on 29 April 1783 on Heath Mill Lane in Deritend, then an industrial suburb of Birmingham. His father was a blacksmith and whitesmith about whom little is known, except that he supplied components such as bayonets and barrels to the Birmingham gun trade. Cox's mother was the daughter of a farmer and miller from Small Heath to the east of Birmingham. Early biographers record that "she had had a better education than his father, and was a woman of superior intelligence and force of character." Cox was initially expected to follow his father into the metal trade and take over his forge, but his lack of physical strength led his family to seek opportunities for him to develop his interest in art, which is said to have first become apparent when the young Cox started painting paper kites while recovering from a broken leg.