Whittaker's exact birth date is not known. He was christened at Quarndon near Derby on 8 February 1813. His father was also named Joseph, and was a labourer who married to Sarah (born Clarke). The son is sometimes reported as being born in Breadsall in 1815.