Carey was the son of Francis Carey, a bricklayer, who came from Celbridge to Dublin, where his son was born in James Street in 1845. James also became a bricklayer, and for 18 years continued in the employment of Michael Meade, builder, of Dublin; he then started business on his own account as a builder in that city, at Denzille Street. In this venture he was successful; he became the leading spokesman of his trade, and obtained several large building contracts.