The castle was held by William's son, also called William, from 1088 to 1138, and then by his son, another William, until his death in 1147. Conisbrough and the earldom then passed through Isabel, William's daughter, to her first husband, William de Blois, and then on to her second husband, Hamelin Plantagenet, whom she married in 1163. Hamelin was the illegitimate half-brother of King Henry II, who had arranged the marriage, and the union brought him great wealth. Hamelin extensively rebuilt the castle around 1180 to 1190, including constructing the stone keep; given his "parvenu" status, he probably hoped to reinforce perceptions of his new elevated rank. King John visited the castle in 1201.