At the time of the Norman invasion, the manor was held by Cnut, son of Karli. Afterwards the manor was in the possession of Count Alan of Brittany, who granted lordship to Geoffrey of Swaffham. By the 13th century, the manor was in the hands of the "Burgh" family of Hackforth. The manor descended with the manor of Hackforth until 1480, at which time they were conveyed to the Crown in the person of Richard, Duke of Gloucester. Some lands in the manor in the 13th century came into the hands of the lords of Middleham and then followed the descent of the manor of Thoralby.