If they were built of wood, this meant they could be burnt down. Also wood rots when it gets wet so the structure needed regular repair. As time went by castles were made of stone, which is much stronger. Palisades were replaced by stone walls, and wooden tower were replaced in stone. 