In the mid-twelfth century, Count of Hainaut Baldwin IV "the builder" surrounded the town of Quesnoy with ditches and ramparts and also built in 1150 the castle which became an important the center of the fortifications of the town (now the "Centre Cernay" and the fire station). This castle had a tower which together with the rest make up a fortress. Alice of Namur, wife of Baldwin IV endowed the castle with a chapel dedicated to St. John the Baptist. The castle had a park called "Bois du Gard" in which encountered deer, fallow deer and wild game. The park extended to the southeast (to Beaudignies and the edge of it is met with a mill near wetlands known as "the Pond du Gard". Desiring to populate his new fortified town, the Count enacted in 1161 a charter granting privileges to many people: the town prospered and there embraced a ... Mayor, aldermen, men of fiefs, (lawyers), a hostel, a hospital and outside, a leper to accommodate lepers (the disease of leprosy had been reported by the Crusaders from the East). Baldwin and his wife were still living, according to the scrolls, in 1169 in Le Quesnoy. The son of the Count (later Baldwin V, Count of Hainaut) married the said year 1169 in Le Quesnoy Margaret of Alsace, sister of Thierry of Alsace Count of Flanders: the wedding was gorgeous and the Holy Roman Emperor Frederick Barbarossa was present in person.