In 1366, Red Ruthenia, of which Hrubieszów, then called Rubieszow, was a part, was annexed by the Kingdom of Poland. Some time in the late 14th century, a wooden castle was built here, as a residence of a local governor. Probably in 1400 Rubieszow received a town charter from Poland's king Władysław II Jagiełło, who visited it in 1411, 1413 and 1430. A castle and church were later added. Casimir IV ordered the construction of a route from Lublin to Lviv passing through Rubieszow. The town was destroyed several times by Crimean Tatars, who raided this area in the 15th, 16th and 17th centuries, and by the rebellious Cossacks.