After the Russian invasion in 1655, the state began to weaken, and that also affected the castle. In August 1655, Vilnius was captured by the Muscovite army. The Polish army recaptured the city six years later, by which time the palace had been destroyed by fire. The palace was greatly damaged and its treasures were plundered. After the recapture of the city of Vilnius in 1660–1661, the palace was no longer a suitable state residence, and stood abandoned for about 150 years. In the late 18th century, after the fall of the Polish–Lithuanian Commonwealth, several families lived in parts of the ruined palace. Soon after the Grand Duchy of Lithuania was incorporated into the Russian Empire, officials ordered the demolition of the remaining sections of the palace. The structure was almost completely demolished in 1801, the bricks and stones were sold, and the site was bowered.