After the Grand Duchy of Lithuania joined the Kingdom of Poland to form the Polish–Lithuanian Commonwealth in 1569, the castles remained a royal property, but the town's importance gradually declined, with the nearby Vilnius and the political center of the Commonwealth in Kraków being far more important. Nevertheless, it continued to be the seat of local Sejmik. In Polish sources the town name was started to be referred to as "Troki". In 1477, the castle on the lake was a meeting place of King Casimir IV with Venetian envoys. After that the castle became a luxurious prison for political prisoners. Sigismund I the Old imprisoned the members of Goštautai family, believed to be conspiring with Michael Glinski. Also Helena, widow of King Alexander was kept there in order to prevent her escape to the Grand Duchy of Moscow. The castle was refurbished by King Sigismund I the Old, who set up his summer residence there; however, after his death in 1548 the castle gradually fell into disrepair.