The town was in the center of a conflict between Grand Duke Jogaila (later to become King of Poland) with his uncle Kęstutis. In 1382 Jogaila's and Kęstutis's armies met near Trakai, but Jogaila tricked Kęstutis and imprisoned him in Kreva. A few weeks later Kęstutis died in captivity and Jogaila transferred the castles to his brother Skirgaila, who became the governor of Lithuania Proper. However, his rule was briefly interrupted when in 1383 joint forces of Kęstutis's son Vytautas and the Teutonic Knights captured the town. In 1392, Vytautas and Jogaila signed the Astrava Agreement ending their quarrel. Vytautas became the Grand Duke of Lithuania while Jogaila technically remained his superior. Vytautas also regained his father's lands, including Trakai. Despite his official capital being in Vilnius, Vytautas spent more time in Trakai. In early 15th century he replaced the older, wooden fortress with a stone-built castle. Some design elements were borrowed from the castles of the Teutonic Knights as Vytautas spent some time with the Teutons forming an alliance against Jogaila in earlier years.