Rylsk was first mentioned in a chronicle in 1152 as one of the Severian towns. It had become the seat of an appanage principality by the end of the 12th century before coming into the hands of Lithuanian rulers sometime in the 14th century. The Polish king Casimir IV made a grant of it to Dmitry Shemyaka's son Ivan, who had settled in Lithuania. Ivan's son Vasily defected to the Grand Duchy of Moscow, but Lithuanians held the town until 1522.