The name of the town means "white" in Russian, although it is unknown how or why this name originated. The fortress of Bely is first mentioned in a chronicle in 1350, since it was conquered by the Grand Duchy of Lithuania. It was located on the border between Lithuanian and Russian lands, and intermittently changed affiliation between Lithuania (later Poland) and the Grand Duchy of Moscow. In the 15th century, it became the seat of the Belsky branch of the ruling House of Gediminas. The town was overrun by the Grand Duchy of Moscow in 1503. Three years later, Muscovites built a formidable castle, which the Lithuanians laid a siege to in 1508. The town was again subordinated to the Polish–Lithuanian Commonwealth between 1618 and 1654, after which it finally went under Moscow.