The current territory of the town was permanently settled around the 13th century. The first written reference of settlement comes from 1263 and is included in the Doksany Prior's purchase roll. The history of the town is associated with Chvala ze Žitavy of the Ronovci family, who founded Lipý Castle, and his grandson Henry of Lipá (Jindřich z Lipé) (1270–1329), a significant royal aristocrat. As rolls have not been preserved, it is not clear who exactly founded the castle and surrounding settlement. Thus Lipý Castle became another fortified seat in North Bohemia on the contemporary trade routes. There was a Slavonic colony near the castle, later renamed as Stará Lípa (now a neighbourhood of the town). There is a reference to Arnold, said to come from Stará Lípa, who was a citizen of Kravaře in 1263. Historians have deduced from the reference to Stará Lípa that Lipý Castle and accompanying settlement were established around that time. Henry of Lipá moved to Moravia in 1319, after he had sold the castle with its surroundings to his cousin Hynek Berka. According to other sources, in 1327 Hynek Berka z Dobé, also a member of the Ronovci ancestry, was the lord of Lipá and its wider surroundings. When he died in 1348, his son of the same name took the title, and after his death his second son Jindřich succeeded him. Then his nephew Hynek Berka z Dubé reigned the area. The eldest town charter, which he issued on 23 March 1381, states that discretions were granted to the town of Lipá and that it was he who contributed the most to the boom of the town and the castle bearing the same name.