During the 1810s in Italy Kozlovsky married Giovanetta Rebora from Milan who bore him a son Charles, and in 1817 a daughter, Sophie (Sofka) Koslowska (died in 1878). Sophie was in close contact with the French novelist Honoré de Balzac, who followed Kozlovsky as the lover of Frances Sarah Lovell (1804–1883), wife of Count Emilio Guidoboni-Visconti. One sister, Maria Borisovna (1788–1851), a poet in her own right, was the mother of the composer Alexander Sergeyevich Dargomyzhsky (Александр Сергеевич Даргомыжский), another, Daria Borisovna, was married to the poet, translator and state councillor Michail Sergeyevich Kaisarov (1780–1825, Михаил Сергеевич Кайсаров)