He was the son of Andrzej Poniatowski, the brother of the last king of Poland Stanislaus II Augustus (born Stanisław Poniatowski), and a field marshal in the service of Austria. His mother was Countess Maria Theresia Kinsky von Wchinitz und Tettau, a lady in the court of Maria Theresa belonging to the influential Czech-Austrian aristocratic family. His father died when Józef was ten, Stanislaus Augustus then became his guardian and the two enjoyed a close personal relationship that lasted for the rest of their lives.