Ferrary's mother, the Duchess of Galliera, born Maria de Brignole-Sale, was the great-niece of the Princess of Monaco and daughter of the Marquis Antoine de Brignole-Sale, ambassador of the Kingdom of Sardinia in Paris, under the Restoration and during the reign of Louis-Philippe. After the death of Ferrary's father, the Duchess proposed that Philippe, Count of Paris (heir apparent to the French throne) take up residence at the Rue de Varenne. He came to occupy the ground floor of the Hôtel Matignon. The Duchess soon became disenchanted with the adverse social environment for the monarchists, quit Paris, and left Hôtel Matignon to the Austro-Hungarian Emperor, who made it his embassy in France. Upon the death of his father, Ferrary renounced all of the titles. He was then adopted by the Austrian Count de La Renotière von Kriegsfeld and he adopted Austrian nationality. It is said that Ferrary was illegitimate and that he was adopted by his natural father. Thereafter, he preferred the name "Ferrary"; his calling card reads "Philipp von Ferrary". Collectors and dealers usually refer to him simply as "Ferrary".