Ferrary started collecting in his youth, and then he inherited a great fortune of approximately 120,000,000 French francs (£5 million), which he dedicated to the purchase of rare stamps and coins. His collection is believed to have been the greatest ever assembled, and it may never be equalled. Amongst his extremely rare stamps were the unique Treskilling Yellow of Sweden and the 1856 one-cent "Black on Magenta" of British Guiana, which he bought in 1878 for £150 and which after his death was sold at the third bid of his collection, in 1924, at Paris for 36,000 US dollars. He also owned the only unused copy of the Two Cent Hawaii Missionary of 1851, for which its owner, Gaston Leroux, had been murdered by a fellow collector. Another piece owned by Ferrary was the only known cover featuring both values of the first Mauritius "Post Office" stamps, which has been called "the greatest item in all philately".