In 1992, he joined Lazard Frères as managing partner and quickly became one of the firm's star bankers and heir-apparent. He tried to reduce overhead and bring in younger partners but clashed with Michel David-Weill, the bank's head and his father-in-law. He quit Lazard Frères in 1997 and set up his own investment fund, Investment Real Returns (IRR). He owned half and the remainder was held by Eurazeo, a Lazard holding company and Mainz Holdings Ltd., a U.S. Virgin Islands firm that Stern wholly owned. He maintained cordial relations with David-Weill, who invested $300 million in IRR.