After resigning as Prime Minister, Takahashi still retained the position of president of the "Rikken Seiyūkai". He resigned his seat in the House of Peers in 1924, and was elected to a seat in the Lower House of the Diet of Japan in the 1924 General Election. When Katō Takaaki became the prime minister and set up a coalition cabinet in 1924, Takahashi accepted the post of Minister of Agriculture and Commerce. He divided the department into the Ministry of Agriculture and Forestry and the Ministry of Commerce and Industry. Takahashi resigned from the "Rikken Seiyūkai" in 1925.