Freeson's seat was renamed Brent East in 1974, and he returned as Minister for Housing and Construction in the new Department of the Environment after the February 1974 general election, in a period of high interest rates and rapidly rising house prices. He later added responsibility for new towns, planning, land and local government to his portfolio. He retained his ministerial office when James Callaghan succeeded Harold Wilson as Prime Minister in 1976, becoming a privy counsellor that year, and retained his office until Labour's defeat at the 1979 general election. He remained on the Labour frontbench in opposition, as spokesman on Health and Social Security, but was demoted by Michael Foot in 1981. He later served on the Environment Select committee.