Griffiths' defeat of Gordon Walker resulted in Harold Wilson claiming in the House of Commons that Griffiths should "serve his term here as a parliamentary leper". Conservatives urged the Speaker, Harry Hylton-Foster, to force Wilson to withdraw the comment. While the Speaker objected to such language, he refused to censure the Prime Minister, and order in the Commons chamber was not restored for ten minutes. In his maiden speech in the Commons, Griffiths pointed out the problems faced by local industry and drew attention to the fact that 4,000 families were awaiting local authority accommodation. Griffiths remained an alderman in Smethwick until 1966. He both supported and arranged for Smethwick council to purchase a row of houses with the intention of letting them exclusively to white families. The government's Housing minister, Richard Crossman, was able to block this proposal by refusing the council permission to borrow the money required.