The couple moved into a very basic ground-floor apartment in London, on Clarendon Road, Notting Hill. Given how hard it was to find decent housing, they decided instead to share with Helge Rubinstein and her husband, the literary agent Hilary Rubinstein, who at the time was working for his uncle, Victor Gollancz. In 1955 the four of them bought a four-storey, seven-bedroom house in Phillimore Place, Kensington, for £6,800, a home they lived in together for 14 years. Williams described it as one of the happiest periods of his life.