Barker visited Smart often in London where she worked. She became pregnant again, and was fired from the Ministry of Information. Their affair produced two more children (Sebastian, born 1945, and Rose Emma, born 1947). Through it all Barker, who was Catholic, said he would leave his wife for Smart, but this never happened (he was to have fifteen children by several different women). They lived a bohemian lifestyle and associated with many of the 'Soho' artists. Christopher Barker writing in the Guardian about this period noted: "On many occasions through the early Sixties, writers and painters such as David Gascoyne, Paddy Kavanagh, Roberts MacBryde and Colquhoun and Paddy Swift lived downstairs from Smart and his wife, Agnes, wrote cookbooks with Smart would gather at Westbourne Terrace in Paddington, our family home at that time. They came for editorial discussions about their poetry magazine, "X"."