His mother was Janetta Aylward (known as Netta), an actress whom Roger met in Paris; the two returned to London together. They had an intermittent relationship, and three years later, in 1895, she gave birth to a son, Peter, then Joe a year later, and Nancy in 1899. According to Joe's maternal Aunt Bunny, Peter's birth, and likely all of them, were "accidents." She told him, "Your father happened to have run out of French letters that day," (when Peter was conceived). His father set up a household with his mother starting in 1903, after which the children saw him more regularly. His business did very well, and the family had a "butler, a gardener, and, evidently, a very good table."