In 1944 Fenner met Bobbie Roberts. Ellen Margaret 'Bobbie' Roberts was a trained midwife and nurse. During World War II she worked on malaria with the Australian Army Nursing Service and through her work met Fenner. Shortly after they met they married in a Catholic ceremony (despite Fenner being an atheist). While keen to have children, the couple were infertile and with Bobbie having a hysterectomy in 1949 they decided to adopt. They subsequently adopted two children. Marilyn Aldus Fenner was born on 27 June 1950. Her biological parents were unknown. Victoria Fenner (born 1 March 1943) was adopted later at the request of her father (Fenner's younger brother Tom) after the death in a fire of Victoria's mother (Beverley Slaney).