Gilbert was born at 17 Southampton Street, Strand, London. His father, also named William, was briefly a naval surgeon, who later became a writer of novels and short stories, some of which his son illustrated. Gilbert's mother was the former Anne Mary Bye Morris (1812–1888), the daughter of Thomas Morris, an apothecary. Gilbert's parents were distant and stern, and he did not have a particularly close relationship with either of them. They quarrelled increasingly, and following the break-up of their marriage in 1876, his relationships with them, especially his mother, became even more strained. Gilbert had three younger sisters, two of whom were born outside England because of the family's travels during these years: Jane Morris (b. 1838 in Milan, Italy – 1906), who married Alfred Weigall, a miniatures painter; Mary Florence (b. 1843 in Boulogne, France – 1911); and Anne Maude (1845–1932). The younger two never married. Gilbert was nicknamed "Bab" as a baby, and then "Schwenck", after the surname of his great aunt and great uncle, who were also his father's godparents.