Eliza Rosanna Gilbert was born into an Anglo-Irish family, the daughter of Elizabeth ("Eliza") Oliver, who was the daughter of Charles Silver Oliver, a former High Sheriff of Cork and member of Parliament for Kilmallock in County Limerick, Ireland. Their residence was Castle Oliver. In December 1818, Eliza's parents, Ensign Edward Gilbert and Eliza Oliver, met when he arrived with the 25th Regiment. They were married on 29 April 1820, and Lola was born the following February, in the village of Grange in the north of County Sligo, refuting persistent rumours that her mother was pregnant with her at the time of the wedding. The young family made their residence at King House in Boyle, County Roscommon, until early 1823, when they journeyed to Liverpool, England, and later departed for India on 14 March.