The fourth son of nine children of Thomas Henry Baker (1824–1904), J.P., of Owletts, a gentleman farmer and director of the Kent Fire and Life Insurance Company, by his wife Frances Georgina (née Davis), Herbert was from the outset exposed to a tradition of good craftsmanship, preserved through isolation in the neighbourhood of his home in Kent. As a boy, walking and exploring the historical ruins found in the area were his favourite pastimes. Here he observed and learned to appreciate the time-honoured materials of brick and plaster, and the various aspects of timber use, especially in roof construction—tie-beam and arch-braced collar-beam trusses. He was profoundly influenced by the stone construction used in Norman cathedrals and Anglo-Saxon churches, as well as the ornamentation and symbolism of the Renaissance buildings in Kent. This early influence is apparent in the churches, schools and houses he later designed in South Africa.