His younger brothers were the sailor Charles Herbert and the botanist William Herbert. Another brother, Algernon Herbert was an antiquary.
Charles Herbert was the sailor among his younger brothers, while William Herbert studied botany. Algernon Herbert, another sibling, focused on antiquities.