Brooke's family were dogged by scandal. His sister, Elizabeth Brooke, was married to Sir Thomas Wyatt but lived openly in adultery with another man. Allegedly she attracted the attention of Henry VIII in 1542, and Eustace Chapuys, the Imperial ambassador, thought that had she tried she could have become Henry's sixth wife. His daughter, Elisabeth Brooke, Marchioness of Northampton, was also prone to scandal as from 1543 she had lived with her future husband William Parr, 1st Marquess of Northampton whilst he was separated from his adulterous wife Anne Bourchier, 7th Baroness Bourchier. They eventually married during the reign of Edward VI, but this was declared invalid by Mary I. In the reign of Elizabeth I, their marriage was finally confirmed as valid.