He had an elder brother, James Butler, 5th Earl of Ormond, and a younger brother, Thomas Butler, 7th Earl of Ormond, as well as two sisters, Elizabeth Butler, who married John Talbot, 2nd Earl of Shrewsbury, and Anne Butler (d. 4 January 1435), who was contracted to marry Thomas FitzGerald, 7th Earl of Desmond, although the marriage appears not to have taken place.
The younger sibling of the subject had an older brother, James Butler, 5th Earl of Ormond, and a younger brother, Thomas Butler, 7th Earl of Ormond, in addition to two sisters, Elizabeth Butler, who married John Talbot, 2nd Earl of Shrewsbury, and Anne Butler (d. 4 January 1435), who was betrothed to Thomas FitzGerald, 7th Earl of Desmond.