Joel B. Mayes was one of Samuel Mayes's many sons. His elder brothers were George Washington Mayes (1824–1894), John Thompson Mayes (1826–1863) and Francis Asbury Mayes (1833–1863). Both the latter accompanied their father driving cattle to California circa 1852. While John Thompson Mayes returned with his father to Indian Territory, Francis Asbury Mayes remained in California for several years, but when returning to Oklahoma in 1863, was reported ambushed by hostile tribesman and killed. The eldest brother, Wash Mayes, would serve as high sheriff of the Cherokee Nation for many years. A younger brother Samuel Houston Mayes (1845–1927), also became a principal chief of the Cherokee Nation, and the namesake of Mayes County, Oklahoma. Other younger brothers were William Henry ("Tip") Mayes (1842–1918) and Wiley B. Mayes (1848–1934).