Hogg was the daughter of Sarah Ann "Sallie" Stinson and James Stephen "Big Jim" Hogg, later attorney general and governor of the state. Ima Hogg's first name was taken from "The Fate of Marvin", an epic poem written by her uncle Thomas Hogg. She endeavored to downplay her unusual name by signing her first name illegibly and having her stationery printed with "I. Hogg" or "Miss Hogg". Although it was rumored that Hogg had a sister named "Ura Hogg", she had only brothers. Hogg's father left public office in 1895, and soon after, her mother was diagnosed with tuberculosis. When Sarah died later that year, Jim Hogg's widowed elder sister moved to Austin to care for the Hogg children. Between 1899 and 1901, Hogg attended the University of Texas at Austin; she then moved to New York City to study piano and music theory for two years. After her father's death in 1906, she traveled to Europe and spent two years studying music under Xaver Scharwenka in Vienna. When she returned to Texas, she established and managed the Houston Symphony Orchestra and served as president of the Symphony Society.