Tinnitus is one of the main hearing impairments often associated with pure-tone hearing loss, and typically manifested in the perception of phantom sounds. Nevertheless, tinnitus has traditionally been studied in isolation without necessarily considering auditory ghosting and hearing loss as part of the same syndrome. Hence, in the present neuroanatomical study, we attempted to pave the way toward a better understanding of the tinnitus syndrome, and compared two groups of almost perfectly matched individuals with (TIHL) and without (NTHL) pure-tone tinnitus, but both characterized by pure-tone hearing loss. The two groups were homogenized in terms of sample size, age, gender, handedness, education, and hearing loss. Furthermore, since the assessment of pure-tone hearing thresholds alone is not sufficient to describe the full spectrum of hearing abilities, the two groups were also harmonized for supra-threshold hearing estimates which were collected using temporal compression, frequency selectivity und speech-in-noise tasks. Regions-of-interest (ROI) analyses based on key brain structures identified in previous neuroimaging studies showed that the TIHL group exhibited increased cortical volume (CV) and surface area (CSA) of the right supramarginal gyrus and posterior planum temporale (PT) as well as CSA of the left middle-anterior part of the superior temporal sulcus (STS). The TIHL group also demonstrated larger volumes of the left amygdala and of the left head and body of the hippocampus. Notably, vertex-wise multiple linear regression analyses additionally brought to light that CSA of a specific cluster, which was located in the left middle-anterior part of the STS and overlapped with the one found to be significant in the between-group analyses, was positively associated with tinnitus distress level. Furthermore, distress also positively correlated with CSA of gray matter vertices in the right dorsal prefrontal cortex and the right posterior STS, whereas tinnitus duration was positively associated with CSA and CV of the right angular gyrus (AG) and posterior part of the STS. These results provide new insights into the critical gray matter architecture of the tinnitus syndrome matrix responsible for the emergence, maintenance and distress of auditory phantom sensations.