The SAGA (Spt-Ada-Gcn5 acetyltransferase) complex is a transcriptional co-activator that both acetylates and deubiquitinates histones. The histone acetyltransferase (HAT) subunit, Gcn5, is part of a subcomplex of SAGA called the HAT module. A minimal HAT module complex containing Gcn5 bound to Ada2 and Ada3 is required for full Gcn5 activity on nucleosomes. Deletion studies have suggested that the Ada2 SWIRM domain plays a role in tethering the HAT module to the remainder of SAGA. While recent cryo-EM studies have resolved the structure of the core of the SAGA complex, the HAT module subunits and molecular details of its interactions with the SAGA core could not be resolved. Here we show that the SWIRM domain is required for incorporation of the HAT module into the yeast SAGA complex, but not the ADA complex, a distinct six-protein acetyltransferase complex that includes the SAGA HAT module proteins. In the isolated Gcn5/Ada2/Ada3 HAT module, deletion of the SWIRM domain modestly increased activity but had negligible effect on nucleosome binding. Loss of the HAT module due to deletion of the SWIRM domain decreases the H2B deubiquitinating activity of SAGA, indicating a role for the HAT module in regulating SAGA DUB module activity. A model of the HAT module created with Alphafold Multimer provides insights into the structural basis for our biochemical data, as well as prior deletion studies.