In this study, experiments and finite element analysis (FEA) were used to evaluate the impact of interference-fit sizes on CFRP hybrid bonded-bolted (HBB) joint damage during bolt insertion. The specimens were designed in accordance with the ASTM D5961 standard and bolt insertion tests were performed at selected interference-fit sizes (0.4%, 0.6%, 0.8%, and 1%). Damage to composite laminates was predicted using the Shokrieh–Hashin criterion and Tan’s degradation rule via the user subroutine USDFLD, while damage to the adhesive layer was simulated by the Cohesive Zone Model (CZM). The corresponding bolt insertion tests were performed. The variation of insertion force with interference-fit size was discussed. The results showed that matrix compressive failure was the main failure mode. With the growth of the interference-fit size, more failure modes appeared, and the failure region expanded. Regarding the adhesive layer, it did not completely fail at the four interference-fit sizes. This paper will be helpful in designing composite joint structures and especially for understanding CFRP HBB joint damage and failure mechanisms.