In this investigation, purified attapulgite (ATT) and polyvinyl alcohol (PVA) were utilized to fabricate nanocomposite hydrogels and a xerogel, with a focus on studying the impact of minor additions of ATT on the properties of the PVA nanocomposite hydrogels and xerogel. The findings demonstrated that at a concentration of 0.75% ATT, the water content and gel fraction of the PVA nanocomposite hydrogel reached their peak. Conversely, the nanocomposite xerogel with 0.75% ATT reduced its swelling and porosity to the minimum. SEM and EDS analyses revealed that when the ATT concentration was at or below 0.5%, nano-sized ATT could be evenly distributed in the PVA nanocomposite xerogel. However, when the concentration of ATT rose to 0.75% or higher, the ATT began to aggregate, resulting in a decrease in porous structure and the disruption of certain 3D porous continuous structures. The XRD analysis further affirmed that at an ATT concentration of 0.75% or higher, a distinct ATT peak emerged in the PVA nanocomposite xerogel. It was observed that as the content of ATT increased, the concavity and convexity of the xerogel surface, as well as the surface roughness, decreased. The results also confirmed that the ATT was evenly distributed in the PVA, and a combination of hydrogen bonds and ether bonds resulted in a more stable gel structure. The tensile properties exhibited that when compared with pure PVA hydrogel, the maximum tensile strength and elongation at break were achieved at an ATT concentration of 0.5%, indicating increases of 23.0% and 11.8%, respectively. The FTIR analysis results showed that the ATT and PVA could generate an ether bond, further confirming that ATT could enhance the PVA properties. The TGA analysis showed that the thermal degradation temperature peaked when the ATT concentration was at 0.5%, providing further evidence that the compactness of the nanocomposite hydrogel and the dispersion of the nanofiller was superior, contributing to a substantial increase in the mechanical properties of the nanocomposite hydrogel. Finally, the dye adsorption results displayed a significant rise in dye removal efficiency for methylene blue with the increase in the ATT concentration. At an ATT concentration of 1%, the removal efficiency rose by 103% compared with that of the pure PVA xerogel.