The coefficient of static friction between solids normally increases with the time they have remained in static contact before the measurement. This phenomenon, known as frictional aging, is at the origin of the difference between static and dynamic friction coefficients but has remained difficult to understand. It is usually attributed to a slow expansion of the area of atomic contact as the interface changes under pressure. This is however challenging to quantify as surfaces have roughness at all length scales. In addition, friction is not always proportional to the contact area. Here we show that the normalized stress relaxation of the surface asperities during frictional contact with a hard substrate is the same as that of the bulk material, regardless of the asperities’ size or degree of compression. This result enables us to predict the frictional aging of rough interfaces based on the bulk material properties of two typical polymers: polypropylene and polytetrafluoroethylene.