Near the coast of China, a large amount of sediment is produced during construction work. In order to mitigate the environmental damage caused by sediment and enhance the performance of rubber-modified asphalt effectively, solidified silt material and waste rubber were prepared to modify asphalt, and its macroscopic properties, such as viscosity and chemical composition, were determined via a routine physical test, DSR, Fourier Transform Infrared Spectroscopy (FTIR), and Fluorescence Microscopy (FM). The results show that, with the increase in powder particles and the addition of a certain amount of hardened mud, the mixing and compaction temperature of modified asphalt can be significantly increased—still reaching the design standard. In addition, the high thermal stability and fatigue resistance of the modified asphalt were clearly better than those of the ordinary asphalt. From the FTIR analysis, rubber particles and hardened silt only exhibited mechanical agitation with the asphalt. Considering that excessive silt might result in the aggregation of matrix asphalt, the addition of an appropriate amount of hardened solidified silt material can eliminate the aggregation. Therefore, the performance of modified asphalt was optimum when solidified silt was added. Our research can provide an effective theoretical basis and reference values for the practical application of compound-modified asphalt. Therefore, 6%HCS(6:4)-CRMA have better performance. Compared to ordinary rubber-modified asphalt, the composite-modified asphalt binder has better physical properties and a more suitable construction temperature. The composite-modified asphalt uses discarded rubber and silt as raw materials, which can effectively protect the environment. Meanwhile, the modified asphalt has excellent rheological properties and fatigue resistance.