Nowadays, machining products, especially by turning methods, are more and more popular and require high-quality. With the development of science and technology, especially numerical computing technology and control technology, the application of these technological achievements to improve productivity and product quality has become increasingly essential. This study applies a simulation method considering the affecting factors of the vibration of the tool and the surface quality of the workpiece during turning. The study simulated and analyzed the characteristics of the cutting force and oscillation of the toolholder when stabilizing; at the same time, the study also simulated the behavior of the toolholder under the effect of cutting force and determined the finished surface quality through simulation. Additionally, the study utilized a machine learning model to examine the relationship between the toolholder length, cutting speed, feed rate, wavelength and surface roughness. The study found that tool hardness is the most crucial factor, and if the toolholder length exceeds the critical length, it leads to a rapid increase in roughness. In this study, the critical toolholder length was determined to be 60 mm, and this resulted in a corresponding surface roughness (Rz) of approximately 20 µm.