Real-time hybrid testing (RTH) is a test method for dynamic loading performance evaluation of structures, which is divided into digital simulation and physical testing, but the integration of the two may lead to problems such as time lag, large errors, and slow response time. The electro-hydraulic servo displacement system, as the transmission system of the physical test structure, directly affects the operational performance of RTH. Improving the performance of the electro-hydraulic servo displacement control system has become the key to solving the problem of RTH. In this paper, the FF-PSO-PID algorithm is proposed to control the electro-hydraulic servo system in real-time hybrid testing (RTH), which uses the PSO algorithm to operate the optimized PID parameters and the feed-forward compensation algorithm to compensate the displacement. First, the mathematical model of the electro-hydraulic displacement servo system in RTH is presented and the actual parameters are determined. Then, the objective evaluation function of the PSO algorithm is proposed to optimize the PID parameters in the context of RTH operation, and a displacement feed-forward compensation algorithm is added for theoretical study. To verify the effectiveness of the method, joint simulations were performed in Matlab/Simulink to compare and test FF-PSO-PID, PSO-PID, and conventional PID (PID) under different input signals. The results show that the proposed FF-PSO-PID algorithm effectively improves the accuracy and response speed of the electro-hydraulic servo displacement system and solves the problems of RTH time lag, large error, and slow response.