Device-to-device (D2D) communication is a promising wireless communication technology which can effectively reduce the traffic load of the base station and improve the spectral efficiency. The application of intelligent reflective surfaces (IRS) in D2D communication systems can further improve the throughput, but the problem of interference suppression becomes more complex and challenging due to the introduction of new links. Therefore, how to perform effective and low-complexity optimal radio resource allocation is still a problem to be solved in IRS-assisted D2D communication systems. To this end, a low-complexity power and phase shift joint optimization algorithm based on particle swarm optimization is proposed in this paper. First, a multivariable joint optimization problem for the uplink cellular network with IRS-assisted D2D communication is established, where multiple DUEs are allowed to share a CUE’s sub-channel. However, the proposed problem considering the joint optimization of power and phase shift, with the objective of maximizing the system sum rate and the constraints of the minimum user signal-to-interference-plus-noise ratio (SINR), is a non-convex non-linear model and is hard to solve. Different from the existing work, instead of decomposing this optimization problem into two sub-problems and optimizing the two variables separately, we jointly optimize them based on Particle Swarm Optimization (PSO). Then, a fitness function with a penalty term is established, and a penalty value priority update scheme is designed for discrete phase shift optimization variables and continuous power optimization variables. Finally, the performance analysis and simulation results show that the proposed algorithm is close to the iterative algorithm in terms of sum rate, but lower in power consumption. In particular, when the number of D2D users is four, the power consumption is reduced by 20%. In addition, compared with PSO and distributed PSO, the sum rate of the proposed algorithm increases by about 10.2% and 38.3%, respectively, when the number of D2D users is four.