Phased transducer arrays (PTA) can control ultrasonic waves to produce a holographic acoustic field. However, obtaining the phase of the corresponding PTA from a given holographic acoustic field is an inverse propagation problem, which is a mathematically unsolvable nonlinear system. Most of the existing methods use iterative methods, which are complex and time-consuming. To better solve this problem, this paper proposed a novel method based on deep learning to reconstruct the holographic sound field from PTA. For the imbalance and randomness of the focal point distribution in the holographic acoustic field, we constructed a novel neural network structure incorporating attention mechanisms to focus on useful focal point information in the holographic sound field. The results showed that the transducer phase distribution obtained from the neural network fully supports the PTA to generate the corresponding holographic sound field, and the simulated holographic sound field can be reconstructed with high efficiency and quality. The method proposed in this paper has the advantage of real-time performance that is difficult to achieve by traditional iterative methods and has the advantage of higher accuracy compared with the novel AcousNet methods.