The presence of manufacture error in large mirrors introduces high-order aberrations, which can severely influence the intensity distribution of point spread function. Therefore, high-resolution phase diversity wavefront sensing is usually needed. However, high-resolution phase diversity wavefront sensing is restricted with the problem of low efficiency and stagnation. This paper proposes a fast high-resolution phase diversity method with limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithm, which can accurately detect aberrations in the presence of high-order aberrations. An analytical gradient of the objective function for phase-diversity is integrated into the framework of the L-BFGS nonlinear optimization algorithm. L-BFGS algorithm is specifically suitable for high-resolution wavefront sensing where a large phase matrix is optimized. The performance of phase diversity with L-BFGS is compared to other iterative method through simulations and a real experiment. This work contributes to fast high-resolution image-based wavefront sensing with a high robustness.