X-ray phase-contrast imaging offers enhanced sensitivity for weakly-attenuating materials, such as breast and brain tissue, but has yet to be widely implemented clinically due to high coherence requirements and expensive x-ray optics. Speckle-based phase contrast imaging has been proposed as an affordable and simple alternative; however, obtaining high-quality phase-contrast images requires accurate tracking of sample-induced speckle pattern modulations. This study introduced a convolutional neural network to accurately retrieve sub-pixel displacement fields from pairs of reference (i.e., without sample) and sample images for speckle tracking. Speckle patterns were generated utilizing an in-house wave-optical simulation tool. These images were then randomly deformed and attenuated to generate training and testing datasets. The performance of the model was evaluated and compared against conventional speckle tracking algorithms: zero-normalized cross-correlation and unified modulated pattern analysis. We demonstrate improved accuracy (1.7 times better than conventional speckle tracking), bias (2.6 times), and spatial resolution (2.3 times), as well as noise robustness, window size independence, and computational efficiency. In addition, the model was validated with a simulated geometric phantom. Thus, in this study, we propose a novel convolutional-neural-network-based speckle-tracking method with enhanced performance and robustness that offers improved alternative tracking while further expanding the potential applications of speckle-based phase contrast imaging.