Given the low accuracy of the traditional digital image correlation (DIC) method in complex deformation measurement, a color DIC method is proposed using a prism camera. Compared to the Bayer camera, the Prism camera can capture color images with three channels of real information. In this paper, a prism camera is used to collect color images. Relying on the rich information of three channels, the classic gray image matching algorithm is improved based on the color speckle image. Considering the change of light intensity of three channels before and after deformation, the matching algorithm merging subsets on three channels of a color image is deduced, including integer-pixel matching, sub-pixel matching, and initial value estimation of light intensity. The advantage of this method in measuring nonlinear deformation is verified by numerical simulation. Finally, it is applied to the cylinder compression experiment. This method can also be combined with stereo vision to measure complex shapes by projecting color speckle patterns.