Due to its excellent mechanical properties and high aspect ratio, graphene can significantly improve the water and chloride ion permeability resistance of cementitious materials. However, few studies have investigated the effect of graphene size on the water and chloride ion permeability resistance of cementitious materials. The main issues are as follows: How do different sizes of graphene affect the water and chloride ion permeability resistance of cement-based materials, and by what means do they affect these properties? To address these issues, in this paper, two different sizes of graphene were used to prepare graphene dispersion, which was then mixed with cement to make graphene-reinforced cement-based materials. The permeability and microstructure of samples were investigated. Results show that the addition of graphene effectively improved both the water and chloride ion permeability resistance of cement-based materials significantly. The SEM (scanning electron microscope) images and XRD (X-ray diffraction) analysis show that the introduction of either type of graphene could effectively regulate the crystal size and morphology of hydration products and reduce the crystal size and the number of needle-like and rod-like hydration products. The main types of hydrated products are calcium hydroxide, ettringite, etc. The template effect of large-size graphene was more obvious, and a large number of regular flower-like cluster hydration products were formed, which made the structure of cement paste more compact and thus significantly improved the resistance to the penetration of water and chloride ions into the matrix of the concrete.