Polysaccharides can be used as fat replacers in ice cream, as they contribute to an increase of viscosity. However, no research has clarified the exact role of viscosity from that of the structure of the polysaccharides on the properties of ice cream. In this study, the effect of polysaccharide structure on different properties of low-fat ice cream was investigated. The polysaccharides taken into consideration varied from flexible (locust bean gum and guar gum) to rigid (xanthan gum and iota carrageenan). Relationships between rheological properties of ice cream mixes and microstructural characteristics and sensory perception of the final ice cream were established. To separate the effect of the polysaccharide structure from that of viscosity, two series of ice cream were prepared: one in which the mix viscosity of the various samples was similar (approximately 68.3 mPa· s), and one in which the serum phase viscosity was similar (approximately 15563 mPa· s). Flexible polysaccharides showed a lower degree of shear-thinning and a more liquid-like viscoelastic behavior compared with rigid polysaccharides. In addition, flexible polysaccharides led to higher overrun (47–58%) than other samples (approximately 30%), which resulted in lower hardness of the ice cream (<3.2 MPa). Rigid polysaccharides caused gelation of the serum phase, which made the ice cream more difficult to scoop. Based on the results of the sensory evaluation, flexible polysaccharides could provide higher softness and creaminess-related properties, while rigid polysaccharides resulted in higher coldness and grittiness. Therefore, polysaccharides with a flexible structure are a better choice for improving the textural and sensory properties of low-fat ice cream.