The effect of botanical origin, the flour particle size, and the content of damaged starch on flour pasting properties, dough behavior during a uniaxial deformation test, and bread characteristics were evaluated on rice and buckwheat flours. The rice flour with a median particle size D(0.5) of 60.2, 70.6, 106.8, and 189.4 μm, and buckwheat flour with a D(0.5) of 56.4, 68.4, and 95.8 μm were prepared using the same milling technology. The botanical origin of the flours was the strongest factor influencing the flour pasting properties, stress accumulated in dough during the uniaxial deformation test, loaf characteristics, texture, and sensory characteristics of breads. The flour particle size significantly influenced mainly the flour pasting properties. The effect of the content of damaged starch was the weakest among the studied factors. The flour particle size and the content of damaged starch were closely related. The flour botanical origin was the strongest factor; therefore, it seems not to be possible to predict the bread-baking potential of gluten-free flours based on the results obtained for flour of a different botanical origin. More research on flours from different plants prepared by the same milling process is required to support this hypothesis.