There were as many gold diggers as the other people of Inari put together.
The statement suggests that the number of people in Inari who are motivated by material gain is equivalent to or surpasses the entire population.