Gorgs are people much bigger than Fraggles. They call their house "the castle" and the rest of their land "the universe." The show never says whether they really are rulers or whether they are poor farmers who like to pretend they are rulers.