Presumably in the 13th century, a rich family from Bruges received the monopoly to levy taxes on gruit, and built a storage for it. The building was changed in the early fifteenth century by Jan IV van der Aa to a luxury house for his family, which subsequently changes its name to "Van Gruuthuse" ("From the Gruit house"). His son Louis de Gruuthuse adds a second wing to the house, and in 1472 a chapel. This connects the house to the adjacent Church of Our Lady, Bruges.