The family lived in the mill until 1820, when they moved into a brick house that had been built nearby.
The passage describes a change of residence for a family, moving from a mill dwelling to a brick house.