The palace, much altered over the generations, was generally used as a residence only at times of dynastic importance or danger, when Holyrood, down in the lowland, was difficult to defend.
The palace was only a house when there was a threat.
The palace was always the primary residence.