It can also be presumed that it was left out because there was a place, to which they should naturally return (for instance, the neighborhood of the Emperor's Palace).
当然帰るべき所（例：天皇の宮の周囲）が決められていたので、省略されたと推測するか。