When he was on holiday, he went to the office to do some work.
You cannot go on holiday while you are in the office.
People usually go on holiday to a different place from where they work.
When a person is on holiday, they don't go to work.