that was during the latter days of the occupation , but they 'd stayed on in japan through '54 while lieutenant colonel richardson served as adviser for the rearming of what would be the japanese self defense forces .