After 1919, Ponsonby-Fane became a permanent resident of Japan. In later life, he was always photographed with a long woolen scarf. It was hand-knit by Dowager Empress Teimei, the widow of Emperor Taishō.