Ullmann was born in Tokyo, Japan, to Norwegian parents. The family moved to Toronto when she was two. Her father was an engineer and after his death in 1946, Ullmann returned to Norway with her mother and older sister, where they settled in Trondheim. 