Abbe was born on 15 December 1915 in a village in Tokushima Prefecture, on the island of Shikoku, Japan. He was the son of Toshizo Abbe, who was the local school's headmaster and a kendo instructor, and his wife, Kote Abbe. The couple had four sons and five daughters, and Abbe was the youngest child in the family. On 4 September 1919, when Abbe was not yet four years old, his father drowned in a flash flood while training in the mountains. A young schoolteacher, Manpei Hino, subsequently became a father figure for the young boy and introduced him to the martial arts, including sumo. Abbe went on to become a local school champion in sumo.