Buster Crabbe grew up in Hawaii. He started swimming on the beach at Waikīkī as a young boy. When his father became the manager of a pineapple plantation on Oahu, the whole family moved away from the beach. Buster and his brother had to stop swimming. A year after the move, Mr. Crabbe got a real estate job in Hilo, Hawaii. The family moved to Hilo, and the boys started swimming again when their father joined a yacht club. With all the moving about, it was not easy for the Crabbe boys to go to school. Buster was eight and a half years old when he joined the first grade in Hilo Union School.