The Djabugay people (particularly the Yirrgay group) believe it is the resting place of Gudju Gudju, a Rainbow Serpent The discovery of gold on the mainland west of the area assured that Double Island became well known in the 1880s and 1890s and it is said that, around this time, a mining company at Irvinebank used the island as a resort for employees.
The island had been known to, and used by, the local Australian Aborigines (predecessors of the Djabugay people), for many tens of thousands of years.