The original inhabitants of James Bay were the Swenghwung people who were part of the Lekwungen people of the Coast Salish and whose descendants today are known as the Songhees First Nation. Even after the Indigenous inhabitants allegedly sold the land to the Hudson's Bay Company, remains of fortifications at Holland Point and of burial grounds at Laurel Point remained. The neighbourhood takes its name from the shallow inlet James Bay that forms part of Victoria's Inner Harbour, named for James Douglas. Settled early after the establishment of Fort Victoria in 1843, much of the present day neighbourhood was originally part of Ogden's Fields Farms, subsequently known as Dutnall's Farm and then Beckley Farm.