Morgan was born in Perth, Western Australia in 1951 as the eldest of five children. She was raised by her mother Gladys and her maternal grandmother Daisy. Her mother, a member of the Bailgu people of the Pilbara region of Western Australia, grew up in the Parkerville Children's Home as part of the Stolen Generations. Her father, William, a plumber by trade, died after a long-term battle with post-war experience post-traumatic stress disorder. Of her siblings, Jill Milroy is an academic, Helen Milroy is a child psychiatrist who was the first indigenous Australian to become a medical doctor, David is a playwright, and William has worked as a senior public servant.