The blaze broke out at a home on Llangwm estate, near Penplas, in Penlan, just after midnight on Monday.
The house was initially too unsafe to search, but the body of the woman who died - named locally as Linda Merron - has now been moved out of the property.
A man and woman were taken to the city's Morriston Hospital with burns and smoke inhalation. A pet dog was also rescued.
Floral tributes have been left on the front gate by neighbours.
The cause of the blaze is being investigated by police and Mid and West Wales Fire and Rescue Service.
Neighbour Gina Norwood said the woman who lived at the house had been in poor health and had moved to the area within the last two years after her daughter came to study in the city.
"She loved her animals, her dogs - she was a lovely lady. She always had time for you, to say hello to all her neighbours," she told BBC Wales.
"My son woke me up screaming to get out of the house. I looked out of the window and I thought the flat was on fire.
"My son came out of the front and it was all blowing out, the flames."
Ms Norwood said she called the fire service initially and later an ambulance to treat the woman's daughter, whom she said had received burns to her body.
Her son Karl said he had heard a series of bangs which alerted him to the fire just after midnight.
A body has been recovered after a fire at a house in Swansea.