The fire covers more than 22,000 acres and residents of about 1,500 homes near Santa Clarita have been evacuated.
A body has been found in a burned-out car but it is not clear if the death is fire-related.
Strong winds have fanned the flames and blown smoke and ash across Los Angeles.
"It started consuming houses that were non-defendable," said Los Angeles County Deputy Fire Chief John Tripp.
He said the flames had charged through the terrain "like a freight train".
The blaze, known as the Sand Fire, broke out on Friday in the Sand Canyon area near Santa Clarita, on the edge of the Angeles National Forest.
It came as southern California was gripped by a heatwave.
"It's not a one-direction type of fire,'' said Nathan Judy, a spokesman for the Angeles National Forest.
"It's going in different directions depending on which way the wind is blowing.''
Sand Canyon resident Juliet Kinikin said there was "panic" as the sky darkened with smoke and flames moved closer to her home on Saturday.
She and her family fled their house but were able to return on Sunday "breathing a big sigh of relief".
A wildlife preserve owned by actress Tippi Hedren, who starred in the 1981 film Roar, has also been evacuated.  Hendren's daughter Melanie Griffith tweeted that her mother and the 40 big cats at The Shambala Preserve were safe.
About 900 firefighters are battling the flames, helped by helicopters and aeroplanes dumping water and fire retardant.
Forecasters say there could be some relief in sight with cooler temperatures due to move in.
Another fire has also broken out further north in mountains near Big Sur. More than 1,000 homes have been evacuated in the communities of Palo Colorado and Carmel Highlands, California's forestry department said.
Fire crews in California are struggling to contain a wildfire that has destroyed 18 homes and threatens hundreds of others in mountains north of Los Angeles.