It happened at Coed Cae, Pontardawe, Neath Port Talbot, at about 18:20 GMT on Saturday.
South Wales Police said children in the house at the time. It is thought at least one of the gang had a knife.
Det Insp Phil Sparrow said: "This is a concerning incident which has left the victims traumatised and shocked."
An unknown number of men walked into the house and demanded the family's car keys but they put up a fight so they did not get them.
The masked man with the weapon is described as being black with a slight build. He spoke with a soft English accent which was possibly from Birmingham.
He was wearing a dark hooded top or jacket, a ski mask which covered his face except for his eyes and black gloves.
The family raised the alarm and neighbours called the police. No-one was seriously hurt.
"They put up an extremely brave resistance which I am in no doubt caused the men to make off without the car they were looking for," said DI Sparrow.
Meanwhile, officers in Swansea are investigating a break-in at a flat in Cromwell Street at about 10:30 on Saturday when two men attacked the occupants with what is thought to be a baseball bat.
One was wearing a grey hooded top and the other was wearing a brown hooded jacket.
Officers believe there was a third man driving a blue Ford Focus who dropped them in Rhondda Street before the attack.
They stole two mobile phone. Nobody was seriously hurt. Police are not linking the cases.
A knife-wielding gang threatened a family in their own home before stealing a mobile phone and cash.