Rock music was found to be particularly distracting for men.
The study, from Imperial College and the Royal College of Music, played music into headphones while people tried to play the Operation board game.
The research teams have been examining how music can alter performance.
Families might have arguments about whether teenagers should be listening to music when studying.
But there are also professional settings where music is sometimes played in the background - including operating theatres.
This study, published in the Medical Journal of Australia, used a different type of operation - the board game where players have to try to extract a piece with tweezers without setting off a buzzer.
The 350 people in the study were monitored as they carried out tasks while listening to pieces of music by Mozart and AC/DC and then the background sound of an operating theatre.
Women's ability to concentrate on the task did not seem to be affected by either types of music or the sound of the operating theatre.
But men were slower and more likely to make mistakes when they listened to rock music.
When men listened to Mozart, there appeared to be an improvement - but this seemed to be associated particularly with people who liked to listen to classical music - and when this was taken into account, there was no significant difference in performance.
The researchers do not have any clear explanation for why women should be less affected - but suggest it might be a greater susceptibility among men to "auditory stress" - where perception is affected by "loud or discordant music".
Lead author of the research Dr Daisy Fancourt said the study using the Operation board game was in some ways "tongue in cheek" but was part of wider research into how music could alter performance, including in settings such as operating theatres.
There were questions about whether different types of music could influence the speed and accuracy of how teams worked, she said.
"This study suggests that for men who are operating or playing a board game, rock music may be a bad idea," said Dr Fancourt.
Women are less likely to have their concentration disrupted by listening to music, says a study examining whether music impairs or enhances the ability to focus on a task.