Dwyer was the fifth child born to his parents. He had three older brothers, Brian, Kenny and Peter, and an older sister, Maureen. His siblings were significantly older than Dwyer and his brothers had already moved out of the family home by the time he was born. Of his three brothers, Kenny died at the age of 36 from diabetes, while Peter joined the armed forces. His other brother, Brian, was a keen sportsman and played football at amateur level for local side St Patrick's. He also represented Wales in boxing at the Empire Games.