Dwyer was born on 28 October 1953 in the house where his family resided on Penarth Road in Grangetown, Cardiff. The area was in the midst of an economic downturn following the end of the Second World War and a falling demand for coal, one of the main industries of the region, which also affected the local docks. Dwyer's family home, which he described as having "tiny living quarters", had no bathroom, meaning the family bathed in front of the fire in a metal bath with water boiled on the stove. The house also had no central heating leading to "very harsh" nights in winter. His parents, Ted and Constance, had met and married in Tonypandy where his father had worked as a miner. In the hope of improving their prospects, the pair moved to Cardiff where his father found work at a foundry in Tremorfa. His mother also took up work in the canteen of an aluminium manufacturing factory.