Daniel Skelton, 44, was hit with such force that he crumpled and fell to the ground, smashing his head on the pavement.
The High Court in Glasgow heard that he later died in hospital when his life support machine was switched off.
Daniel McGuire, 25, admitted the culpable homicide of Mr Skelton in Queensbury Street on 1 August.
He also pled guilty to two further assaults on the same night.
Judge Lord Bannatyne deferred sentence on McGuire, who has two previous convictions for assault, and ordered a background report on him.
As he was led to the cells a member of his victim's family shouted at him: "You've ruined our family."
The court was told that Mr Skelton, from Bellshill, Lanarkshire, was staying in the area while working for a construction company building the new Dumfries hospital.
He spent the night of 31 July drinking in the Hole-in-the-Wa' pub with his friend, Gary Wilson.
After midnight they came across McGuire, of Dumfries, and his friends and chatted to them.
Advocate depute Alan Nicol, prosecuting, said: "A conversation began between the accused and Mr Skelton.
"There are varying accounts as to precisely what happened next but it would seem likely that prior to the assault, there was some discussion between them.
"However, following a comment by Mr Skelton, the accused took offence, lost his temper and punched Mr Skelton a single blow to the head.
"Mr Skelton was seen to crumple before striking his head on the ground."
The court heard that when Mr Wilson approached McGuire and asked why he had hit his friend, he too was punched.
McGuire then walked away and went to the N'Dulge pub, in St Andrew's Street, where he punched Aidan Shuttleworth, who was drunk, and left him knocked out on the floor.
The court heard that Mr Skelton was transferred to the Western General Hospital in Edinburgh where he underwent an emergency operation for a blood clot on his brain.
He was placed on a life support machine, but a CT scan on August 6 revealed that he was brain dead.
Mr Nicol said: "The family of Mr Skelton are naturally devastated at his untimely death."
A father-of-four died after a night out in Dumfries when he was hit with a single punch.