It will replace the current GP surgery on Pen y Graig Road, Mayhill, which was deemed not fit for purpose in 2007, along with the family centre.
Swansea council said the hub would "transform health and social care for thousands".
The new building should open in October 2017.
In Mayhill every one in two children live in poverty, according to council statistics and Welsh Communities Secretary Carl Sargeant said the new centre would give children the "best start in life".
Surveys indicate the current surgery had a backlog maintenance figure of £127,000 in 2010, with the current 30-year-old family centre vulnerable to vandalism.
Plans for the joint health and family centre were first put forward in October 2012 but hit numerous delays due to funding uncertainties.
Councillors accepted a recent offer of £167,000 of extra grant funding from the Welsh Government for the scheme, on top of a £400,000 grant awarded back in 2012.
The project is being delivered by a partnership between Swansea council and Abertawe Bro Morgannwg University Health Board, who will fund the running of the GP surgery.
Council leader Rob Stewart said: "As well as providing maintained access to expert medical advice in a modern, spacious building, the new complex will also lead to a one-stop-shop for families, offering them advice on parenting, childcare and benefits," he said.
"The current family centre and GP surgery buildings in Mayhill are no longer fit for purpose, so I'm sure residents in that community will be thrilled that work will soon start on constructing the new, integrated complex for their benefit."
A £2m new health hub in one of the most deprived areas of Swansea has been given the go ahead eight years after it was marked as a "priority".