Gandon had married Eleanor Smullen in 1770; sadly, he was widowed shortly after his invitation to Dublin, but while they were in London the couple had six children. James Gandon died in 1823 at his home in Lucan, County Dublin, having spent forty-two years in the city. He was buried in the church-yard of Drumcondra Church, in the same grave as his friend the antiquary Francis Grose. It seems that already by the time of his death his reputation was undergoing a re-evaluation, for his tomb-stone reads: – ""Such was the respect in which Gandon was held by his neighbours and friends from around his home in Lucan that they refused carriages and walked the 16 miles to and from Drumcondra on the day of his funeral"."