n 16 April 1804, he was married to Grace Purves, eldest daughter of Sir Alexander Purves, 5th Baronet, of Purves, by his second wife Mary Home, daughter of Sir James Home, 3rd Baronet of Coldingham. Their children included advocate and geologist David Milne-Home and Admiral of the Fleet Sir Alexander Milne, 1st Baronet. His wife died at Bordeaux, in France, on 4 October 1814, where she had gone for the recovery of her health. The admiral brought her body back to Scotland, and she was buried at Inveresk in East Lothian on 8 February 1815.