Milne was married to Rachel Cowie, daughter of Charles Cowie, Esq. of Aberdeen, at St. Leonard's on 4 August 1812. They soon departed from Portsmouth for the Cape of Good Hope on 4 September 1812. The Milnes stayed at the Cape of Good Hope with John Campbell for a few days, and they did not arrive in Macau until 4 July 1813. Milne, with his wife and infant child were expelled by the Roman Catholic priests there after three days, and he left for Guangzhou, where he was able to begin study of the Chinese language. His observation regarding the difficulty of an English-speaker acquiring the Chinese tongue has been frequently repeated: