During Barry's first posting abroad to Cape Town in South Africa, Barry became a close friend of the Governor, Lord Charles Somerset, and his family. It has been suggested that Lord Charles discovered Dr Barry's secret and that the relationship was more than friendship. Their closeness led to rumours and ultimately an accusation briefly appearing on a bridge post in Cape Town on 1 June 1824 saying that the writer had "detected Lord Charles buggering Dr Barry", which led to a court trial and investigation, as homosexuality was at that time strictly illegal. Despite these allegations, if Somerset was aware of Barry's sex, he did not reveal it.