She had much correspondence with Elizabeth Fry about the female convicts, and did what she could to ameliorate their lot. In 1841 the convict ship "Rajah" arrived loaded with convict women who had been supplied with sewing materials organised by Lydia Irving of Fry's convict ship committee. The quilt is now one of the most treasured textiles in Australia. She was accused of using undue influence with her husband in his official acts but there is no evidence of this. When Franklin was recalled at the end of 1843, they went first to Melbourne by the schooner "Flying Fish" and then to England by way of New Zealand on board, coincidentally, the barque "Rajah".