In February 1790, Latrobe married Lydia Sellon, and they lived a busy social life in London. The couple had a daughter (Lydia Sellon Latrobe) and a son (Henry Sellon Boneval Latrobe), before she died giving birth during November 1793. Lydia had inherited her father's wealth, which in turn was to be left to the children through a trust with the children's uncles, but never ended up going to the children. In 1795, Latrobe suffered a breakdown and decided to emigrate to America, departing on November 25 aboard the "Eliza".