It's subtitled "A Novel in the Form of a Memoir", because the letters and other personal documents are quoted by another character, Apley's biographer.
The novel is titled "A Novel in the Form of a Memoir" because it includes quoted letters and personal documents from Apley's life, which are narrated by another character, Apley's biographer.