( Vanity Fair reports that Peter actually was abandoned by his mother, too; he was raised by his grandmother during Harrer's absence.
Peter was actually abandoned by his mother too, he was raised by his grandmother during Harrer's absence, as reported by Vanity Fair.
His grandmother never raised Peter, because she was too old.