The Harem housed the private quarters of the sultan, his mother, and his many wives and concubines.
The sultan and his mother lived in the Harem.  
The sultan did not live in the harem with his mother.  