Throw out the baby with the bath water
A slightly different explanation focuses on the loss of the essential while holding on to what is not essential.  In other words, the idiom also makes sense when someone might throw out the baby and keep the bath water. Idioms are a common stumbling block for learners of a language.