Hell, you could say the guy in the pew in front is "turning his back on you."  That's nutty.  

The priest facing the altar in the Latin Mass was not interpreted as turning his back on the people.  "Don't turn your back on me!" never would have been the peoples' response.  A wacky few who didn't know the Mass might have responded that way, maybe, but in general, it was not perceived as a social slight.  Nor was I aware that I was excluded until post Vat II liturgists told me I was.  Pre Vat II, I never thought the Mass was about me; it was about all of us --- including the priest -- facing God.  That the priest was the main figure -- the "celebrant" --- among us humans didn't bother me.  I knew my place.  In that time and place --- the Holy Sacrifice of the Mass --- the priest was certainly above me.  Maybe certain Vat II liturgical changes were spurred by the sense that the laity didn't like the priest's being "above" them.  If so, the changes were spurred by spoiled children.