I think you should just talk to him about this and be honest with him about how it made you feel.
Ngakho ongakwenza yikuya kuye ufike ukhulume laye ngomusa umtshele ukuthi okwenzakeleyo kukuphathe njani.