It's okay if things don't work out between you and a partner even if it might seem like something little to the other person.
You both understand that it may be something small to them but it's a dealbreaker for you.