Readers and commenters may be quick to judge and condemn, just because it doesn't follow the conventional expectations of a happy or acceptable relationship.  Don't listen to them.  From my past experience in a similar situation, and from reading your story, you are just fine.  As long as it enriches your life and all parties agree, let it be and enjoy it to the fullest.