it never bothered hugh because he saw james all the time , and so he didnt think anything of him being in our house , or phoning up , even of the two of us going out for the evening if hugh was late at work .