I think he is trying to let you down easy.  He is probably just using it as an excuse to do his own thing.  I think he wants you to think "do i really want to be the one to hold my boyfriend back".  Think about it? Or maybe you are smothering him to much and he wants his space.