I fancy he wants to give some of it back. It was possibly this remark which induced the driver to move away.
Perhaps the driver moved away because he wanted to give some of it back.
Even though he didn't want to give some of it back, the driver reluctantly moved away.