Is 'He said to me, You have to go, he said to me.' a fitting logical lead-up to 'He didn't say anything to me about having to go.'
He said to me, You have to go, he said to me.###He didn't say anything to me about having to go.