Does 'I was old enough at the time that I knew what was going on.' offer a contrasting position to 'By that age, I knew what was happening.'
I was old enough at the time that I knew what was going on.###By that age, I knew what was happening.