Does 'That's the second time you've made that sort of remark.' offer a contrasting position to 'That is the second occurrence of that sort of remark.'
That's the second time you've made that sort of remark.###That is the second occurrence of that sort of remark.