I don't know, really. In 1977  the NP got 64.8% of the white vote after which its popularity declined as the popularity of far right parties increased. So it seems like there was a least a majority that did support it based on those numbers. If one was white and had the opportunity to vote for another party but chose either not to or not vote at all does that not make one complicit? So, maybe he's not talking about you in which case there's no need to take offense. But some humility might not hurt, apartheid being a crime against huminity and all.