First, are these critics aware of what the definition of "fiction" actually describes?  It's 'made up' stories or art or music.  

If I am writing about actual lived experiences and try to assume that I can write a first-person account of a Black Woman living in the USA under Trump, I have every expectation to be vilified for my misguided attempt at writing about something I have no first-hand experience with.  But if I tried to imagine that situation in fiction, I might get laughed at or find there is no audience for my rather implausible story.  

However, I do believe the problem is not cultural appropriation but financial appropriation -- when some old white guy writes better stories about a culture than someone from that culture, the problem is not the appropriation but that being of a culture or race or gender or whatever doesn't necessarily give you a privileged place to write from and worse, is no guarantee that it makes you a good or interesting writer.