Anonymous Group Challenges Statistical Validity of Fireside Report




The article’s many examples include:




Who are you?




Another comment:



Bear in mind this quote from the Fireside Report —






62 thoughts on “Anonymous Group Challenges Statistical Validity of Fireside Report

  1. Greg Hullender:

    “No, the SFF editors are not making it worse–not to a significant degree, anyway. If there were massive, systematic discrimination, both studies would have detected it. They did not.”

    And if discrimination existed and was not massive or not systematic? Should it then be ignored? Everything is allright then?

    And do you think that study is good enough to prove that no discrimination exists that will affect people from minorities, that the data is not skewed in any way depending on who has answered?

  2. Greg Hullender: If there were massive, systematic discrimination, both studies would have detected it. They did not. More subtle (or just less-effective) discrimination might still exist, but it would take more data to detect it.

    And you are missing my point again; at this point, I can only presume that this is deliberate on your part.

    Your first sentence here is a false claim. The second sentence is the root of the problem.

    The original study, while of interest for numerous reasons, is massively unsuited (as Fireside admits) for drawing any sort of concrete conclusions, and certainly not the claims of “fact” you have been making.

    How many minority authors have submitted short SFF fiction to genre venues? How many pieces did they submit? What is the actual rejection rate? There is no way to know. The Fireside report is based on data from the ones they knew about and those they could identify through outreach. It is hardly surprising that they would know about published minority genre authors to contact — after all, they got published, so they were noticed. The non-published ones? not so much.

    You are making utterly unsubstantiable claims based on evidence which does not — can not, by its very insufficiency — support those claims.

    Just stop it.

  3. Greg Hullender on March 19, 2017 at 6:58 pm said:

    Although a number of people have claimed that the article isn’t valid, none of them showed any evidence of having read or understood the document.

    I’ve found it very difficult to go through mainly because it has a very odd relationship (to put it politely and assuming no malice on Bronstein et al’s part) with the original report.

    So with the first section when it states “The report assumes that 13.2% of speculative fiction submissions come from black authors,” I could recall from the last time I’d read the report that the ‘assumption’ was purely for the purpose of using a binomial distribution as a comparison rather than FR more generally assuming that this was (or should be) the case. OK, maybe a poor use of “assumes” there but I’m just on the first proper section and I’m already having to compare what the Bronstein report is saying versus what FR actually says.

    The section on Tokenisation is similar. It leads with a quote from FR which implies it was a conclusion of the report. It is actually just a comment on the FR data spreadsheet explaining why they listed the magazines in order.
    Full quote from FR “Technically, this list overall would provide a more accurate presentation if it was ordered by the percentage of stories by black authors per
    zine rather than the number — 1 of 7 means something far different than 1 of 58. However, we decided to put them in order by number because we are seeing far too many “exactly ones,” which is suggestive of tokenization.”

    Just two sections in (where I was when I began asking the authors questions about the study on medium) and I was finding that the Bronstein report appeared to be misreading the FR report in a way I found odd – as in it appeared to be presenting things in FR as conclusions or core assumptions by FR and attempting to show those conclusions were false when they were not actually conclusions of FR.

    I didn’t ask a question about the tokenisation bit because I wanted to work through what they had done first (then everything went kablooey). I didn’t assume their approach for testing tokenisation via the FR data was invalid but I was confused by what they were attempting to do.

    Now, maybe it was just really poor framing on their part. Assume the analysis of tokenization is correct* – if so then if they had presented what they were doing as using the FR data to investigate the issue that would have made more sense rather than an apparent attempt to identify errors.

    Bronstein has stated in their introduction that “At the same time we find the report to be fraught with error.” and two sections in I was seeing stuff that was not uninteresting but which did not support the idea that FR was ‘fraught with error’.

    So two sections in and the Bronstein report looked like it was building up and knocking down strawmen rather than engaging with the FR report.

    I didn’t look at the section on Terraform in any depth. That left the criticism of FR using a binomial distribution as a comparison.

    As things progressed ‘missing the point’ began to look like to charitable an interpretation of the Bronstein report.

    *[I suspect it is flawed because it tests the hypothesis using the same total number of stories by black authors against the same number of magazines, whereas if tokenism was not present we’d expect either more stories or less stories by black authors I think- but I haven’t done any numbers on what difference that would make]

  4. I’m generally not inclined to give the benefit of the doubt concerning the motivations of people who are unwilling to publicly stand behind their alleged analysis.

  5. Camestros Felapton on March 19, 2017 at 8:12 pm said:

    OK – less worried about that now. It just looked like an odd result that the proportion didn’t seem to matter.

  6. Pingback: Rebuttal of the 2016 Cecily Kane Fireside Article | Lela E. Buis

  7. This is a couple of days late and I guess will be buried, but given that I have some math background, I feel like writing a comment.

    0. I welcome the archival of the pseudonym report by various sources including Fireside.

    1. The pseudonym report is correct to point out that it’s practically impossible to determine on which level the discrimination is happening when we have data points only the baseline population of black authors and the end results (underrepresentation of PoC authors in publications). To determine to which extent the discrimination done by editors, as everyone has pointed out, one really would need slush pile statistics.

    Of course, their choice of baseline from statistics of professional writers is somewhat arbitrary, too, but it does has a point: the number demonstrates that it’s very well possible that bulk of the systemic racism filtering happens far before any texts end up on someone’s desk in a publishing house. Both numbers are poor proxies for the real base rate and thus not worth much, except for illustrative purposes (that choice of base rate matters in these models).

    It’s also possible that maybe there are dynamical relationships: there are less professional black authors than proportion of black people in general population would suggest because black authors have more difficult time getting published combined with other reasons (for example, rate of literary hobby). Rejection rate in publishing industry might have an effect on the distribution of demographics of authors the submitting stories. But estimating such models (especially the contribution of systemic racism in publishing vs other reasons) is difficult with only this kind of data.

    2. A charitable interpretation is that some editors felt like they were unduly attacked of being racists by the Fireside report and the commentary that followed it. It’s kind of reasonable that someone in the publishing might feel like they were being accused of racist publication policy, when the conclusion of the Fireside report is “Speculative short fiction publishing is rife with antiblackness, and white speculative fiction writers and publishers need to stop pretending otherwise”.

    3. As an aside, I think it’s worthwhile to notice that numbers do not tell us the who is doing the discrimination and why. Someone above in the comments pointed out the some of the discrimination could be caused by simply because of the differences in the life experience: the editor thinks that in their personal opinion, a story written by a PoC person is “not interesting” because it’s about stuff they can’t relate to. Which is both sad, and a reasonable response from the editor, which makes the situation doubly sad. It is also very problematic, especially if all the editors in the industry are from the same social segment and thus PoC authors writing about PoC experiences find it more difficult to get published than they should, so it should be fought against.

    But at the same time I’d find it is terrible to accuse publishers of racism on these grounds. The data at hand does not answer which kind of filtering effects – reasonable or unreasonable – are taking place. On the other hand, it’s certainly possible that there’s happening something more disgusting like, say, throwing manuscripts into rubbish bin because of the name of the author on the first page does not sound “white” enough. Or (most likely) combination of different effects. But we don’t know, at least with this information alone.

    4. I think pseudonym was also correct to point out that the data is consistent with Poisson process instead of ‘tokenism’. However, curious thing is that when I Ctrl-F’d “token” in the original report on the Medium website I didn’t find anything. No mention of tokenism. Where did the claim of tokenism originate?

    5. As a math person I’m inclined like pseudonym report more because it elaborates on the statistical methodology more. Like the multiple hypothesis part about Terraform or the effect of established authors writing most of the stuff being published and having easier time having their stories published (instead of i.i.d. story model).

    Pseudonym’s act of taking the report down was certainly weird. Both charitable and uncharitable interpretations are possible. However, we do not gain more knowledge by attacking the results of the pseudonym by accusing that their motivations for writing the report (whatever they are) might have been racist, if the contents of the claims itself are correct. Given the accusations of trolling and worse, I sort of feel sympathetic to their decision to remain anonymous.

  8. a.n.: Given the accusations of trolling and worse, I sort of feel sympathetic to their decision to remain anonymous.

    Given that the so-called rebuttal spent a considerable amount of verbiage countering claims that the Fireside report did not actually make, I think that the accusations of trolling are fair. There is a serious amount of strawmanning going on there.

    I am rarely inclined to give the benefit of the doubt to people who engage in strawman arguments — and the fact that they did so anonymously just makes them look even worse.

  9. I don’t find anonymity inherently suspicious. Lots of people use varying degrees of pseudonymity depending on the situation; it doesn’t automatically imply they’re up to no good.

    I do find arguing with statements not made off-putting. To me, it implies that someone is arguing with an assumption rather than the people actually involved. When posting a rebuttal on an emotive topic it helps to double-check to make sure that isn’t what you’re doing.

  10. Pingback: Pixel Scroll 3/24/17 No, Mr. Bond, I Expect You To Scroll | File 770

  11. Pingback: Loose-leaf Links #36 | Earl Grey Editing

Comments are closed.