- The authors have reproduced a well-known result in the literature--left political bias in ChatGPT and in LLMs in general--using the "coarse" (their description) methodology of passing a binary stance classifier over ChatGPT's output. The observation that language models reproduce the biases of the corpora on which they're trained has been made at each step of the evolution of these models, from word2vec to BERT to ChatGPT, and so it's unclear why this observation needs to once again be made using the authors "coarse" methodology.
- The authors' decision to filter by divisive topic using introduces an unacknowledged prior: for most of the topics given in the appendix (immigration, the death penalty, etc.), the choice to bother writing about the topic itself introduces bias. This choice will be reflected in both the frequency with which such articles appear and in the language used in those articles. The existence of the death penalty, for example, might be considered unproblematic on the right and, for that reason, one will see fewer articles on the subject in right-leaning newspapers. When they do appear, neutral language will be employed. The opposite is likely true for a left-leaning publication, for whom the existence of the death penalty is a problem: the articles will occur with more frequency and will employ less neutral language. For these reasons, it's not surprising that the authors' classifier, which was annotated via distant supervision using political slant tags assigned by Wikipedia, will tend to score most content generated by an LLM as left-leaning.
- The authors have reproduced a well-known result in the literature--left political bias in ChatGPT and in LLMs in general--using the "coarse" (their description) methodology of passing a binary stance classifier over ChatGPT's output. The observation that language models reproduce the biases of the corpora on which they're trained has been made at each step of the evolution of these models, from word2vec to BERT to ChatGPT, and so it's unclear why this observation needs to once again be made using the authors "coarse" methodology.