- A number of claims from this paper would benefit from more in-depth analysis.
- There are still some methodological flaws that should be addressed.
### Main questions/comments Looking at the attached dataset files, I cannot work out whether the data is noisy or if I don't understand the format. The 7th example in the test set has three parts separated by [SEP] which I thought corresponded to the headlines from the three sides (left, center, right). However, the second and third headlines don't make sense as stand-alone texts. Especially the third one which states "Finally, borrowers will receive relief." seems like a continuation of the previous statements. In addition to the previous question, I cannot find the titles, any meta-information as stated in lines 187-189, nor VAD scores which I assumed would be included.
Part of the motivation is that scaling the production of neutral (all-sides) summaries is difficult. It would be good to quantify this if possible, as a counterargument to that would be that not all stories are noteworthy enough to require such treatment (and not all stories will appear on all sides).
Allsides.com sometimes includes more than one source from the same side -- e.g. https://www.allsides.com/story/jan-6-panel-reportedly-finds-gaps-trump-white-house-phone-records has two stories from Center publishers (CNBC and Reuters) and none from the right. Since the inputs are always one from each side (Section 3.2), are such stories filtered out of the dataset? Of the two major insights that form the basis of this work (polarity is a proxy for framing bias and titles are good indicators of framing bias) only first one is empirically tested with the human evaluation presented in Section 5.1.3. Even then, we are missing any form of analysis of disagreements or low correlation cases that would help solidify the argument. The only evidence we have for the second insight are the results from the NeuS-Title system (compared to the NeuSFT model that doesn't explicitly look at the titles), but again, the comparison is not systematic enough (e.g. no ablation study) to give us concrete evidence to the validity of the claim.
Related to the previous point, it's not clear what the case study mentioned in Section 4.2 actually involved. The insights gathered aren't particularly difficult to arrive at by reading the related literature and the examples in Table 1, while indicative of the arguments don't seem causally critical. It would be good to have more details about this study and how it drove the decisions in the rest of the paper. In particular, I would like to know if there were any counterexamples to the main points (e.g. are there titles that aren't representative of the type of bias displayed in the main article?).
The example in Table 3 shows that the lexicon-based approach (VAD dataset) suffers from lack of context sensitivity (the word "close" in this example is just a marker of proximity). This is a counterpoint to the advantages of such approaches presented in Section 5.1.1 and it would be interesting to quantify it (e.g. by looking at the human-annotated data from Section 5.1.3) beyond the safeguard introduced in the second paragraph (metric calibration).
For the NeuS-Title model, does that order of the input matter? It would be interesting to rerun the same evaluation with different permutations (e.g. center first, or right first). Is there a risk of running into the token limit of the encoder?
From the examples shared in Table 3, it appears that both the NeuSFT and NeuS-Title models stay close to a single target article. What strikes me as odd is that neither chose the center headline (the former is basically copying the Right headline, the latter has done some paraphrasing both mostly based on the Left headline). Is there a particular reason for this? Is the training objective discouraging true multi-headline summarisation since the partisan headlines will always contain more biased information/tokens?
### Minor issues I was slightly confused by the word "headline" to refer to the summary of the article. I think of headlines and titles as fundamentally the same thing: short (one sentence) high-level descriptions of the article to come. It would be helpful to refer to longer text as a summary or a "headline roundup" as allsides.com calls it. Also there is a discrepancy between the input to the NeuS-Title model (lines 479-486) and the output shown in Table 3 (HEADLINE vs ARTICLE).
Some citations have issues. Allsides.com is cited as (all, 2021) and (Sides, 2018); the year is missing for the citations on lines 110 and 369; Entman (1993) and (2002) are seemingly the same citation (and the 1993 is missing any publication details); various capitatlisation errors (e.g. "us" instead of "US" on line 716) It's not clear what the highlighting in Table 1 shows (it is implicitly mentioned later in the text). I would recommend colour-coding the different types of bias and providing the details in the caption.
- A number of claims from this paper would benefit from more in-depth analysis.