More SOTU Scaling

A couple of days ago the Monkey Cage featured Ben Lauderdale's one-dimensional scaling model of US State of the Union addresses. In this post, I replicate the analysis with a closely related model, ask what the scaled dimension actually means, and consider what things would look like if we added another one.



First the replication materials. Historical State of the Union addresses are available in machine readable form from the American Presidency Project. Like Ben, I'm only going to go back to 29 years to 1986. From these addresses I construct a cross-tabulation of words and the Addresses they were used in, trim it a bit, and scale it using correspondence analysis (CA).

If you've listened to almost any talk I've given in the last two years (or maybe read this) you'll know that CA is, among other things, a least-squares approximation to the text scaling model known to political scientists as Wordfish. Ben fitted a 'variant of Wordfish' so we should expect very much the same results here. And we get them:




CA models are cheap to fit and we've just seen that they recover what a more intensive Wordfishing approach would give us, so let's go a bit further and ask what would happen if we fitted a model with more than one dimension. For identification reasons CA requires that the second dimension be orthogonal to the first, so it's sufficient to show how the Addresses would line up in this second dimension:


and just for completeness, here they are plotted together:



Distinctive vocabulary





energi, solar, children, entrepreneur, profit, job, 
busi, parent, manufactur, fact, consum,



"Coca cola, sometimes war"


Each address at the American Presidency Project gets its own web page so it is the work of a moment to curl the ones we want onto a local disk. In each web page the president's words are wrapped in a span of class "displaytext", so we can pull them out using some XPath and collapse all the paragraphs together, because text scaling methods ignore all that sort of thing anyway.


## parse the web page
doc <- htmlTreeParse(filename, useInternalNodes=T)
## extract just the speech bits using xpath
## merge all the paragraphs together
## remove references to applause





Creative Commons License