The political geography of human development

The research I did for the previous post on the inadequacy of the widely-used term ‘Global South’ led me to some surprising results about the political geography of development.

Although the relationship between latitude and human development is not linear, distance from the equator turned out to have a rather strong, although far from deterministic and not necessarily causal, link with a country’s development level, as measured by its Human Development Index (HDI). Even more remarkably, once we include indicators (dummy variables) for islands and landlocked countries, and interactions between these and distance from the equator, we can account for more than 55% of the variance in HDI (2017). In other words, with three simple geographic variables and their interactions we can ‘explain’ more than half of the variation in the level of development of all countries in the world today. Wow! The plot below (pdf) shows these relationships.



In case you are wondering whether this results is driven by many small counties with tiny populations, it is not, When we run a weighted linear regression with population size as the weight, the adjusted R-squared of the model remains still (just above) 0.50. On a sidenote, including dummies for (former) communist countries and current European Union (EU) member states pushed the R-squared above 0.60. Communist regime or legacy is associated with significantly lower HDI, net of the geographic variables, and EU membership is associated with significantly higher HDI.

The next question to consider is whether the relationship between geography and development has grown weaker or stronger over time. There are many plausible ideas we might have about the influence of globalization, the spread of information and communication technologies, wars, and financial crises on the links between geography and development. When we look at the data, however, it turns out that the strength of the link has remained roughly the same since 1990. Wow! Despite of all global social and political transformations over the past 30 years, geography still play the same, rather larger role in constraining and enabling human development. The gif below shows the same plots for 1990, 2000, 2010, and 2017. While overal development grows over time, the relationship with distance from the equator remains roughly the same, as indicated by the slopes of the linear regression lines.



Note that the way the HDI is constructed (HDI) makes changes in development over time not quite comparable (the index is capped at 1.0, so if you are an already highly developed country, there is not much scope to improve further your index). Also, the sample of countries for which there is available data is smaller in 1990 (N=144) than in 2017 (N=191).

Since we mentioned population size, let’s consider the link between the population size of a country and its level of HDI. Are small countries more successful? Does it pay off to be a large state? Maybe countries with populations that are neither too big nor too small perform best?

As the plot below (pdf) shows, there is no clear relationship between population size and HDI. The linear regression line slopes slightly downwards but the ‘effect’ is not significant and it is not really linear. The loess fit meanders up and down without a clear pattern. It turns out there is no sweet spot for population size when it comes to human development. Small populations can be just as good, and just as bad, and bigger ones. There are tiny states that are successful, and ones that do pretty badly. The same for mid-sized, big, and enormous countries (not in terms of area, but population).



This lack of relationship is quite remarkable, but there is another surprise when we look at the change in development between 2000 and 2017. As the plot below (pdf) shows, more populous countries have been more successful in improving their HDI over the past 18 years. It is not a huge difference, but given the overall small scale of the observed changes, it is significant and important.



To sum up, while in general population size is not related to development, during the past two decades more populous countries have been more successful in improving their development index. This is of course good news, as it means that more people live longer, study longer, and enjoy higher standards of living.

For now, this concludes my exploits in political geography, which turned out to harbor more insights that I expected, even when I have only explored a total of five variables. If you want to continue from here on your own, the R script for the figures is here and the datafile is here.

Books on data visualization

Here is a compilation of new and classic books on data visualization:


Scott Murray (2017) Interactive Data Visualization for the Web 

Elijah Meeks (2017) D3.Js in Action: Data Visualization with JavaScript 

Alberto Cairo (2016) The Truthful Art: Data, Charts, and Maps for Communication 

Andy Kirk (2016) Data Visualization 

David McCandless (2014) Knowledge is Beautiful 


Edward Tufte (2006) Beautiful Evidence  

Edward Tufte (2001) The Visual Display of Quantitative Information 

Edward Tufte (1997) Visual Explanations: Images and Quantities, Evidence and Narrative 

Edward Tufte (1990) Envisioning Information 

Olympic medals, economic power and population size





Olympic medals and economic power



Olympic medals and population size



Putting everything together




Model 1 Model 2
Population size – 0.20 – 0.09
GDP + 0.04 + 0.03
GDP squared – 0.00000008 /
GDP*Population / -0.0000008
Sum of errors 1691 1678
Adjusted R-squared 0.83 0.81






country 2016 medals 2016 medal points predicted medal points model error
Great Britain 67 221 68 153
Russia 56 168 87 81
Australia 29 83 30 53
France 42 118 68 50
Kenya 13 49 0 49
New Zealand 18 52 4 48
Hungary 15 53 6 47
Netherlands 19 65 22 43
Jamaica 11 41 0 41
Croatia 10 36 2 34
Cuba 11 35 2 33
Azerbaijan 18 36 4 32
Germany 42 130 98 32
Uzbekistan 13 33 2 31
Italy 28 84 54 30
Kazakhstan 17 39 10 29
Denmark 15 35 7 28
Ukraine 11 29 5 24
Serbia 8 24 2 22
North Korea 7 21 0 21
Sweden 11 31 12 19
Belarus 9 21 4 17
Ethiopia 8 16 0 16
Georgia 7 17 1 16
South Korea 21 63 47 16
China 70 210 195 15
South Africa 10 30 15 15
Armenia 4 14 0 14
Greece 6 20 7 13
Slovakia 4 16 4 12
Spain 17 53 41 12
Colombia 8 24 14 10
Czech Republic 10 18 8 10
Slovenia 4 12 2 10
Switzerland 7 23 13 10
Bahamas 2 6 0 6
Bahrain 2 8 2 6
Ivory Coast 2 6 0 6
Belgium 6 18 13 5
Fiji 1 5 0 5
Kosovo 1 5 0 5
Tajikistan 1 5 0 5
Lithuania 4 6 2 4
Burundi 1 3 0 3
Grenada 1 3 0 3
Jordan 1 5 2 3
Mongolia 2 4 1 3
Niger 1 3 0 3
Puerto Rico 1 5 2 3
Bulgaria 3 5 3 2
Canada 22 44 43 1
Moldova 1 1 0 1
Romania 5 11 10 1
Vietnam 2 8 7 1
Afghanistan 0 0 0 0
American Samoa 0 0 0 0
Andorra 0 0 0 0
Antigua and Barbuda 0 0 0 0
Aruba 0 0 0 0
Barbados 0 0 0 0
Belize 0 0 0 0
Benin 0 0 0 0
Bermuda 0 0 0 0
Bhutan 0 0 0 0
British Virgin Islands 0 0 0 0
Burkina Faso 0 0 0 0
Cambodia 0 0 0 0
Cameroon 0 0 0 0
Cape Verde 0 0 0 0
Cayman slands 0 0 0 0
Central African Republic 0 0 0 0
Chad 0 0 0 0
Comoros 0 0 0 0
Congo 0 0 0 0
Cook Islands 0 0 0 0
Djibouti 0 0 0 0
Dominica 0 0 0 0
DR Congo 0 0 0 0
Eritrea 0 0 0 0
Estonia 1 1 1 0
Gambia 0 0 0 0
Guam 0 0 0 0
Guinea 0 0 0 0
Guinea-Bissau 0 0 0 0
Guyana 0 0 0 0
Haiti 0 0 0 0
Honduras 0 0 0 0
Iceland 0 0 0 0
Kiribati 0 0 0 0
Kyrgyzstan 0 0 0 0
Laos 0 0 0 0
Lesotho 0 0 0 0
Liberia 0 0 0 0
Liechtenstein 0 0 0 0
Madagascar 0 0 0 0
Malawi 0 0 0 0
Maldives 0 0 0 0
Mali 0 0 0 0
Malta 0 0 0 0
Marshall Islands 0 0 0 0
Mauritania 0 0 0 0
Micronesia 0 0 0 0
Monaco 0 0 0 0
Montenegro 0 0 0 0
Mozambique 0 0 0 0
Nauru 0 0 0 0
Nepal 0 0 0 0
Nicaragua 0 0 0 0
Palau 0 0 0 0
Palestine 0 0 0 0
Papua New Guinea 0 0 0 0
Poland 11 25 25 0
Rwanda 0 0 0 0
Saint Kitts and Nevis 0 0 0 0
Saint Lucia 0 0 0 0
Samoa 0 0 0 0
San Marino 0 0 0 0
Sao Tome and Principe 0 0 0 0
Senegal 0 0 0 0
Seychelles 0 0 0 0
Sierra Leone 0 0 0 0
Solomon Islands 0 0 0 0
Somalia 0 0 0 0
South Sudan 0 0 0 0
St Vincent and the Grenadines 0 0 0 0
Suriname 0 0 0 0
Swaziland 0 0 0 0
Tanzania 0 0 0 0
Timor-Leste 0 0 0 0
Togo 0 0 0 0
Tonga 0 0 0 0
Trinidad and Tobago 1 1 1 0
Tunisia 3 3 3 0
Tuvalu 0 0 0 0
Uganda 0 0 0 0
US Virgin Islands 0 0 0 0
Vanuatu 0 0 0 0
Yemen 0 0 0 0
Zambia 0 0 0 0
Zimbabwe 0 0 0 0
Albania 0 0 1 -1
Bangladesh 0 0 1 -1
Bolivia 0 0 1 -1
Bosnia and Herzegovina 0 0 1 -1
Botswana 0 0 1 -1
Brunei 0 0 1 -1
Cyprus 0 0 1 -1
El Salvador 0 0 1 -1
Equatorial Guinea 0 0 1 -1
FYR Macedonia 0 0 1 -1
Gabon 0 0 1 -1
Ghana 0 0 1 -1
Ireland 2 6 7 -1
Latvia 0 0 1 -1
Mauritius 0 0 1 -1
Namibia 0 0 1 -1
Paraguay 0 0 1 -1
Sudan 0 0 1 -1
Syria 0 0 1 -1
Costa Rica 0 0 2 -2
Dominican Rep. 1 1 3 -2
Guatemala 0 0 2 -2
Libya 0 0 2 -2
Luxembourg 0 0 2 -2
Panama 0 0 2 -2
Turkmenistan 0 0 2 -2
Uruguay 0 0 2 -2
Angola 0 0 3 -3
Lebanon 0 0 3 -3
Myanmar 0 0 3 -3
Ecuador 0 0 4 -4
Morocco 1 1 5 -4
Sri Lanka 0 0 4 -4
Argentina 4 18 23 -5
Finland 1 1 6 -5
Israel 2 2 7 -5
Oman 0 0 5 -5
Qatar 1 3 8 -5
Thailand 6 18 23 -5
Norway 4 4 10 -6
Portugal 1 1 7 -6
Algeria 2 6 13 -7
Brazil 19 59 66 -7
Malaysia 5 13 20 -7
Venezuela 3 5 12 -7
Iran 8 22 30 -8
Pakistan 0 0 8 -8
Peru 0 0 8 -8
Philippines 1 3 11 -8
Singapore 1 5 13 -8
Austria 1 1 11 -10
Chile 0 0 10 -10
Hong Kong 0 0 11 -11
Nigeria 1 1 13 -12
India 2 4 17 -13
Iraq 0 0 13 -13
Japan 41 105 119 -14
U.A.E. 1 1 18 -17
Egypt 3 3 21 -18
Turkey 8 18 37 -19
Chinese Taipei 3 7 29 -22
Mexico 5 11 49 -38
Indonesia 3 11 51 -40
Saudi Arabia 0 0 44 -44
United States 121 379 431 -52

Visualizing asylum statistics




Asylum applications and recognition rates



Who takes the brunt of the asylum burden?





Are asylum application shares responsive to the recognition rate?







Predicting movie ratings with IMDb data and R

It’s Oscars season again so why not explore how predictable (my) movie tastes are. This has literally been a million dollar problem and obviously I am not gonna solve it here, but it’s fun and slightly educational to do some number crunching, so why not. Below, I will proceed from a simple linear regression to a generalized additive model to an ordered logistic regression analysis. And I will illustrate the results with nice plots along the way. Of course, all done in R (you can get the script here).

The data for this little project comes from the IMDb website and, in particular, from my personal ratings of 442 titles recorded there. IMDb keeps the movies you have rated in a nice little table which includes information on the movie title, director, duration, year of release, genre, IMDb rating, and a few other less interesting variables. Conveniently, you can export the data directly as a csv file.

Outcome variable
The outcome variable that I want to predict is my personal movie rating. IMDb lets you score movies with one to ten stars. Half-points and other fractions are not allowed. It is a tricky variable to work with. It is obviously not a continuous one; at the same time ten ordered categories are a bit too many to treat as a regular categorical variable. Figure 1 plots the frequency distribution (black bars) and density (red area) of my ratings and the density of the IMDb scores (in blue) for the 442 observations in the data.


The mean of my ratings is a good 0.9 points lower than the IMDb scores, which are also less dispersed and have a higher peak (can you say ‘kurtosis’).

Data-generating process
Some reflection on how the data is generated can highlight its potential shortcomings. First, life is short and I try not to waste my time watching bad movies. Second, even if I get fooled to start watching a bad movie, usually I would not bother rating it on IMDb.There are occasional two- and three-star scores, but these are usually movies that were terrible and annoyed me for some reason or another (like, for example, getting a Cannes award or featuring Bill Murray). The data-generating process leads to a selection bias with two important implications. First, the effective range of variation of both the outcome and the main predictor variables is restricted, giving the models less information to work with. Second, because movies with a decent IMDb ratings which I disliked have a lower chance of being recorded in the dataset, the relationship we find in the sample will overestimate the real link between my ratings and the IMDb ones.

Take one: linear regression
Enough preliminaries, let’s get to business. An ordinary linear regression model is a common starting point for analysis and its results can serve as a baseline. Here are the estimates that lm provides for regressing my ratings on IMDb scores:

summary(lm(mine~imdb, data=d))

(Intercept)  -0.6387     0.6669  -0.958    0.339    
imdb          0.9686     0.0884  10.957   ***
Residual standard error: 1.254 on 420 degrees of freedom
Multiple R-squared: 0.2223,	Adjusted R-squared: 0.2205

The intercept indicates that on average my ratings are more than half a point lower. The positive coefficient of IMDb score is positive and very close to one which implies that one point higher (lower) IMDb rating would predict, on average, one point higher (lower) personal rating. Figure 2 plots the relationship between the two variables (for an interactive version of the scatter plot, click here):


The solid black line is the regression fit, the blue one shows a non-parametric loess smoothing which suggests some non-linearity in the relationship that we will explore later.

Although the IMDb score coefficient is highly statistically significant that should not fool us that we have gained much predictive capacity. The model fit is rather poor. The root mean squared error is 1.25 which is large given the variation in the data. But the inadequate fit is most clearly visible if we plot the actual data versus the predictions. Figure 3 below does just that. The grey bars show the prediction plus/minus two predictive standard errors. If the predictions derived from the model were good, the dots (observations) would be very close to the diagonal (indicated by the dotted line). In this case, they are not. The model does a particularly bad job in predicting very low and very high ratings.


We can also see how little information IMDb scores contain about (my) personal scores by going back to the raw data. Figure 4 plots to density of my ratings for two sets of values of IMDb scores – from 6.5 to 7.5 (blue) and from 7.5- to 8.5 (red). The means for the two sets differ somewhat, but the overlap in the density is great.


In sum, knowing the IMDb rating provides some information but on its own doesn’t get us very far in predicting what my score would be.

Take two: adding predictors
Let’s add more variables to see if things improve. Some playing around shows that among the available candidates only the year of release of the movie and dummies for a few genres and directors (selected only from those with more than four movies in the data) give any leverage.

 summary(lm(mine~imdb+d$comedy +d$romance+d$mystery+d$"Stanley Kubrick"+d$"Lars Von Trier"+d$"Darren Aronofsky"+year.c, data=d))

(Intercept)           1.074930   0.651223   1.651  .  
imdb                  0.727829   0.087238   8.343  ***
d$comedy             -0.598040   0.133533  -4.479  ***
d$romance            -0.411929   0.141274  -2.916  ** 
d$mystery             0.315991   0.185906   1.700  .  
d$"Stanley Kubrick"   1.066991   0.450826   2.367  *  
d$"Lars Von Trier"    2.117281   0.582790   3.633  ***
d$"Darren Aronofsky"  1.357664   0.584179   2.324  *  
year.c                0.016578   0.003693   4.488  ***
Residual standard error: 1.156 on 413 degrees of freedom
Multiple R-squared: 0.3508,	Adjusted R-squared: 0.3382

The fit improves somewhat. The root mean squared error of this model is 1.14. Moreover, looking again at the actual versus predicted ratings, the fit is better, especially for highly rated movies – no surprise given that the director dummies pick these up.


The last variable in the regression above is the year of release of the movie. It is coded as the difference from 2014, so the positive coefficient implies that older movies get higher ratings. The statistically significant effect, however, has no straightforward predictive interpretation. The reason is again selection bias. I have only watched movies released before the 1990s that have withstood the test of time. So even though in the sample older films have higher scores, it is highly unlikely that if I pick a random film made in the 1970s I would like it more than a random film made after 2010. In any case, Figure 6 below plots the year of release versus the residuals from the regression of my ratings on IMDb scores (for the subset of films after 1960). We can see that the relationship is likely nonlinear (and that I really dislike comedies from the 1980s).


So far both regressions assumed that the relationship between the predictors and the outcome is linear. Needless to say, there is no compelling reason why this should be the case. Maybe our predictions will improve if we allow the relationships to take any form. This calls for a generalized additive model.

Take three: generalized additive model (GAM)
In R, we can use the mgcv library to fit a  GAM. It doesn’t make sense to hypothesize non-linear effects for binary variables, so we only smooth the effects of IMDb rating and year of release. But why stop there, perhaps the non-linear effects of IMDb rating and release year are not independent, why not allow them to interact!

summary(gam(mine ~ te(imdb,year.c)+d$"comedy " +d$"romance "+d$"mystery "+d$"Stanley Kubrick"+d$"Lars Von Trier"+d$"Darren Aronofsky", data = d)) 

PParametric coefficients:
                     Estimate Std. Error t value Pr(|t|)    
(Intercept)           6.80394    0.07541  90.225   ***
d$"comedy "          -0.60742    0.13254  -4.583   ***
d$"romance "         -0.43808    0.14133  -3.100   ** 
d$"mystery "          0.32299    0.18331   1.762   .  
d$"Stanley Kubrick"   0.83139    0.45208   1.839   .  
d$"Lars Von Trier"    2.00522    0.57873   3.465   ***
d$"Darren Aronofsky"  1.26903    0.57525   2.206   *  
Approximate significance of smooth terms:
                  edf Ref.df     F p-value    
te(imdb,year.c) 10.85  13.42 11.09

Well, the root mean squared error drops to 1.11 and the jointly smoothed (with a full tensor product smooth) variables are significant, but the added predictive value is minimal in this case. Nevertheless, the plot below shows the smoothed terms are more appropriate than the linear ones, and that there is a complex interaction between the two:


Take four: models for categorical data
So far we treated personal movie ratings as if they were a continuous variable, but they are not – taking into account that they are essentially an ordered categorical variable might help. But ten categories, while possible to model, would make the analysis rather unwieldy, so we recode the personal ratings into five categories without much loss of information: 5 and less, 6,7,8,9 and more.

We can first see a nonparametric conditional destiny plot of the newly created categorical variable as a function of IMDb scores:

The plot shows the observed density for each category of the outcome variable along the range of the predictor. For example, for a film with an IMDb rating of ‘6’, about 35% of the personal scores are ‘5’, a further 50% are ‘6’, and the remaining 15% are ‘7’. Remember that the plot is based on the observed conditional frequencies only (with some smoothing), not on the projections of a model. But the small ups and downs seem pretty idiosyncratic. We can also fit an ordered logistic regression model, which would be appropriated for the categorical outcome variable we have, and plot its predicted probabilities given the model.

First, here is the output of the model:

summary(polr(as.factor(mine.c) ~ imdb+year.c,  Hess=TRUE, data = d)
        Value Std. Error t value
imdb   1.4103   0.149921   9.407
year.c 0.0283   0.006023   4.699

    Value   Std. Error t value
5|6  9.0487  1.0795     8.3822
6|7 10.6143  1.1075     9.5840
7|8 12.1539  1.1435    10.6289
8|9 14.0234  1.1876    11.8079

Residual Deviance: 1148.665 
AIC: 1160.665

The coefficients of the two predictors are significant. The plot below shows the predicted probability of the outcome variable – personal movie rating – being in each of the five categories as a function of IMDb rating and illustrates the substantive scale of the effect.


Compared to the non-parametric conditional density plot above, these model-based predictions are much smoother and have ‘disciplined’ the effect of the predictor to follow a systematic pattern.

It is interesting to ponder which of the two would be more useful for out-of-sample predictions. Despite the fact that the non-parametric one is more faithful to the current data, I think I would go for the parametric model projections. After all, is it really plausible that a random film with an IMDb rating of 5 would have lower chance a getting a 5 from me than a film with an IMDb rating of 6, as the non-parametric conditional density plot suggests? I don’t think so. Interestingly, in this case the parametric model has actually corrected for some of the selection bias and made for more plausible out-of-sample predictions.

In sum, whatever the method, it is not very fruitful to try to predict how much a person (or at least, the particular person writing this) would like a movie based on the average rating the movie gets and covariates like the genre or the director. Non-linear regressions and other modeling tricks offer only marginal predictive improvements over a simple linear regression approach, but bring plenty of insight about the data itself.

What is the way ahead? Obviously, one would want to get more relevant predictors, but, unfortunately, IMDb seems to have a policy against web-scrapping from its database, so one would either have to ask for permission or look at a different website with a more liberal policy (like Rotten Tomatoes perhaps). For me, the purpose of this exercise has been mostly in its methodological educational value, so I think I will leave it at that. Finally, don’t forget to check out the interactive scatterplot of the data used here which shows a user’s entire movie rating history at a glance.

As you would have noted, the IMDb ratings come at a greater level of precision (like 7.3) than the one available for individual users (like 7). So a user who really thinks that a film is worth 7.5 has to pick 7 or 8, but its average IMDb score could well be 7.5. If the rating categories available to the user are indeed too coarse, this would show up in the relationship with the IMDb score: movies with an average score of 7.5 would be less predictable that movies with an average score of either 7 or 8. To test this conjecture, a rerun the linear regression models on two subsets of the data: one comprising the movies with an average IMDb rating between 5.9 and 6.1, 6.9 an 7.1, etc., and a  second one comprising those with an average IMDb rating between 5.4 and 5.6, 6.4 and 6.6, etc. The fit of the regression for the first group was better than for the second (RMSE of 1.07 vs. 1.11), but, frankly, I expected a more dramatic difference. So maybe ten categories are just enough.

The evolution of EU legislation (graphed with ggplot2 and R)



EU laws over time