CBO Budget Projections: After Two Years No Better Than Throwing Darts

This article is more than 2 years old.

The latest Senate Republican healthcare reform plan, like its House predecessor, appears now to have failed. It did so for many reasons, some good and some not so good. Among the not-so-good reasons the proposal failed were the projections of the Congressional Budget Office. As I show here, CBO projections more than two years into the future in the domain it perhaps studies most -- federal deficits -- are little better than random guesses.

Look at the chart below. It shows the deterioration of CBO predictive accuracy when it comes to the federal budget deficit. The circular markers show how accurate these predictions have been since 2009 in forecasting how large the federal budget deficit would be. As one can see, although CBO estimates are very good for the same year in which they are made, accuracy deteriorates rapidly. By year 3, CBO projections have actually been no better than luck.

Graphic created by Seth J. Chandler

The accuracy metric used in the chart above is an industry-standard metric known as "Adjusted R Squared," which, to put it roughly, shows what percentage of variation in what you are trying to predict (the budget deficit) can be explained by your model relative to "throwing darts." So, when we say that the Adjusted R Squared of CBO predictions two years out is 0.57, that means that only 57% of the variation in the actual budget deficit has been predicted by the CBO. (Those interested in technical details should look at the notes below).

The inaccuracy of CBO budget deficit forecasts is, in my view, relevant to the accuracy of the CBO's forecasts with respect to the Republican healthcare alternatives. First, though, let me not be guilty of overstatement. This doesn't mean we shouldn't make forecasts or cite their results. Random results may actually be better than deliberately wrong projections. It doesn't mean we shouldn't have a CBO or multiple good sources of economic research: Congress should have dependable sources of information. It doesn't mean the Republican bills were good ideas or that they wouldn't have increased -- at least somewhat -- the number of persons without useful health insurance. What it does suggest, however, is that politicians should stop citing these CBO forecasts as some sort of infallible oracle. Journalists should no more cite them as fact than they would a forecast, even by a reputable organization, about the Dow Jones average several years hence. We all need to recognize that, even with the best of intentions, economic forecasting is extremely difficult. The CBO is neither immune from politics nor oracular, particularly when it comes to projections more than two years into the future.

We should not think that CBO scoring of the healthcare bills is any better than its scoring of the deficit. Indeed, while some of the error we see in CBO budget deficit forecasting is simply the result of computational irreducibility -- an irreducible inability to forecast political and economic events far into the future -- some of the error with respect to healthcare is indefensible pigheadedness.

Let's look more closely at the much-bandied (examples, here, here and here) 22-million-more-uninsured figure used to bludgeon the Better Care Reconciliation Act. This figure is actually the result of a CBO subtraction. One takes the estimate of the number of persons projected to be insured under the Affordable Care Act and subtracts from that the number of persons projected to be insured under the comparison bill. So, if one overestimates the number of people insured by the ACA, one exaggerates the loss of insurance created by the comparison bills. That's math. And, yet, that is exactly what the CBO has done. It has "estimated" for example that 15 million people are currently insured on the Exchanges under the ACA when, in fact, about 10 million people have such insurance.  It projects that 18 million people will be insured next year on the Exchanges when absolutely no one thinks the number will be that large. And, to get its final number about the comparative number of uninsureds, it projects that 25 million will be covered on the Exchanges by 2026. Why this unwarranted optimism? Because a year and a half ago the CBO made that projection (which was wrong even then) and it explicitly refuses (see page 15 of this June 26, 2017 document) to update its number on light of actual events, such as insurer withdrawals from the marketplace, further 20%+ increases in gross premiums, and a court ruling that payment of a key Obamacare subsidy is illegal. In the rest of the world, an accounting error like that might get you thrown in prison.

Or consider what the non-partisan CBO did in 2014 when scoring the ACA. I know we are going deep into ancient weeds here. But if journalists are willing to look at details in things such as the Trump-Russia scandal, they ought to be willing to look at details in a broader domain. Critical thinking should not be reserved for use only against people and programs one is predisposed to dislike.

Back in 2014, the issue in question was something called the Risk Corridors program. I recount it more fully here, here, and here. It was a part of Obamacare (section 1342) under which insurers who lost money on the Exchanges were supposed to be bailed out by insurers who made money on the Exchanges. Congress never explicitly appropriated any federal money to pay for it. The problem was that there weren't many insurers who actually were making money and, so, there wasn't going to be enough money to pay off the insurers who had set prices way too low (to the uncritical applause of many ACA supporters) and thus wanted Risk Corridor cash. The Obama administration was proposing to divert other funds in the HHS budget to keep the losing insurers afloat. Republicans got wind of this and, ultimately, proposed either scrapping the Risk Corridors program or (Marco Rubio) stopping Executive branch use of other money to "bail out" insurers. At that point, the non-partisan CBO intervened and suddenly changed an earlier forecast and claimed that the winning insurers would ultimately make $8 billion more than the losing insurers. The finding was promptly and predictably seized upon by Democrats who, based on uncritical acceptance of the work of this non-partisan organization, could argue that scrapping Risk Corridors would actually cost the government money; there was thus no need to prevent the Obama administration from diverting money. A few months later, after the threat to scrap Risk Corridors had essentially died, the CBO reversed course again and admitted that, perhaps, the Risk Corridors program wouldn't make any money for the government. Presumably, many of the people engaged in this bizarre forecasting zigzag are still at the CBO.

The inexplicable CBO changes in its forecast had consequences. Ultimately, as many sensible people had foreseen at all times throughout these CBO forecast changes, insurers lost money on the Exchanges. There wasn't nearly enough coming in from the minority of profitable insurers to cover losing insurers for their losses. This shortfall contributed to the demise of many of the co-ops and the withdrawal of other insurers from the Obamacare marketplace. Had the CBO admitted that Obamacare was going poorly or recognized that it had made a mistake, and that insurers were losing money, Congress would have at least had a better opportunity to debate whether to change the law.

I also note that the CBO projected that Title VIII of the Affordable Care Act would net $70 billion over 10 years. In fact, as many people predicted at the time, that part of the ACA, the so-called CLASS Act, was so poorly structured that its provisions did not yield the government a dime. CBO scoring is vulnerable to manipulation by Congress who, at least when populated by intelligent staffers, can structure legislation precisely so that CBO is almost impelled to make misleading estimates. (Other writers have argued for yet more flaws with CBO analyses. Examples here, here, and here.)

I recognize that this the tone of this entry bears some resemblance to post-game interviews of losing sports figures who played poorly and then blame the referees. Referee error (and economic forecaster) error is an unavoidable human failing. And, too often, supposed malevolence is really just incompetence and time pressure.  The idea, however, is to play so well -- and write such good bills -- that one succeeds despite typical referee and forecasting errors. So, if I were using the argument above to say that the Republican bills were great ideas and that they should have passed, I would be guilty of the same lack of maturity. But I'm not saying that. I am simply saying that bad forecasting and attributions of infallibility to flawed forecasts harms political debate. It's as if a travel website put the wrong prices on hotel stays: you might sometimes end up at a decent hotel, but you'd still be making bad decisions.

In the recent healthcare debate, overestimates of the number of persons likely to be insured by the ACA are leading us (apparently) to stick with that law in little-changed form. (I could be wrong, but I don't think repeal will pass and I don't think we yet have quite enough immediate pain in the middle class to induce Democrats and Republicans to come together for more modest reforms of health insurance). Our political inability to modify the ACA at the federal level continues even after the number of persons it insures fail to climb as originally projected and the gross premiums charged by insurers continue to rise well beyond expectations. And, yes, some of this is due to actions by President Trump that would have been exceptionally difficult to forecast. But not all of it.

The end does not justify the means. Cheating or incompetence in financial analysis should not be ignored or condoned just because it favors one's intuition about the "correct result" in a particular case. That kind of reasoning will, I promise, sooner or later, bite one in the behind, perhaps on something even more important than healthcare.  I, at least, will celebrate any supporter of the ideas of the Affordable Care Act or yet more liberal plans who is enough of a sportsperson to acknowledge plain referee errors in one's favor. We all need better and more objective work from the CBO.


1. I chose to use "Adjusted R Squared" rather than the somewhat more common "R Squared" for good reason. When sample sizes are small, as they are here, R Squared without adjustment tends to overstate the success of a model in making predictions. This is so because one obtains a positive R Squared statistic even if one simply guesses results. The "adjustment" in "Adjusted R Squared" roughly compensates for this issue.

The figure below shows the problem in more detail. The circular markers show how accurate these predictions have been since 2009 in forecasting how large the federal budget deficit would be. The rectangular markers show how large an "accuracy number" one would obtain if one just threw mathematical darts and made essentially random predictions for the same years. As one can see, although CBO estimates are very good for the same year in which they are made, by year 3, random guesses and CBO estimates have converged.

Graphic created by Seth J. Chandler

2. My methodology here was to gather up eight years worth of reports from the Congressional Budget Office stored at various PDF documents at a set of URLs on the CBO website:

  • "https://www.cbo.gov/sites/default/files/111th-congress-2009-2010/reports/03-20-presidentbudget.pdf"
  • "https://www.cbo.gov/sites/default/files/111th-congress-2009-2010/reports/03-05-apb.pdf"
  • "https://www.cbo.gov/sites/default/files/112th-congress-2011-2012/reports/2011-03-18-apb-preliminaryreport.pdf"
  • "https://www.cbo.gov/sites/default/files/112th-congress-2011-2012/reports/03-16-APB1.pdf"
  • "https://www.cbo.gov/sites/default/files/113th-congress-2013-2014/reports/44173-apb0.pdf", and
  • "https://www.cbo.gov/sites/default/files/114th-congress-2015-2016/reports/51383-apbonecol.pdf".

I looked at Table 1 in each of those documents, which lists the prior year's actual deficit and then various deficit projections extending about a decade into the future. I thus created a table showing the year the projection was made, the year that was the subject of the projection, the predicted deficit in the subject year, and the actual deficit in the subject year (where it is known). I then grouped the dataset according to the difference between the year in which the projection was made and the subject of the projection. I performed a linear regression one each grouping ("the actual grouped data") and computed the "R Squared" statistic for each regression.

To determine the R Squared one would obtain by luck, I created 1000 n x 2 matrices for integer values of n between 3 and 8 inclusive. Each element of these matrices was a random draw from a normal distribution. I performed linear regressions on the data within each of these matrices and computed an R Squared statistic. I then found the mean R Squared for n=3, n=4, n=5, ... n=8. I then mapped each value of n into the number of entries in my CBO dataset. Thus, if I had five rows of data for situations in which the CBO had projected 3 years into the future, the "luck value" for projections 3 years into the future would be the mean R Squared for n=5. (Here's another example of this sort of simulation.)

3. The Congressional Budget Office might be as non-partisan as the Supreme Court. Law and financial modeling both require judgment. It's difficult to disentangle one's predispositions about they way the world works -- and the way one wants it to work -- from one's supposedly objective work about either the law or financial projections. In both domains, there's room, however, to at least somewhat reduce the role of politics.

4. This blog entry was inspired by a blog entry from journalist Jamie Dupree.  I checked his data (which was correct), augmented it with figures from additional years and performed the analysis myself.  Conclusions and any errors are my responsibility, not his.

5. My apologies for any criticism of real-world dart throwers. Some of them are pretty accurate.

Follow me on Twitter or LinkedIn. Check out my website.
