When Policy Meets Statistics: The Reinhart and Rogoff Study on Excessive Debt

Knowing how many of us economists toil away in obscurity on most research, I’m always intrigued by what catches the press’s and public’s attention. Take, for example, the significant attention paid to a 2010 study by Harvard economists Carmen Reinhart and Kenneth Rogoff that concluded that countries with debt levels above 90 percent of GDP began showing slower rates of growth. When Thomas Herndon, Michael Ash and Robert Pollin, scholars at the University of Massachusetts at Amherst, recently had trouble replicating Reinhart and Rogoff’s results, the debate played out in national news outlet.

Unfortunately, this discussion quickly devolved from substance to politics to arguments ad hominem. Without getting into the extent to which I or others can validate Reinhart and Rogoff’s (R&R’s) original findings, I offer six cautions for anyone witnessing this or a similar statistical debate with significant policy implications: (1) statistics should never be interpreted as showing more than simple but potentially useful correlations; (2) healthy skepticism is required for all social science research, which seldom gets replicated for validity; (3) all empirical economic work is based on history that will not repeat in the exact same form; (4) research can certainly contradict conventional wisdom but not reason; (5) arguments ad hominem, particularly by those with their own agendas, are unhelpful; and (6) be careful with labels and straw men.

  1. Correlation versus causation. It’s long been stressed that statistical tests never prove causation, not simply because they can’t but because researchers make many choices and assumptions, often of statistical convenience. This doesn’t mean statistics are useless. Just accept that any result merely shows that A and B seem to occur together even after trying to account for other influences under a huge range of never-fully-tested assumptions.
  2. Skepticism. A growing body of “research on research” shows that few social science experiments, and even many medical studies, are replicated. Also, positive results get published; negative results usually do not. R&R’s study was replicated mainly because it got an unusual amount of attention.
  3. History. The past never repeats itself exactly—or, as Heraclitus warns, “You could not step twice into the same river.” That historical interpretations are contained and sometimes couched in data analyses doesn’t mitigate this well-known caution.
  4. Reason versus conventional wisdom. The R&R debate mainly revolves around two reasonable notions. One is the simple arithmetic conclusion that debt can’t rise forever relative to national income, along with the related economic conclusion that higher levels of debt can and have been shown in many places to have consequences for investment, interest rates, ability to borrow, and how government revenues are spent. The other is that institutions, times, places, and circumstances matter greatly, and they affect how one should interpret past data, such as those presented by both R&R and their critics.
  5. Ad hominem arguments. R&R published some of their work with the Peterson Institute for International Economics, so some of those who attacked R&R may have considered the authors guilty by association because of Peter George Peterson’s concern about deficits and his contributions to that Institute. Peterson’s own “guilt” on budget issues seems to be that he became rich on Wall Street, although he favors higher taxes on what he calls “fat cats” like himself. Still, while many of those who engage in these attacks themselves fail to represent their own or their institution’s sources of funding, let’s be honest. Much social science research is funded in ways that doesn’t necessarily bias how the research is done, but rather what is researched in the first place. So, unless one fully engages and thoroughly analyzes every study, even the careful academic reader must often try to determine trustworthiness in other ways, such as whether the researchers report results only consistent with some special interest or political party.
  6. Labels and “straw men.” While we all use labels and straw men at times to set up the stories we tell, they at best simplify greatly. In this case, R&R are identified as advocates of “austerity” and their opponents as “Keynesians” advocating stimulus, both of which are nothing more than labels. Most budget analysts I know worry about deficits but vary widely in whether they would engage in more short-term stimulus or in how strongly they believe that a path toward long-term balance even requires austerity. For instance, is reducing some rate of growth of spending, no matter what its level, austerity? Is the Congressional Budget Office an advocate of austerity or Keynesian when it asserts that sequestration hurts the economy in the short run, but has long-run benefits relative to doing nothing about deficits?

The bottom line: use extreme caution no matter which economist you read or believe.

Full disclosure: I have spent most of my career at the Urban Institute or the Treasury Department, brief periods each at the Brookings Institution, American Enterprise Institute, and the Peter G. Peterson Foundation (which differs from the Peterson Institute for International Economics). I also serve or have served on many advisory groups and boards for such organizations as the Committee for a Responsible Federal Budget and the Comptroller General of the United States. As a consequence, je m’accuse of being among the many economists limited more than I would like by what research is supported by those institutions or their funders.


One Comment on “When Policy Meets Statistics: The Reinhart and Rogoff Study on Excessive Debt”

  1. I would also add that it is important to look outside of the normal sources for new discoveries. Read my study.


Leave a Reply to Michael Bindner Cancel reply

Your email address will not be published. Required fields are marked *