Rogoff-Reinhart and Spreadsheet Errors: Is it Different This Time?

     It may just be. For so long their paper of of 2010 was supposed to show the terrible dangers of public debt for growth-Krugman insists that we should make a distinction between their book and the paper; the latter has been given way too much credibility by the former; now the opposite is likely to happen.

     "It’s important to make a distinction between the R-R book “This time is different” and the paper. The paper got undeserved credibility from the book; now the book may be devalued by the paper. But they’re quite different."

    "The book had a sound empirical strategy: it focused only on extreme events, then described what happened around those events. Because of the severity of the shock, it was reasonable to infer that whatever happened around crises was in fact crisis-related, so problems of causation were sidestepped."

    "The paper didn’t do any of that — it just looked at simple correlations, without making any effort to untangle causation. It wasn’t worthy of the authors. And they behaved badly by digging in when critiques surfaced, rather than responding with a good-faith effort to sort out what was really happening."

     http://krugman.blogs.nytimes.com/2013/04/17/further-further-thoughts-on-death-by-excel/

     Obviously the pun in the title is intended. 

     Paul Ryan actually appealed to their authority in his budget. It was the what Austerians-many of these Austerians are also Austrians; it's interesting how similar the two words are-used to insist that austerity was a pressing issue now, not when the economy was healthy again.

    It's now fallen apart and a major story line is it's all due to a simple Excel formatting error:
   
     "Read Mike Konczal for the whole rundown, but I'll just focus on the spreadsheet part. At one point they set cell L51 equal to AVERAGE(L30:L44) when the correct procedure was AVERAGE(L30:L49). By typing wrong, they accidentally left Denmark, Canada, Belgium, Austria, and Australia out of the average. When you fix the Excel error, a -0.1 percent growth rate turns into 0.2 percent growth.*

    "This is literally the most influential article cited in public and policy debates about the importance of debt stabilization, so naturally this is going to change everything

 http://www.slate.com/blogs/moneybox/2013/04/16/reinhart_rogoff_coding_error_austerity_policies_founded_on_bad_coding.html

      This is different than one very common criticism of their result: that they ignore that causation very may well run the other way: not from high debt or the magical 90% debt to low GDP but the other way around. What's at work here though is that the correlation itself has been questioned. An ongoing problem with the paper is that subsequent researchers have been unable to replicate the results. It turns out that they can be replicated provided you make the same formatting error they did: leaving those five countries out that very much belie the correlation-Canada, Denmark, Australia, Austrian, and Belgium.

     Of course, a cynical soul might find it a little too convenient that the countries they left out just happened to be the ones who would make their whole premise not work if they were included. Mike Konczal is also such a cynical soul. The spreadsheet is not their only error. Two very important errors was:

    "Reinhart-Rogoff use 1946-2009 as their period, with the main difference among countries being their starting year. In their data set, there are 110 years of data available for countries that have a debt/GDP over 90 percent, but they only use 96 of those years. The paper didn't disclose which years they excluded or why.

Herndon-Ash-Pollin find that they exclude Australia (1946-1950), New Zealand (1946-1949), and Canada (1946-1950). This has consequences, as these countries have high-debt and solid growth. Canada had debt-to-GDP over 90 percent during this period and 3 percent growth. New Zealand had a debt/GDP over 90 percent from 1946-1951. If you use the average growth rate across all those years it is 2.58 percent. If you only use the last year, as Reinhart-Rogoff does, it has a growth rate of -7.6 percent. That's a big difference, especially considering how they weigh the countries."

     "Reinhart-Rogoff divides country years into debt-to-GDP buckets. They then take the average real growth for each country within the buckets. So the growth rate of the 19 years that the U.K. is above 90 percent debt-to-GDP are averaged into one number. These country numbers are then averaged, equally by country, to calculate the average real GDP growth weight."

     "In case that didn't make sense, let's look at an example. The U.K. has 19 years (1946-1964) above 90 percent debt-to-GDP with an average 2.4 percent growth rate. New Zealand has one year in their sample above 90 percent debt-to-GDP with a growth rate of -7.6. These two numbers, 2.4 and -7.6 percent, are given equal weight in the final calculation, as they average the countries equally. Even though there are 19 times as many data points for the U.K."

    "Now maybe you don't want to give equal weighting to years (technical aside: Herndon-Ash-Pollin bring up serial correlation as a possibility). Perhaps you want to take episodes. But this weighting significantly reduces the average; if you weight by the number of years you find a higher growth rate above 90 percent. Reinhart-Rogoff don't discuss this methodology, either the fact that they are weighing this way or the justification for it, in their paper."

     http://www.nextnewdeal.net/rortybomb/researchers-finally-replicated-reinhart-rogoff-and-there-are-serious-problems

     So the coding eror isn't their only one. Krugman says that he didn't want to question their intellectual honesty at first but their answers to the questions raised has been very poor:

     "So how do R-R respond?"

     "First, they argue that another measure — median growth — isn’t that different from the Herndon et al results. But that is, first of all, an apples-and-oranges comparison — the fact is that when you compare the results head to head, R-R looks very off. Something went very wrong, and pointing to your other results isn’t a good defense."

    "Second, they say that they like to emphasize the median results, which are much milder than the mean results; but what everyone using their work likes to cite is the strong result, and if R-R have made a major effort to disabuse people of the notion that debt has huge negative effects on growth, I haven’t noticed it."

    "Third, they point out that even cleaned-up data do show a negative association between debt and growth. Yes, but that’s where the issue of reverse causation comes in. More on that in a second."

    "Finally, while they acknowledge the issue of reverse causation, they seem very much to be trying to have it both ways — saying yes, we know about the issue, but then immediately reverting to talking as if debt was necessarily causing slow growth rather than the other way around."

    "So, about the slow growth/debt connection: I’ve done a quick and dirty mini-RR for the period 1950-2007 (starting 1950 because that’s where the Total Economy Database starts), focusing only on the G7. If you look at the scatterplot, there does seem to be an association between high debt and slow growth:

     "But I’ve coded the points by country — and if you look at it, you see that most of the apparent relationship is coming from Italy and Japan; Britain didn’t seem to suffer much from its high debt in the 1950s. And it’s quite clear from the history that both Italy and (especially) Japan ran up high debts as a consequence of their growth slowdowns, not the other way around."

     "So this is really disappointing; they’re basically evading the critique. And that’s a terrible thing when so much is at stake."

    http://krugman.blogs.nytimes.com/2013/04/16/reinhart-rogoff-continued/

     So will this change the conversation? Yglesias is pessimistic but I actually think it will-over time. As I mentioned in a discussion with Greg yesterday-he gets a HT for making me aware of Yglesias' post-I think our monetary and fiscal understanding is evolving and I think we're headed in the right direction-not as quickly as we would like but this is evolution. Many of the worst ideas have been ferreted out. Paul Ryan-both in his defeat as part of the Romney ticket-and the black eye he took from his convention speech-which wasn't constrained by fact checkers-has lost a lot of public stature-even among the VSP.

    I'd say things are being learnt at the slowest pace in Europe. I think we're slowly getting there. This paper by R&R was a kind of Bible for the VSP and seeing it disgraced will have a long term effect. Again, evolution. This is also true on gun control as Greg Sargent is arguing.

    http://diaryofarepublicanhater.blogspot.com/2013/04/wall-street-journal-editorial-page.html

    http://www.washingtonpost.com/blogs/plum-line/wp/2013/04/17/the-morning-plum-if-gun-proposal-dies-what-happens-when-theres-another-shooting/