Are Think Tanks At Their End?

by Andrew Gillen

Bruce Bartlett has a terrific piece in Forbes titled “The End Of The Think Tank” concerning the history behind and the troubles with modern think tanks.

Their rise was spurred by those

frustrated by the slow, plodding style of AEI and Brookings, which tended to publish their research in books that often took years to complete…

The idea was to have an institution that wouldn't take years to study an issue to death and not deliver its research until it was too late, but that would produce its research on a much faster schedule, in time to influence congressional debate…

This is problematic:

The increasing impact of think tanks brought in new money as corporations realized that think tank studies were highly effective ways of influencing legislation... Unfortunately, the additional money brought increased donor pressure to produce bottom line results--getting bills passed or defeated--and had a corrupting effect on the think tanks…

Mark Thoma comments:

I think these organizations -- think tanks -- have done great damage to economics… there has also been a blurring of lines between academic research and think tank research … The main problem, I think, is the he said - she said presentation of academic work in the media alongside the papers that think tanks put out as though there is an equivalence (or a similarly structured debate on, say, CNN). Much of the think tank work (but not all) is junk and no such equivalence exists, but the work is often given equal footing in the press…

See also comments by Holly Yeager

I don’t have too much to add, but working at a think tank, I do want to make a couple of points.

First, while I’ve certainly come across some of the shoddy “research” produced by some think tanks, there are problems with peer reviewed academic research too. This article describes how many seminal articles were rejected, sometimes over and over again (and it’s in a peer-reviewed journal, so you know it’s good :) You can also see here, or the second link here, or here, or this story.

Of course, none of this makes think tank research any better, and I still put more weight behind a journal article than the policy brief, but it does illustrate that it’s a bit more complicated than “academic research = good, think tank research=bad.”

Second, the focus and process of academic work and think tank work is very different. To vastly oversimplify, journals are interested in publishing original theoretical ideas, or rigorous evaluations of ideas that have been implemented (which requires lots of data, which means the program must have survived for awhile). It also often takes a few years from the first draft of a paper to publication.

In contrast, think tanks are interested in how to apply ideas, and in arguing for the continuation or elimination of policies for which there is not yet overwhelming evidence. Their time frame is much shorter, since it is driven much more by the political process.

In other words, academics generally come up with the ideas and evaluate them if implemented, while think tanks figure out how to implement them and whether they are promising enough to continue when piloted, or whether those resources have better uses elsewhere. However, they must do this latter part before there is really sufficient data to answer the question to an academic standard, which means that there will be lots of conflicting conclusions. Looking back after the fact, much of think tank research will be wrong. This higher error rate is not because think tankers are inherently charlatans (though some are), or because their research is junk as Thoma would have it, but because they are asked questions to which the answer is not (yet) clear. It’s one thing if they get it wrong given the information available at the time, but it seems too harsh to me to go back later and call their work junk if it's wrong only with the benefit of more and better information that time provides.

To illustrate, consider that when academics are asked the types of questions think tankers are routinely tasked with answering, there response is in the style of Zhou Enlai’s response when asked in the 1970’s if the French Revolution was a good thing:

It is too soon to say.

While that might be a reasonable answer for an academic, that is not a sufficient answer for those making policy decisions right now (and therefore for think tankers trying to provide them with information).

Given the different focus, timeline, and process, it is no more surprising that there are different yardsticks for evaluating academic vs. think tank research than it is that there are different yardsticks for evaluating pure research as opposed to development in R&D. Of course, pointing out these things doesn’t make think tank research any better, but it does illustrate why it is more often wrong than academic work, and how simply adopting the quality control practices of the academic world won’t really work.