The Graduation Study Is Useful

by Andrew Gillen

Over at The Education Optimists, Sara Goldrick-Rab takes issue with the new study on graduation rates.

[schools] differ tremendously in the students they serve...

all this sorting (selection bias) has to be properly accounted for if you want to isolate the contributions that colleges make to graduation rates... if you want to isolate institutional practices that ought to be adopted, you first have to get your statistical models right.

Unfortunately, I don’t think the AEI authors have done that...I don’t see how this approach is moving the ball forward...Their methods don’t begin to approach the gold standard tools needed to figure out what works (say, a good quasi-experimental design).

If we want better answers, we need to start by investing in better data and better studies.

I support 110% her call for more and better data, but I think she's a little too harsh here for 3 reasons.

First, it is very useful to have basic, if not perfect reports like this if for no other reason than that they are quite useful in framing and shifting the debate. In a sort of Hegelian dialectic, the debate over graduation rates will move along in steps, getting closer and closer to the truth. Schools used to claim that their graduation rates are low because they were open admission. This report accounts for that, and still finds huge differences. The schools will counter that you also need "(socioeconomic background, high school preparation, need for remediation, etc)" which will be accounted for in another study, moving the debate forward.

Second, there are lots of people with very strongly held beliefs on virtually any topic. For instance, my 1,200 word op-ed on an arcane student loan issue elicited 4,500 words in comments, many of them quite strong. There are arguments over basic facts - so the publication of facts is helpful to separate legitimate from asinine disagreements.

Third, not everything needs gold standard analysis. An excerpt from a terrific article making this point:

The perception that parachutes are a successful intervention is based largely on anecdotal evidence...

One of the major weaknesses of observational data is the possibility of bias, including selection bias and reporting bias, which can be obviated largely by using randomised controlled trials. The relevance to parachute use is that individuals jumping from aircraft without the help of a parachute are likely to have a high prevalence of pre-existing psychiatric morbidity. Individuals who use parachutes are likely to have less psychiatric morbidity and may also differ in key demographic factors, such as income and cigarette use. It follows, therefore, that the apparent protective effect of parachutes may be merely an example of the "healthy cohort" effect. Observational studies typically use multivariate analytical approaches, using maximum likelihood based modelling methods to try to adjust estimates of relative risk for these biases... no such analyses exist for assessing the presumed effects of the parachute.