Wednesday, October 04, 2006

US News Rankings and Learning: Statistical Evidence

By Richard Vedder

As readers know from previous epistles, I very much liked the Intercollegiate Studies Institute (ISI) study of civic learning in American universities, because it is one of the very few attempts to explicitly report, school by school, the "value added" by university training with respect to one area of knowledge. I would like the testing expanded both in terms of content and the number of schools examined, and hope to explore with other educational reformers whose thinking mirrors my own ways of making this happen.

I put the CCAP Whiz Kids (Jonathan Leirer, Matt Denhart, James Woodward) to work on doing some statistical analysis of the relationship between learning in college (as measured by the ISI study) and the ranking of the school in the US News & World Report (hereafter USN&WR) rankings. We looked specifically at the 29 schools for which we had ISI data that were on the national rankings list for national universities or liberal arts colleges. Did the top ranked schools (e.g., Princeton, Harvard, Yale, Williams, Amherest) show higher levels of "value added" learning by their students than the kids going to lower ranked (but generally pretty good) schools?

We found no statistically significant relationships between the USN&WR rankings and student learning in college. To be sure, there is a bias against the prestigious highly ranked schools in using the raw ISI test results. Those schools get students as freshman who do relatively well on the test initially, and thus have less knowledge left to learn in order to have a perfect comprehension. But even after correcting for that in various ways, there is no statistically significant relationship observed. Kids at the top USN&WR schools did somewhat better on the ISI test in their senior year than students at other schools, but that is solely because they knew more as freshman. Dollar for dollar, learning was far greater at the lesser ranked schools than at the prestige universities.

To be sure, the ISI test only examined one area of learning, albeit an important one. The sample of schools examined was pretty small. The testing might be criticized because the same students were not examined three (or four) years apart, first as freshman and then as seniors -- different students were being tested. Nonetheless, the results show that the metrics typically used to measure collegiate excellence, such as the USN&WR rankings, are very poor, and we need to take the ISI methodological approach and expand it to provide useful information to parents and policymakers about our universities --we need to create a good "bottom line."

The Spellings Commission liked that idea, as does, at least rhetorically, Secretary of Education Spellings herself (and presumably the Bush administration). I hope that the private sector can do an expanded version of the ISI test in the hopes of shaming the body politic and the universities into developing a truly national system measuring "value added" in college. This would be a giant step forward in making the universities more accountable, and making them suffer market consequences in cases where learning is meager in relation to the costs of college. This type of "value added" testing also would be a great way that private for-profit providers could demonstrate they offer a quality product and deliver learning results.


Ken D. said...
This comment has been removed by a blog administrator.
Ken D. said...

Here is an idea for a high-level metric for assessing productivity of Higher Ed. institutions: the ratio of the combined starting salaries of the graduating class to the annual operating budget of the university.

So, for example, if the graduating class of 1000 students had an average income of $30k, and the operating budget of the school that year was $30m, the ratio would equal 1. This metric could be used to compare different schools or the same school over time.

Askinstoo said...

Hi, i was looking over your blog and didn't
quite find what I was looking for. I'm looking for
different ways to earn money... I did find this though...
a place where you can make some nice extra cash secret shopping.
I made over $900 last month having fun!
make extra money

Big Blue said...

Ken - Please cut the brake lines on your car and go for a drive.

Lady Justice said...

For all of the criticism of the US News rankings, doesn't anyone find it worth noting that the brightest schools are the ones who know how to manage the data so that they rank higher? Has anyone seen the huge jump that University of Chicago just made by reporting information that hadn't previously been included?

Anyone with a clue knows that Chicago should easily be among the top ten in the nation, but because they are so brainy and scholarly over there, they just weren't paying attention to the rankings until someone complained internally that they were being undervalued. They took a look at things, reported the data more completely, and they jumped multiple spots up in the rankings.

Surely knowing how to game the rankings is at least somewhat indicative of how good a school it truly is, assuming they are really paying attention, as in the recent case of U Chicago, no?

Scott A. Edwards said...

Attention Blog Owners...!!! WARNING: A new immensely powerful viral marketing system has been released, which allows marketers to broadcast messages directly to people's desktops. Learn more about it here: earn residual income site. It pretty much covers earn residual income related stuff and it is FREE to join.