Why U.S. News & World Report Changed Its Rankings Methodology
For prospective MBAs, the most difficult step towards a business school degree is knowing where to start. And for many, the most obvious is researching the near-endless amount of college ranking sites. After years of scandals and much-publicized doubts, however, the authority of these rankings may be in question.
In a recent Philadelphia Magazine op-ed, writer Sandy Hingston asks, simply, “Can We All Agree Now That College Rankings Are Bunk?”
Hingston’s critique, which highlights in relative inexactness of the college ranking sites, is not the first of its kind. In 2011, popular New Yorker writer Malcolm Gladwell wrote wrote about his issue with college rankings. In “The Order of Things,” Gladwell notes that the outcomes of students sometimes do not match college reputations.
Principally, Gladwell, the author of the massively popular 2008 book Outliers: The Story of Success, found fault in how publications like U.S. News & World Report conducts rankings. He writes, “the magazine sends a survey to the country’s university and college presidents, provosts, and admissions deans (along with a sampling of high-school guidance counselors) asking them to grade all the schools in their category on a scale of one to five.”
In regards to the scoring system, Gladwell cites numerous incidents that say reputation weighs too heavily. This is important considering that personnel often do not know what other schools are actually like. To illustrate, he brings up a different ranking from the time, with much different results:
“In an article published recently in the Annals of Internal Medicine, Ashwini Sehgal analyzed U.S. News’s ‘Best Hospitals’ rankings, which also rely heavily on reputation ratings generated by professional peers. Sehgal put together a list of objective criteria of performance—such as a hospital’s mortality rates for various surgical procedures, patient-safety rates, nursing-staffing levels, and key technologies. Then he checked to see how well those measures of performance matched each hospital’s reputation rating. The answer, he discovered, was that they didn’t. Having good outcomes doesn’t translate into being admired by other doctors. Why, after all, should a gastroenterologist at the Ochsner Medical Center, in New Orleans, have any specific insight into the performance of the gastroenterology department at Mass General, in Boston, or even, for that matter, have anything more than an anecdotal impression of the gastroenterology department down the road at some hospital in Baton Rouge?”
He goes on to say, “reputational ratings are simply inferences from broad, readily observable features of an institution’s identity, such as its history, its prominence in the media, or the elegance of its architecture. They are prejudices.”
In addition, Gladwell notes important economic distinctions that many college ranking sites ignore. For the sake of simplified ranking, small private schools are compared to massive public institutions and scored equally. But doing so overlooks the economic makeup of enrolled students, funding, and much more.
A Modern Change for U.S. News
Seven years later after Gladwell’s critique, U.S. News & World Report continues to refine its rankings method. The publication’s new changes aim to emphasize economic mobility. This comes just one year after Politico revealed that the upper-echelon schools continually reward students that are already wealthy. That discovery may not be “groundbreaking,” but the data is still jarring. Many of the country’s most well-known institutions enroll more students that come from the top 1 percent of income-earning families than the bottom 60 percent of earners.
Why is this important and why did U.S. News & World Report need to change? Like Gladwell said seven years prior, prejudices feed the rankings and the economics factors are biased. And the amount of inequality it expanded may have been profound.
The Politico study outlined these key factors in the previous U.S. News ranking formula:
- Student Performance
- Lower Acceptance Rates
- Higher Performance on Surveys
- Alumni Donations
These factors weigh heavily for students from more affluent backgrounds. U.S. News, like many of its popular ranking competitors—such as the Financial Times and Bloomberg—rely on similar scoring systems, which often left the economically disadvantaged out of the equation.
The aforementioned 2017 Politico report points to the lengths some schools and state governments went to meet the U.S. News standards:
“Colleges go to great lengths to rise in the rankings. The president of one school ranked in the top 20 said the college caps classes at 19 students, simply because the rankings reward schools for keeping classes under 20 students. In some states, the rankings are built into accountability systems for university presidents. Arizona State trustees put a bonus pegged to the rankings in the president’s contract. In Florida, Gov. Rick Scott set a goal to ‘achieve the first top-10 research public university and a second-ranked public university in the top 25.'”
Robert Morse, U.S. News Chief Data Strategist, says of the changes; “There is an active and ongoing debate about how to best measure quality in education, and we pay close attention to that debate. Over time, our ranking model has put more emphasis on outcomes measures … As part of this evolving process, we’ve wanted to measure whether schools were successful at serving all of their students, regardless of economic status.”
The publication will begin accounting more for social mobility and rewarding of Pell Grants. As well, Morse says, schools that enroll a higher percentage of lower-income students will get more credit for graduation rates. In total, “13 percent of a school’s rank is now dependent on the economic diversity of its campus.”
Will Other College Ranking Sites Follow?
The changes for U.S. News‘s rankings may cause a slow sea change for its competitors. For its undergraduate ranking, SAT/ACT scores now factor less in the overall score. This, like many of the highlights indicated above, is because standardized tests are highly correlative with income. The acceptance rate boost is being completely removed as well.
The dismissive stances from the likes of Hingston and Gladwell likely will not be re-mediated immediately. After all, college ranking sites will likely remain among most popular resource for school research. However, a more modern evaluation can pay dividends in the long-term. And for U.S. News‘ prime competitors, following suit may be an unavoidable remedy.
The next annual U.S. News ranking of the country’s best business schools will arrive early next year. Whether the new changes will weigh heavily in the rankings remains to be seen.
