Maclean's ranking revisions offer transparency at a cost
By Natasha Marar
News Editor
November 14, 2007
 The University of Windsor improved its grades in seven categories, but finished in 10th place once again, in this year’s edition of the Maclean’s university rankings.
McGill University claimed the number one spot in the medical doctoral category, Victoria in the comprehensive category, and Acadia and Mount Allison were tied in first place among the primarily undergraduate schools.
The annual national survey of university performance marked its seventeenth issue with a reformed set of indicators and research methodology.
Maclean’s has decided to rely solely on publicly available information for the report, rather than obtaining data from universities directly. Information is gathered from Statistics Canada, the three largest federal granting agencies, the Canadian Association of Research Libraries, and information published on university web sites.
“All of the information that we have in the rankings is highly transparent, completely objective and completely fair,” said Tony Keller, managing editor, special projects for Maclean’s.
“I think it’s a good idea to rely on information that has been in someway audited by a third party,” explained Keller, who said that in the past universities sometimes exaggerated records in their favour.
“It’s harder for anyone to gain the [ranking] system or undermine the system.”
“Nobody has to supply us with the information…and no one can boycott the rankings,” added Keller in reference to last year’s university boycott.
Although it was not the first time, controversy surrounded the rankings last year when 26 universities refused to offer information they had gladly released to Maclean’s in the 15 years prior.
[Universities] suddenly decided was dangerous for the public to know the entering average at their university or the graduating rate at their university,” said Keller, who stressed the need for public scrutiny. “It’s information that has to be made public.”
Universities have also been hesitant about releasing institutional information for the Maclean’s rankings, such as class size, which Keller explained is not public anymore. “There is probably less information publicly available about university performance and operation…than there has been in the past.”
“Universities knows about their institution but they don’t know about their peer institutions, and the public doesn’t know,” said Keller, who added, “The public should know given that they are the ones that pay for universities.”
The decision to use public information eliminates the need for university cooperation or support, and has resulted in less controversy for this year’s rankings.
Although the research changes for the 2007 survey are meant to portray a more accurate representation of Canada’s universities, differences in the measured categories can end up disadvantaging schools.
Keller claimed, however, “We are using slightly different data to measure the same things we always measure.”
A quick glance between the 2006 and 2007 rankings will tell any reader that not all factors are being measured the same way. Rosemary Zanutto, director, Institutional Analysis for the University, was quick to point out the numerous changes in this year’s survey.
Comprehensive and primary undergraduate universities are ranked using only 13 indicators, while last year’s survey assessed schools using 23 indicators. This means that some areas where schools score favourably were given less importance or were eliminated all together.
Two new indicators, however, student/faculty ratio (10 per cent) and total research dollars (6 per cent), were added to this year’s rankings.
Zanutto explained that by introducing new indicators and changing the weighting of existing indicators Maclean’s has changed the focus of the survey. “Students in classes were 40 per cent of their ranking, now its 20 [per cent],” said Zanutto.
Some of the indicators eliminated included class sizes (14 per cent), average entering grade (11 per cent), alumni support (3 per cent), student retention (2 per cent), class sizes (7 per cent), and faculty with PhDs (3 per cent). In total, the 12 disregarded categories constituted 43 per cent of the weighting in the 2006 survey.
The reputation composite saw the greatest weight increase from 16 per cent to 22 per cent—a category in which Windsor came in last place among 10 other comprehensive universities.
“The composite has changed a lot,” University of Windsor president, Ross Paul, said disapprovingly of the reputation survey’s increase weight in the rankings. “And so many people only look at that.”
“Even though we are 10th [overall] again, we went up in all the key areas,” remarked Paul. “The problem is [Maclean’s] increased the weight for the reputation survey…[and] they took out parts we did better on.”
“Our reputation survey is asking people who we feel have a reason to be in the know,” defended Keller, who explained that only certain individuals who are familiar with the school in question, such as high school administrators, corporate recruiters, and university officials are asked to comment.
While Keller believes the Maclean’s method for determining university reputation is sound, he also disagrees with the extent of its importance for the survey. “I would like to see the weight placed on reputation reduced. I think the reputation survey is valid and useful, but I wouldn’t want to rely too heavily on it.”
Keller recommends instead that more importance be placed on “student feedback information and student outcomes information,” which he admitted is not publicly available.
“I don’t have any problem with Maclean’s, I just have a problem with the composite. It’s unfair and misleading,” said Paul.
“Quite frankly, there isn’t a lot of overall difference in the quality of universities in Canada. It’s not like in the [United] States where you have a big range,” said Paul, who claimed that, “The [Maclean’s] rankings give an kind of artificial distance.”
“I think we should be measured by what we do and the student experience,” said Paul, who believes the University’s reputation will eventually improve along these indicators. “Ultimately the test is the student experience.”
Paul also offered a plan aimed at discovering why students, especially first-years, do not rate their university experience favourably. “I’m going to do some open meetings on campus [next semester]…and really just get out and talk to students.”
The decreased number of respondents Maclean’s receives for the reputation survey also makes the legitimacy and objectiveness of that indicator questionable. The response rate has fallen by almost half, to 7 per cent, over the past five years.
“We get 1,600 [responses] just from our own students telling us their experience,” claimed Zanutto of the greater effectiveness of other national surveys the University participates in, such as the National Survey of Student Engagement (NSSE) and Common University Data Ontario (CUDO).
“How many [respondents] are from Windsor they don’t tell us,” pointed out Zanutto at the number or individuals that participate in the reputation composite.
Despite its reputation ranking, the University did score well in several categories, including student support, library acquisitions, library holdings per student, and grants given to faculty. “We went to first place in student services, and we went from 8th to 5th [place] in student scholarships,” boasted Paul. Both subcategories, which comprise the student support indicator, are calculated based on the budget per cent allotted to each service.
The University also rose from 7th to 4th place in medical/science grants, and from 10th to 7th place in social sciences and humanities grants.
An increase was also seen in terms of entering averages, an indicator that was published by Maclean’s, but which did not count towards the University’s actual ranking.
It is hard to determine, especially from Maclean’s position, whether universities are more supportive of the rankings since the decision to use public information. Keller does not believe university approval is needed to conduct the rankings.
“The objective of this whole exercise is to serve readers, so our objective is not to satisfy universities. Universities are the subject of our coverage not our audience,” said Keller.
“I don’t think it’s a question of supporting Maclean’s. Maclean’s has every right to use public information,” remarked Paul. “If that helps promote universities, great.” Though this right exists,” Paul maintained, “I don’t think [universities] should spend a lot of time and effort assembling information for Maclean’s; they are out to sell magazines. None of us should be managing to Maclean’s.”
|
