Academic peer review
A detailed description of the methodology behind the academic peer review aspect of the THE - QS World University Rankings
The Academic Peer Review is the centrepiece of the Times Higher - QS World University Rankings carrying a weighting of 40%. It is the component that attracts the greatest interest and the greatest scrutiny and that which sets this ranking most clearly apart from any other.
The results are based on the responses to a survey distributed worldwide both to previous respondents and subscribers to two key databases: The World Scientific (www.worldscientific.com)from which 180,000 email addresses are drawn and the International Book Information Service (IBIS) a service from Mardev (www.mardev.com).The online survey can be previewed on the following link, but it is important to know that for a genuine respondent the survey is responsive to their early responses and only appropriate questions are asked.
Click here to preview Academic Peer Review Survey
The scores that contribute 40% to the overall score are compiled from five sets of results in the following five subject areas.
- Arts & Humanities
- Engineering & IT
- Life Sciences & Biomedicine
- Natural Sciences
- Social Sciences
For each subject area a regional weigthing is applied to ensure equal representation from our 3 "super regions" of Americas; Europe, Middle East & Africa and Asia Pacific.
In 2007 for the first time, respondents were prevented from selecting their own institution.
In 2008 the questions respondents are asked regarding universities inside their own country and outside were separated. These are recombined after the responses are in using methods designed to reduce any response bias by country.
The five subject areas are combined with equal weighting to form the overall score.
Ultimately the Academic Peer Review results in over 100,000 and climbing individual data points, that we use to evaluate universities for the Times Higher - QS World University Rankings.
Peer Review is an effective way to evaluate universities. It takes smart people to recognize smart people
Sir Richard Sykes
(Former) Rector, Imperial College
Frequently Asked Questions: Academic Peer Review
-
Why is the Academic Peer Review given a weighting of 40%?
-
Weightings are a subjective area of this research exercise, but in a scenario where multiple factors are being used, an essential component of how the rankings are compiled. In this context they are the responsibility of the Times Higher Education, but there are some clear reasons why this aspect carries such a significant proportion of the overall weighting.
Firstly, it's worth noting that unlike our citations per faculty score (which many commentators feel should carry greater weight) the peer review component offers an even handed perspective on the various broad subject areas - with institutional strengths in Arts & Humanities and Social Sciences able to contribute significantly to the overall view of an institution.
Secondly, the weight carried by the Peer Review would be likely to be reduced in a scenario where additional reliable indicators of institutional quality were identified.
Some commentators have suggested that a higher weighting ought to be placed on the Peer Review and Recruiter Reviews as they are aspects of the research most unique to this study and ergo most interesting.
Ultimately, in 2007, the Peer Review was compiled from over 5000 academic responses, each identifying an average of around 20 institutions as excellent in their given field providing over 100,000 distinct data points be which to evaluate the qualifying institutions. As the level of response improves the statistical strength of this measure will only develop.
Category: Methodology: Academic Peer Reviewxx
Submitting your vote...
-
How many responses are considered for the Academic Peer Review?
-
A three year latest response mechanism is used - so we use three year's worth of responses, but if anyone responds in more than one of those three years, only their latest response is considered. This led to 5,101 responses in 2007 and 6,354 responses in 2008
Category: Methodology: Academic Peer Reviewxx
Submitting your vote...
-
How are the peers selected?
-
For the 2004 Academic Peer Review a large database of respondents was purchased from the World Scientific and 1,300 respondents formed the universe for the first rankings.
Since then, things have moved on a bit... and the process runs as follows...
1. Previous respondents are invited to respond again - to update their opinions based on anything they may have learned since their prior response
2. We select 180,000 addresses from the World Scientific database based on balance of field prepresentaion and geography. Previous respondents to the survey are excluded.
3. We select slightly over 13,000 addresses from the International Book Information Service (IBIS) operated by Mardev (a division of Elsevier). These supplement the shortfalls of the World Scientific database - particularly in the Arts & Humanities. Again, prior respondents are excluded.
There are two ideas in play to boost the response in the future - one is an academic sign up facility to enable people to volunteer to participate in the survey and the other is an institution specific version of the survey that we would encourage university leadership to distribute to appropriate academics. Watch this space.
Respondents are not able to select their own university.
Responses are screened for junk and test entries and for any attempted manipulations from individual institutions or countries.
Category: Methodology: Academic Peer Reviewxx
Submitting your vote...
-
How are differing response levels from different countries handled?
-
Category: Methodology: Academic Peer Reviewxx
Submitting your vote...
-
The 2008 survey included separate questions for international and domestic universities. How are the results combined?
-
Category: Methodology: Academic Peer Reviewxx
Submitting your vote...
News: Peer Review
2008 Academic Peer Review Response Analysis
The Academic Peer Review currently forms 40% of the weightings applied by Times Higher Education to...
2007 Peer Review Response Analysis
The Peer Review currently forms 40% of the weightings applied by the Times Higher Education...
2006 Peer Review Response Analysis
The Peer Review currently forms 40% of the weightings applied by the Times Higher Education...