The NRF rating system and its appeals: some facts and some

fallacies

Duncan Mitchell

● I am a theoretical physicist and the panel consists only of experimental physicists, so they can’t assess me

● My h index is better that that of five B-rated researchers I know, but the committee gave me a C

● I was elected Vice-Chairman of the International Society of Orchid Botanists

so I must have an international reputation

● My research has improved but my rating has gone down

Reviewers

● The only assessors of research output

● Your choices and the specialist committee’s

choices

● Excluding potential reviewers

Pick a likely reviewer and write your

application for her/him

The reviewers’ focus

● Your best outputs of the review period

● Peer-reviewed journal articles

● Other legitimate research outputs

● Postgraduate students

Be careful out about conference proceedings

Teaching textbooks are not research outputs

Scoring the reviewers’ reports

● Familiarity with the research field

● Evidence-based assessment of the applicant’s

research in the review period

● Bias and hostility

● Halo effect

● Congruence with data in the application

Good reports count more

Arriving at the rating

● Enough usable reports?

● Search for the language:

…… coherent body of work?

…… promise, leadership?

…… significant international impact?

● Agreement between committee, assessor and

chair

Count up the A, B, C, P, Y, RU reports (weighting the good reports)

The Appeals Committee

● Mandate

● Membership

● Evidence: ……. appeal letter

..…... RRAP sheets

……. documents that served before the

specialist committee/ Executive

Evaluation Committee

……. and it has the right to call for more

reports

The Appeals Committee’s deliberations

● Is the appeal legitimate?

● What issues in the letter need a response?

● Did the rating process follow the rules?

● Were there enough usable reviewers’ reports?

● Was the set of reviewers (with usable reports)

balanced?

Does the Appeals Committee agree with the rating decision?

The difficult issues

● Applicants on a cusp

● Established early-career researchers

● Members of large research teams

● Researchers competent in more than one field

of research

● Researchers still active but past their peaks

● I am a theoretical physicist and the panel consists only of experimental physicists, so they can’t assess me

● My h index is better that that of five B-rated researchers I know, but the committee gave me a C

● I was elected Vice-Chairman of the International Society of Orchid Botanists

so I must have an international reputation

● My research has improved but my rating has gone down