validity of the ukcat in applicant selection and ... · pdf fileseptember 2013 ournal of...

September 2013 ■ Journal of Dental Education 1159

Validity of the UKCAT in Applicant Selection and Predicting Exam Performance in UK Dental StudentsRizwana Lala, M.D.P.H.; Duncan Wood, Ph.D.; Sarah Baker, Ph.D.Abstract: The United Kingdom’s Clinical Aptitude Test (UKCAT) aims to assess candidates’ “natural talent” for dentistry. The aim of this study was to determine the validity of the UKCAT for dental school applicant selection. The relationship of the UK-CAT with demographic and academic variables was examined, assessing if the likelihood of being offered a place at a UK dental school was predicted by demographic factors and academic selection tools (predicted grades and existing school results). Finally, the validity of these selection tools in predicting first-year dental exam performance was assessed. Correlational and regression analyses showed that females and poorer students were more likely to have lower UKCAT scores. Gender and social class did not, however, predict first-year dental exam performance. UKCAT scores predicted the likelihood of the candidate being offered a place in the dental course; however, they did not predict exam performance during the first year of the course. Indeed, the only predictor of dental exam performance was existing school results. These findings argue against the use of the UKCAT as the sole determinant in dental applicant selection, instead highlighting the value of using existing school results.

Ms. Lala is Clinical Lecturer, Unit of Dental Public Health, University of Sheffield, United Kingdom; Dr. Wood is Senior University Teacher, Academic Unit of Restorative Dentistry, University of Sheffield, United Kingdom; and Dr. Baker is Reader, Unit of Dental Public Health, University of Sheffield, United Kingdom. Direct correspondence and requests for reprints to Ms. Rizwana Lala, School of Clinical Dentistry, University of Sheffield, 19 Claremont Crescent, Sheffield, S10 2TA, United Kingdom; [email protected].

Keywords: admissions test, aptitude test, applicant selection, UKCAT, exam performance, dental school admissions, United Kingdom

Submitted for publication 3/2/12; accepted 8/6/12

The United Kingdom’s Clinical Aptitude Test (UKCAT) was introduced in 2006 by the University Consortium Ltd., an organization

consisting of twenty-six medical and dental schools in the United Kingdom. Currently, applicants to study medicine or dentistry in any university member of this consortium are required to take the test unless they are residing in a country where there is no test center; an application without UKCAT results is rejected.1

The test is taken over one and a half hours and is comprised of four individually timed sub-tests: Quantitative Reasoning (QR), Decision Analysis (DA), Verbal Reasoning (VR), and Abstract Reason-ing (AR). QR assesses the ability to solve numerical problems; DA assesses the ability to deal with various forms of information and infer relationships; VR tests the ability to read and think logically about written passages; and AR aims to test convergent and diver-gent thinking along with critical evaluation skills.2-4 Taking the UKCAT costs between £65 and £100.5 This cost is borne directly by the student, although bursaries are available for some students who meet relevant need-based criteria.6

The rationale behind the introduction of the UKCAT was that it would widen university participa-tion, alleviate concerns raised with using predicted school grades for applicant selection such as bias, unreliability, and grade inflation, and help select students who would make the best future clinicians.1,7 Students from poorer backgrounds are underrepre-sented in higher education, including dentistry.8-10 Half of the population in England is in lower socio-economic groups,11 but in 2008 only 18 percent of successful applicants to dental schools were from the lower social classes.9 The UK government has reduced the central teaching grant to universities by £940 million, leading to an increase in tuition fees. Dental schools may charge up to £9000 per year, rais-ing further concerns about access to courses for these underrepresented groups.12 Widening participation is therefore a highly topical issue.8,13,14

Universities have aimed to broaden participa-tion by increasing the intake of students from lower socioeconomic groups and ethnic minorities through widening participation schemes. Such schemes are designed to assist students identified through objec-tive markers of deprivation such as address or pa-

1160 Journal of Dental Education ■ Volume 77, Number 9

The UKCAT Consortium argues that it is not only students with the strongest academic record who make the best clinicians and that the UKCAT helps select students who have “natural talent” and show a range of mental abilities, attitudes, and professional behavior considered desirable in future clinicians.1,3,4 Internationally, aptitude tests have long been used to select dental students,23 but may lack predictive abil-ity.24 Aptitude tests have also attracted controversy in studies of British students, with evidence showing that the verbal and mathematics parts of the tests are biased towards males,25 that attainment is higher for wealthier students than for their poorer counterparts with comparable school grades, and that students from independent schools do better.26 Therefore, the biases that aptitude tests are trying to avoid still exist, and as a consequence, their role in widening participation and fair student selection is unsupported by current research.26

In previous research, the UKCAT has dem-onstrated the same biases as predicted grades,7 and the evidence assessing the reliability, validity, and predictive ability of the UKCAT is equivocal. One study found the UKCAT to have significant pre-dictive ability for university exam performance in years one and two for Newcastle medical students, with the exception of performance in clinical and communication skills.27 Another study showed the UKCAT and its subtests failed to predict university exam performance in the first year at two medical schools.28 Yet another study found that even though the overall predictive value of the UKCAT was low, the individual subtests may have value.29 Currently, there are no published studies about the validity and reliability of the UKCAT amongst dental students.

The Higher Education Steering Group has ar-gued that the admissions process should be evidence-based with robust validation and reliability testing and should be supported by statistical evidence.30,31 However, there has been no evidence published on the validity of the UKCAT for selecting dental students to date. Therefore, dental schools using the UKCAT without such evidence may be subject to legal challenge.32 Medical and dental students have also expressed concerns about the lack of transpar-ency regarding how the test is used and the test’s validity, cost, and fairness.33,34

Given the lack of previous research addressing the validity of the UKCAT in dentistry, the objectives of our study were to address three research questions: 1) do demographic factors and academic selection

rental occupation or students with no family history of access to higher education. It was thought the UKCAT would help widen university participation further: it claims to be an objective test of clinical aptitude because it includes no curriculum content and allegedly measures natural talent.1 Therefore, perceived advantages gained from private tuition and fee-paying schools available to students from wealthier backgrounds would be made redundant, and dental applicants would be selected on the basis of merit irrespective of background.

In the UK, candidates apply for university study through a centralized service and can apply to up to six institutions. Predicted school grades (prediction of their performance in the final school examinations, or A-levels) are used as one tool to select students. Other tools include existing school results, a per-sonal statement by the applicant, references from the school, and interviews. A-level grade predictions are made by the students’ school; candidates have not completed A-level examinations at the time of university application and interview, so actual grades are unavailable. It has been noted that these predicted grades are unreliable,15,16 causing specula-tion in the media, academic literature, and public bodies that they are biased towards males of higher socioeconomic groups and thus unfairly disadvan-taging women and poorer students in the university selection process.7,17-19 The alleged advantage of the UKCAT over the use of predicted grades is that it is an actual achieved score, rather than the perceived ability of the student by an external party that has not been validated and therefore may be subject to bias.

In addition to the unreliability of predicted grades, examination passes at A-level have improved over the years with more students obtaining the top A grade.20 This has made it difficult for universities to discriminate among candidates. There are also concerns about the A-levels themselves and whether they are a reliable tool to predict university perfor-mance.21 School students in the UK also obtain grades for a variety of subjects when they are examined at the age of sixteen (General Certificate of Secondary Education [GCSE] results). These existing school results are available to universities during selection and have been shown to have predictive value.22 Therefore, the use of the UKCAT, the cost of which is borne directly by students, can only be justified if it adds validity to the selection process. Universities need to be certain that the use of the UKCAT is fair and that it truly does widen participation.


The overall marks for each semester’s summa-tive examinations were obtained. Re-sit examina-tion results were not included in the analyses. One hundred and sixty-four students were in the course out of the 434 students offered a place. (Not all the students offered a place accepted the offer, and some students did not meet the conditions of the offer, i.e., did not attain the required A-level grades.) A total of 135 out of 164 students (82.3 percent) were included in the analyses. This percentage was not 100 percent because twenty-nine students could not be found on the admissions database. (This was confirmed by a second independent person; some of these may be students repeating a year.) Of the final included sample, seventy-five (55.6 percent) were female, and twelve (8.9 percent) were part of a WPS.

Admissions Procedures and First-Year Exams

In the interest of fairness, the dental school in making admissions decisions assessed the results of only eight existing school results (GCSEs) because some state schools offer only eight subjects. An initial shortlist of applicants for interview was created by admissions tutors using these existing school results, as well as the predicted grades, personal statements, and references. This shortlist was further refined using the total UKCAT score. There is no formal minimum UKCAT score required, and UKCAT subtest scores are not assessed separately. Personal statements, references, and interviews were assessed subjectively by admissions tutors; consequently, these selection tools had no quantifiable data avail-able for analyses.

The first year of this dental school is divided into two themes: The Human Body, which is exam-ined in the first semester, and The Oral Cavity in Health and Disease, which is examined in the second semester. The themes focus on anatomy and physi-ology as well as pathologic processes underlying disease. Each theme was analyzed separately due to differences in curriculum content and to assess the temporal robustness of the UKCAT.

Variables AssessedTwo demographic variables were assessed:

gender (males=0; females=1) and WPS (yes=1; no=0). For academic variables, the best eight existing school results were included in the analysis. These were quantified as follows: A grade=12, B=10, C=8,

tools predict UKCAT results? 2) to what degree does the UKCAT predict acceptance to dental school? and 3) to what degree does the UKCAT predict first-year exam performance in dental school?

MethodsThis study included data for the 2008-09 and

2009-10 entries to one UK (Sheffield) dental school. Formal ethical approval was not needed for the use of routinely collected anonymized data. Applicants taking the UKCAT gave consent for the use of their anonymized data for the appraisal of the UKCAT and educational research.

Applicant and Examined SampleThe UKCAT results were downloaded from the

UKCAT website by two people and matched to the Sheffield applicants using an applicant identification number. Independent spot checks were carried out manually for one in every twenty-five applicants by checking the Sheffield data against the original UKCAT downloaded spreadsheet.

The total sample consisted of 2,118 applicants (100 percent of the applicant sample), of whom 51.7 percent were female. Only 7.7 percent of the sample was part of a Widening Participation Scheme (WPS). Sheffield dental school recruits students from two widening participation schemes: Access to Dental Occupation: Practice and Tutoring (ADOPT) and Compact. Compact is Sheffield university’s general scheme, open to all students across the university applying for any course, and ADOPT is specifically for those wanting to apply for dental courses. WPS students are guaranteed an interview to an appropri-ate course at Sheffield, with the proviso that mini-mum academic standards are met. Sometimes WPS students are given a conditional offer with lower A-level grade attainment. Applicants who were not undertaking A-levels (e.g., applicants with Scottish entrance qualifications since Scottish school-leaving examinations are different from English final ex-aminations) were excluded from the analysis. This resulted in a sample of 1,809 with predicted grades (predicted A-level grades) for analysis (85.4 percent). Of these, the majority (1,145; 63.3 percent) were predicted all A grades. Of the sample, 20.5 percent (n=434) were given a conditional offer of a place in the dental course and would be accepted depending on the A-level grades they attained.


classified as small (0.02-0.14), moderate (0.15-0.34), or large (≥0.35).35

The second research question was as follows: to what degree does the UKCAT predict acceptance to dental school? To address this question, a logistic regression was carried out with five predictor vari-ables (gender, being part of a WPS, existing school results, predicted grades, and mean UKCAT score) with offer/reject as the outcome variable. As with the analysis for the first question, further exploratory regression analyses were carried out replacing mean UKCAT score with the UKCAT subtest scores.

The third research question was as follows: to what degree does the UKCAT predict first-year exam performance in dental school? To address this question, a multiple regression was carried out with five predictor variables (gender, being part of a WPS, existing school results, predicted grades, and mean UKCAT scores) and overall first- and second-semester marks in first-year dental exams separately as the outcome variables. Exploratory analyses of the UKCAT subtests could not be carried out due to the small sample size.

Results Table 1 shows the study variables. All continu-

ous variables were normally distributed with the ex-ception of existing school results, which were slightly negatively skewed. However, with reasonably large samples, as seen in this study, skewness did not make a substantive difference in the analysis.36

Pearson’s r and Spearman’s rho intercorrela-tions are shown in Table 2. Cohen’s guidelines were used to assess the strength of the associations: weak (0.1-0.29), moderate (0.3-0.49), and strong

D=6, and E=4. Predicted grades fell into four catego-ries (BBC, ABB, AAB, AAA) and were scored 0-3. A random 10 percent of the sample was assessed for accuracy by a second independent person.

The AR score on the UKCAT was missing for the 2008-09 cohort due to a nationwide error.4 Therefore, to eliminate this disparity between the 2008-09 and 2009-10 entries, the mean UKCAT score was calculated using three subtests for the 2008-09 cohort and all four subtests for the 2009-10 cohort. All the subtests QR, DA, VR, and AR were included in the analyses, but the AR analysis was limited to one year due to the missing data. For the university performance variable, the overall marks for each semester were analyzed.

Data AnalysesInitial descriptive and preliminary analyses

were carried out to ensure there were no violations of the assumptions of normality, linearity, multicol-linearity, and homoscedasticity, to identify outliers, and to describe the data. Statistical analyses were then carried out for each of the three research ques-tions as follows.

The first research question was as follows: do demographic factors and academic selection tools predict UKCAT results? To address this question, a multiple regression was carried out with four predic-tor variables (gender, being part of WPS, existing school results, predicted grades) and mean UKCAT score as the outcome variable. Given previous evidence that suggests contradictory findings for the UKCAT subtests,28,29 separate regression analyses were carried out for the four UKCAT subtest scores. Cohen’s effect sizes (f2) were calculated from the R

square output from the regression model and were

Table 1. Descriptives of the study variables

N Range Mean Std. Deviation

Gender 2,118 – – –WPS 2,116 – – –Predicted grades 1,809 – – –Existing school results 1,799 102.00 97.37 12.47Mean UKCAT score 1,905 467.50 591.25 64.25QR 1,903 550.00 624.54 81.55DA 1,903 600.00 585.51 99.85VR 1,903 560.00 564.36 83.37AR 982 590.00 594.87 80.73Offer/Reject 2,115 – – –


(0.5-1.0).37 There was a weak significant association between gender and existing school results, with women likely to do better. Weak but significant associations between gender and mean UKCAT results, QR, and VR were also seen, with men more likely to perform better in these components. Being part of a WPS was significantly related to existing school results, mean UKCAT scores, QR, DA, and VR; those students from a WPS were more likely to perform poorly in these school and UKCAT assess-ments. Existing school results were related to mean UKCAT scores, VR, and DA: those with higher existing school results were moderately more likely to obtain higher mean UKCAT, VR, and DA scores. Existing school results were also related to QR and AR subtests of the UKCAT.

Overall, being offered a place in the dental course was associated with UKCAT scores, and applicants with higher mean UKCAT scores and existing school results were moderately more likely to receive an offer. Each subtest of the UKCAT was associated with being accepted. Finally, females were more likely to be accepted than males.

Predictive Value of Demographic Factors and Academic Selection Tools for UKCAT Results

On the first research question, the multiple regression model was significant (F[1521]=88.55, p<0.0001), with 18.9 percent of the variance in mean UKCAT scores explained by gender (β=-0.11, p<0.001), whether the student was part of a WPS (β=-0.06, p<0.01), and existing school results (β=0.43, p<0.001). Predicted grades were not a significant predictor of UKCAT scores (β=-0.03, p=0.18). Existing school results made the strongest unique contribution in predicting the mean UKCAT score. Indeed, the part correlation coefficient of the existing school results squared (0.4032) demonstrated that 16.2 percent of the total variance (18.9 percent) was explained by existing school results. Calcula-tion of Cohen’s effect size showed a medium effect (f2=0.23).35

Further exploratory regression analyses of the UKCAT subtests were significant, explaining 11 percent (QR), 12 percent (DA), 12 percent (VR), and 4 percent (AR) of the variance in each of the subtest scores. As with the overall UKCAT score, existing school results were a significant predictor in all sub-tests. In addition, gender significantly predicted QR and VR, and WPS significantly predicted DA. Over- Ta

ble

2. I

nter

corr

elat

ions

bet

wee

n st

udy

vari

able

s


In analyses of the UKCAT subtests, the binary logistic regression showed gender, being part of a WPS, existing school results, predicted grades, QR, DA, VR, and AR significantly predicted who would get an offer to attend Sheffield dental school (X² [8, N=796]=133.10, p<0.000). The model as a whole explained 25.2 percent (Nagelkerke R square) of the variance, less than the mean UKCAT alone. The model correctly identified 18.2 percent of the applicants who were accepted and 95.9 percent of the applicants who were rejected. The positive predictive value of the model was 49.1 percent, and the negative predictive value of the model was 84.3 percent.

In summary, WPS students, women, and those with better existing school results and UKCAT scores were more likely to be accepted for the course. Unusually, applicants with lower predicted grades were more likely to receive an offer. Students with higher UKCAT subtest results were also more likely to receive an offer, but the reduced sample size in the exploratory analyses meant that gender, WPS, and predicted grades were not significant predictors. However, there was a trend (p=0.051) that those from a WPS were more likely to be accepted (see Table 4).

all, predicted school grades did not predict UKCAT results; however, demographic variables (gender, being part of a WPS) and existing school results were statistically significant predictors of UKCAT results.

Predictive Value of UKCAT for Acceptance to Dental School

On the second research question, the results of the binary logistic regression indicated that gen-der, being part of a WPS, predicted grades, existing school results, and mean UKCAT scores significantly predicted who was accepted or rejected for the dental course (X²[5, 1438]=289.92, p<0.000). The model explained 27.5 percent (Nagelkerke R square) of the variance. The model correctly identified 23.5 percent of the applicants accepted and 93.5 percent of those rejected. The positive predictive value of the model was 52.9 percent, and the negative predictive value was 77.8 percent. All the predictors were significant. The odds ratio of a student from a WPS being ac-cepted was 1.90 (CI=3.22-1.12), for females 1.32 (CI=1.78-1.00), predicted grades 1.02 (CI=1.01-1.03), existing school results 1.06 (CI=1.04-1.08), and mean UKCAT score 1.02 (CI1.01-1.02) (see Table 3).

Table 3. Factors predicting acceptance to University of Sheffield dental school and effect sizes

95% C.I. for Exp(B)

B S.E. Wald df Sig. Exp(B) Lower Upper

Gender -0.286 0.144 3.952 1 0.047 0.752 0.567 0.996WPS -0.639 0.269 5.644 1 0.018 0.528 0.312 0.894Predicted grades 0.016 0.005 11.077 1 0.001 1.016 1.007 1.026Existing school results -0.056 0.010 33.623 1 0.000 0.945 0.927 0.963Mean UKCAT -0.017 0.002 123.631 1 0.000 0.983 0.980 0.986

Table 4. UKCAT cognitive subtest scores predicting acceptance to University of Sheffield dental school and effect sizes

95% C.I. for Exp(B)

B S.E. Wald df Sig. Exp(B) Lower Upper

Gender -0.200 0.217 0.846 1 0.639 0.358 0.535 1.254WPS -0.716 0.408 3.072 1 0.051 0.080 0.220 1.088Predicted grades 0.019 0.007 7.447 1 0.155 0.006 1.005 1.034Existing school results -0.043 0.013 10.247 1 0.000 0.001 0.933 0.983QR -0.003 0.002 4.688 1 0.000 0.030 0.994 1.000DA -0.004 0.001 11.002 1 0.000 0.001 0.993 0.998VR -0.005 0.001 10.237 1 0.000 0.001 0.992 0.998AR -0.006 0.001 15.817 1 0.000 0.000 0.991 0.997


effect size of the model was moderate, f2=0.19. In summary, only existing school results predicted first-year exam performance in both semesters.

DiscussionIn this sample, predicted school grades did not

predict UKCAT performance; however, demographic variables (gender, being part of a WPS) and existing school results were important. WPS applicants and women were most likely to be offered a place in the dental course, but existing school results and UKCAT results also significantly predicted acceptance to the school. In addition, of all of the variables, only existing school results significantly predicted exam performance in the first year of the course. These results are discussed for each research question.

First Research QuestionThe first research question sought to determine

if demographic factors and academic selection tools predict UKCAT results. Existing school results were the strongest predictor of the UKCAT and its subtests, followed by gender and being part of a WPS. Appli-cants with better existing school results were more likely to have higher UKCAT scores, including all the subtests. Women and WPS students were more likely to have lower mean UKCAT scores. Differ-ences were also seen in individual subtest scores, with WPS students doing less well in DA and women obtaining lower scores in QR and VR.

The differences in performance by socioeco-nomic status may be due to an inherent bias of the UKCAT as shown by previous research7 or due to the differences in preparation and availability of

Predictive Value of UKCAT for First-Year Exam Performance

See Table 5 for the descriptives of exam perfor-mance. One hundred and twenty-six students (93.3 percent) passed the first semester. The failure rate for the second semester was slightly higher: 123 students (91.1 percent) passed. The intercorrelations between exam performance and study variables are shown in Table 6. There was a statistically significant relation-ship of moderate strength between existing school results and first-year dental exam performance. Those students with higher existing school results obtained higher marks in both semesters’ examinations. Stu-dents with higher DA scores were also significantly more likely to get better marks in both semesters of the first year.

The multiple regression model of the overall first semester results was significant (F[83]=4.4, p<0.0001) and showed 20.9 percent of the variance in the overall first semester marks was explained by gender, whether the student was part of a WPS, exist-ing school results, and predicted grades. However, existing school results were the only statistically significant predictor (β=0.42, p<0.001), explaining 16.2 percent (0.4022) of the total variance. Cohen’s effect size of the model was moderate, f2=0.26.

The multiple regression model of the overall second semester results was significant (F[83]=3.12, p<0.012) and showed 15.9 percent of the variance in the overall second semester marks was explained by gender, whether the student was part of a WPS, existing school results, and predicted grades. How-ever, existing school results were the only statistically significant predictor (β=0.33, p<0.005), explaining 9.7 percent (0.3112) of the total variance. Cohen’s

Table 5. Descriptives (mean, SD) of exam performance

N Range Mean Std. Deviation

Gender 135 – – –WPS 135 – – –Existing school results 125 76.00 102.51 10.31Predicted grades 93 – – –Mean UKCAT score 127 265.83 631.08 44.51QR 127 390.00 654.80 67.75DA 127 460.00 626.69 76.30VR 127 360.00 611.89 69.01AR 65 390.00 635.54 75.481st semester overall 135 53.00 63.03 10.042nd semester overall 135 46.00 63.18 10.78


resources between the groups. It has been argued that applicants cannot prepare for the UKCAT as it is an objective test of clinical aptitude without a curriculum; however, the UKCAT consortium’s own data show that a quarter of the students attend courses and three-quarters purchase books.33 It may be that wealthier students do not necessarily have a greater clinical aptitude but have managed to pur-chase effective preparation strategies. There is also evidence suggesting that aptitude test performance can be improved with coaching.38

Our study also found that women were less likely to do well on the UKCAT overall and on VR and QR. This is consistent with the results of a previ-ous study demonstrating the UKCAT to be inherently biased towards males.7 It also mirrors the controversy seen in a large British study that found women were less likely to do well in aptitude tests, particularly the verbal and mathematics components.25,39

In our study, predicted school grades (predicted A-level grades) did not make a significant contribu-tion in predicting UKCAT performance. This may be because the UKCAT is an actual achieved score and thus a more accurate reflection of the candidate’s ability; hence, the inherent error in predicted school grades was reflected in their inability to predict actual UKCAT results. This may indicate that the UKCAT may help alleviate some of the biases associated with inaccurate school grade prediction. As well as inac-curacies in school grade prediction, grade inflation has also been a concern.20

Another justification for using the UKCAT has been its ability to select the best candidates.27 Most applicants in our study achieved the maximum pre-dicted school grades, and there are concerns about bias with predictions. On the contrary, a wide range was seen with the UKCAT results, giving them po-tential discriminatory power. However, the existing school results also showed a wide range and are avail-able to admissions tutors at the time of applicant se-lection; hence, they are immune to the biases of grade prediction. In this case, the use of the UKCAT scores could only be justified if they are fair and contribute additional information to university selection that existing school results do not provide. Differences in UKCAT performance with respect to gender and socioeconomic status have been discussed, and the results from our analysis showed existing school results made the greatest significant unique contri-bution in predicting mean UKCAT scores as well as the UKCAT subtest scores. This means that those students with better existing school results were also Ta

ble

6. I

nter

corr

elat

ions

bet

wee

n ex

am p

erfo

rman

ce a

nd s

tudy

var

iabl

es


likely to receive an offer. This may be because ap-plicants from a WPS are given dispensation in the se-lection process, and the correlational analysis showed that these students were more likely to have lower predicted grades. The evidence of those from lower social classes having lower predicted school grades due to systematic bias7,15,18,19 must also be kept in mind. It may be that although these applicants have lower predicted grades, their actual grades may not be low.

Third Research QuestionThe third research question asked to what

degree the UKCAT predicted first-year exam perfor-mance. The UKCAT did not predict first-year dental exam performance; however, higher DA scores were significantly related to better marks in both semes-ters. Only existing school results predicted first-year examination performance, while predicted grades, gender, and being part of a WPS did not.

The UKCAT results in this study had a wide range; therefore, they were initially considered ben-eficial in being able to discriminate between students. However, our analysis did not find evidence for mean UKCAT results predicting performance in the overall marks of the first two semesters. This is consistent with recent research amongst British students, where an aptitude test showed a wide range25,26 but was not beneficial in identifying the most able students.22 There is no previous published research amongst dental students; however, Lynch et al. also found that the UKCAT failed to predict exam performance in the first two years of medical school.28

Although the UKCAT did not have predictive ability in our study, recent research has concluded that individual subtests do have predictive value for specific courses.29 Unfortunately, further exploratory analysis of the UKCAT subtests in our study could not be carried out due to the small sample size. Nev-ertheless, correlational analyses showed significant associations between the decision analysis subtest and the overall marks in the first two semesters. Those with higher DA scores were more likely to get higher overall marks in the first and second semesters. This may be because, unlike the other UKCAT subtests,40 the DA component was specifically developed for clinical practice.7 Thus, instead of using the overall UKCAT score, universities may be advised to look at individual subtest scores, in particular the DA subtest score.

This is even more important given that, with the exception of DA scores, the remainder of the UKCAT is a generic psychometric test used for various profes-

more likely to do well on the UKCAT and its subtests. An explanation for this may be that the UKCAT is testing similar elements of ability as school subjects and is not providing additional information for the selection process.

Second Research QuestionThe second research question asked about the

degree to which the UKCAT predicts acceptance to dental school. We found that the applicants with higher mean UKCAT and UKCAT subtest scores were more likely to be accepted to the course. This would be expected as the UKCAT is a selection tool used by the dental school. Existing school results also predicted an offer to study dentistry. The weakest predictor of an offer was predicted grades. Demo-graphic variables such as socioeconomic group and gender were the strongest predictors.

WPS students were nearly twice as likely to obtain an offer to study dentistry, presumably be-cause they were offered dispensation in the selection process, i.e., they were guaranteed an interview and were sometimes offered a place with lower school grades. Therefore, although these students did less well on the UKCAT, they were not discriminated against in the selection process. The differences in performance on the UKCAT do raise concerns that it has the potential to be counterproductive and reduce the opportunity of poorer students to be accepted, as not all eligible students would be part of a WPS due to the limited number of places.

In addition, women were more likely to be offered a place in the course despite having poorer performance on the UKCAT. This may be because females were found to be more likely to have better existing school results than men (see Table 2).

The weakest predictor of an offer to study den-tistry was predicted grades. This may be attributed to grade inflation, which is a national issue,20 and that most candidates were predicted to receive all A grades. Dentistry has very high grade requirements, and students with low predicted school grades may not have been encouraged to apply. If the advantage of the UKCAT is that it is immune to teachers’ pre-conceptions of students’ ability and consequently fairer than predicted grades, students with lower grades but good UKCAT results should be encour-aged to apply. However, currently at Sheffield, stu-dents need a strong academic record as well as good UKCAT results on application.

Despite this, it was unusual to see that it was those with lower predicted grades who were most


In addition, being part of a WPS did not predict first-year dental exam performance. This highlights the success of widening participation schemes and is consistent with previous evidence showing that although students from poorer backgrounds may have lower school grades (and in our study also lower UKCAT results), this may not necessarily be translated into less than satisfactory achievement in university education.22, 47

Conclusion and LimitationsWhilst our study has highlighted some im-

portant findings, there were a number of limitations of the research. The AR scores were only available for half the sample. The sample size for the exam performance analysis was small and limited to the first year of one dental school; therefore, exploratory analyses of the UKCAT subtests could not be carried out. Detailed information about the social class and ethnicity of the applicants was unavailable. There was also no data about some of the selection tools used—namely, interview performance, references, and personal statements. Further research is needed in collaboration with dental school members of the university consortium using more detailed demo-graphic data as well as all the information used for applicant selection to ascertain the most valid tools for dental student selection. Despite these limitations, the findings indicated that the UKCAT is unable to predict first-year dental exam performance.

Although the UKCAT consortium claims the test evaluates mental abilities, attitudes, and profes-sional behavior desirable in future clinicians,1,3,4 cur-rently the test does not assess attitude or professional-ism, and in this study the mental abilities tested failed to determine first-year dental exam performance. It is also unlikely the UKCAT would widen participation, not only because of the differences in performance with respect to socioeconomic status but because the extra direct and indirect financial costs of taking the test would pose an additional barrier for poorer students as only the poorest ones would receive a bursary to take the test.13 Dental schools are now charging up to £9,000 tuition fees per annum. That coupled with the costs of the UKCAT and the length of dental training may deter poorer students from ap-plying. The UKCAT also raises concerns about equity of access for women because there were differences in UKCAT performance with respect to gender, with females less likely to do well.

sions. Hence, it was not developed to assess clinical aptitude, and biases relating to gender and ethnicity have been identified.40 As a result, there are concerns about the generalizability of the UKCAT for clinical students, and specifically for dental students, and also the fairness of the test.

The existing school results of the students in our study had a wide range, giving discriminatory potential, and were found to be the most important predictor of first-year exam performance. This is consistent with the results from a recent five-year study that showed this school attainment alone even without A-level achievements was an important pre-dictor of degree performance and that aptitude tests did not add anything extra.22 This may be because existing school results not only have a wide range like the UKCAT, but they also have a wide diversity in the range of subjects studied; therefore, they may provide a broader assessment of the candidate’s cog-nitive abilities.41 In the United States, where aptitude tests are used routinely, the high school record has been found to be a better predictor of degree per-formance than aptitude tests.39 Specifically for U.S. dental students, aptitude tests have not demonstrated predictive ability.24

In the UK, a great deal of emphasis is placed on predicted school grades in university selection, regardless of the course. Surprisingly then, predicted grades did not predict first-year dental exam perfor-mance in our study. This finding is not unprecedent-ed: there have been concerns raised in the literature about the reliability of these grades (A-levels) in predicting university performance.21 The narrow range seen in the predicted grades may be another reason why they were not beneficial in predicting first-year exam performance as most students were predicted to earn the maximum grades.

Finally, gender did not predict first-year dental exam performance, yet gender did predict the mean UKCAT score and subtests QR and VR. Since the UKCAT claims to assess clinical aptitude, it could be argued that the UKCAT is not biased against women and the differences seen are merely highlighting that women may not have the clinical aptitude for dentistry. However, if this were the case, it ought to have been reflected in dental exam performance, which was not the case in our study, and the evidence from previous research does not support this assertion either.42-46 In fact, the evidence suggests that aptitude tests consistently underpredict female higher educa-tional achievement.39


14. Denham J. Universities need to have access to our most gifted students. The Times, October 20, 2008. At: www.timesonline.co.uk/tol/comment/columnists/guest_ contributors/article4974308.ece. Accessed: January 12, 2011.

15. Snell M, Thorpe A, Hoskins S, Chevalier A. Teachers’ per-ceptions and A-level performance: is there any evidence of systematic bias. Oxford Rev Educ 2008;34 (4):403-23.

16. New proposals to make applying to higher education fairer, 2005. At: www.dcsf.gov.uk/pns/DisplayPN.cgi?pn_id=2005_0098. Accessed: February 21, 2011.

17. British Broadcasting Corporation. Timing of A-level ex-ams could be changed. January 15, 2011. At: www.bbc.co.uk/news/education-12199669. Accessed: January 20, 2011.

18. Improving the higher education consultation process: government response to consultation, 2006. At: www.education.gov.uk/consultations/downloadableDocs/Im-proving%20the%20HE%20Applications%20Process%20-%20Government%20Response%20.pdf. Accessed: February, 24, 2011.

19. Hayward G, Sturdy S, James S. Estimating the reliability of predicted grades. Cheltenham, UK: Universities and Colleges Admissions Service, 2005.

20. Tomlinson M. 14-19 curriculum and qualifications reform: final report of the working group on 14-19 reform, 2004. At: www.dcsf.gov.uk/14-19/documents/Final%20Report.pdf. Accessed: March 11, 2011.

21. Smith J, Naylor R. Schooling effects on subsequent university performance: evidence for the UK university population. Coventry, UK: University of Warwick, 2000.

22. Kirkup C, Wheater R, Morrison J, Durbin B, Pomati M. Use of an aptitude test in university entrance: a validity study, 2010. At: www.bis.gov.uk/assets/biscore/higher-education/docs/u/10-1321-use-of-aptitude-test-university-entrance-validity-study.pdf. Accessed: May 3, 2011.

23. Smith RV. Aptitudes and aptitude testing in dentistry. J Dent Educ 1943;8(1):55-70.

24. Gray SA, Deem LP, Straja SR. Are traditional cognitive tests useful in predicting clinical success? J Dent Educ 2002;66(11):1241-5.

25. Kirkup C, Schagen I, Wheater R, Morrison J, Whetton C. Use of an aptitude test in university entrance: a validity study—relationships between SAT scores, attainment measures, and background variables, 2007. At: www.education.gov.uk/publications//eOrderingDownload/RR846.pdf. Accessed: March 11, 2011.

26. Kirkup C, Wheater R, Schagen I, Morrison J, Whetton C. Use of an aptitude test in university entrance: a validity study—2008 update: further analyses of SAT data. At: www.bis.gov.uk/assets/biscore/corporate/migratedD/publications/D/DIUS_RR_09_02. Accessed: May 2, 2011.

27. Wright S, Bradley PM. Has the UK clinical aptitude test improved medical student selection? Med Educ 2010; 44:1069-76.

28. Lynch B, MacKenzie R, Dowell J, Cleland J, Prescott G. Does the UKCAT predict year 1 performance in medical school? Med Educ 2009;43:1203-9.

29. Yates J, James D. The value of the UK clinical aptitude test in predicting preclinical performance: a prospective cohort study at Nottingham Medical School. BMC Med Educ 2010;10:55.

Finally, it was not the UKCAT but existing school results that provided some of the solutions to the problems seen with using predicted grades for applicant selection.7,15,17-20,31 These results are exist-ing achieved marks, which are therefore robust to any bias in grade prediction. Existing school results also showed a wide range and therefore useful in discriminating among the most able applicants and are consistently beneficial in predicting first-year dental exam performance.

AcknowledgmentsNone of the authors has a financial interest in

the UKCAT industry.

REFERENCES1. UK Clinical Aptitude Test. Background. At: www.ukcat.

ac.uk/pages/details.aspx?page=background. Accessed: January 20, 2011.

2. UK Clinical Aptitude Test. Overview. At: www.ukcat.ac.uk/pages/details.aspx?page=overview. Accessed: Janu-ary 27, 2011.

3. UK Clinical Aptitude Test. UKCAT 2008 annual report. At: www.ukcat.ac.uk/pdf/Annual%20Report%202008.pdf. Accessed: January 27, 2011.

4. UK Clinical Aptitude Test. UKCAT 2007 annual report. At: www.ukcat.ac.uk/pdf/UKCAT%20Annual%20 Report%202007.pdf. Accessed: January 27, 2011.

5. UK Clinical Aptitude Test. Registration questions. At: www.ukcat.ac.uk/pages/details.aspx?page=registrationQuestions#. Accessed: January 20, 2011.

6. UK Clinical Aptitude Test. Bursaries. At: www.ukcat.ac.uk/pages/details.aspx?page=Bursaries. Accessed: August 18, 2011.

7. James D, Yates J, Nicholson S. Comparison of A-level and UKCAT performance in students applying to UK medical and dental schools in 2006: cohort study. BMJ 2010;340:c478.

8. Higher Education Funding Council for England. Widen-ing participation. At: www.hefce.ac.uk/widen/. Accessed: January 12, 2011.

9. Dental School Council. Evidence for the panel on fair ac-cess to the professions: dentistry, March 2009. At: www.dentalschoolscouncil.ac.uk/documents/DentistryFair AccesstotheProfessionsMarch2009_000.pdf. Accessed: January 20, 2011.

10. Widening participation in higher education. London, UK: National Audit Office, The Stationary Office, 2008.

11. Young participation in higher education: report. Bristol, UK: Higher Education Funding Council for England, 2005.

12. Shephard J. Private school yield more than a quarter of medical and dentistry students. The Guardian, March 16, 2011. At: www.guardian.co.uk/education/2011/mar/16/quarter-medical-dentistry-students-private-schools. Accessed: May 19, 2011.

13. Hoare AG, Aitchison RL. PQA: pretty questionable as-sumptions. Hige Educ Rev 2009;42:17-24.


review. The Sutton Trust, 2001. At: www.suttontrust.com/research/aptitude-testing-for-university-entrance/. Accessed: March 10, 2011.

40. McDonald AS. Profiling for success: user’s guide. At: www.profilingforsuccess.com/ReasoningTestUserGuide_update_2010. Accessed: July 30, 2011.

41. Bell JF. Methods of aggregating GCSE results to predict A level performance. Paper presented at British Educa-tional Research Association Annual Conference, Cardiff University, Education-Line, September 7-10, 2000.

42. Ferguson M, James D, Madeley L. Factors associated with success in medical school. BMJ 2002;324:952-7.

43. Ferguson E, Sanders A, O’Hehir F, James D. Predictive validity of personal statements and the role of the five-factor model of personality in relation to medical training. J Occup Organiz Psychol 2000;73(3):321-44.

44. Mercer A. Selecting medical students: an Australian case study. Murdoch, WA: Murdoch University, 2007.

45. Silva ET, Nunes M, Queiroz MG, Leles CR. Factors influencing students’ performance in a Brazilian dental school. Braz Dent J 2010;21(1):80-6.

46. Sawair FA, Baqain ZH, Al-Omari IK, Wahab FK, Rajab LD. Effect of gender on performance of undergraduate dental students at the University of Jordan, Amman. J Dent Educ 2009;73(11):1313-9.

47. Hoare A, Johnston R. Widening participation through admissions policy: a British case study of school and uni-versity performance. Stud Higher Educ 2011;36(1):21-41.

30. Supporting professionalism in admissions: admissions tests. At: www.spa.ac.uk/admission-tests/index.html. Accessed: March 13, 2011.

31. Fair admissions to higher education: recommendations for good practice. The Schwartz Report. Nottingham, UK: Department for Education and Skills, Admissions to Higher Education Steering Group, 2004.

32. McManus IC, Ferguson E, Wakeford R, Powis D, James D. Predictive validity of the biomedical admissions test: an evaluation and case study. Med Teach 2011;33(1):53-7.

33. UK Clinical Aptitude Test. UKCAT candidate sur-vey, 2009. At: www.ukcat.ac.uk/pdf/Executive%20 summary%20for%20website.pdf. Accessed: January 20, 2011.

34. Cleland JA, French FH, Johnston PW. A mixed-methods study identifying and exploring medical students’ views of the UKCAT. Med Teach 2011;33(3):244-9.

35. Cohen J. Quantitative methods in psychology: a power primer. Psychol Bull 1992;112(1):155-9.

36. Tabachnick BG, Fidell LS. Using multivariate statistics. 5th ed. Boston: Pearson Education, 2007.

37. Cohen JW. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associ-ates, 1988.

38. Agbor-Baiyee W. A study of cognitive achievement in a special paramedical program. Coll Stud J 2009;43:1-36.

39. McDonald AS, Newton PE, Whetton C, Benefield P. Aptitude testing for university entrance: a literature

validity of the ukcat in applicant selection and ... · pdf fileseptember 2013 ournal of...

Documents