Abstract

Introduction

Selection of UK medical students depends mainly on grades achieved in school leaving examinations, such as A levels.¹ Few long term studies have validated such selection measures,² and their theoretical underpinning is unclear. Examinations measure achievement, accomplishment, or attainment and assess whether students have mastered an academic subject. In contrast, measures of ability or aptitude assess cognitive ability independently of cultural content and educational experience and are typified by measures of intelligence (general mental ability³) (see www.bmj.com). Whereas intelligence shows stability through life,⁴ achievement tests depend mainly on recent educational experience.

Although seldom articulated, three arguments underpin selection with achievement tests:

To distinguish such positions we need to relate career outcomes to achievement and intellectual ability. In 2002 we followed up a cohort of clinical students who had taken a standard intelligence test when they entered Westminster Medical School between 1975 and 1982. We had four outcome measures:

For practical reasons we could not assess doctor-patient interaction.

Method

PRF administered the AH5 (a timed “high grade” intelligence test⁸) to clinical students entering the Westminster Medical School from 1975 to 1982. The test has measures of verbal and reasoning ability (part I, “verbal”) and spatial ability (part II, “spatial”). Students were informed that the test was confidential and for research and that results would not be available to teachers or examiners.

In 1988, PRF and ICM collated the results with date of birth, sex, A levels, intercalated degree results, finals performance, and performance in preregistration posts.⁹ A levels were scored as A=5 to E=1 and O/F=0, and summarised as the mean. Performance at finals was recorded as 4=distinction, 3=pass all first time, 2=pass after resits, 1=fail. Preregistration performance was the average consultant rating (4=outstanding, 3=good, 2=satisfactory, 1=inadequate).

In 2001 we used the Medical Register and Directory to trace the graduates. In January 2002 we sent a questionnaire to those on the 2001 UK Medical Register; non-respondents were sent two reminders. The questionnaire asked about career, qualifications, interests, and personality.^10,11 We assessed stress with the general health questionnaire (GHQ-12) and an abbreviated Maslach burnout inventory (aMBI),¹² with additional questions on satisfaction with medicine (see www.bmj.com). Statistical analysis used SPSS 10.0 and LISREL 8.51.

Results

The mean total AH5 score (fig A, bmj.com) of 40.4 was similar to norms,⁸ as were verbal and spatial scores (table A, bmj.com). The mean A level score (fig A, bmj.com) was 4.00, equivalent to grade BBB. AH5 score and A level grade were correlated (Pearson r=0.285, P < 0.001; fig B, bmj.com).

Dropouts from Medical Register—All 511 students registered with the General Medical Council, but only 464 were on the 2001 Medical Register. The 47 doctors who left the register (a mean of 11.1 years after qualifying; SD 5.9; range 2-23) had lower A level grades but not lower AH5 scores (table A, bmj.com); see www.bmj.com for ROC analysis. Two doctors subsequently returned to the register. Of the remainder, three had died, contact details were available for 35, and no information was available for seven.

Questionnaire response—Of the 464 doctors on the register, 349 (73%) replied to the questionnaire. Non-respondents had lower AH5 scores but did not have different A levels results (table A, bmj.com).

Career choice and career progression—Of 332 doctors for whom we had usable information, 173 worked in hospital (149 were consultants) and 131 in general practice (116 were principals). Of the remainder, four were not working, five had non-medical posts, and 19 had other medical posts. Hospital doctors had higher A level grades and AH5 scores (see table A on bmj.com), each effect being significant after we accounted for the other (A levels: Student's t test, t₂₉₉ = 2.674, P=0.008; AH5: t₂₉₉ = 2.059, P=0.040). Remaining analyses therefore took differences in speciality into account. Figure 1 shows the career progression of hospital doctors and general practitioners. Qualifications are grouped into memberships (MRCP, FRCS, etc), diplomas (or equivalent, often offered by Royal Colleges), and academic degrees (PhD, MD, masters, or bachelors degree). A levels had a highly significant effect on years to membership (table, Cox regression, P < 0.001; fig 2), even after we accounted for AH5 (P=0.001). AH5 had a significant simple effect on years to membership (P=0.049) but not after we accounted for A levels (P=0.401). Other effects of A levels and AH5 were not significant after we accounted for multiple testing.

Fig 1

Careers of doctors in hospital medicine and general practice (career grade above and acquisition of memberships, diplomas, and academic degrees below)

Fig 2

Kaplan-Meier plots for percentage of doctors obtaining membership in relation to A level grade, after taking hospital/general practice differences into account

Structural modelling of educational achievement—We modelled academic and professional achievement using structural equation modelling with causal order mainly determined by temporal order, except that we regarded AH5 score before A levels. Goodness of fit was excellent (χ²=4.90, df=8, P=0.768; GFI (goodness of fit index)=0.995; AGFI (adjusted goodness of fit index)=0.988). Each stage predicted the subsequent stage, and A level grade and finals performance had additional direct effects on time to membership (fig 3).

Fig 3

Path model of causal associations between different educational achievements of doctors. Coefficients represent standardised path coefficients (β coefficients) with their associated significance levels

Research publications—In total 138 doctors (40%) had not published any research papers, 44 (13%) had published 1-2 papers, 36 (11%) 3-5 papers, 30 (9%) 6-10 papers, 39 (11%) 11-20 papers, 18 (5%) 51-100 papers, 29 (9%) 21-50 papers, 8 (2%) had published more than 100 papers. Regression of normal scores (ranked normal deviates; normal order statistics) showed differences and between hospital doctors and general practitioners (P < 0.001) but no effect of A levels or AH5 score (table).

Table 1

Effects of mean A level grade, total AH5 score, and AH5 verbal and spatial subscores on various outcome measures. All effects are simple effects that do not take other variables into account; all analyses, however, take differences in general practice/hospital ...

Stress, burnout, and satisfaction with medicine as a career—Sixty two doctors (18%) scored ≥ 4 on the general health questionnaire, indicating “caseness” for stress. General practitioners scored higher than hospital doctors on measures of emotional exhaustion, depersonalisation, and personal accomplishment in the Maslach burnout inventory but did not differ on the general health questionnaire (0-1-2-3 scoring) or on satisfaction with a medical career. No measure showed any association with A level grades or AH5 score (table).

Discussion

Few studies have attempted to validate the selection procedures for medical students, although in such studies the effect size of academic measures for postgraduate performance is 0.48.² Despite A levels being the basis for selection in the United Kingdom,¹ little evaluation has taken place, and although occasional comments suggest that A levels are “completely unpredictive” they actually predict early dropout from medical school.^13,14 For university degrees overall, A levels also predict degree class, dropout, and repeated years, particularly for science.^13,14 We have shown that A level results, which are measures of achievement, can predict time taken to gain membership qualifications, choosing to become a general practitioner, and leaving the register. In contrast the AH5, which measures ability, cannot independently predict membership qualifications or dropout.

A levels therefore have validity in selection, with a validity coefficient of about 0.3 (see www.bmj.com), although care should be taken in generalising the results to other examinations in other countries. Intelligence does not predict careers, thus rejecting the ability argument. A levels predict because they assess achievement, and the structural model shows how past achievements predict future achievement. Our data cannot distinguish the achievement argument and the motivation argument, although the long term, direct effect of A levels on membership examinations (fig 3) suggests that motivation might be important.

Despite their predictive ability, A levels are probably not the only predictors² and should not be the sole basis for selection.¹⁵ Some of our other outcomes were not predicted by A levels but were correlated with measures of personality (see www.bmj.com) and would probably also be predicted by learning styles.^16,17 West answered Smith's editorial question of “Why are doctors so unhappy?”¹⁸ by suggesting that burn out because they are overqualified for a repetitious job.⁷ The causes of stress and burnout in doctors are complex,¹² but our data suggest that excess intellectual ability is not one of them.

Supplementary Material

Notes

References