Identifying the Medical X Factor or the XY Factor: Health Professions Admission Test (HPAT) for Irish medical school entry

Carol Linehan^*

Introduction

‘In the sick room, ten cents worth of human understanding equals ten dollars worth of medical science.’ (Fabing & Marr, 1944).

It is hardly controversial that we expect good doctors to have technical knowledge but also an ‘x’ factor, what Fischer calls ‘human understanding’ or more colloquially a ‘bedside manner’. What is less well accepted, and indeed quite controversial, is whether we can identify this x factor and use it to reliably select between applicants to medical school. The fanfare surrounding the introduction, in 2009, of the new Health Professions Admission Test (HPAT) suggested it would do just that, and more.

The HPAT is a supplement to the Leaving Certificate to select candidates for entry to medical degrees. It is a 2.5 hour multiple choice test containing three sections which claim to assess non-academic abilities in the areas of logical reasoning and problem solving, non-verbal reasoning, and interpersonal understanding. The new test accounts for approximately 33% of the weighting for entry to medicine. There appears to be two justifications for its introduction.

1. To ensure that those accepted to study medicine would have the requisite non-academic abilities to perform well as doctors (that elusive x factor)
2. To broaden access to medical training by reducing the requirement for near perfect leaving certificate performance to win a medical school place.

So what has happened since the introduction of the HPAT?

‘Welcoming more men to medicine’

In its first year of use there was uproar as many students of higher academic ability lost a place in university to those who had lower leaving certificate attainment but higher HPAT scores. Female students were particularly disadvantaged as a result of its introduction with the percentage of medical undergraduate places won by females dropping from 60% in 2008 to 52% in 2009. Data from one Irish university indicates that in 2010 the gender breakdown was 56% (female) and 44% (male). These changes were welcomed by many as ‘more even gender balance’. However others were concerned that such tests introduce a systematic gender bias in selection. For example, a recent article in the Irish Medical Journal highlights that the MCAT (a similar test in the United States) under-predicts female performance in future medical exams and also that the UKCAT (the UK aptitude test) appears to also have a favourable bias towards males (Halpenny et al, 2010). If we are really interested in more ‘well rounded’ medics Lumsden et al (2005) demonstrated that female applicants to Scottish medical schools score higher on measures of empathy than males.

Broadening access or introducing barriers?

What is the objective of widening access to medical training? I would take this to mean that those from lower socio-economic groups would have an increased chance of attaining medical places. Baxter (2010) cautions that over half of the test items in the interpersonal ability section are based on medical scenarios which in his view would disproportionately advantage the children of medics who may have grown up hearing of such cases. He also questions the concentration of items measuring visuo-spatial abilities (on which males tend to perform at the higher and lower extremes) and their relevance to many areas of medical practice.

A frequent criticism levelled at psychometric tests is that one can improve scores through test practice. Stories have emerged in Ireland of students significantly improving their scores on HPAT second time around. Whether repeating the HPAT or practicing for it has statistically significant positive effects on scores remains to be seen (and should be monitored) however in the fiercely competitive points race any slight advantage will be played. A quick Google search for ‘HPAT’ produces many services offering sample hpats, workshops etc. for a fee. For example, one provider offers a two-day preparatory workshop for €570. With competition for medical places so tough it is easy to see why students and their parents would consider paying for such courses. What is harder to see is how expensive preparatory courses (which those on higher incomes can more easily afford) and the tentative evidence that practice may increase scores will help to widen access to those groups of applicants who can ill afford grinds (additional extracurricular classes) for traditional Leaving Certificate examinations and now face the additional burden of extra cost in the HPAT grind business.

Does it do what it says on the tin?

Since the first cohort of HPAT entrants are now just finishing second year it is perhaps premature to assess the relationship between HPAT and medical school performance. Similarly, time will tell if ‘widened access’ means anything more substantive than welcoming more boys back into medical education. But whatever our concerns about possible systematic bias, be it on gender or socio-economic grounds there remains two simple questions. Firstly as a selection tool does HPAT improve on:

1. Does HPAT have incremental validity as a selection device to deliver better results than existing tools (i.e. conversion of Leaving Certificate results into Central Applications Office points for access to higher education courses)?
2. Does HPAT have predictive validity to select applicants who excel at undergraduate level and/or go on to become excellent ‘well rounded’ doctors?

In the field of ‘selection testing’ validity assesses whether or not a test measures what it is supposed to measure. That is, there should be a clear relationship between candidates’ scores on the selection test and their later performance on elements of the job. Let us look first at the relationship between performance on aptitude tests and medical school performance. A recent study of the United Kingdom Clinical Aptitude Test (UKCAT) found that scores on that test did not predict (even at a small to moderate magnitude) the first year performance in medical school (Lynch et al, 2009). Studies of similar aptitude tests used in the US and Australia seem to indicate some incremental validity (beyond grades alone) in predicting students’ first year medical school performance.

Moving beyond university examination halls, the next issue is whether such tests identify more suitable doctors. Standard advice in best practice recruitment is to use selection tools that reliably and validly discriminate between candidates on job relevant criterion. Do good doctors score better on HPAT than bad ones? Here is where it gets sticky. How is a ‘good doctor’ to be defined? To what degree has patients’ views on medical practitioners’ communication skills, empathy and the like been gathered and correlated with practicing doctors’ scores on HPAT (a concurrent test of validity).

As a first step Quinn et al (2010) administered a modified HPAT to a sample of surgeons, non-consultant hospital doctors (NCHDs) and medical students. No statistically significant difference was found between the three groups’ scores on HPAT. This is a troubling finding as one might reasonably expect to find for example better scores on problem solving for medical scenarios amongst those with greater medical experience. It raises the question as to what exactly HPAT is measuring and why it was introduced before such concurrent validity tests were run and analysed. Without such evidence the introduction of an additional selection tool is at best a waste of money and effort and at worst deeply suspect.

Finally, let’s not forget that one can ‘make’ as well as ‘buy’ talent. If the quest is truly to improve doctors’ interpersonal and communication skill sets surely such skills could be fostered during their training both at medical school and in work environments. Internationally there have been trends to try to instill more patient centred attitudes (rather than doctor/disease centred attitudes) in medical students and thus improve the quality of their communication and interaction with patients. Interestingly, a recent study measuring levels of patient-centred attitudes amongst Swedish medical students showed that females score much higher on measures of such attitudes than do males and while females scores increase across the years of medical training, males do not (Wahlqvist et al, 2010).

Which brings me back to the only data publicly available thus far on actual systematic effects attributable to the HPAT, which is that males improved their share of medical school places from 40% in 2008 to 48% in 2009. Consider for a moment if the effect had favoured females that is, that their representation rose from 60% (2008) to 68% (2009) following the HPAT introduction – would the test have been so warmly welcomed? The HPAT has yet to be proven to help in identifying the medical ‘x factor’ but it certainly helps (perhaps unintentionally) those with the XY factor.

References

References
1. Baxter, G. HPAT-Ireland: Too far too fast? Irish Medical Times Feb 26th 2010. Available at HYPERLINK "http://www.imt.ie/opinion/guests/2010/02/hpat-ireland-too-far-too-fast.html"http://www.imt.ie/opinion/guests/2010/02/hpat-ireland-too-far-too-fast.html [Accessed on 24/06/2011].
2. Fabing, H. & Marr, R. (1944), (eds) Fischerisms, Springfield: Charles C. Thomas Accessed on http://www.todayinsci.com/F/Fischer_Martin/FischerMartin-Quotations.htm
3. Halpenny, D., Cadoo, K., Halpenny, M., Burke, J., Torreggiani, W. (2010). The health professions admission test (HPAT) score and leaving certificate results can independently predict academic performance in medical school: Do we need both tests? Irish Medical Journal, 103: 10.
4. Lumsden, M.A., Bore, M., Millar, K., Jack, R., & Powis, D. (2005) Assessment of personal qualities in relation to admission to medical school. Medical Education, 39(3), 258-265.
5. Lynch, B., MacKenzie, R., Dowell, J., Cleland, J., Prescott, G. (2009) Does the UKCAT predict Year I performance in medical school? Medical Education, 43: 1203-1209.
6. Quinn, A., Corrigan, M.A., Broderick, J., McEntee, G., Hill, A.D.K. (2010). A comparison of performances of consultant surgeons, NCHDs and medical students in a modified HPAT examination. Irish Medical Journal, 103: 6.
7. Wahlqvist, M., Gunnarsson, R., Dahlgren, G., & Nordgren, S. (2010). Patient-centred attitudes among medical students: Gender and work experience in health care make a difference. Medical Teacher, 32: e191-e193.