BY RCGP – DECEMBER 2012
General questions about the MRCGP
Q: How confident can the RCGP be about the validity and reliability of the AKT and the CSA?
A: We are confident about the validity of these two assessments, which are designed to test applied knowledge and the integration of knowledge in clinical scenarios that are specific to general practice. Both modules have good reliability, with that of the CSA increased to 13 cases in 2010. MRCGP assessment procedures have been well researched and have been approved by the General Medical Council as the Regulator. There have been several external reviews over the last four years by leading international assessment experts, which have led to helpful modifications.
Q: Can we predict in any way those doctors that will go on to fail the AKT and/or CSA?
A: Work done looking at the correlations between various parts of the selection process and how candidates perform in the AKT and CSA, shows a strong correlation in performance between the two assessments, and is awaiting publication at present. A paper published by Richard Wakeford in June 2012 demonstrated that the differential failure rate of IMGs in the AKT and the CSA could be predicted by looking at selection data. See Wakeford R: Education for Primary Care 2012:23:148-152
Q: What is the RCGP doing to help candidates who have been identified as likely to have difficulty with the exit exam?
A: We work closely with the Deaneries who have responsibility for the standards of selection and training in general practice, and all have in place support schemes for trainees in difficulty. Many RCGP examiners also hold posts in their respective Deaneries and are closely involved in helping them develop supportive programmes and courses. The Deaneries are all aware of the examiners in their area and the special skills they have which can be tapped into as necessary. We also plan to work closely with the GP National Recruitment Office to update the Wakeford study, which will be of help to Deaneries in identifying those who will need early and sustained additional support.
Q: Is the trend for failure rate different for the CSA and AKT?
A: No, the trend (i.e. increasing failure rate for IMGs) is very similar for both AKT and CSA. However, the pass differential is higher for CSA performance than for AKT performance, although IMGs also do worse in the AKT than UK graduates.
The reasons for this are still being researched and Kings College London will be publishing its findings next year from a research project, which is looking at sociolinguistic aspects of the CSA by detailed analysis of 40 consultations.
Q: Is it true that UK graduates of Asian origin fail more often than those of white British origin?
A: Yes. Analysis of ethnicity differences between UK graduates shows that those of Asian origin do perform less successfully in both the AKT and CSA. The failure rates for UK trained Asians are three times higher than those for white candidates in both of the modules. It is important to remember that the AKT is machine-delivered and marked and is ‘colour-blind’ as far as examiner-input is concerned. This is a pattern that is repeated amongst both undergraduate and other postgraduate exams having been extensively investigated by Woolf and McManus. BMJ2011; 342doi: http://dx.doi.org/10.1136/bmj.d901 (Published 8 March 2011)
Q: Can you also break down the failure rate between men and women?
A: Yes. Reference to our MRCGP annual reports provides data on this. There is a paper about gender in the AKT published in 2012. Comparing performance among male and female candidates in sex-specific clinical knowledge in the MRCGP Source: British Journal of General Practice, Volume 62, Number 599, June 2012, pp. e446-e450 (5)
Q: Is the MRCGP a test of use of ‘proper’ English language?
A: No. While it is important to remember that this exam is a test of whether a trainee is competent to enter independent practice in the United Kingdom without further supervision and is conducted in English, the CSA tests clinical and communication skills, not language skills. There is no right way of speaking; regional accents of any sort are completely acceptable, the key is being able to communicate clearly and effectively with the role player.
The CSA tests the candidate’s ability to discover the reason for a patient’s attendance, identify his/her view of the problem, make a diagnosis or differential diagnosis and devise a suitable plan for management of the problem that is in line with current British general practice. This requires fairly sophisticated professional conversation with the patient as well as a knowledge and ability to apply primary care medicine to a problem, in partnership with the patient. These consulting skills are in line with GMC requirements for doctors practising in the UK and are described in the GMC’s document ‘Tomorrow’s Doctors’ www.gmc-uk.org/education/undergraduate/tomorrows_doctors.asp which is set at the level of a graduating doctor from a UK medical school.
Q: How can a candidate do exceptionally well in the Work Place-Based Assessment and then fail the CSA?
A: The MRCGP is a tripos (WBPA, AKT and CSA), which is blueprinted as a whole across the GP curriculum. The three modules all test different areas of the curriculum although there is some overlap between all three modules. This means that success in one component does not necessarily predict success in another module.
Q: Some organisations have said that 300 IMGs have had to leave training because they have not been allowed more than four attempts at the CSA. Is this true?
A: No. This does not match up with our data that shows that in the last 5 years 133 candidates have failed the CSA four or more times. Many of these have also failed the AKT four or more times.
Q: Is there any evidence about IMGs and the medical schools in which they trained?
A: No: The College has spent time looking at data relating to all candidates and their medical schools. Analysis of overseas medical schools shows that, as with UK medical schools, there are some marked differences between the best and the less well performing. We will be undertaking further work to investigate whether there are particular types of undergraduate training, which predict success in the MRCGP.
Q: Is there evidence of high failure rates for IMGs in other postgraduate exams?
A: Yes there is evidence of high failure rates amongst other postgraduate examinations. For example:
Royal College of Physicians examination (MRCP) ‘Analysis of predictors of success in the MRCP (UK) PACES examination in candidates attending a revision course’ Postgrad Med J. 2006 February; 82(964): 145–149. Bessant Et al. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2596701/
Summary:
A questionnaire survey of candidates attending an unselected PasTest PACES revision course was undertaken and results correlated with subsequent pass lists published by the Colleges of Physicians (MRCP face to face examination)Of 483 candidates who took the examination immediately after the course, 219 (45.3%) passed.
UK graduates were more likely to pass (67.0%) than overseas graduates (26.2%).
For UK graduates, pass rates were higher for white candidates (73%) than for ethnic minorities (56%) and for those who passed at the first attempt in the MRCP (UK) part 2 written paper.
For overseas graduates, those who had been qualified for less than eight years were more likely to pass.More overseas (45.7%) than UK (30.8%) graduates were confident that they would pass, but confidence did not predict success.
There was a lower pass rate for ethnic minority than white UK trainees in the PACES examination.
Royal College of Psychiatrists examination – MRCPsych
A report from the RCPsych from August 2011, demonstrates very similar patterns of differential performance to that seen in the MRCGP and can be viewed here http://www.rcpsych.ac.uk/pdf/MRCPsych%20Cumulative%20Results%20Report%20-%20August%202011.pdf
Q: Is there any evidence that foreign graduates taking exams in other countries have comparable failure rates compares to non-foreign graduates?
A: There is some informal information, from other countries such as Australia, Canada and the USA, that international medical graduates perform less well than local medical graduates for those countries. However, there is very little in the published literature on comparable assessments to the CSA (MRCGP) examination – it is mostly about entrance examinations to practice medicine in one of the above countries (similar to the PLAB test).
A House of Representatives Committee in Australia published a report on the induction, training and employment of IMG doctors: ‘Lost in the Labyrinth’ http://www.aph.gov.au/Parliamentary_Business/Committees/House_of_Representatives_Committees?url=haa/overseasdoctors/report.htm which does not specifically report on differential pass rates, but which clearly outlines IMG issues in the Australian context.
Q: What is the current makeup of the MRCGP exam panel?
A: The current make-up of the examiner panel includes 4% who are non-UK graduates, 86% who are white-British and 36% who are women. We would like to recruit more examiners who are non-UK graduates or who are from a BME or EU background. If you are interested in becoming an examiner for the MRCGP please contact exams@rcgp.org.uk
Q: How can I become an examiner?
A: Being an examiner is an interesting and stimulating job which entails ensuring your knowledge is up to date, and as such is excellent for personal development. The daily fee for examiners is the same as throughout the RCGP when GPs work for the college. Rates of reimbursement are published on our website. If you are interested but worried about the selection process please do contact us and we can talk you through it.
Clinical Skills Assessment (CSA)
Q. How many examiners mark each CSA?
A: Each CSA circuit is made up of 13 cases and each of these is marked by an examiner who marks the same case all day. Therefore, each candidate is marked by 13 different examiners. The examiners are all NHS general practitioners from across the UK.
Q: How can you be sure that the CSA is standardised?
A: There are two ways in which we strive to ensure ‘equivalence’ of CSA circuits:
Equivalence from day-to-day: the cases for each circuit are chosen according to a pre-specified assessment blueprint that is on the RCGP (CSA) website http://www.rcgp.org.uk/gp-training-and-exams/mrcgp-exam-overview/mrcgp-clinical-skills-assessment-csa.aspx
This ensures that there is a set mix of cases – the mix includes a specified gender split, age range, diversity, health promotion, psychosocial aspects to ensure that one day is not significantly more difficult than another (we use a ‘difficulty index’ based on individual case performance to ensure the latter). In addition, the pass mark for each day of the CSA is standard set using the Borderline Group method, which addresses any slight differences between one day of cases and another.
Standardisation of case performance on the day of an examination: Each case is calibrated by the three examiners and three role players using the case for that day. Calibration starts (at present) at 8.30am each morning, and continues until just before the start of the morning examination at 10.00am. Calibration follows strict guidelines, and each role player has to role-play the case, using each examiner as a ‘candidate’.
The examiners discuss aspects of the case, its clinical content, the marking schedule, and how they would expect the case to run. Any differences in the way the role players may interpret the case are ironed out during this calibration exercise, and the group meet regularly at coffee time, lunchtime and tea time to discuss how the case is running on the three circuits. Role players move to a different circuit and different examiner after lunch so that examiners can check that performances are truly standardised.
In addition, two cases per circuit are quality assured for role player performance, with an additional examiner watching the case in action, checking that role player performance is indeed standardised across the three circuits. This exercise is recorded, reports on role player performance are fed back to the CSA Core Group via our Lay Adviser and fed back to the role player agency, ‘Interact’.
Four examiners per day undergo marker training, with examiner trainers monitoring the marking of these examiners during the morning and discussing marking protocols and actual marking with the examiners during the afternoon. All examiners also undergo in-depth training at the annual MRCGP Examiners’ Training Conference.
Cases are written by a panel of examiners from across the UK, who have been selected for their proven abilities in writing for assessments. Case content is chosen from the MRCGP curriculum blueprint and mapped to learning outcomes from the Curriculum Statements. Gaps in the case bank are identified by mapping it to the MRCGP curriculum blueprint, and cases are specifically written to fill these gaps. The case bank now numbers over 650 cases. They are continually under review for their performance and to be updated in line with current British general practice.
Q: How is the CSA quality assured?
A: All aspects of the CSA are quality assured. The main areas are in case writing and case maintenance, examiner and role player training, and day-to-day monitoring during the examination. The process for case writing has been explained in FAQ “How can you be sure that the CSA is standardised?”
In addition to this, each time a case is used, the three examiners and three role players meet at the end of the day to fill out an evaluation form about the case – how it performed, if there were any issues or problems with running the case, and suggestions for updating/ referencing/ improving the use of the case.
Examiners are selected extremely carefully. If it is more than 5 years since first acquiring the MRCGP, potential examiners are required to sit the AKT examination. If they pass this, then they are invited to a selection day, during which their ability to understand assessment methodology, rate candidate performance and work effectively in teams is assessed. On the basis of this selection, they are invited to join the Panel of Examiners.
Examiners’ marking performance is continuously monitored and each examiner’s marking is compared with that of the others on the Panel. Training is also ongoing, with both small group training based on the real-time marking of candidates taking place on each day the CSA is run as well as whole Panel training including regular Equality and Diversity training at the Examiners’ Training Conference.
There is also a short training session each morning of the CSA for those examiners attending on the day, to get them calibrated as a cohort and thinking about the examination in its correct context.
Role players are trained actors, who are registered with an agency specialising in this field (i.e. playing the role of patients for examinations). They have experience of role play in a variety of different professions, including the police, teachers, and lawyers. They undergo an induction/ training process that includes Equality and Diversity training, as well as learning how the CSA is conducted, and how they are expected to behave and perform the cases in a manner that will allow candidates to demonstrate their consulting skills. The quality assurance of day-to-day role player performance has been described in the question above.
The actors are observed to ensure that their role-playing is the same throughout the day and that any variability is identified and addressed immediately.
Overall quality assurance is monitored by our psychometrician and reported as part of the MRCGP Annual Report, published on the MRCGP website. It includes analysis of the demography of candidates sitting each day of each session, and specifically describes overall candidate performance by gender, age, Deanery, country of origin and Institution of primary medical qualification.
Q: What does the CSA exam involve?
A: The CSA is taken in the ST3 (third year of specialist training) stage of training. The CSA involves thirteen different examiners and thirteen different role-players delivering and assessing candidates on a carefully balanced and equated selection of thirteen worked up cases having good reliability data for an exam of its type.
Each case lasts for 10 minutes and is marked in three domains: Data Gathering, Clinical Management, and Interpersonal Skills. The GP clinical marker does not involve the role player in the marking or in a discussion about the marks prior to marking.
There is information on the CSA website that describes the exam, describes how cases are written and constructed, gives tips on how to pass the exam and explains how feedback is given.
Q: Do IMGs fail the CSA in any particular area compared to non-IMGs?
A: No, they don’t. The feedback profiles from the CSA in the last few years have previously shown that IMG candidates lose marks in the same areas as non-IMG but to a greater extent. Both groups perform less well in Clinical Management skills and IMGs perform less well in the Interpersonal Skills domains.
Q: Have you asked candidates what they think about the CSA examination?
A: Yes, we regularly ask candidates what they think.
The RCGP undertook an exit survey in February 2011 achieving a 95% response rate (933/977). In answer to the binary question “Is the CSA fair?” 92% answered ‘yes’. However of the failed candidates about twice as many thought it was unfair (8.2%) as the passing candidates.
Overall the attitude to the CSA, as determined by a number of questions relating to the specifics of the exam, was very positive but with some small differences between IMGs and UK graduates.
The next candidate survey is due to be conducted in February 2013.
Q: Could the higher failure rate of some IMGs be due to unconscious bias or otherwise of the examiners?
A: No, as all doctors entering general practice in the UK must now complete the MRCGP exam as part of their training, the RCGP takes any suggestion of bias, subconscious or otherwise extremely seriously, and this is therefore an issue on which we have done a considerable amount of research.
The GMC is looking at requiring Medical Royal Colleges to routinely monitor data relating to examiner and candidate ethnicity; up to now only the RCGP and the RCP have published on this (see attached paper Wakeford and Denney) finding no evidence that examiners substantially favoured their own gender or ethnic group.
The same researchers also recently reviewed 52,000 cases (paper in preparation) and their finding is that there are no substantial effects of gender or ethnicity on examiner/candidate interactions.
Q: How can the CSA examiner control for intra-observer bias if she/he is the only examiner covering the case 26 times in the day?
A: There are a number of ways that CSA examiners can control for intra-observer bias, despite examining only one case all day. These have been covered in detail elsewhere in this document, but in summary include the following: The calibration exercise at the start of the day.
This is a 90-minute exercise with the other examiners for the case and the three role players. Using the case specific mark sheet and by role playing the case themselves, the examiners discuss the three plays and agree on their marking strategy for the day. They meet again at every break to recalibrate and discuss how the case is proceeding.
At lunchtime, role players from the case swap examiners so that examiners can also check that the role players are not being allowed to drift away from the agreed performance of the case over the day.
Q: Could the role players act differently with different types of candidates?
A: We do not feel this is the case though we understand that there have been some concerns and comments about this issue. The argument some give is that it is difficult to determine between differences in behaviour due to poorer clinical and communication skills and other more subtle reasons. The actors are professionals and very skilled.
Since the start of the CSA the College has run a quality assurance process for role players involving examiners following them, and assessing whether they are playing the case as scripted and according to decisions made at the calibration meeting. Feedback is given immediately and the overwhelming number of observations have no concerns attached.
A paper describing this process was published in July 2011. (Russell, Simpson and Rendel): ‘Standardisation of role players for the Clinical Skills Assessment of the MRCGP’ Education for Primary Care, Volume 22, Number 3, May 2011 , pp. 166-170(5) http://www.ingentaconnect.com/content/rmp/epc/2011/00000022/00000003/art00008
The RCGP has refined the process for the February 2013 examination to look specifically at the issue of whether any difference in behaviours between candidates can be observed using a semi-structured questionnaire and an immediate review.
A research project undertaken by Kings College London (KCL) looking at sociolinguistic aspects of the CSA by detailed analysis of 40 consultations, is also going to look at this in detail.
Q: Is there any research around why IMG’s fail the CSA more often that non-IMGs?
A: Yes, the full outcomes of the sociolinguistic study conducted by KCL will be available in 2013. The study is a detailed qualitative analysis of factors that affect performance in the CSA with a view to addressing the gap between success rates of UK and IMG candidates.
While it is too early to say why one type of candidate fails over another type, the reasons are clearly often due to a number of different factors. This research has so far uncovered issues with being able to explain medical concepts to patients in an understandable manner.
These findings have been presented to the Medical Educators’ Group at the recent RCGP Conference in Glasgow, and also at the AiT session at the same conference.
The sociolinguistic research is also proposing to look in-depth at role player behaviour with candidates as well. The researchers are confident that one of the outcomes will be a toolkit to help trainees and trainers in the difficult area of giving explanations to patients.
Q: We understand that the examination rooms have video cameras in them. When are these used?
A: Videos of consultations are used for training, quality assurance and security. The College does not retain them or use them for routine double marking, which is exactly in line with the approach taken by similar organisations that use this system.
Q: Why does the RCGP not use video cameras to record examinations that could later be used for appeal processes?
A: We consider that the quality assurance processes applied to the CSA, and the frequent calibration and training of CSA examiners is more than sufficient to ensure that the academic judgement of the markers is valid and fair. The routine examiner training sessions involve double marking of cases by a Senior Examiner, or Trainer, and discussion of marking decisions based on those cases in a small group setting, with written reports.
In addition, markers can be observed at any time by the Floor Marshal or Senior Marshal if there are any concerns about marker performance. The examiner/ role player pairing also acts as an informal check on both of them so that deficiencies in performance would be speedily reported to the Floor Marshal.
Videos of the consultation do not show everything that happens within the consulting room – for example, the angle of the camera can obscure non-verbal communication between doctor and role player patient, and can also miss important clinical examination sections of the consultation. They are therefore not necessarily a good record of a consultation in the CSA.
Applied Knowledge Test (AKT)
Q: Can you give more details about the AKT exam?
A: The Applied Knowledge Test is a summative assessment of the knowledge base that underpins independent general practice in the United Kingdom within the context of the National Health Service. Trainees who pass this assessment will have demonstrated their competence in applying knowledge at a level which is sufficiently high for independent practice.
The test takes the form of a three-hour computer-delivered and marked test comprising 200 items. It is delivered at up to 150 professional testing centres around the UK. Approximately 80% of question items will be on clinical medicine, 10% on critical appraisal and evidence-based clinical practice, and 10% on health informatics and administrative issues. All questions address important issues relating to UK general practice and focus mainly on higher order problem solving rather than just the simple recall of basic facts.
The AKT standard is set using the internationally recognised process (the modified Angoff process), where a group of judges review the paper scoring each item for the just passing candidate. This process is repeated at intervals of no more than 3 years, and the standard of each AKT in between is maintained by a process of linear equating using anchor items, which have been used in the AKT the previous year. This standard setting process is long-established and well proven and is used widely throughout the educational system.
Q: Has the pass rate for AKT changed in recent years?
A: Yes, the pass rate for AKT has been showing a downward trend over the last 4 years and this is most significant for doctors trained overseas including in Europe (IMGs).
Q: How is the AKT marked?
A: The AKT is an anonymously computer marked exam and as such could never be influenced by any knowledge of gender or ethnicity.
Click Here to get the RCGP – FAQs.