What should you expect when you graduate?
Monday, August 23rd, 2010By Kevin Gillmann
University of Newcastle
kevin.gillmann(a)wanadoo.fr
“Is it legal for employment to be refused as a result of the outcome of a 3 station OSCE devised by a Trust when a 5 year programme, quality assured by the GMC has provided evidence that the applicant is fit for purpose?” [1]
Such was Professor Tony Weetman, Chair of the Medical Schools Council’s comment, reflecting on the pitfalls underlying the British Government decision to reform the medical career pathway.
Medical Career Pathway in the UK
Since the introduction of the Modernising Medical Careers programme, in 2005, British medical careers are made up of 4 steps. The medical degree, typically consists of 5 years of study in one of the 32 medical schools of United Kingdom. After graduating, junior doctors enter a 2-year multidisciplinary clinical training called the foundation program. Once they have completed this, doctors can apply for specialty registrar positions, and train to a specialty. This training varies in length depending on the chosen specialty, and gives access to hospital consultancy or general practice.
Figure 1: Modernising Medical Careers’ organisation with average duration of each step.
A flawed selection process?
While the Modernising Medical Careers programme has made transitions between career steps more harmonious, it is now the access to foundation programme positions that is questioned. The issue was raised in 2008, when a NHS study “The Next Stage Review: A High Quality Workforce” stated that “New work needs to be undertaken to develop more reliable and valid selection tools for recruitment”. Following this declaration, the British department of health took immediate action, committing a panel of international experts to analyse and reflect upon the current selection methods. [2]
The current selection is based upon 2 scores. The first one, a knowledge score, is obtained by ranking medical students on the basis of their medical school results. From this sorted list, each medical school divides its own students into four quartiles, and scores are allocated to students of each quartile. Students from the first quartile receive 40 points, and all those from the second, third and fourth quartiles receive 38, 36 and 34 points respectively. This allocating method has been depicted unfair and two crucial issues were highlighted. First, since students from the first quartile of any medical school get the same amount of points, some students may theoretically be lead to choose a less competitive medical school to rank in the first quartile more easily and thus be more likely to obtain the foundation placement of their choice. [3] Then, there are currently enough foundation positions to ensure employment to every junior doctor. However, it may not always be the case, and the day the ever-increasing number of junior doctors will overcome the number of posts available, some will ultimately be refused. And refusing employment on the basis of a selection process, which is not considered as sound, transparent and fair would open the way to a myriad of potential legal challenges. [2, 4]
The second part of the current selection process is a professionalism score. Students’ professional traits are assessed by a series of open-ended ‘white space’ questions they answer during their spare time at the end of their final year. Their answers are marked by a panel of doctor from the applicant’s first choice base unit. The criticisms on this method are numerous. Firstly, it has been identified that the ‘white space’ format for answers compromised the objectivity of the assessment. [4] Then, despite the fact a random selection of 10% of applicants is asked to provide evidences of their answers, this method is said to open the way to cheating and plagiarism. [4, 5] Finally, the panels responsible for the marking of the forms complain about the time-consuming aspect of the task. [6]
Options study
During the 9 months following these findings, the panel of experts assessed various selection methods, and proposed the “Improving Selection to the Foundation Programme Project”. In order to define the ideal selection criteria, they firstly raised the crucial question of what expectations hospitals have of ‘good doctors’. Three attributes were retained: clinical knowledge, clinical skills and professional traits. [7] They then considered the traditional options available to assess such criteria: structured interviews, multiple-mini interviews, a national examination of clinical skills and knowledge, a national test of professional traits, and a standardised score from medical schools. All these selection methods were assessed upon several criteria, ranging from transparency to fairness and cost. As a result of this feasibility study, interviews were dismissed for their inherent lack of objectiveness and their cost, just like national examinations of which the OSCE component was considered to be too expensive, and the assessment of knowledge too redundant with students’ finals. [8]

Figure 2: Results of the feasibility study showing the cost of every method over five years on the ordinate and their reliability score in arbitrary units on the abscissa. (Adapted from Option Review in Paul O’Neil’s presentation: Pilots for FP Recruitment and Selection 2013) [8]
Proposed selection process : what is going to change?
At the end of the study, the panel reached a conclusion and proposed a recommendation framework. As other methods to assess clinical knowledge and skills had been ruled out, experts decided that medical school should keep the responsibility of this assessment. However, the quartile distribution would be replaced by a more granular process allowing standardisation of every student’s grades in a fair and reliable way: the Educational Performance Measure (EPM).
As far as ‘white space’ questions are concerned, they would be replaced by another professionalism assessment, which would this time be a national computer-marked test: the Situational Judgement Test (SJT). The Medical Schools Council describes it as “similar to GP short-listing tests”, and it is very likely to be on the form of small scenarios and identifying the most appropriate action to take. It should also put an emphasis on solving ethical and professional dilemma rather than focusing on students’ extra-curricular achievements. The rest of the process should on the whole not be reformed. [8] In practical terms, for students this reform would mainly replace the professionalism “take-home essays” by a National Situational Test, and change the way their medical school achievements are converted into a standardised score.
Figure 3: Proposed Selection Process. (Adapted from «Proposed Selection Process» in Denis Shaughnessy’s presentation: Improving the Selection Process) (7)
Figure 4: Situational Judgement Test, sample question. A suggested answer to the above question is: A-D-C-B-E. (2)
This new selection process is currently in its stage of development and still has to be tested, legally reviewed and approved by the ministry before it can go live.
Dr Richard Price, careers coordinator at Newcastle University adds: “All we know currently is that the planned changes will be piloted in selected medical schools alongside the existing system for the next two years, and will be thoroughly evaluated prior to their full introduction.”
These trials are to start in October 2010 on a sample of a few hundred applicants, before expanding to a few thousand by March 2011. These pilots will be followed by a legal review, an approval, and a preparation phase that could last until autumn 2012. This last stage is an unavoidable part of the process since it is during this preparation that will be decided how to combine the EPM and SJT to produce a representative overall score.
Thereby, the new selection process should not go live until the academic year 2012-2013, at the earliest. However, experts recommend to continue with the current method unless the pilots show sound reasons for change. [8]
Figure 5: Introducing the changes: a timing overview. (Adapted from “Timing Overview” in Denis Shaughnessy’s presentation: Improving the Selection Process) [7]
So how do other countries sort the matter?
Generally, four main methods are broadly accepted internationally to determine selection of junior doctors’ first placements after graduation. Most countries opt for one of these solutions, or a close alternatives.
1. A national examination ranks applicants according to their results and allocates everyone to his first available choice, starting from the top of the list. [9]
2. Applicants get systematically appointed to the university hospital the graduated from. [10]
3. The government decides where junior doctors are appointed to, depending on current needs. [11]
4. Applicants are scored against various criteria before an algorithm matches them with to their most highly ranked choice. [12]
France : Concours National Classant (National ranking examination)
In France, students do not graduate until they have finished their internship. But the 6-years lecture-based studies are separated from the clinical internship by an examination. This latter is a national examination after which students are ranked upon their score. Students may then choose their positions in the order of their ranking. There are generally more positions than applicants, however applicants with the lowest scores may not have the choice of their specialty. [9]
Italy : Tirocinio Post-Lauream (Post-degree placement)
In Italy, after graduation junior doctors complete a three-month unpaid supervised placement. It consists of two months in the university hospital they graduated from and a month shadowing a general practitioner in a local surgery. After they successfully complete these, a national examination must be passed to obtain full license to practise. [10]
India : Compulsory Rural Service
In India, the degree of Bachelor of Medicine and Bachelor of Surgery (MBBS) is awarded to students once they have completed five years and a half at medical school. However, in an attempt to improve health care standards in the most rural communities, some states have decreed that junior doctors should, as a priority, be appointed to tribal areas. Thereby, young doctors are now bound to practice medicine in rural settings for a year after graduation, before they can apply to any other post. [11]
United-States : The Match and the Scramble
During their last year of medical school, American students apply to residency programmes via the online Electronic Residency Application Service (ERAS). These programmes then short-list the applicants they want to interview. Criteria used for such a selection can vary widely. A study undertaken in 2000 shows these variations as 94% residency programmes reported to base their selection on examination scores, 87% on potential letters from deans, 85% on application forms, and only 61% on personal statements. [13] After the interview stage, programmes rank their interviewee by order of preference, and applicants similarly rank the programmes they applied to. An algorithm, the National Residency Matching Program, is then responsible to match programmes and applicants with their most highly ranked choice. In 2008, of the almost 36,000 applicants, 20,940 matched. [14] For the unmatched applicants, it is during a period known as the ‘Scramble’, during which the applicants who did not match try to secure a remaining available position. But this process is barely structured and offers minimal chances of success. Dr. Freedman, an American leader in medical education, describes the situation: “Jammed fax machines and lines of communication make this process challenging. […] In 2008 by 6pm the day after the Scramble, only 179 positions of the 1,388 positions remained open.” The Association of American Medical Colleges has, however, acknowledged the issue and is currently leading a study to give a more defined structure to the Scramble. [12]
References
1- Pr. Weetman T., Written Evidence Submission – The Evaluation Of the Foundation Programme, Medical Education England.
2- Improving Selection to Foundation Programme Briefing Pack – Stage 2 of Project, Medical Schools Council: 2010.
3- Kelly C., Should UK Medical Students Sit a National Qualifying Exam?, Student BMJ: 2008 – 16:184|17.
4- Pr. Paice E., Dr. Rainsberry A., House of Commons, Health Committee, Modernising Medical Careers, Volume II, Written evidence: October 2007 – p127.
5- Medical Schools Council, Characteristics of the Current System for Selection into the Foundation Programme, Appendix B.
6- KSS FP Evaluation Final Report from KSS Deanery: 2006 – p25-28.
7- Shaughnessy D., Improving The Selection Process, Presentation: March 2010.
8- O’Neill P., Pilots For FP Recruitment and Selection 2013, Presentation, UKFPO: 2010.
9- Postes et Rangs Pour l’Examen National Classant, Journal Officiel, Direction de l’Information Légale et Administrative: 2009.
10-Pr. Torre G.C., Corso di Laurea in Medicina e Chirurgia, Facoltà di Medicina e Chirurgia, Università degli Studi di Genova.
11- Sai Gopal M., Opinion Divided Over Compulsory Rural Service, The Hindu: February 2010.
12- Freedman J. MD, The Residency Scramble: How It Works and How It Can Be Improved, The Student Doctor Network: March 2009.
13- Adams L. J. MSW, Brandenburg S. MD, Blake M. MS, Factors Influencing Internal Medicine Program Directors’ Decisions about Applicants, Academic Medicine: May 2000 – Volume 75 – Issue 5 – p 542-543.
14- Advance Data Tables For The 2008 Main Residency Match, National Residency Matching Program, March 2008 – p3.













