Competence versus excellence

Competency-based curricula are increasingly being introduced in medical education, as outlined in Chapter 6, and there is an increased focus on technical qualifications. Some believe that this approach undermines medical professionalism and fails to provide the appropriate learning platform for continuing professional development. Instead, it fosters a ‘tick box’, ‘can do’ mentality. Trainees perceive that they have achieved competence when they pass an assessment and do not recognise that they need to improve their skills further through apprenticeship and practice. The aim to ‘do better’ may be lost. The United Kingdom Royal College of Physician’s (RCPs) report (RCP, 2005) reviewing the challenges to the medical profession in the twenty-first century emphasised the need to aspire continually to excellence. It raised serious concerns that current assessment trends towards competency were failing us. Even more seriously, there is a risk of generating ‘incompetency’ if learners perceive prematurely that they have achieved ‘competency’. They can fail to recognise that the judgement of competency pertains only to a point in time. The need to set goals for further development is lost (Hodges, 2006). That tension has to be acknowledged as you read through the content of this chapter.

To achieve a new developmental culture and move away from end point examinations and final competency assessments, we need to bring assessment more into the centre of the learning process (Shepard, 2000), as argued above. That would concur with Vygotsky’s (1978) concept of a ‘zone of proximal development’. Each learner is on a progressive, developmental pathway of expertise. At a particular moment in time, with a mentor’s guidance, they need to understand not only what they are capable of performing independently but also how they can improve. Continually resetting their goals can foster an aspiration to do better. This requires a change in assessment culture for both students and tutors. In a competitive assessment world, high scores are of paramount importance; in a more developmental world, which encourages educational feedback, students will not necessarily score high. In the current culture, they can be uncomfortable with low scores. Tutors who are used to linking judgements with a numerical grade that can be defended will have to review their approach to assessment. Emerging evidence shows many tutors have difficulty giving constructive feedback rather than assigning a score or ticking off a competency. A fundamental change in practice is needed, which requires time and training resources.

There is increasing recognition, according to Lave and Wenger’s theory (1991), that learning and the development of an identity of mastery occur simultaneously as students become increasingly adept at participating in a community of practice. If we are to understand why some students struggle to achieve competence and how to motivate others towards goals of excellence, then attention should be paid to the concept of legitimate peripheral participation. More theoretical understanding is needed of how students socialise into and communicate within a community of practice and how they develop competence as part of a social group. This almost certainly impacts on how they develop the institutional socially acceptable language, become accustomed to explaining their reasoning, and receive the feedback necessary to move across competency towards excellence. Herein may also lie an explanation of differences seen in performance across gender and ethnicity (Wass et al, 2003a).

Novice to expert

Clinical competence is a complex construct. We define competency as ‘the ability to handle a complex professional task by integrating the relevant cognitive, psychomotor, and affective skills’. Inevitably, by the very nature of these three components, ‘knowledge, skills, and attitudes’, multiple assessment methods must be used. The resulting observations can then be compared and integrated to create an overall picture of an individual’s competence: a process termed ‘triangulation’. This is not that easy, particularly if assessment is set in the context of continuous professional development, as discussed above.

Vygotsky’s concept of a zone of proximal development highlights the importance of interactive assessments to explore where the student is placed in their progression from novice to expert. The pathway of learning diagnostic reasoning is, to a significant extent, dependent on expertise resultant on clinical experience. The move from a novice to an expert involves a complex cognitive process (Schmidt and Norman, 2007). The rather laborious data gathering process, on which hypothetico-deductive diagnostic reasoning is initially built, changes with time. A shift in the structuring of knowledge is intrinsic to the growth of expertise. For novices, a clinical case appears as a series of isolated symptoms and signs, which must be related to finite pathophysiological concepts. Similarly, a skill is learnt as a stepwise series of distinct procedural steps. Knowledge then becomes progressively assimilated into diagnostic labels, a process termed ‘encapsulation’. As more clinical experience is gained, a further shift occurs as patterns of disease recognition, called ‘illness scripts’, develop. These are more narrative than factual, derive from everyday clinical practice, and almost completely ‘bypass’ the original building stones with which a novice works. They are cognitive entities containing relatively little basic knowledge and scaffolded from direct patient encounters, a product of growing clinical experience. Not surprisingly, ‘illness scripts’ can be very individual as clinical experience is so varied. If, for example, you have witnessed a significant thromboembolic episode in a young girl on the contraceptive pill, then pattern recognition when dealing with patients with headaches taking the pill will weigh towards more serious causes than for clinicians who have not. Similarly when learning skills, practice is essential to allow the initial rather pedantic stepwise checklist of tasks to become automatic and integrated.

A learner’s position on the novice to expert pathway affects both their performance and the judgements of those assessing them. A medical student just starting clinical placements will be in the first stage of diagnostic reasoning, using a checklist of items to gather data, examine a patient, or demonstrate a skill. An experienced assessor will have developed a global, integrated approach and have difficulty reverting to explicit analysis of inexpert performance. As Eraut (2004) points out: ‘expert performance is often tacit; identifying aspects of different disciplinary practices is in itself a significant task’. A useful model in understanding the novice to expert pathway, which provides a framework for assessing across the undergraduate postgraduate continuum, is that of Dreyfus and Dreyfus (1986). Table 14.1 outlines the stages and implications for testing as knowledge and skills move from basic knowledge, where there is little situational perception and situational judgement, through to the holistic, intuitive approach of experts. This ‘mismatch’ between reasoning processes of a novice and those of an expert explains some of the dilemmas faced when assessing learners in medicine.

Table 14.1 Novice to expert: stages of progression

Assessment methods must be tailored to accommodate shifts in reasoning which develop implicitly with experience. Although medical students and residents are among the most frequently tested groups in higher education, it is surprising how often the assessment methods still focus primarily on low-level skills. If we expect our future clinicians to be excellent, we must begin to test across the education continuum at increasingly higher skill levels or, indeed, use assessment to drive its development. It is important both to identify a trainee’s position on the novice to expert pathway and encourage a higher level of reasoning.

Bloom’s taxonomy of knowledge (Figure 14.1) provides a means of illustrating why this is important. Bloom (1965) defines increasing levels of complexity; initially defining basic facts, then applying these to clinical situations, before progressing towards analysis, and finally evaluating clinical events. It is relatively easy to test basic factual knowledge; it is not uncommon to find postgraduate examinations still assessing at this level. Yet, even at undergraduate level, novices need encouragement to ‘interpret’ basic facts in a clinical context and apply them to simple clinical scenarios. Assessment should be designed at increasingly higher levels of the taxonomy towards ‘synthesis’ and ‘evaluation’. In the postgraduate arena, audit and critical appraisal become more appropriate assessment tools. Designing assessments to achieve this can be challenging. It is essential to ask the question repeatedly: ‘Is this assessment appropriate to the cognitive level of the trainee as they progress towards clinical expertise?’

Figure 14.1 • Hierarchy of knowledge: Bloom’s taxonomy.

Summative versus formative

There are conflicting commitments within medical curricula. Throughout undergraduate and postgraduate ‘training’, there is a strong obligation on the part of institutions to ensure that their learners become ‘safe’ to practise in health care. There is an absolute necessity to protect patients. Minimum levels of competency for progression in training are required. This summative approach is potentially threatening and, as we have discussed, may be counter-productive in supporting learners’ progression from novice to expert. Some are concerned that ‘licensing tests’ can impact negatively on learning because they detract from the safe educational environments we strive to create. Examinations drive learning too narrowly towards the licensing test. Training towards minimum competency can conflict with the overriding commitment to education and lifelong learning. The latter requires a more ‘formative’ approach to assessment, with regular constructive feedback to trainees to ensure that they set personal goals for improvement.

The focus in the past has been very much on serial examinations and the summative assessment of learners. The need for a social constructivist approach to the assessment of clinicians is increasingly acknowledged (Rust et al, 2005). This view argues that knowledge evolves and is shaped through increasing participation within communities of practice (Lave and Wenger, 1991). Assessment needs to be constructively aligned with the environment in which trainees learn, that is, the apprenticeship foundation of medicine. We face a compelling need to change to a formative approach and foster continuous professional development in workplaces. There has been a strong initiative to replace summative examinations with more formative assessment programmes, which reflect the socio-cultural milieu of the curriculum (van der Vleuten and Schuwirth, 2005). The socio-cultural approach creates a further tension for tutors within the pressures of their working day. Tutors are being asked to collaborate formatively with trainees and give regular feedback on performance. At the same time, they may be asked to make summative judgements on the trainee’s competence to practise. These roles entail very different power relations, which can be difficult to enact side by side, a dilemma that cannot be ignored.

There have been two significant consequences of this increasing move towards formative assessment. First, in an attempt to acknowledge assessment as developmental, confusion has evolved in the assessment discourse. Appraisal (see Chapter 13), essentially a confidential process focused on personal professional development, is increasingly seen as a potential vehicle for summative assessment. This is potentially damaging, threatening the educational potential of a feedback culture across the medical education continuum. The fundamental principle of supporting learning through feedback is in the systematic, supportive manner in which it is facilitated (Jamtvedt et al, 2003). Introducing a summative element threatens this. Learners can be confused by a language that uses appraisal outcomes, understood as confidential, to inform assessment processes such as revalidation. Whereas assessment can usefully inform appraisal, we believe appraisal cannot realistically inform assessment, by the very nature of its confidentiality. A second consequence is the use of ‘formative’ and ‘summative’ as dichotomous terms. Formative assessment risks are aimed only at the generation of educational feedback to support and develop learners, with little emphasis on psychometric principles. Psychometric preoccupation with reliability is viewed as the preserve of summative assessment – ensuring that pass/fail decisions are robust. As a result, feedback on performance is rarely given after summative assessment beyond a pass/fail decision. Continuing to view formative and summative assessment as separate processes is not progressive. Learners, whether they pass or fail, are entitled to expect detailed feedback on their performance. We argue that all assessments must be able to demonstrate reliable psychometric properties, whatever their purpose, so more attention must be paid to the development of psychometrically robust formative tools. If the main aim is to provide feedback, then the information given must be both valid and reliable. Research is urgently needed to address this cultural change as we move away from the formative–summative dichotomy.

Validity and reliability

The fourth challenge lies in addressing tensions that exist when ensuring that an assessment is both valid and reliable. There are trade-offs between reliability and validity, which inevitably impact on developing more formative assessment programmes. It is important to explore theoretical understanding of these two concepts and how they relate to each other. Intrinsically, by the very nature of assessment, a test cannot be valid unless it is reliable. Yet, without validity, the purpose and value we place on the educational impact of assessment would have little meaning.

Reliability

Reliability is, by comparison, a more ‘concrete’ parameter. It can be measured quantitatively, as described later in the chapter. It can be defined as the reproducibility of assessment data or scores, over time or occasions (Downing, 2004). Although usually distinguished from validity, reliability is a pre-condition for validity. If an assessment is not sufficiently reliable, then the evidence required to establish validity becomes inaccurate. Reliability is usually (and correctly) said to be a necessary but not ‘sufficient’ condition for validity. Measurements or judgements may be reliable in the sense of being consistent over time or over judges but still be off-target (or invalid) if they have not actually measured what you set out to measure. Reliability and validity are, in a sense, intertwined and need to be balanced against each other, along with other parameters discussed above (Figure 14.2).

Figure 14.2 • Balancing development of education and judgement of competency.

  Page 235 

The traditional, artificial separation of validity and reliability fuels the formative versus summative debate. Where educational supervision and feedback are of paramount importance, as in formative assessment, attention to the validity of judgements made by tutors and the constructive potential of their feedback takes precedence. Yet, this must not be entirely to the exclusion of reliability. The emphasis on improvement must be based on robust as well as valid feedback. If a decision-making function is also placed on the process, then the defensibility required of the judgements places greater weight on reliability. Little is known to date on how assessments can be combined (if indeed they can) to balance validity with reliability. The importance of trying to achieve a greater understanding of this might well emerge as we move away from the defensibility of examinations to explore more developmental assessment programmes in workplaces.

These theoretical concepts (Figure 14.2) should be kept in mind as the chapter proceeds to outline the fundamentals of assessment design. There is no gold standard for the assessment of any facet of clinical competence. At best, robust frameworks balance the formative approach needed to inform self-monitoring with the stringent judgements required to assure that trainees are competent for safe clinical practice. Increasingly, programmes must be designed to offer more flexibility, more formative feedback, and longitudinal assessment. Testing must adapt to levels of expertise and involve all stakeholders in setting standards and quality assurance.

Context specificity

Perhaps the most important evidence to emerge over the years is that all competencies are context bound and not generic, a principle that is fundamental to assessment design. It was once assumed that some skills, such as communication, were generic. If you communicated well on one occasion, you were ‘a good communicator’. The ability to communicate was a personality trait. There is now ample evidence that this is not the case. Professionals do not perform consistently from task to task. Explaining a medical condition to a patient may be done well when one particular patient has one particular diagnosis. With varying diagnoses and different patients (e.g. explaining a more complex diagnosis to an angry patient), the individual’s performance will vary. Performance on one case does not generalise to the ‘universe’ of cases and patients. The concept of a ‘generically good communicator’ does not exist.

Given our definition of competency, which emphasises integration of cognitive, psychomotor, and affective skills, this is not surprising. Communication skills and professional behaviours are contextually bound by the knowledge required for that particular situation. An individual’s performance on a particular problem or situation only weakly predicts their performance on a different problem or situation. Although most of us believe that individuals have stable personality traits, research has clearly demonstrated that it is not true. The ‘traits’ individuals exhibit are context-dependent. The situation (i.e. the context) should be considered a better predictor of behaviour than personality.

Two important lessons have been learnt. First, sampling of content must be wide. That was the main catalyst for developing OSCEs and for the demise of the long case. We now understand why sufficient testing time is essential in order to achieve adequate case specificity (reliability across cases), as shown in Figure 14.3, based on data published by Swanson et al (1995). A generalisability coefficient of 1.0 would indicate 100% test reproducibility (see below in section on measuring reliability). In reality, as Figure 14.3 demonstrates, that is rarely achieved because other variables such as examiner performance or poor standardisation of cases impact on tests. A coefficient of >0.8 is the pragmatic aim for most high-stakes tests of clinical competency. Based on that level, at least 14 OSCE cases are needed. Increasing the number of judges improves reliability to a significantly lesser extent (Figure 14.3). Second, more psychometric analyses of traditional methods, such as long cases and orals, have been published, which show that all methods are relevant provided the test is long enough to sample adequately. Sufficient testing across a range of contexts is essential (Figure 14.4). We have been released from the stringencies of examinations. Multiple assessment tools can, if planned appropriately using a sufficient range of contexts, be combined to assess individual performance more validly grounded in the reality of workplaces. (Wass et al, 2001c; van der Vleuten and Schuwirth, 2005).

Figure 14.3 • The reliability of the OSCE. Statistics demonstrating how reliability (generalisability coefficient) improved as station number was increased and the number of raters on each station was increased

(from Swanson et al, 1999, with thanks to David Swanson).

Figure 14.4 • Reliability of different assessment methods over 4-hour testing time (MCQ, Norcini et al, 1985; Short essay, Stalenhoef-Halling et al, 1990; PMP, Norcini et al, 1985; Orals, Swanson, 1987; Long case, Wass and Jolly, 2001; OSCE, Newble and Swanson, 1996).

Blueprinting

It follows that tests must be carefully planned against intended learning outcomes and curriculum content, a process known as ‘blueprinting’. Assessments must validate all the objectives set by a curriculum because students inevitably focus on learning what is tested. Clinical competency is multifaceted. The learning outcomes required, sometimes defined as knowledge, skills, and attitudes, cannot realistically be assessed by a single test format. To assess clinical competence, a battery of different tests is needed. No single one can be valid, given the complexity of clinical competency itself. All tests must use a method that is appropriate to the learning outcome being tested. A multiple choice question (MCQ) may be a valid test of knowledge but not of communication skills, where an interactive test is required. Context specificity must be addressed and integrated into the framework to ensure an appropriate range of case scenarios are used. A blueprint is essential to overcome this complexity (Boursicot et al, 2007). It specifies not just what to assess but also how to assess it. For undergraduate curricula, where core content is more homogeneous, that is easier than for more broadly defined postgraduate curricula (Tombeson et al, 2000). To improve assessments, there is now a move for more detailed definition of their content. Conceptual frameworks to plan assessment programmes are essential and can be designed even for generalist collegiate tests. The assessment programme of the UK Royal College of General Practitioners (Rughani, 2007) provides an example.

Miller’s model of competence assessment

Over the past 20 years, ‘Miller’s triangle’ has provided an invaluable framework for planning clinical competency assessments (Miller, 1990). The model, shown in Figure 14.5, considers progression towards expertise as learners become more knowledgeable. They begin by assimilating knowledge (‘knows’), then apply knowledge (‘knows how’), before acquiring skills (‘shows how’), and finally move to ‘does’: that is, performance in the workplace. Miller’s model can be used to demonstrate the appropriateness of testing methods to competencies being measured. Mapping assessment tools against ‘the triangle’ has increasingly highlighted the lack of robust methods to assess performance (the apex of the triangle). As discussed above, the need to adjust test methodology to developing expertise has tended to be overlooked. There has been reconsideration of Miller’s model. It implicitly assumes that competence, demonstrated under controlled conditions, predicts practice in the real world. Several studies have shown differences between doctors’ performance in controlled high-stakes assessments and the reality of their actual practice (Rethans et al, 1991; Southgate et al, 2001). Knowledge is a pre-requisite for competence (Ram et al, 1999b), but not sufficient alone to predict application in practice. Other influences related both to the individual (stress, health) and systems (conditions of practice) also have a significant impact on performance. Competence (‘shows how’) sheds light on performance but does not fully illuminate it, which Miller’s model lacks the flexibility to reflect. Rethans et al (2002) suggested inverting the triangle to place greater weight on performance and modify it to highlight additional individual and system-related influences. They offered a useful reflection of the challenges faced when moving assessment from the relatively controlled environment of the medical school to the reality of postgraduate training in the workplace.

Figure 14.5 • Miller’s triangle (modified).

Our version of Miller’s triangle is a three-dimensional pyramid (Figure 14.5), emphasising that blueprints must acknowledge ‘domain-independent’ skills – ones that are not specific to medicine but nevertheless essential facets of professional behaviour. We have added the third dimension because it is increasingly recognised that the two-dimensional (triangle) knowledge model favoured the use of unduly simply summative methodologies. It served as a good planning design but pulled us away from integration. Flexner’s definition of an ideal test (Flexner, 1910) captures the essence of the problem. ‘There is only one sort of licensing test that is significant, namely a test that ascertains the practical ability of the student confronting a concrete case to collect all relevant data and to suggest the positive procedure applicable to the conditions disclosed. A written examination may have some incidental value; it does not touch the heart of the matter.’ Miller has offered a model of identifying how assessments can be combined at different levels of cognition to achieve Flexner’s aim. Professional values must be integrated into the model in order to effectively capture performance.

The utility equation

No assessment tool is perfect. When designing assessment, it is important to acknowledge and weigh logistics. The practicalities of delivery cannot be ignored. The need to balance the concepts described above (Figure 14.2) to address the purpose of a test and reflect the philosophy of the training programme is essential. Choice is generally a compromise. The ‘utility equation’ (van der Vleuten, 1996) provides useful guidance for matching test method to the competency being assessed in the context of the curriculum:

The equation highlights the importance of balancing educational impact with reliability and validity when designing an assessment. Summative tests place more weight within the equation on reliability; formative tests lean more towards educational impact and validity. In addition, van der Vleuten has highlighted the inevitability of addressing the feasibility of implementation. To achieve optimum reliability and validity, an individual needs theoretically to be assessed ad infinitum. That is never possible. Feasibility is intrinsic to assessment delivery. Reliability at this aspirational level is impractical. The time available, recruitment of assessors, administrative requirements, and financial restrictions all impact on test design. Similarly, test validity is undermined if collective acceptance of its theoretical framework is lost due to poor implementation. Constraints of practicality should be explicitly acknowledged. At the same time, assessment proposals must be acceptable to those involved: students, assessors, teachers, and institutions. Acceptability needs to be addressed when balancing assessment decisions.

No test can score uniformly high on all five parameters. Some trade-off is inevitable to ensure the purpose of an assessment is achieved. Utility counterbalances the selection of assessment methods, supporting delivery of a comprehensive, robust, and educationally transparent assessment package. We emphasise again the importance of educational impact. This should be at the heart of the endeavour. Too often, the educational benefits of assessments are lost. All assessors have a responsibility to make assessment a source of learning. Its place in the equation is fundamentally important.

The assessment of ‘knows, and ‘knows how’

From the start of undergraduate curricula, increasing emphasis is being placed on testing ‘knows how’ – that is, applying knowledge to problem solving – rather than ‘knows’ – that is, straight, factual knowledge recall (Figures 14.1 and 14.5). Problem solving is not a generic skill. It is closely linked to knowledge and is context-specific. A candidate’s problem-solving ability is inconsistent across different tasks. As in all areas of clinical competence testing, inter-case reliability becomes an issue (van der Vleuten, 1996). This is more easily solved for written formats, where a large number of questions can be covered relatively quickly. As more complex approaches to assessment have evolved, pure knowledge tests have been criticised for their lack of validity. Yet, they remain a reliable and feasible way of assessing core knowledge. They retain a firm place in undergraduate and postgraduate training programmes and, arguably, in continuing professional development. Scores on pure knowledge tests can predict clinical performance. Studies by the American Board of Internal Medicine (ABIM) (Ramsey et al, 1989) showed a correlation between performance on the ABIM certification examination (a pure multiple-choice test) and performance in practice 10 years later, as assessed by peer ratings. More recently, Norcini et al (2002) linked data on all heart attack cases in Pennsylvania to certification by the ABIM. Physicians certified by this knowledge test had a 19% lower case fatality rate than those who were not. There was a 0.5% increase in mortality in their practice for every year post graduation. That is consistent with knowledge decreasing as experience lengthens (Choudhry et al, 2005). Performance on pure knowledge tests appears to relate positively to health outcomes and negatively to experience.

The assessment of ‘shows how’

The assessment of ‘does’

There have been significant advances over the last decade in workplace-based assessment (WBA) (Norcini and Burch, 2007), a change with potential for significant educational advantage. Increased curriculum integration encourages more contact with patients from the start of medical school. A patient-centred approach to care can improve health outcomes (Beck et al, 2002; Little et al, 2001). It makes sense to design test methodology to drive patient-centred learning and increase holistic understanding of patients’ needs. To achieve this, assessment must emulate the reality of a patient’s journey through the health services. We need to know how learners perform in the working world and not in a pre-planned controlled environment. At the same time, there is compelling evidence that giving formative feedback to learners, while assessing them in workplaces, can ultimately improve achievement (Norcini and Burch, 2007). Intrinsic to WBA is the need to embed multiple assessment tools in the curriculum and ensure that learners are assessed at the appropriate level of Miller’s model. As the concept of in-training assessment programmes has evolved (van der Vleuten and Schuwirth, 2005), there has been reluctance in some arenas to conceptualise assessment instruments as part of an overall programme. Moving from individual assessment instruments to programmes has created new challenges that are only just beginning to be explored (Davies et al, 2009).

Interestingly, modifications of more traditional methods are coming to the fore. Assessment of clinical competencies in the UK Foundation post-graduation 2-year residency programme is entirely workplace-based. The methods used are, essentially, adaptations of the observed long case (mini-CEX), OSCE stations (direct observation of procedures (DOPs)), and an oral (case-based discussion) (Modernising Medical Careers, 2008). In a sense, there is a swing away from OSCEs back to more traditional methods modified to address context specificity, the issue which originally led to their demise. Most knowledge tests can be improved to test at the ‘knows how’ rather than ‘knows’ level within a traditional written paper format. It remains difficult, though, to assess synthesis and evaluation (Figure 14.1). WBAs, such as audit projects and portfolios, may well prove the answer to assessing a trainee’s ability to apply knowledge at those higher levels of Bloom’s hierarchy. With adequate sampling to cover a range of contexts, assessors, and methods, workplace assessment should theoretically be able to realise adequate reliability. Given the difficulties of standardising content and training assessors, it remains to be seen whether these methods can ever achieve more than medium-stakes reliability. Other biases such as score inflation due to the formative nature of the assessment are of concern (Govaerts et al, 2007).

Self-assessment and self-monitoring

Most studies define self-assessment as an individual’s ability to identify their own strengths and weaknesses compared to those perceived by others. It has long been hypothesised that engaging the learner in self-assessment places them at the centre of the process. More reliance on self-assessment should theoretically increase the educational impact. Yet, evidence suggests the process is poor (Ward et al, 2002). It is informed by culture and gender rather than a shared reality. Self-assessment appears to be methodologically flawed (Eva and Regehr, 2005). Social psychologists view our behaviour and performance as shaped by our unconscious minds (Bargh, 1999). These are self-serving, primarily focusing on self-preservation. For example, when learners are given negative feedback, they blame external factors and distance themselves from any personal responsibility (Alimo-Metcalfe, 1998). Self-assessment must currently be viewed with caution as an assessment tool.

Self-assessment should not be confused with self-monitoring and reflection. Self-monitoring is the ability to respond to situations shaped by our own capability in a particular context. Reflection, often used interchangeably with self-assessment, represents a very different and important concept. It is a conscious and deliberate process focusing on understanding events and processes to bring about self-improvement (Mann et al, 2009), discussed in more detail in Chapter 2. Reflection is clearly important and may be part of self-monitoring (Eva and Regehr, 2005). Yet, although intuitive, there is little evidence that we achieve a better understanding of ourselves through reflection (Mann et al, 2009). Learners should seek the views of others external to themselves (Eva and Regehr, 2007), which places even greater emphasis on delivering high-quality assessments to ensure valid and reliable feedback is delivered and learners are appropriately supported (Archer, 2010).

These important concepts remain central to our ability to learn from external feedback on assessment performance. Eva and Regehr argue that the ‘health professional community should predominantly be concerned with identifying contextual factors that influence self monitoring behaviours in the moment of action rather than worrying about the accuracy of generic and broader self assessments of ability’ (Eva and Regehr, 2007). This is not to say all individuals respond to feedback in the same way. Individual receptiveness is complex and related to self-awareness, demographic similarity, and acquaintance, and it appears to fall with age (Archer, 2010). A better understanding of these concepts is required if we are to harness successfully the formative feedback, which external assessment offers learners.

Setting standards

Quality inferences about, and expectations of, levels of performance are critical to any assessment. Well-defined and transparent procedures must be set in place to achieve that. When assessment is used for summative purposes, a defensible pass/fail level must be agreed. In reality, there are no hard and fast scientific parameters for achieving one. It is essential that people setting standards are well acquainted with the learners and the expectations of the curriculum. At the same time, learners must fully understand what is expected of them. Various methods are available to support the process. It has to be acknowledged that there are no gold standards. The standard setting process should be viewed as a triangulation of methodologies to justify and defend the parameters being set. It cannot occur in isolation. Final decisions are inevitably informed partly by prior knowledge of the programme and learners and, unavoidably at times, by political influences. We offer guidelines for the methods available.

Measuring validity

As discussed above, validity is a complex and fluid concept. Assessment methods are never valid or invalid. Assessors must collate evidence and build a case for validity. Traditionally, to help build a framework for achieving this, a range of ‘facets’ was identified (Table 14.2). These acknowledge that appraising the validity of a test requires multiple sources of evidence (Wass and van der Vleuten, 2006). Many educators still refer to these. ‘Face’ validity represents an overview of the assessment measuring whether the educational intentions have been honoured. ‘Content’ validity requires analysis of whether subject matter has been comprehensively covered. It is also important to evaluate whether the test was appropriate to learners’ level of expertise. That is termed ‘construct validity’. These three facets of validity are relatively easy to evaluate. ‘Predictive validity’ is more challenging. This requires longitudinal follow-up to measure whether trainees’ subsequent clinical performance correlates positively with their previous assessment grades: that is, do the top scoring candidates on knowledge tests subsequently do best in the health care arena? There is some evidence, for example, that UK school ‘A’ level grades are the best predictors of subsequent postgraduate performance (McManus et al, 2003). Finally, to ensure emphasis is placed on the educational impact of assessment, the consequences of the test, ‘consequential validity’, should be appraised (Wass and van der Vleuten, 2006). ‘Does the test produce the desired educational outcome?’ is an important question to ask of any assessment.

This initial, rather pragmatist view of validity has changed. In its contemporary conceptualisation, one unitary concept, ‘construct validity’, is offered (Downing, 2003b). We believe that this redefining of validity is essential to the development of new assessment methodology. The single term embraces the multiple sources of evidence (Table 14.5) needed to evaluate and interpret the outcomes of an assessment. It enables us to move away from the traditional individual descriptive ‘facets’ outlined above and focus more on the sources of evidence needed to validate an assessment process. Five sources of evidence have been identified (Downing, 2003b), which can be used to support or refute the validity of assessment outcomes (Table 14.5). By weighing evidence from both score measurements and quality assurance processes, a more logical, theoretically based, and defensible conclusion can be drawn about whether the original ‘hypothetical’ intentions have been met or not. The context of an assessment, that is, whether it is a ‘high-stakes’ summative or a ‘low-stakes’ more formative process, is also integral to any final conclusions on test validity. Scrutinising assessments against this framework (Table 14.5) offers a useful approach to evaluating validity. Intrinsic to this lies the reality, emphasised above, that quality data are needed to evaluate validity. The integrity of the results (response process), performance of test items (internal structure), and correlations with other test outcomes are all concrete measurements that should be robust. The process loses all value if the test data itself has not reliably ranked candidate performance. A test cannot be valid unless it is also reliable.

Table 14.5 Sources of evidence for measuring validity

Adapted form Downing (2003b).

Measuring reliability

Reliability has been defined as the reproducibility of assessment data or scores, over time or occasions (Downing, 2004). We have chosen briefly to outline three approaches to measuring test reproducibility: internal consistency, classical theory, and generalisability (G) theory. References are given for further detailed reading. A plea for different psychometric approaches, which depend less on these rather reductionist measurement of reliability and validity, has been made (Schuwirth and van der Vleuten, 2006). Currently, though, generalisability theory remains the gold standard (Brennan, 2001).

Classical test theory

Classical test theory assumes that each learner has a true score (‘the signal’) that would be obtained if there were no errors in measurement attributable to external factors (‘the noise’). In reality, as discussed, there is always some variance or ‘noise’ attributable to examiner performance, conditions of testing, etc. A true score is a learner’s average score if measured an infinitely large number of times, when all measurements are independent and stable across different observers and situations. The difference between the observed score and the learner’s true score is the measurement error. The candidate score on any test inevitably does not totally reflect their true ability. The difference between the true score and the observed score is the result of errors in measurement.

Classical test theory can only estimate reliability by evaluating one source of potential error at a time. It rests on a ‘crossed test’ design. Learners’ scores are measured against a single source of error; for example, ‘the test item’. The learners must all take identical test items. This has significant limitations:

Case specificity

As already discussed, competency is context-specific. One case does not generalise to another. Classical test theory cannot be applied to tests, such as traditional vivas or long cases, where the cohort of candidates have not been assessed on the same questions under the same conditions. Even when identical cases are used, as in OSCEs, failure to standardise patients or role play to ensure identical conditions can result in significant error variance.

  Page 249 

Inter-rater reliability

Observers often vary in their interpretation of an assessment even when assessing a learner simultaneously. Inter-rater differences can cause significant error variance. Assessor performance needs to be monitored and training is essential. It helps to build a common understanding of the process and the standard expected. However, there is limited evidence that training assessors effectively reduces inter-rater variance (van der Vleuten and Swanson, 1990).

Intra-rater reliability

Assessors do not necessarily rate consistently. Checklists have evolved in an attempt to reduce this error although evidence now shows that global ratings may be more reliable than checklists (Reznick et al, 1998). Reliability is expressed as a coefficient that represents the proportion of true to error variance:

An example of a fully crossed design is when all candidates take the same written paper, which is marked by the same examiner. There are two potential sources of error: (1) test items; (2) examiner ratings. Only two facets are calculable using classical test theory; candidate ability (true variance) and examination items (error variance) In reality, the additional error of examiner inconsistency (intra-rater error) cannot be measured (Figure 14.6).

Figure 14.6 • A. Variance in crossed design, p = true variance, i = item variance, pi,e = un-attributable (error). B. Variance in nested design, p = true variance, i,pi,e = un-attributable (error).

Standard error of the measurement

Reliability, the probability that an identical pass/fail decision will be made on retesting, is reported in these studies as a coefficient between 0 and 1. A coefficient of 1 represents 100% reliability; in reality, never achieved. The level to aim for is debatable (Downing, 2004). In ‘high-stakes’ testing, where important progression decisions need to be made, setting robust reliability is essential to achieve defensible pass/fail decisions on assessment scores. At a minimum, coefficients of 0.8–0.89 should be aimed for (Downing, 2004). If the stakes are less high, then lower levels are acceptable although less confident and subsequently less defensible. It is increasingly accepted that the precision of measurement – that is, 95% confidence limits – must be defined by applying the standard error of measurement (SEM) at the cut score. The square root of the measurement error (√measurement error) constitutes the SEM. Ninety-five percent confidence intervals are determined by the SEM multiplied by 1.96. Cronbach and Shavelson (2004) ultimately argued that reliability coefficients are more honestly and usefully expressed as confidence limits. When using test–retest theory, this requires a correction for the spread of scores:

By applying confidence limits to pass/fail scores, as determined by standard setting procedures, decisions become more defensible. Raising the agreed cutoff scores by two standard errors of measurement assures 95% confidence that all passing candidates would still be judged competent if retested. The pass/fail score can be adjusted according to the level of confidence required. Using standard errors of measurement, learners can be evaluated in terms of their proximity to the pass mark. Those approaching the pass score will increasingly have confidence intervals that cross it. There is uncertainty about their accurate placement on the one side or the other of the pass mark. More data on their performance needs to be accrued. This is good educational practice. Identifying and understanding the needs of individuals who are potentially struggling is clearly beneficial.

Generalisability (G) theory

Generalisability (G) theory is an extension of classical test theory designed to overcome these limitations in measuring error variance. It enables several sources of variance error within a test; for example, cases, assessors, and candidates to be quantified individually (Brennan, 2001). A ‘nested’ rather than ‘crossed’ assessment design is used. The candidate score is taken as the main facet and analysed individually against each potential source of variance. A single OSCE case can be ‘nested’ within a rater who sits at one OSCE station throughout the examination. By maintaining the rater and the station as a constant, the variance attributable to the candidate can be estimated. All other variance remains as unattributable error. To assess variance due to assessor ratings, two assessors would need to be nested within a single OSCE station throughout the examination. By keeping the item constant, the impact on candidate score of assessor ratings (inter-rater error) can be estimated. From the variance components, a G coefficient can be calculated, using the same classical test formula:

Generalisability theory integrates the discriminating ability of the test and the reproducibility of the result. The coefficient will always be lower than with the classical approach as it takes into account all possible sources of error simultaneously.

Decision (D) studies

The generalisability coefficient is an estimate based on the actual assessment data from which it is formulated. It is also possible to predict test reliability if one of the facets were to be theoretically changed; for example, if the number of cases or number of raters (or indeed both) were to be increased. Decision studies use statistical modelling to modify assessment designs to improve reliability. Generalisability theory was employed in this way, for example, to predict OSCE reliability for a varying number of stations and raters in the example offered previously in this chapter (Figure 14.3). Such predictions can be used to inform the design of future assessments modifying test length, number of items or assessors to identify combinations that achieve acceptable levels of reliability. Although generalisability and decision studies are powerful, prediction does not replace measurement. Decision studies rely on the data presented to them. Modelling assumes that any new sample of subjects, assessors, and items is identical. It is always good practice to re-evaluate reliability.

Item response theory

Classical and generalisability test theories are sample-dependent. They centre on candidate scores. Assessment of item performance is always confounded by the characteristics of subjects taking the items at that point in time. Yet, for test design, it is important to isolate the performance of items per se, for example, level of difficulty, to both understand their individual properties and apply items independently of a particular cohort of examinees. This theoretical approach allows error variance due to candidate cohort ability to be excluded. Only then can items stand alone. A constant independent measurement scale for items can be generated. This permits both scaling of difficulty and legitimate comparison of performance; for example, of changes in student ability if they take the item sequentially over time. Three common item response theory models exist and are reviewed by Downing (2003a). Although technically difficult to apply, awareness of this model is important to understand computer-adaptive testing where the candidate follows a testing pathway determined by their personal capability. The theory can also be used for adjusting rater error in clinical performance assessments.