Chapter 14 Finding and Appraising the Evidence
Overview
The aim of this chapter is to explore ways of both locating and evaluating the evidence that underpins practice. Developing the skills of evidence-based occupational therapy (EBOT) is vital in today’s political climate. Without a sound appreciation of the evidence, or lack of evidence, to support occupational therapy interventions and actions then practitioners will be unable to justify the rationale for these interventions and actions.
The chapter will begin by reminding the reader of the background and context of evidence-based practice (EBP) and how EBP has developed into EBOT. The main focus of the chapter will be on exploring both the nature and sources of ‘evidence’ with practical examples of searches. Having located some evidence, the next step is to appraise the rigour and value of that evidence; this process will then be discussed. However, finding and appraising evidence is often difficult for the busy practitioner, the chapter will, therefore, discuss ways of including appraisal into everyday practice.
Historical context
The need to find and appraise evidence is central to the concept of EBP. The term ‘evidence-based practice’ is probably very familiar to most readers. However, it is worth spending a little time reviewing the definitions and history of EBP and EBOT in order to provide a context for the remainder of the chapter.
EBP has its roots in the evidence-based medicine (EBM) movement, which began at McMaster University in Canada in the 1980s (Taylor 2007), linked to the use of a problem-based approach to the teaching of medicine. By the mid-1990s EBM had been defined as:
the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients (Sackett et al 1996: 71)
By the late 1990s EBP had begun to seep into the consciousness of the occupational therapy profession, especially in the UK and Canada, with special issues of both the British Journal of Occupational Therapy (College of Occupational Therapists 1997, 2001) and the Canadian Journal of Occupational Therapy (Canadian Association of Occupational Therapists 1998) as well as books (e.g. Law 2002, Taylor 2007) and articles (e.g. Brown and Rodgers 1999, Bennett and Bennett 2000) on the implementation of EBP in occupational therapy. It is also worth noting that the 1997 Casson Memorial Lecture in the UK (Eakin 1997), the 2000 Eleanor Clarke Slagle Lecture in the US (Holm 2000) and the 2001 Sylvia Docker Lecture in Australia (Cusick 2001) all explored EBP and EBOT and encouraged practitioners to find and use evidence to support their everyday practice.
This interest in EBOT also led to the development of a number of definitions of EBOT, the best known of which defines it as:
Client-centred enablement of occupation, based on client information and a critical review of relevant research, expert consensus and past experience (Canadian Association of Occupational Therapists et al 1999: 267).
However, the essence of EBOT is probably best expressed by Cusick (2001: 103),
When we practice with evidence, it means we should ask ourselves the following question: ‘am I doing the right thing in the right way with the right person at the right time in the right place for the right result – and am I the right person to be doing it?’
The challenge for the evidence-based occupational therapist is to find and appraise the relevant evidence. However, the challenge does not end once the evidence has been located. The evidence may indicate that a particular intervention has been shown to be effective. The evidence may also show that there is little or no evidence to support the effectiveness of a chosen intervention and the evidence-based occupational therapy then needs the courage to change practices, that may be long established, and to ensure that occupational therapy is seen as an effective and relevant profession (see Chapter 15). Thus EBOT is a way of thinking critically about every intervention and action and, as such, is just one of the tools of clinical reasoning and reflective practice. However, because of the use of up-to-date best evidence, evidence-based practice is a powerful tool.
Why is appraisal of evidence important?
Whilst the principles of EBOT and a questioning approach to the value of our interventions and activities as practitioners might appear to be a valuable philosophy, and very useful in the abstract, we must also remind ourselves of the reasons that EBOT and the need to appraise evidence are currently seen as vitally important for all practitioners.
EBP has been enshrined into health and social care policy for some time (English examples include The New NHS: Modern, Dependable (DoH 1997; National Service Frameworks, DoH, 2000, 2001) and into professional practice at both statutory (Health Professions Council, 2003) and professional levels (College of Occupational Therapists 2003, 2005). However, for many practitioners there has been a resistance to this drive for EBP. Dubouloz et al (1999) found that occupational therapists have been slow to integrate research evidence into their clinical decision-making, although there is some more recent evidence that occupational therapists now believe that EBP is important to practice (Bennett et al 2003).
For many practitioners their own previous experiential evidence has been key to intervention decision-making (Pringle 1999, Upton 1999a, 1999b). EBP does not seek to deny the value of experiential evidence; it seeks to ensure that experiential evidence is not the sole source of evidence for competent decision-making. To be able to make competent decisions about our interventions and action we must be able to incorporate critical reflection on our own experiential knowledge with critical appraisal of the theoretical and research evidence relevant to that intervention or action.
The stages of EBOT
Whilst EBOT can be described as a way of thinking about practice and asking, as Cusick (2001) puts it, the ‘right’ questions; EBOT is also a problem-solving process, which mirrors both the research process and the occupational therapy process. Sackett et al (1995) articulated the five stages of EBP as:
Whilst other chapters focus on the 4th and 5th stage (Chapters 16 and 6 respectively), this chapter will focus on the first three stages of this process:
However, the process of EBOT should not be viewed as a one-off activity: it should run parallel to the process of clinical reasoning and can be applied at each stage of the occupational therapy process to ensure that every action from initial assessment to final evaluation is effective and based on sound reasoning and evidence. Bennett and Bennett (2000) have developed a framework that shows how EBP can be used at every stage of the occupational therapy process. Rappolt (2003) has developed a model of a client-centred EBOT process, which has integrated three strands of evidence (from the client, from research and from professional expertise) into the occupational therapy process in the hope of overcoming the perception that EBP and client-centred practice are mutually exclusive (Taylor 1997).
Asking good questions
The first stage of the EBOT process is to identify and articulate a question that will guide the search for evidence to be appraised. Because of the original medical focus of EBP, questions are often thought of as ‘clinical’ questions relating to diagnosis, prognosis or treatment (Rosenberg and Donald, 1995) and are phrased in terms of:
What is the evidence for the effectiveness of x (the intervention) for y (the outcome) in a patient with z (the problem or diagnosis)?
This might fit very nicely into medical practice when thinking about whether treatment with aspirin and warfarin will reduce the risk of stroke in an elderly lady with hypertension, but how can it relate to the complexities of occupational therapy practice?
Herbert et al (2005: 12) have expanded the application of the clinical question to include:
Whilst broadening the idea of the clinical question beyond assessing the potential effectiveness of an intervention, this approach still does not address the totality of occupational therapy practice. However, if we adopt Cusick’s (2001) approach of asking the right questions, we can utilise an evidence-based approach to all stages of the occupational therapy process, e.g.:
The anatomy of a well-built question
The task for the evidence-based occupational therapist is to devise a question. This question will arise from practice, possibly by asking oneself one of the ‘right’ questions outlined above or by articulating or reflecting on a particular incident or client to develop a scenario to help you to develop the final EBP question. Box 14.1 outlines two scenarios, which will be used as illustrations throughout this chapter.
Box 14.1 Identifying a scenario
Scenario 1
The Stroke Unit where you are working is keen to use ‘constraint induced therapy’ (CIT) as an intervention. You are unsure of the potential value of this approach, especially in relation to other more established intervention approaches, and decide to explore the literature to find out what evidence exists to demonstrate the effectiveness of CIT.
Scenario 2
You have been asked to co-facilitate a fatigue management group for people living with multiple sclerosis (MS). The central philosophy for the group is self-help and empowerment and to use strategies that are informed by experience as well as research evidence. You identify some relevant anecdotal knowledge but want to adopt a more evidence-based approach. You decide to collect evidence from a range of sources to inform your planning of the fatigue management group. You also hope that the evidence will give you ideas about assessing the outcomes and the success of the group.
Having identified and articulated a scenario, it is then possible to refine this information further to create and structure a specific evidence-based question. The clearer the structure of the EBP question, the more focused and, hopefully, successful the search for evidence.
As we noted above, an evidence-based question usually consists of a number of elements:
Richardson et al (1995) refer to these elements as the ‘anatomy’ of a well-built question, and add a further element: a comparative intervention. The inclusion of a comparative intervention is common practice in the context of EBM where the effectiveness of drug x may be compared with that of drug y. The inclusion of a comparative intervention may be perceived as having less relevance to many for EBOT, although it may be of value in the context of some EBP questions.
Whilst this approach, commonly referred to as PICO (problem, intervention, comparative intervention, outcome) may seem of limited relevance to EBOT, where there is rarely a comparative intervention and outcomes are not always clearly identified, as Table 14.1 illustrates the elements can easily be applied to occupational therapy especially if the elements are renamed:
This structure can then be applied to any scenario, as Box 14.2 illustrates.
Box 14.2 Developing an evidence-based question
Scenario 1
person – people experiencing hemiparesis, especially upper limb involvement, following a stroke
action – constraint induced therapy
comparative action – all other current approaches to stroke intervention
application – improved upper limb (UL) function, independence in activities of daily living (ADL), increased meaningful occupational behaviours
What is the evidence for the effectiveness of constraint induced therapy, in comparison to existing approaches to stroke rehabilitation, in improving UL function/independence in ADL or increasing meaningful occupational behaviours in clients who experience hemiparesis following stroke?
Scenario 2
person – people experiencing fatigue as a consequence of MS
action – living with fatigue; fatigue management
comparative action – not relevant for this question
application – guidelines for facilitating a fatigue management group
What information can be derived from a range of evidence into the experience of living with MS-related fatigue, in order to facilitate a fatigue management group?
Searching for the evidence
Having articulated the evidence-based question the next task is to search for and locate the evidence to address this question. However, before we can embark on the search it is necessary to understand the concept of ‘evidence’ and the scope of evidence within EBOT.
What do we mean by ‘evidence’?
Sackett et al’s (1996) definition of evidence-based practice talks about the need to use ‘best evidence’ to support the decision-making process. However, the nature of ‘best evidence’ is perhaps the most contentious and debated area within the concept of evidence-based practice (Sackett and Wennberg 1997, Taylor 2007). The traditional view, drawn from evidence-based medicine, has been to adopt a hierarchy, or levels, of research evidence. Table 14.2 outlines the hierarchy of evidence for evidence-based medicine.
Table 14.2 Levels of evidence within EBM
However this approach ignores two major factors. The first is that there is a breadth of potential research approaches, which might be appropriately viewed as ‘best evidence’. The second is that many definitions of evidence-based practice include not only research but also the therapist’s experiential knowledge and the client’s perspective as potential sources of evidence. Table 14.3 identifies the potential range of research approaches that might be seen as evidence.
Table 14.3 Types of research evidence
The question for the evidence-based occupational therapist is, which of this long list might be the ‘best evidence’ to use to address a particular evidence-based question? As Sackett (1998), Sackett and Wennberg (1997) and Taylor (2007) argue, the nature of the ‘best evidence’ depends upon the type of evidence-based question being asked and as we have seen the potential varieties of EBOT questions are diverse. Table 14.4 gives an overview of the types of research evidence that might be appropriate for the different types of evidence-based question.
Table 14.4 Appropriate research evidence for particular types of evidence-based questions
| Effectiveness of interventions |
| Systematic reviews of RCTs |
| RCTs |
| Other experimental designs, e.g. controlled clinical trials |
| Single subject design studies |
| Client’s experiences and perceptions |
| Qualitative research studies |
| Systematic reviews of qualitative research |
| Descriptive research studies, e.g. surveys |
| Appropriateness of assessments |
| Cross-sectional studies |
| Measurement studies |
| Prognosis and life-course of a particular condition or group of people |
| Cohort studies |
| Longitudinal studies |
| Qualitative research studies |
| Correlational studies |
Having established a list of the potential types of research evidence to address a particular question, the evidence-based occupational therapist’s next task is to decide whether there is a particular order or hierarchy of the types of evidence, to ensure that the ‘best’ evidence is found.
Developing a hierarchy of the most appropriate evidence for the effectiveness of interventions is relatively straightforward. The hierarchy, outlined in Table 14.2, was developed to show the value and weighting of evidence for the effectiveness of interventions, with systematic reviews of RCTs seen as the most rigorous and reliable form of evidence. Similar hierarchies have been developed for
The appropriateness of a similar hierarchy for qualitative research is much more questionable, with many authors (e.g. Barbour 2001, Pawson et al 2003) arguing that, whilst it might be possible to critically appraise the rigour and strength of a particular qualitative study is it neither possible nor appropriate to locate different types of qualitative studies within a hierarchy.
The identification of the ‘best’ evidence is a complex task. The potential range of evidence is broad and should not solely focus on research evidence. Using specific types of research evidence to address particular evidence-based questions may seem the most useful approach. However, it may also act as a constraint to the development of a broad perspective on the ‘best evidence’ with which to answer evidence-based questions. Certainly an RCT or a systematic review should provide powerful evidence for the effectiveness of a particular intervention, it should not, however, be the only evidence required for clinical reasoning and decision-making. Pawson et al (2003) argue, from a social care perspective, that evidence should include:
thus acknowledging the breadth and complexity of evidence to be considered, especially within a social care context.
Evidence from research studies can only give a partial answer to any evidence-based question. The research evidence must be balanced with information from the client about their values and perspectives, as well as the therapist’s experiential knowledge. The intervention or action decision will also be influenced by contextual factors such as service priorities and resources, as well as local and national policies. Evidence should not be seen in terms of a hierarchy but in terms of pieces of a complex jigsaw, which together provide the ‘best evidence’ to answer any evidence-based question.
Structuring a search
Having clarified the elements of the evidence-based question, the next stage in the process is to use these elements to provide a clear structure for the task of searching for some evidence, which will then be appraised for its value in the context of the evidence-based question.
Once the evidence-based question has been articulated, there is often a temptation to go straight on to the Internet to find some evidence. However, if you start searching without doing sufficient thinking and planning your search will probably be very frustrating and ultimately unsuccessful. It is much more useful to spend some time thinking about the structure of your search, so that any time spent at the computer or in the library is usefully spent and might also result in the successful location of some relevant and valuable evidence.
Developing a search strategy consists of a number of stages:
Identifying the key words and search terms is an important task if the search is to be successful. It is always useful to look at the search terms and key words that you have identified and then to think of synonyms or alternative terms, as well as different spellings (many terms have different spelling depending on whether the usage is English or American).
Most databases do not readily understand ‘natural language’, the colloquial language that we use every day. However, the majority of databases have a thesaurus of accepted terms and key words. Databases use what are known as Boolean operators to help the searcher refine the search question. Operators can be used to focus the search. The most common Boolean operators are:
It is also sensible to set limits to any search to avoid being swamped by information that is inappropriate, irrelevant or inaccessible. Common limits are the language the article is written in, the age of the article and the type of article.
Table 14.5 provides a summary of the pointers to assist in structuring a search and Box 14.3 illustrates how the questions identified in the previous scenarios can be developed into search strategies.
Table 14.5 Structuring the search
Box 14.3 Developing search strategies
Scenario 1
What is the evidence for the effectiveness of constraint induced therapy, in comparison to existing approaches to stroke rehabilitation, in improving UL function/independence in ADL or increasing meaningful occupational behaviours in clients who experience hemiparesis following stroke?
the question focuses on the effectiveness of an intervention, so the search needs to look for systematic reviews and RCTs
“constraint induced therapy” OR “constraint induced movement therapy” AND stroke
Scenario 2
What information can be derived from a range of evidence into the experience of living with MS-related fatigue, in order to facilitate a fatigue management group?
the question focuses on people’s experiences of dealing with illness as well as the effectiveness of an intervention, therefore a range of evidence including both qualitative research studies and RCTs/systematic reviews will be sought
Where to look
Rather than wandering vaguely around the library randomly selecting interesting-looking books and journals, the evidence-based occupational therapist should focus on the most appropriate resources in the search for relevant evidence. Bibliographic indexes and databases have been developed to help the evidence-based practitioner to locate the most suitable information. However, many databases are only available by subscription, so it is worth checking with your library to see which databases you might be able to access. The databases most relevant to occupational therapy are identified in Table 14.6. The searches linked to the evidence-based questions outlined above are discussed and illustrated in Table 14.7 and Box 14.4 and the reader might be able to draw some conclusions as to the relative merits of the various databases for their own evidence-based searches.
Table 14.6 Databases relevant to the evidence-based OT
| General databases | Specialist evidence-based databases |
|---|---|
| OT databases | Web-based resources |
Box 14.4 Overview of search examples
Scenario 1
What is the evidence for the effectiveness of constraint induced therapy, in comparison to existing approaches to stroke rehabilitation, in improving UL function/independence in ADL or increasing meaningful occupational behaviours in clients who experience hemiparesis following stroke?
The first choice of databases for this question would be those databases that have a specific focus on systematic reviews and RCTs, such as the Cochrane Library and OTSeeker, with more general databases such as AMED, CINAHL and MEDLINE/PubMed providing access to other types of research evidence and the web-based resources possibly giving access to a range of opinion-based information.
This search only identified RCTs, it did not locate the systematic reviews that were found in the OTSeeker search.
This was a highly successful search as all 16 of the citations were potentially relevant to the evidence-based question. One of the strengths of OTSeeker is that all RCTs are critically appraised and rated for their research rigour by a team of researchers, so much of the EBOT’s has already been done.
Whilst the AMED search resulted in the fewest hits, it was probably the most focused although all of the studies identified in the OTSeeker search were replicated in the AMED search. There was also considerable replication between all of these searches, but also some useful additional material that was not identified in either the Cochrane Library, OTSeeker or AMED searches. However, for both the CINAHL and MEDLINE/PubMed searches the reader would have to spend some time sifting through a number of apparently irrelevant citations, such as a paper on the effects of d-amphetamine injections on the motor training of squirrel monkeys (Barbay 2006). Narrowing the search by including the search terms ‘occupational therapy’ and ‘activities of daily living’ resulted in the identification of a smaller number (8) of highly relevant research papers, reminding the reader of the need to have clear and specific key words to avoid being swamped by the search results.
These resulted in a large number of hits but little evidence that was of value for the evidence-based question.
Scenario 2
What information can be derived from a range of evidence into the experience of living with MS-related fatigue, in order to facilitate a fatigue management group?
The search for this evidence-based question is rather less straightforward than the previous search. This question has a primary focus on the lived experience of MS-related fatigue but is also interested in the effectiveness of fatigue management as an intervention. The search, therefore, needs to begin with the general databases such as AMED, CINAHL and MEDLINE/PubMed in order to access the qualitative research and then needs to look at OTSeeker and the Cochrane Library for systematic reviews and RCTs.
In terms of the search for qualitative research evidence the MEDLINE/PubMed search was the most effective as all 3 ‘hits’ were highly pertinent to the evidence-based question. Replication of citations was again common across the various databases. The AMED search provided some useful supporting material on the nature of MS-related fatigue and the effectiveness of fatigue management strategies.
Not all of the 10 ‘hits’ were relevant to this evidence-based question, but this search did identify relevant papers that the other searches did not find.
This search produced a mixed bag of results. 7 Cochrane reviews were identified, although only 2 were of relevance to the evidence-based question. A further review was identified on DARE (Database of Reviews of Effectiveness) but this proved to be a review of pharmacological interventions. A total of 93 RCTs were identified, however the majority of these were not relevant to the evidence-based question and those that were had already been located in other searches.
These searches did reveal some useful material, but only after considerable time and effort had been spent scrutinising the numerous citations.
Databases have different areas of focus. MEDLINE, PubMed (the free access version of MEDLINE) and EMBASE have a very medical focus, whilst CINAHL and AMED focus more on allied health. ASSIA, PsychLit and SocioFile draw heavily on social science literature and ERIC has an educational focus. A number of specific EBP databases now exist. These databases, unlike general databases, employ quality checks, and so will only include citations for work that meets their standards for good evidence. There are also a number of occupational therapy-specific databases, which might be of value. OTSeeker (www.otseeker.com; Bennett et al 2003, McClusky 2006, McKenna 2004, 2005) is a particularly valuable resource as it also incorporates an evidence-based approach and does not require a subscription. Web-based search engines (such as Google, Google Scholar and Ask) are also databases of a sort, although they are less regulated and, therefore, less rigorous than more specific databases. Intute (www.intute.ac.uk) is a relatively new free online service, which aims to give students access to good web-based resources that have been evaluated by a network of subject specialists. Table 14.7 gives an overview of the results of the searches for the two scenarios (constraint induced therapy and MS-related fatigue), in terms of numbers of ‘hits’ and Box 14.4 provides a commentary on these searches.
Appraising the evidence
Once suitable research evidence has been identified and located the job of the EBOT is not finished. Research evidence should not be accepted at face value. Whilst many journals use a rigorous process of peer review and critical evaluation prior to the publication of any article, it does not mean that a published piece of research is flawless or that it can be applied to any practice situation.
Having located relevant research evidence, the next stage in the EBOT process is to critically appraise the evidence. Critical appraisal can be defined as
‘the ability to read original and summarised research, to make judgements on its scientific value and to consider how its results can be applied in practice’ (Taylor 2003: 102).
Critical appraisal provides a structured framework in order to assess the value and trustworthiness of a piece of research within the context of practice. The structure of the critical appraisal process allows the evidence-based occupational therapist to review not just the findings of the research but the whole research process. Critical appraisal should give the reader the opportunity to identify both the strengths and the weaknesses of a particular research paper. The weaknesses of the research should be weighed and evaluated carefully but should be viewed in the context of whether they make you question the conclusions of the researchers. Critical appraisal should be positive rather than negative and viewed with an open mind and the ability to challenge your own, as well as the researcher’s, ideas and assumptions.
Adopting a critical appraisal approach to reading the research evidence will encourage the reader not to ignore, or skip over, the ‘complicated’ sections of a research paper, such as the results section. Skipping over parts of an article may lead to misinterpretation of findings or erroneously accepting (or rejecting) the author’s conclusions. Critical appraisal is often more interesting if it is not a solitary activity, so that the findings and ideas can be discussed and ideas and comments can be challenged and reviewed. Journal clubs can be used as a way of sharing critical appraisals of research literature.
The key questions
Critical appraisal is a process of asking a series of structured questions about the rigour and applicability of the research. Any critical appraisal, irrespective of the type of research being appraised, is structured around three essential questions:
All of these questions can be broken down further to explore the research in depth.
The first group of questions addresses the rigour of the research and helps the reader to appraise how good the research is. The second group of questions helps the reader to focus on the research findings and to assess the significance, both statistical and clinical, or strength of the findings. The final group of questions gives the reader the opportunity to explore whether the research and its findings can be used in the context of practice, does the research give any evidence to support or challenge current practice.
Appraising different types of evidence
Whilst the three questions outlined above give an overall structure to any critical appraisal, they have to be adapted to suit different types of research methodology. A common mistake when starting to critically appraise research evidence is to adopt a standard series of questions, in a one-size-fits-all approach to appraisal. This can lead to certain research papers and research methodologies being viewed as less rigorous because the wrong questions have been used to appraise them. Thus, qualitative research papers need to be appraised with a different set of questions from quantitative research papers and systematic reviews should not be appraised with questions relevant to RCTs. Any appraisal calls for some level of understanding and knowledge of the research methodology being appraised.
It is beyond the scope of this chapter to give a thorough and detailed discussion of the questions used to appraise different research types and methodologies. Critical appraisal is usually best facilitated by the use of an appraisal checklist, which will outline the key questions for a particular research type. Numerous examples of checklists and lists of questions are available. Table 14.8 outlines some of the most useful examples.
Table 14.8 Identifying a useful appraisal checklist
| Taylor (2007) gives worked examples of appraisal using separate checklists for: |
| Humphris (2005) gives a brief overview of the questions that might be used to appraise: |
| Dickson (2005) gives a brief overview of the questions to use when appraising systematic reviews |
| Craig and Smyth (2002) outline the types of questions to ask when appraising a range of types of evidence, including: |
| Greenhalgh (2006) gives a thorough overview of the questions to ask when appraising a variety of medical research approaches, including: |
| Helewa and Walker (2000) give a basic overview of the questions to ask when appraising a variety of research articles: |
| Herbert et al (2005) provide questions for assessing the validity of a number of research types: |
| CASP [www.phru.nhs.uk/Pages/PHD/CASP.htm] have a range of checklists, which can be downloaded from their web site, for: |
| McMaster University School of Rehabilitation Science [www.srs-mcmaster.ca/Default.aspx?tabid=546] have general guidelines for reading and appraising research, which can be downloaded from their web site, for: |
| Netting the evidence [www.shef.ac.uk/scharr/ir/netting/] has weblinks to a wide range of appraisal resources |
Including appraisal in everyday practice
Evidence-based occupational therapy is not about doing research it is about using research evidence to underpin interventions and actions within everyday practice. The challenge is to find space in the busy working week to be able to search for and appraise the relevant research evidence. The chapter will now explore a number of ways that appraisal of evidence can be incorporated into everyday practice.
Journal clubs
Journal clubs have been a regular feature of medical education and practice for many years (Linzer 1987). They usually consisted of someone presenting an overview of a paper that they had read and then attempting to stimulate discussion. However, with the advent of EBM the format of journal clubs began to change (Sackett et al 1997) and evidence-based journal clubs began to be established (Phillips and Glasziou 2004). Evidence-based journal clubs also began to be developed not just amongst medical practitioners but for groups of occupational therapists, nurses and other allied health professionals (Bannigan and Hooper 2002, Sherratt 2005, McQueen et al 2006).
Evidence-based journal clubs provide an opportunity for a group of like-minded colleagues to meet together to discuss and appraise the relevant research evidence linked to a clear evidence-based question, probably using an appraisal checklist as a way of structuring and recording their discussions.
Journal clubs are a useful way of both promoting and recording evidence of continuing professional development. They enable participants to maintain and enhance their critical appraisal skills and their knowledge of the research base for practice. They also provide an opportunity to review and reflect on current practice. The outcome of a journal club discussion might be to implement changes to current interventions or actions, or just to enhance confidence in the value of those interventions and actions. The journal club can provide a valuable opportunity for groups of colleagues to explore issues, share ideas, consider different perspectives and participate in shaping and developing departmental policy and practice. Specific guidelines for establishing a journal club are beyond the scope of this chapter, but can be found in a number of publications e.g. Taylor (2007), Sherratt (2005) and Phillips and Glasziou (2004).
As the focus of this chapter is on evidence-based occupational therapy, it would seem appropriate to outline the evidence for the value of journal clubs in changing practice and ensuring that all interventions and actions are evidence-based. Sadly, the majority of the evidence is anecdotal or, at best, expert opinion and consensus (McQueen et al 2006). There is evidence to support the value of journal clubs in developing critical appraisal skills (Parkes et al 2001), but there is little research evidence to support the impact of journal clubs on practice development and change. McQueen et al’s (2006) small exploratory study indicated that there were some changes in the participants’ attitudes and confidence and an increased awareness of the evidence base supporting their interventions and actions. Journal clubs would appear, at face-value, to be useful activities both for developing an evidence-based culture within a department and for ensuring that practice is evidence-based, however it is imperative that the impact and value of journal clubs is researched further.
Evidence-based reflection and case studies
In the discussion on the nature of ‘evidence’ earlier in the chapter it was noted that experience is often reported as the main source of evidence used by practitioners when making decisions about interventions and actions as part of the clinical reasoning process. However, whilst Enkin and Jadad (1998) put forward a sound and coherent argument for the use of experiential, or, as they term it, ‘anecdotal’, evidence within EBP, they do not give any practical suggestions as to its use. Rappolt’s (2003) model of a client-centred EBOT process mentions both ‘client evidence’ and ‘clinical expertise’ but does not discuss how evidence of clinical expertise can be articulated. The evidence-based occupational therapist is, therefore, left wondering how experiential evidence can be brought explicitly into the process of EBOT.
Reflection on action and the use of evidence-based case studies may provide useful tools to help the evidence-based occupational therapist to articulate the experiential evidence used in practice (see Chapter 5). The process of reflection involves describing an event and then looking at the decision-making and reasoning process, which underpin the actions taken within that event. The event concerned can be any interaction with a client. To use these reflections to develop an evidence-based case study the following topics should be addressed:
Bailey et al (2007) and Smallfield and Lou (2006) are two interesting examples of how evidence-based case studies can be developed and used to incorporate EBOT within everyday practice.
Developing critically appraised papers and critically appraised topics
Many practitioners are developing their critical appraisal skills and becoming involved in the critical appraisal of research papers. It is important to avoid duplication of effort and that appraisal of research evidence is shared and disseminated to the wider practitioner community. Two useful tools for disseminating appraisal are critically appraised papers (CAPs) and critically appraised topics (CATs).
A number of journals are now including CAPs regularly (for example the Australian Occupational Therapy Journal, and the Australian Journal of Physiotherapy) whilst journals such as Evidence-based Mental Health and Evidence-based Healthcare contain nothing but CAPs. The importance of these papers is that they include not just a structured summary of a previously published research paper but they also include a commentary, by a practitioner, on the rigour of the research and its usefulness as evidence for practice. CAPs are vital tools in the dissemination of EBOT and wider publication of CAPs should be encouraged within the professional literature.
CATs are short summaries of the research evidence on a topic of interest, usually focused around an evidence-based question. A CAT should be seen as a shorter and less rigorous form of a systematic review; beginning with an outline of the evidence-based question and the search strategy before summarising the best available research evidence on the chosen topic. Usually more than one research study is included in a CAT. CATs are a useful tool for busy practitioners to summarise and share their research appraisals. There are multiple sites on the internet containing CATs. Many have a medical focus, but sites of interest to occupational therapists include:
A major benefit of a CAT is its brevity and simplicity. It is important to note a major limitation of many CATs is the absence of independent peer review, as many CATs published on the Internet are not peer reviewed, it is important to bear this fact in mind if using CATs to guide practice. Readers cannot be confident that a thorough and complete search of the literature has been conducted nor that an accurate interpretation of the methods, results and statistics has been made. However, CATs are useful tools that can be used to guide and inform practice. When appraising the research literature it is worth considering whether the work you are doing can be developed into a CAT.
Summary
This chapter has focused on the first stages of the EBOT process. It began by outlining the nature of EBOT before exploring the nature of ‘evidence’ in the context of EBOT and discussing ways of searching for research evidence. The chapter then discussed the questions to ask when critically appraising a research paper before concluding by presenting a number of ways of incorporating critical appraisal into everyday practice.
Bailey DM, Bornstein J, Ryan S. A case report of evidence-based practice: from academia to clinic. American Journal of Occupational Therapy. 2007;61(1):85-92.
Bannigan K, Hooper L. How journal clubs can overcome barriers to research utilisation. British Journal of Therapy & Rehabilitation. 2002;9(8):299-303.
Barbay S. A single injection of d-amphetamine facilitates improvement in motor training following a focal cortical infarct in squirrel monkeys. Neurorehabilitation & Neural Repair. 2006;20(4):455-458.
Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? British Medical Journal. 2001;322(7294):1115-1117.
Bennett S, Bennett JW. The process of evidence-based practice in occupational therapy: informing clinical decisions. Australian Occupational Therapy Journal. 2000;47:171-180.
Bennett S, Hoffman T, McCluskey A, et al. Evidenced-based practice forum. Introducing OTSeeker (Occupational Therapy Systematic Evaluation of Evidence): a new database for occupational therapists. American Journal of Occupational Therapy. 2003;57:635-638.
Brown GT, Rodgers S. Research utilization models: frameworks for implementing evidence-based occupational therapy practice. Occupational Therapy International. 1999;6(1):1-23.
Canadian Association of Occupational Therapists. Special edition on evidence-based practice. Canadian Journal of Occupational Therapy. 65(3), 1998.
Canadian Association of Occupational Therapists, Association of Canadian Occupational Therapy University Programs, Association of Canadian Occupational Therapy Regulatory Organizations & the Presidents’ Advisory Committee. Joint position statement on evidence-based practice. Canadian Journal of Occupational Therapy. 1999;66:267-269.
College of Occupational Therapists. Special edition on evidence-based practice. British Journal of Occupational Therapy. 60(11), 1997.
College of Occupational Therapists. British Journal of Occupational Therapy, 64. 2001;5.
College of Occupational Therapists. Professional Standards for Occupational Therapy Practice. London: College of Occupational Therapists, 2003.
College of Occupational Therapists. Code of Ethics and Professional Conduct. London: College of Occupational Therapists, 2005.
Craig JV, Smyth RL, editors. The Evidence-based Practice Manual for Nurses. Edinburgh, UK: Churchill Livingstone, 2002.
Cusick A. 2001 Sylvia Docker Lecture: OZ OT EBP 21c: Australian occupational therapy, evidence-based practice and the 21st century. Australian Occupational Therapy Journal. 2001;48(3):102-117.
Department of Health. The New NHS: Modern, Dependable. London: HMSO, 1997.
Department of Health. www.doh.gov.uk/nsf/coronary.htm, 2000.
Department of Health. www.doh.gov.uk/nsf/mentalhealth.htm, 2001.
Dickson R. Systematic reviews. Hamer S, Collinson G, editors. Achieving Evidence-based Practice, 2nd edn. Edinburgh, UK: Baillière Tindall. 2005:43-62.
Dubouloz CJ, Egan M, Vallerand J, et al. Occupational therapists’ perceptions of evidence-based practice. American Journal of Occupational Therapy. 1999;53(5):445-453.
Eakin P. The Casson Memorial Lecture 1997: Shifting the balance – Evidence-based Practice. British Journal of Occupational Therapy. 1997;60(7):290-294.
Enkin MW, Jadad AR. Using anecdotal information in evidence-based health care: Heresy or necessity. Annals of Oncology. 1998;9:963-966.
Greenhalgh T. How to Read a Paper, 3rd edn. Oxford, UK: Blackwell Publishing. 2006.
Health Professions Council. Standards of Proficiency, Occupational Therapists. London: Health Professions Council, 2003.
Helewa A, Walker JM. Critical Evaluation of Research in Physical Rehabilitation. Philadelphia, PA: WB Saunders, 2000.
Herbert R, Jamtvedt G, Mead J, et al. Practical Evidence-based Physiotherapy. Edinburgh, UK: Churchill Livingstone, 2005.
Holm MB. The 2000 Eleanor Clarke Slagle Lecture – Our mandate for the new millennium: Evidence-based practice. American Journal of Occupational Therapy. 2000;54:575-585.
Humphris D. Types of evidence. Hamer S, Collinson G, editors. Achieving Evidence-based Practice, 2nd edn, Edinburgh, UK: Baillière Tindall, 2005.
Law M, editor. Evidence-based Rehabilitation. Thorofare, NJ: Slack Incorporated, 2002.
Linzer M. The journal club and medical education: over one hundred years of unrecorded history. Postgraduate Medical Journal. 1987;63:475-478.
McCluskey A. How and why do occupational therapists use the OTseeker evidence database? Australian Occupational Therapy Journal. 2006;53(3):188-195.
McKenna K. In practice. OT seeker: facilitating evidence-based practice in occupational therapy. Australian Occupational Therapy Journal. 2004;51(2):102-105.
McKenna K. Australian occupational therapists’ use of an online evidence-based practice database (OTseeker). Health Information & Libraries Journal. 2005;22(3):205-214.
McQueen J, Nivison C, Husband V, et al. An investigation into the use of a journal club for evidence-based practice. International Journal of Therapy & Rehabilitation. 2006;13(7):311-316.
Parkes J, Hyde C, Deeks J, et al. Teaching Critical Appraisal Skills in Health Care Settings. Cochrane Database of Systematic Reviews 2001. (Issue 3):2001.
Pawson R, Boaz A, Grayson L, et al. Knowledge Reviews 3: Types & Quality of Knowledge in Social Care. London: SCIE & The Policy Press, 2003.
Phillips RS, Glasziou P. What makes evidence-based journal clubs succeed? Evidence Based Medicine. 2004;9:36-37.
Phillips B, Ball C, Sackett D, et al. Oxford Centre for Evidence-based Medicine Levels of Evidence. 2001. (May 2001) http://www.cebm.net/index.aspx?o=1025 (accessed 28th October 2001).
Pringle E. EBP: is it for me? Therapy Weekly. 1999;25(46):12.
Rappolt S. The role of professional expertise in evidence-based occupational therapy. American Journal of Occupational Therapy. 2003;57(5):589-593.
Richardson WS, Wilson MC, Nishikawa J, et al. The well-built clinical question: a key to evidence-based decisions (editorial). ACP Journal Club. 1995;123:A12-A13.
Rosenberg W, Donald A. Evidence based medicine: an approach to clinical problem-solving. British Medical Journal. 1995;310:1122-1126.
Sackett DL. Shamanism (Was: Pre-test probability). Evidence-based-health. 1998. Accessed 8 June 2000, [online]. Available from http://www.mailbase.ac.uk/lists/evidence-based-health/1998-03/0066.html
Sackett DL, Wennberg JE. Choosing the best research design for each question. British Medical Journal. 1997;315:16-36.
Sackett DL, Rosenberg WMC, Gray JAM, et al. Evidence-based medicine: what it is and what it isn’t. British Medical Journal. 1996;312:71-72.
Sackett DL, Richardson WS, Rosenberg W, et al. Evidence-based Medicine: How to Practice & Teach EBM. New York: Churchill Livingstone, 1997.
Sherratt C. The Journal Club: a method for occupational therapists to bridge the theory-practice gap. British Journal of Occupational Therapy. 2005;68(7):301-306.
Smallfield S, Lou JQ. The effectiveness of low vision rehabilitation on quality of life: an evidence-based practice approach to answer clinical questions. Occupational Therapy in Health Care. 2006;20(2):17-30.
Taylor MC. What is evidence-based practice? British Journal of Occupational Therapy. 1997;60(11):470-474.
Taylor MC. Evidence-based practice: informing practice and critically evaluating related research. In: Brown G, Esdaile SA, Ryan SE, editors. Becoming an Advanced Practitioner. Edinburgh, UK: Butterworth-Heinemann; 2003:90-117.
Taylor MC. Evidence-based Practice for Occupational Therapists, 2nd edn. Oxford, UK: Blackwell Publishing. 2007.
Upton D. Clinical effectiveness and EBP 2: attitudes of health-care professionals. British Journal of Therapy & Rehabilitation. 1999;6(1):26-30.
Upton D. Clinical effectiveness and EBP 2: application by health-care professionals. British Journal of Therapy & Rehabilitation. 1999;6(2):86-90.