Appraising and understanding systematic reviews of quantitative and qualitative evidence

• involve a clear definition of eligibility criteria

• incorporate a comprehensive search to identify all potentially relevant studies (although there are qualitative reviewers that opt for a purposeful sample of papers deemed relevant for generating theory)

• use explicit, reproducible and uniformly applied criteria in the selection of studies for the review

• rigorously appraise the risk of bias within individual studies, and

• systematically synthesise the results of included studies.²

• improve the dissemination of evidence²

• hasten the assimilation of research into practice³

• assist in clarifying the heterogeneity of conflicting results between studies⁴

• establish generalisability of the overall findings²

• improve the understanding of a particular phenomenon or situation, and

• guide decision making.

explore questions such as how do people experience illness, why does an intervention work (or not), for whom and in what circumstances … what are the barriers and facilitators to accessing health care, or what impact do specific barriers and facilitators have on people, their experiences and behavior? … [Qualitative evidence syntheses] can aid understanding of the way in which an intervention is experienced by all of those involved in developing, delivering or receiving it; what aspect of the intervention they value, or not; and why this is so … [it] can provide insight into factors that are external to an intervention including, for example, the impact of other policy developments, factors which facilitate or hinder successful implementation of a programme, service, or treatment and how a particular intervention may need to be adapted for large-scale roll-out.

1. Define the research question and plan the methods for undertaking the review.

2. Determine the eligibility criteria for studies to be included.

3. Search for potentially eligible studies.

4. Apply eligibility criteria to select studies.

5. Assess the risk of bias in included studies.

6. Extract data from the included studies.

7. Synthesise the data.

8. Interpret and report the results.

9. Update the review at an appropriate time point in the future.

Define the research question and plan the methods for undertaking the systematic review

As with any research study, when conducting a systematic review the first place to start is planning the overall project. Planning also ensures that all aspects important to the scientific rigour of a review are undertaken to reduce the risk of bias. Each stage of the review needs to be thoroughly understood prior to moving on to plan the next stage.

The question needs to be clearly focused, as too broad a question will not produce a useful review. The question should also make sense clinically. Involving consumers, at both the health professional and the patient levels, may also be helpful when developing a plan, as this will ensure that areas of concern such as the types of interventions or outcomes that might not otherwise be considered are addressed.¹⁷

Planning a review also requires decisions to be made about the methods for searching, screening, appraisal and synthesis. These decisions should be made before commencing the review itself and ideally written up as a protocol, as this will help to make the whole process systematic and more transparent for all involved.

Determine the eligibility criteria for studies to be included in the review

The use of explicit, pre-defined criteria for including and excluding studies (eligibility criteria) is an important feature of systematic reviews. The eligibility criteria to be defined for, and applied in, a systematic review will depend on the type of review question (for example, whether the review is about effects of intervention or about diagnostic tests or patient experiences) and the population of interest.

Traditionally, eligibility criteria for systematic reviews of the effects of interventions have focused on defining types of participants (that is, people and populations), types of interventions and comparisons and types of studies (that is, the study designs most suitable for answering the review question). Although systematic reviews of the effects of interventions also typically pre-specify types of outcomes (the outcome measures that are important to answer the review question), these usually are not used to include or exclude studies but rather to guide how the findings from the included studies are reported. Generally, systematic reviews of the effects of interventions primarily focus on randomised controlled trials or quasi-randomised controlled trials. However, there are many reviews which include both randomised and non-randomised studies. While these reviews can provide a comprehensive overall picture of the available research, they require the reader to be particularly careful when interpreting the conclusions and to take into account the study types (and their associated strengths/weaknesses) that may have been used to contribute to the conclusion of the review.

Qualitative evidence syntheses generally include all sorts of qualitative study designs, including, for example, phenomenological studies, case studies, grounded theory designs and action research designs. Some review authors opt for the exclusion of opinion papers and editorials and papers without a clear methods and findings section. To date, there is little guidance on how to search for, critically appraise or extract data from these reports, or how to use them in a cross-comparison between papers.¹⁰

Search for potentially eligible studies

The search methodology needs to be developed prior to commencing the review and to be clearly explained and reproducible. All components of the question of the review (such as the participants, interventions, comparisons and outcomes) should be considered when developing the search strategy of the review. Synonyms and relevant Medical Subject Heading (MeSH) terms for key components of the search (for example, participants and interventions) should also be incorporated.¹⁸ The search should not be limited by publication date, publication format or language unless there is a good reason for doing so (such as an intervention only being available after a certain point in time). Searching should occur across multiple databases, as it has been found that search strategies that are limited to one database do not identify all of the relevant studies.¹⁸ The review authors should also ideally contact authors in the field in an attempt to locate other studies that have not already been identified and ongoing or planned studies in the area, and to ask about and obtain written copies of unpublished studies.¹⁹ The reason for doing this is to limit a problem called ‘publication bias’ that can occur due to study authors being more likely to submit studies with statistically significant results and journals being more likely to publish studies with positive results than those with negative outcomes.

As an illustration of how important it is that review authors use a comprehensive search strategy, one study found that 35% of all appropriate randomised controlled trials were not located by computerised searching²⁰ and another reported that 10% of suitable trials for systematic reviews were missed when only electronic searching was conducted.²¹ Greenhalgh and Peacock²² evaluated the appropriateness of standard searching procedures for the retrieval of qualitative research papers and found that only 30% were retrieved through databases and hand-searching. The rest were identified through a snowball procedure of following up on the references of relevant papers and by personal contacts. Hand-searching journals has been identified as important to conducting a high quality systematic review,²³ as not all journals are indexed in electronic databases or may be missed by the search strategy used. It is suggested²⁴ that authors of systematic reviews may perform hand-searching of the major journals that publish in the area relevant to the review question, to increase the comprehensiveness of their search strategy. Citation tracking or use of the references from the studies found may also help the reviewer to locate further studies on the topic and increase comprehensiveness of the search. Attempts could also be made to obtain unpublished studies (including masters and doctoral research and conference proceedings) by searching the CENTRAL database in the Cochrane Library and databases of grey literature (e.g. OpenSIGLE, Open System for Information on Grey Literature, www.opensigle.inist.fr). The same applies for reviewers who are conducting a qualitative evidence synthesis, although, as far as we are aware, specialised registers for qualitative research currently do not exist.

Apply eligibility criteria to select studies

Once the search has been completed and potentially relevant studies identified, the next task is to decide which of the studies should be included in the systematic review. Not all articles that are located during the search will be directly relevant to the review question. Eligibility criteria are established to guide the selection of studies to be included in the review. The criteria specify the studies, participants, interventions and outcomes that are to be included. The selection process should then be carried out by two or more authors independently to minimise bias. Titles and abstracts are screened for relevance and eligibility. At this point some citations will be excluded as it will be clear from the title and abstract that they do not meet the review eligibility criteria. For other citations, the decision on whether to include or exclude will be unclear due to insufficient information in the title and abstract. The full text of these potentially relevant studies is then retrieved so that a more detailed evaluation can be done. A final evaluation of the full-text articles is conducted and a decision is made whether to include or exclude the studies. If the review authors are conducting a Cochrane systematic review, studies that are excluded at this stage of the review process are listed in the review along with the reason why they were excluded.²⁴ A flow diagram summarising the study selection processes may also be appended to the review.

Assess the risk of bias in included studies

The next step is to evaluate the validity of the studies that have been included in the review. Assessing the validity, or ‘risk of bias’, of the results from the individual studies is vital to conducting a high quality systematic review. Including studies with a high risk of bias can serve to magnify the impact of that bias, and thus may raise doubts about the reliability of the systematic review's results and conclusions.²⁵ Where such studies are included in the review, it is important that the risk of bias is clearly communicated for the reader. To guard against errors and increase the reliability of the validity assessment, it is recommended that more than one person independently extract data and assess the risk of bias of the included studies.²⁶ Determining the potential for bias in individual studies can be done in a number of different ways, such as using the approach to appraising randomised controlled trials that was explained in Chapter 4 of this book. From the beginning of 2008, systematic reviews that are carried out through The Cochrane Collaboration have to use the ‘risk of bias’ tool.²⁶ The risk of bias tool includes seven different domains which are shown in Table 12.1, along with a definition of each of these criteria. Implementation of this tool requires two steps:

For the critical appraisal of qualitative studies, a number of different checklists and frameworks have been developed. Among them is the commonly used CASP checklist and QARI tool that were discussed and used in Chapters 10 and 11. The meta-aggregative approach encourages qualitative evidence synthesis review authors to assign an overall judgment to findings in an original article.¹³ This allows the reviewer to decide which insights are valid enough to be considered for inclusion. These judgments are:

Extract data from the included studies

Generally, the data that are collected from quantitative research papers concern participant details, intervention details, outcome measures used, results of the study and the study methodology.²⁴ For qualitative research studies, the data to be extracted include setting, participant details, methodology and methods used, phenomenon of interest, cultural and geographical information, and findings of the study (such as the themes, metaphors or categories reported). Standardised data collection tools or processes are typically established regardless of the type of review so that the data are collected in a consistent manner and are relevant to the research question.

Synthesise the data

Before we explain what review authors do at this stage of the process, you first need to understand the principles of meta-analysis and meta-aggregation in more detail. We will deal with both issues, starting with the principles for conducting a meta-analysis. As we mentioned at the beginning of this chapter, a meta-analysis pools the data obtained from all studies answering the same clinical question included in a systematic review and produces an overall summary effect. The rationale behind a meta-analysis is that, as you are combining the samples of individual studies, the overall sample size is increased, which improves the power of the analysis as well as the precision of the estimates of the effects of the intervention.¹⁸

When review authors plan a systematic review, the process by which they will conduct the meta-analysis should be clearly outlined. It has been suggested that the process should include the following steps:²⁷

There are two stages to doing a meta-analysis. First, the intervention effect, including the confidence intervals, for each of the studies is calculated. The statistics used will depend on the type of outcome data found in the studies, with commonly used statistics including odds ratios, risk ratios (also known as relative risks) and mean differences.¹⁸ As we saw in Chapter 4 when learning how to interpret the results of randomised controlled trials, if the outcome is dichotomous then odds ratios or risk ratios may be used. For continuous outcomes, mean differences may be used.

Table 12.2 lists some of the statistical measures of intervention effect that can be used in a meta-analysis based on the outcome data type as outlined in the Cochrane Handbook for Systematic Reviews of Interventions.²⁷

The second stage in a meta-analysis is calculating the overall intervention effect. This is generally an average of the summary statistics from the individual studies, each weighted to adjust for its sample size and variance. This is described in more detail later in this chapter.

The procedure of meta-aggregation used in qualitative evidence synthesis involves three steps: (1) assembling the findings of studies (variously reported as themes, metaphors, categories); (2) pooling them through further aggregation based on similarity in meaning, to (3) arrive at a set of synthesised statements presented as ‘lines of action’ for practice and policy.¹² In extracting the themes (step 1) identified by the authors of original studies, the reviewer takes the literal descriptions presented in the results sections of original articles into account and maintains representativeness with the primary literature. Similarity of meaning on the level of categories (step 2) is contingent on the reviewer's knowledge and understanding of the included studies. Having read and re-read the articles and extracted the findings from each included study, the reviewer looks for commonality in the themes and metaphors across all studies. Similarity may be conceptual (where a particular theme, metaphor or part thereof is identified across multiple papers) or descriptive (where the terminology associated with a theme or metaphor is consistent across studies). The actual move from findings to categories is similar to the procedures used in primary qualitative research methods (such as constant comparative analysis and thematic analysis). However, in a primary qualitative research project, the end result is a particular framework, matrix or conceptual model. In a meta-aggregation, it is declamatory statements or ‘lines of action’ (step 3) that are arrived at.¹² An adapted version of the process of meta-aggregation²⁸ is illustrated in Figure 12.2.

Interpret and report the results

The results of a systematic review are presented in the text, while the results of a meta-analysis are also displayed visually using a forest plot. Forest plots are made up of tree plots (explained in Chapter 4 and in Figure 4.3) represented in one figure. They display both the information from the individual studies included in the review (such as the intervention effect and the associated confidence interval) and an estimate of the overall effect (see Figure 12.4 later in the chapter). In some systematic reviews it is not proper to statistically combine data (because of heterogeneity, meaning the studies are not similar enough in one or more respects); forest plots may still be presented but without an overall estimate (summary statistic). Forest plots are useful, as they allow readers to visually assess the amount of variation among the studies that are included in the review.¹⁸

The importance of reporting in systematic reviews

Determining the methodological quality or risk of bias of a systematic review depends on how well the details of how the systematic review was conducted are reported. Based on studies which found that systematic reviews were generally poorly reported,⁴⁵ an international group developed a reporting guideline for reporting details of meta-analyses of randomised controlled trials called the QUOROM Statement (QUality Of Reporting Of Meta-analyses).⁴⁶ In 2009, the guideline was updated to encompass all types of systematic reviews about the effects of interventions, not just those using meta-analyses of randomised controlled trials. This reporting statement is called the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, or the PRISMA statement.⁴⁷ Reporting statements improve the quality of reporting of studies⁴⁸ and therefore it is now recommended, or often insisted on, by an increasing number of journals that review authors use the PRISMA statement when writing their reviews.⁴⁹

1. Mulrow, C, Cook, D, Davidoff, F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med. 1997; 126:389–391.

2. Cook, D, Mulrow, C, Haynes, RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997; 126:376–380.

3. Hannes, K, Van den Noortgate, W. Master thesis projects in the field of education: who says we need more basic research? Educational Research. 2012; 3:340–344.

4. Greenhalgh, T. How to read a paper: papers that summarise other papers (systematic reviews and meta-analyses). BMJ. 1997; 315:672–675.

5. Phillips, B, Ball, C, Sackett, D, et al, Levels of evidence. Centre for Evidence-based Medicine, Oxford, 2009. www.cebm.net/index.aspx?o=1025

6. National Health and Medical Research Council (NHMRC), NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. NHMRC, Canberra, 2009. www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf

7. Meijer, R, Ihnenfeldt, D, van Limbeek, J, et al. Prognostic factors in the subacute phase after stroke for the future residence after six months to one year. A systematic review of the literature. Clin Rehab. 2003; 17:512–520.

8. Irwig, L, Tosteson, A, Gatsonis, C, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994; 120:667–676.

9. Mitchell, A. A meta-analysis of the accuracy of the mini-mental state examination in the detection of dementia and mild cognitive impairment. J Psychiatr Res. 2009; 43:411–431.

10. Noyes, J, Popay, J, Pearson, A, et al, Chapter 20: Qualitative research and Cochrane reviewsHiggins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration: 201. www.cochrane-handbook.org

11. Goldsmith, M, Bankhead, C, Austoker, J. Synthesising quantitative and qualitative research in evidence-based patient information. J Epidemiol Community Health. 2007; 61:262–270.

12. Hannes, K, Lockwood, C. Pragmatism as the philosophical underpinning of the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis. J Adv Nurs. 2011; 67:1632–1642.

13. Pearson, A. Balancing the evidence: incorporating the synthesis of qualitative data into systematic reviews. JBI Reports. 2004; 2:45–64.

14. Hannes, K, Aertgeerts, B, Goedhuys, J. Obstacles to implementing evidence-based practice in Belgium: a context-specific qualitative evidence synthesis including findings from different health care disciplines. Acta Clinica Belgica. 2012; 67:99–107.

15. McInnes, E, Askie, L. Evidence review on older people's views and experiences of falls prevention strategies. World Views Evid Based Nurs. 2004; 1:20–37.

16. Allen, IJ, Okin, I. Estimating the time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999; 282:634–635.

17. Jackson, N, Waters, E. Criteria for the systematic review of health promotion and public health interventions. Health Promot Int. 2005; 20:367–374.

18. Akobeng, AK. Understanding systematic reviews and meta-analysis. Arch Dis Child. 2005; 90:845–848.

19. Bhandari, M, Guyatt, G, Montori, V, et al. Current concepts review. Users guide to orthopedic literature: how to use a systematic review. J Bone Joint Surg. 2002; 84A:1672–1682.

20. Hopewell, S, Clarke, M, Lefebvre, C, et al. Handsearching versus electronic searching to identify reports of randomised trials. Cochrane Database Syst Rev. (2):2007.

21. Kennedy, G, Rutherford, G. Identifying randomised controlled trials in the journal. Cape Town, South Africa: Presentation at the Eighth International Cochrane Colloquium; 2000.

22. Greenhalgh, T, Peacock, R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005; 331:1064–1065.

23. Armstrong, R, Jackson, N, Doyle, J, et al. It's in your hands: the value of handsearching in conducting systematic reviews. J Pub Health. 2005; 27:388–391.

24. Lefebvre, C, Manheimer, E, Glanville, J, Chapter 6: Searching for studiesHiggins J, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration: 201. http://handbook.cochrane.org/

25. Juni, P, Altman, D, Egger, M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001; 323:42–46.

26. Higgins, JPT, Altman, DG, Sterne, JAC, Chapter 8: Assessing risk of bias in included studiesHiggins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration: 201. http://handbook.cochrane.org/

27. Deeks, JJ, Higgins, JPT, Altman, DG, Chapter 9: Analysing data and undertaking meta-analysesHiggins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration: 201. www.cochrane-handbook.org

28. The Joanna Briggs Institute. Joanna Briggs Institute Reviewers’ Manual. Adelaide: Joanna Briggs Institute; 2008.

29. Schünemann, H, Oxman, A, Higgins, J, et al, Chapter 11: Presenting results and ‘Summary of findings’ tablesHiggins J, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration: 201. www.cochrane-handbook.org

30. Henken, H, Huibers, M, Churchill, R, et al. Family therapy for depression. Cochrane Database Syst Rev. (3):2007.

31. Joanna Briggs Institute, SUMARI. User's Manual Version 5.0. System for the unified management, assessment and review of information. Joanna Briggs Institute, Adelaide, 2012. www.joannabriggs.edu.au/documents/sumari/SUMARI%20V5%20User%20guide.pdf

32. The Joanna Briggs Institute. Comprehensive Systematic Review Study Guide Module 4. Adelaide: The Joanna Briggs Institute; 2012.

33. McInerney, P, Brysiewicz, P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa. JBI Library of Systematic Reviews. 2009; 7:130–153.

34. Konno, R. Support for overseas qualified nurses in adjusting to Australian nursing practice: a systematic review. Int J Evid Based Healthc. 2006; 4:83–100.

35. McInnes, E, Wimpenny, P. Using qualitative assessment and review instrument software to synthesise studies on older people's views and experiences of falls prevention. Int J Evid Based Healthc. 2008; 6:337–344.

36. Moher, D, Tetzlaff, J, Tricco, A, et al. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007; 4:e78.

37. Oxman, A, Guyatt, G. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991; 44:1271–1278.

38. Oxman, A, Cook, D, Guyatt, G. Users’ guides to the medical literature: VI. How to use an overview. JAMA. 1994; 272:1367–1371.

39. Shea, B, Hamel, C, Wells, G, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009; 62:1013–1020.

40. Greenhalgh, T, Donald, A. Evidence-based health care workbook: understanding research: for individual and group learning. London: BMJ Publishing Group; 2000.

41. Pope, C, Mays, N, Popay, J. Synthesising qualitative and quantitative health research: a guide to methods. Maidenhead (UK): Open University Press; 2007.

42. Stroke Unit Trialists’ Collaboration. Organised inpatient (stroke unit) care for stroke. Cochrane Database Syst Rev. (4):2007.

43. Lamb, M, Buchanan, D, Godfrey, C, et al. The psychosocial spiritual experience of elderly individuals recovering from stroke: a systematic review. Int J Evid Based Healthc. 2008; 6:173–205.

44. Hannes, K, Lockwood, C, Pearson, A. A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qual Health Res. 2010; 20:1736–1743.

45. Sacks, HS, Reitman, D, Pagano, D, et al. Meta-analysis: an update. Mt Sinai J Med. 1996; 63:216–224.

46. Moher, D, Cook, D, Eastwood, S, et al. Improving the quality of reporting of meta-analysis of randomised controlled trials: the QUOROM statement. Lancet. 1999; 354:1896–1900.

47. Liberati, A, Altman, D, Tetzlaff, J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009; 6:e1000100.

48. Simera, I, Moher, D, Hirst, A, et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 2010; 26(8):24.

49. Tao, K, Li, X, Zhou, Q, et al. From QUOROM to PRISMA: a survey of high-impact medical journals’ instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011; 6:e27611.

50. Kalra, L, Eade, J. Role of stroke rehabilitation units in managing severe disability after stroke. Stroke. 1995; 26:2031–2034.

51. Vemmos, K, Takis, K, Madelos, D, et al. Stroke unit treatment versus general medical wards: long term survival. Cerebrovasc Dis. 2001; 11:8.

52. Leonardi-Bee, J, Presenting and interpreting meta-analyses. RLO, School of Nursing and Academic Division of Midwifery, University of Nottingham, 2007. www.nottingham.ac.uk/nursing/sonet/rlos/ebp/meta-analysis2/index.html

53. Hannes, K. The Joanna Briggs Institute Best Practice Information Sheet: the psychosocial and spiritual experiences of elderly individuals recovering from a stroke. Nurs Health Sci. 2010; 12:515–518.