45: Consensus group methods—the Delphi technique
Liam Langford
Learning outcomes
Definition
The Delphi technique is a coordinated process for collating and synthesising knowledge from expert panellists through rounds of anonymised opinions and questionnaires.1
Introduction
As the name alludes to, consensus group methods are as a systematic means for measuring and developing consensus. They are mostly used in conditions where there is uncertainty, conflict or incomplete evidence. Usually, groups comprise of experts relevant to the study aim. Of the consensus group methods, the Delphi technique is most widely used.2
Originally developed in the 1950s by the RAND Corporation to predict the impact of technology on warfare, the Delphi technique has been extensively used across disciplines to assist with creating consensus, group judgments and informing decisions.3 Rounds, consisting of a qualitative, quantitative or combination of questionnaires, are conducted until a consensus is achieved among the expert panellists. During the process panellists are unaware of one another and never interact. At this stage, there is no exact gold standard reporting guideline for Delphi studies and, over time, variations to the process have evolved to meet research needs.
Strengths of the Delphi technique
The Delphi technique has a number of strengths that can assist researchers in designing, collaborating and implementing the method relevant to their research question.4,5 The Delphi technique’s design can be simple and flexible. Depending on the question and purpose it can be qualitative, quantitative or mixed methods. Additionally, the Delphi technique enables access to experts and their insights, while potentially causing them to evaluate their opinions and learn/develop new knowledge. Given their involvement, experts are more likely to accept results and aid in dissemination. When conducting a Delphi technique, the response rates are generally high, with the anonymity and democratic aspects mitigating individuals influencing group dynamics and biasing outcomes. Furthermore, the Delphi technique is a relatively cost-effective method, especially when conducted electronically.
Weaknesses of the Delphi technique
There are, however, some drawbacks and critiques of the Delphi Technique.4,5 For a consensus method, there appears to be little consensus on terminology, design and parameters. Often researchers will conduct a ‘modified’ Delphi technique, despite high ambiguity of its nature and term. Furthermore, there is significant variation and debate in prescribed panel size, consensus levels, rounds and overall approach. Therefore, researchers must be robust in their justification for decisions and variations. From some scholars’ aspects, the Delphi technique is not considered a valid scientific method nor is its theoretical basis well established. It is also asserted that the approach to communication is restrictive and does not facilitate debate and problem identification. Outcomes may also be a result of collective ignorance as opposed to collective wisdom.
When conducting the method, it has been noted that inappropriate expert selection can occur, leading to muddled results. Furthermore, timelines can become protracted if multiple rounds are required due to disagreements. This protraction also increases the risk of participant attrition.
The Delphi technique process
Justification
It is imperative that the aim of the research aligns with the Delphi techniques strengths (refer to above). Additionally, the Delphi technique method can be used when there is limited or conflicting information or for developing new concepts, decision-making and/or agenda setting.6
Establish expert panellists criteria and sampling
The Delphi technique is not concerned with a generalised sample population, but a purposive sample of individuals with expertise on a given topic.7 However, there is no consensus on what constitutes an expert.5 The inclusion criteria of what defines an expert for the study is usually decided by the steering research group.8 Once established, sampling can occur via direct communication or a snowball technique. To ensure rigor and trustworthiness, the criteria and sampling strategy have to be reported.
Determine size of panel
There is no set size required for a panel; however, six experts have been considered the minimum requirement. It is common to have between 10 and 20 panellists. While there may be a sense that increasing the size of the panel thereby increases the validity, it can make the process unstable and time consuming. Consideration and justification of panel size needs to support the aim of the research and be appropriately reported.6
Confirm consensus
There is no set definition or criteria for consensus in the Delphi technique.9 This is determined by the authors or the steering research group before the study begins. Most studies have used a percentage of agreement (mainly 75% or 80%), median score or a combination of both.6 Consensus, however defined by researchers, should be justified and reported.
Questionnaire design
Questionnaire design depends on the aim of the research and preexisting information. It can use both quantitative and qualitative methods. As an example, researchers may develop a Likert-scale questionnaire informed by systematic or literature reviews, a synthesis of already existing guidelines or from a conceptual framework.6 Seven-point Likert scales are considered accurate in measuring a participant’s true evaluation.10
The first round can be unstructured or semi-structured with open questions. This promotes participants’ relative freedom and scope to elaborate on the subject being investigated. Alternatively, a quantitative questionnaire can be derived from a literature review.1
Analysis
A qualitative analysis (e.g. thematic analysis) of the results is then undertaken. This creates a foundation to construct the second and subsequent questionnaires.1 Any qualitative analysis method should aim to limit bias and should be included in the final report. If questionnaires are designed with the Likert scale or other quantitative methods, appropriate analysis must be used.
Number of rounds
As with other aspects of the Delphi technique, there are no hard guidelines for the number of rounds.8 It depends on when consensus is met. Most studies have two to three rounds until consensus is achieved; however, others have been known to achieve consensus with five or more rounds. Increasing the number of rounds can have consequences. Consensus may have been as a result of survey fatigue. That is, participants may not hold a consensus view, but to finish lengthy and numerous rounds participants may concede. Other issues arising from numerous rounds can include higher attrition rates and an increase in the length of the study period.2
Given these issues, some scholars state that the Delphi technique should be limited to no more than three rounds, and with the areas where no consensus was apparent, should also be reported. Either approach (continuous round until consensus or predetermined number) should be justified and reported.
Length of rounds
The duration of a round once again depends on the context of the study. Usually, a round lasts 1 to 4 weeks, although there is a version of the Delphi technique that conducts a round in real-time.
Delphi technique process flowchart
An overview of the Delphi technique’s process is outlined below in Figure 45.1.
Example
Jamie is a paramedic who has been tasked with developing a new paramedic educator role for their organisation. As this is a new role, there are no established capability frameworks that outline the skills, knowledge and attributes that the educators require. To guide the selection process of applicants, budget for their training needs and performance management system, Jamie will need to create a Paramedic Educator Capability Framework. Jamie can use the Delphi technique to facilitate structured dialogue, reduce bias and establish a consensus on the required capabilities for the paramedic educator role.
Jamie begins by establishing the research question and aim: ‘What are the capabilities required for the role of a paramedic educator within the context of the organisation?’
As there are differing opinions and sporadic literature about what capabilities are required for the role and noting that accommodating travel and schedules of experts for face-to-face dialogue and meetings is unfeasible, Jamie can justify using the Delphi technique.
Jamie establishes the criteria to be an expert panellist to be paramedic manager or senior academic with more than 5 years’ experience in health professional education. The panel size they are aiming for is a minimum six members and no more than 12. Sampling of panellists occurs from different networks and by reviewing authors from relevant research articles. A mix of internal, national and international experts are identified.
Based on other studies’ consensus definition and criteria, Jamie decided that 80% of panellists will need to agree on each capability for it to be considered consensus.
As there is no previous research on the given area, Jamie will use an open-ended questionnaire design in the first round to elicit qualitative feedback. After analysis of the qualitative responses, they will design a Likert scale questionnaire with some limited free-text questions for subsequent rounds. Before dissemination, the questionnaires were tested with non-panellist colleagues. Table 45.1 outlines the how Jamie conducted their Delphi technique based on the established parameters and subsequent participants’ input.
| Round 1 | In total, 11 invitations, details, deadlines and questionnaires have been sent to the identified experts. They are given 1 week to respond. Ten panellists complete and return the questionnaire. |
| Analysis | Jamie uses thematic analysis to identify possible capabilities and create the questionnaire for round 2 from the themes. |
| Round 2 | The new questionnaire was distributed to the panellists and they were all completed and within 1 week. |
| Analysis | All capabilities except for one reached 80% agreement. This was sufficient for consensus and completion of the Delphi technique process. During reporting, Jamie will report on the capabilities that did not reach consensus. |
Conclusion
Consensus methods such as the Delphi technique are useful research tools in paramedicine. They are flexible and require little resources. Additionally, they can help identify problems, prioritise actions, build a shared understanding and establish agreements. However, they do have their limitations. Thus their use must be justified appropriately and meet the aim of the study. Furthermore, modifications must be reported on.
Review questions
Suggested further reading
Jünger S, Payne SA, Brine J, Radbruch L & Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review Palliative Medicine. 8, 2017;31: 684-706.
Waggoner J, Carline JD & Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research Academic Medicine 5, 2016;91.
References
1. Powell C. The Delphi technique: myths and realities Journal of Advanced Nursing. 4, 2003;41: 376-382.
2. Humphrey-Murto S, Varpio L, Gonsalves C & Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research Medical Teacher. 1, 2017;39: 14-19 Online. Available: https://doi.org/10.1080/0142159X.2017.1245856.
3. The RAND Corporation. Delphi Method 2016; RAND Online. Available: https://www.rand.org/topics/delphi-method.html.
4. Hasson F & Keeney S. Enhancing rigour in the Delphi technique research Technological Forecasting and Social Change. 9, 2011;78: 1695-1704 Online. Available: https://doi.org/10.1016/j.techfore.2011.04.005.
5. Vernon W. The Delphi technique: a review International Journal of Therapy and Rehabilitation. 2, 2009;16: 69-76.
6. Jünger S, Payne SA, Brine J, Radbruch L & Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review Palliative Medicine. 8, 2017;31: 684-706 Online. Available: https://doi.org/10.1177/0269216317690685.
7. Keeney S, Hasson F & McKenna HP. A critical review of the Delphi technique as a research methodology for nursing International Journal of Nursing Studies. 2, 2001;38: 195-200 Online. Available: https://doi.org/10.1016/S0020-7489(00)00044-4.
8. Foth T, Efstathiou N, Vanderspank-Wright B, Ufholz LA, Dütthorn N, Zimansky M & Humphrey-Murto S. The use of Delphi and Nominal Group Technique in nursing education: A review International Journal of Nursing Studies. 2016;60: 112-120 Online. Available: https://doi.org/10.1016/j.ijnurstu.2016.04.015.
9. Waggoner J, Carline JD & Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research Academic Medicine. 5, 2016;91: Online. Available: https://journals.lww.com/academicmedicine/Fulltext/2016/05000/Is_There_a_Consensus_on_Consensus_Methodology_.22.aspx.
10. Finstad K. Response interpolation and scale sensitivity: Evidence against 5-point scales Journal of Usability Studies. 3, 2010;5: 104-110.