Overview

Evaluation is an integral aspect of all planned activities, enabling an assessment of the value or worth of an intervention. Evaluation also performs several other roles. For practitioners, evaluation helps develop their skills and competences. For funders evaluation demonstrates where resources can be most usefully channelled. For lay people, evaluation provides an opportunity to have their voice heard. There are additional reasons why evaluating health promotion is a key aspect of practice. As a relatively new discipline there is great pressure on health promotion to prove its worth through evaluation of its activities. In addition, the drive in the National Health Service (NHS) to ensure that all practice is evidence-based affects health promotion as well as more clinical activities. In a situation where resources will always be limited, demonstrating the cost-effectiveness of interventions is important. There are thus many factors leading to a demand for evaluation of health promotion practice.

Evaluating health promotion is not a straightforward task. Health promotion interventions often involve different kinds of activities, a long timescale, and several partners who may each have their own objectives. Health promotion is still seen as belonging within the health services, where the dominant evaluation model is quantitative research centred on experimental trials, with randomized controlled trials (RCTs) as the preferred evaluation tool. Health promotion has had to argue its case for a more holistic evaluation strategy encompassing qualitative methodologies and taking into account contextual features.

The focus of this chapter is on evaluating health promotion interventions. Evaluation of research studies is also part of the health promoter’s role and remit, and readers are referred to Chapters 2 and 3 in our companion volume (Naidoo & Wills 2005) for a detailed discussion of this topic. This chapter considers what is meant by evaluation, the range of research methodologies used in evaluation studies, its rationale, how it is done and the role of evaluation in building the evidence base for health promotion.

Defining evaluation

Evaluation is a complex concept with many definitions that vary according to purpose, disciplinary boundaries and values. A comprehensive definition of evaluation is: ‘the systematic examination and assessment of features of a programme or other intervention in order to produce knowledge that different stakeholders can use for a variety of purposes’ (Rootman et al 2001, p. 26). The above definition is useful because it also flags up the importance of the purpose of evaluation, and the fact that there can be many different reasons to evaluate. Evaluation can provide information on the extent to which an intervention met its aims and goals, the manner in which the intervention was carried out, and the cost-effectiveness of the intervention. It is important to be clear at the outset about the purpose of evaluation as this will determine what information is gathered and how information is obtained. The value-driven purpose of evaluation distinguishes it from research (Springett 2001). Evaluation uses resources which might otherwise be used for programme planning and implementation, so a clear purpose is also necessary in order to legitimate and protect this use of resources.

From a practitioner’s perspective, evaluation is needed to assess results, determine whether objectives have been met and find out if the methods used were appropriate and efficient. These findings can then be fed back into the planning process in order to progress practice. Evaluations of interventions are used to build an evidence base of what works, enabling other practitioners to focus their inputs where they will have most effect. From a lay perspective, evaluation helps to clarify expectations and assess the extent to which these have been met. Evaluation may also help determine what strategies had most impact, and why. Without evaluation, it is very difficult to make a reasoned case for more resources or expansion of an intervention. Even when a programme is rolling out an established and effective intervention, specific local features may have an unanticipated impact that will only become apparent in an evaluation. There are sound reasons for evaluating all interventions, although more innovative projects will require more substantial and costly evaluation.

Evaluation covers many different activities undertaken with varying degrees of rigour or reflectiveness. At its simplest level, evaluation describes what any competent practitioner does as a matter of course, that is, the process of appraising and assessing work activities. This includes the process of informal feedback or more systematic review of health promotion interventions. In the example above, noting how the sessions have been received by the men, or soliciting their comments, or those of peers and colleagues, is part of the evaluation process. Evaluation is often used to refer to a more formal or systematic activity, where assessment is linked to original intentions and is fed back into the planning process. For the well-man clinic example, this might involve monitoring vital statistics and doing a before-and-after study of lifestyle behaviours. Health promotion evaluation should integrate core health promotion principles such as equity and participation into the evaluation process. In the well-man clinic example this might be achieved through asking the men what they wanted from participating in the programme and whether they achieved their goals. Comparing the socioeconomic status of participants and non-participants would help determine if the programme was reinforcing or challenging social and health inequalities.

Evaluation research methodologies

Evaluation is often more formally conducted as research using a variety of different methods. The classic scientific method of proof, the experiment, relies on controlling all factors apart from the one being studied and can best be achieved under laboratory conditions. However, this is clearly impossible and unethical to achieve where people’s health is concerned. The RCT is the next most rigorous scientific method of proof. The RCT involves randomly allocating people to an intervention or control group. Random allocation means that the two groups should be matched in terms of factors such as age, gender and social class which are all known to affect health. Any changes detected in the intervention group are then compared to those found amongst the control group. Those changes which occur in the intervention but not the control group can then be attributed to the health promotion programme.

In the well-man clinic example in Activity 20.3 an RCT study would involve randomly allocating all men in the target group to either the intervention group (invited for screening) or the control group (not invited). The two groups would then be compared after the intervention had taken place. If the intervention group showed statistically significant improvements in health status or health-related behaviour over and above those recorded for the control group the intervention would be deemed to be effective.

The degree of scientific rigour necessary to conduct an RCT is hard to achieve in real-life situations. Most health promotion programmes have spin-off effects and indeed are designed to do so. It is impossible to isolate different groups of people or to ensure that programmes do not ‘leak’ beyond their set boundaries. However, the RCT design does mean that changes detected in the input group may be ascribed to the health promotion programme with a greater degree of confidence.

Evaluation research may also use qualitative methods to focus on understanding the processes involved in change. This kind of evaluation provides details on what is happening in interventions and which features have been effective. This is achieved through the use of qualitative methodologies and methods, and the case study is one example of this approach. The health promotion intervention is the ‘case’ that is intensively studied using a variety of methods. This enables the evaluator to get a detailed picture of how the intervention has affected the people involved. Case studies are typically small-scale and findings are expressed in descriptive rather than numerical terms. Each case study is unique and findings cannot be generalized to other situations. Its strength as a method is that there is a high degree of confidence that identified effects are real and result from the programme.

In the well-man clinic example in Activity 20.3, a qualitative case-study approach might involve indepth interviews with a sample of men who took up the screening opportunity and the practice nurse. The interviews would aim to explore what motivated the men to accept the invitation to attend the clinic, how they found the experience and how (if at all) it has affected them.

Both the RCT and the case study are valid methods which can be used to isolate the effects of health promotion interventions. There are also many other methods that lie between these two extremes, e.g. surveys which aim to identify significant trends. In practice methods often overlap or are combined. The RCT fits into a scientific, quantitative medical model of proof, has higher status and is generally regarded as more respectable and credible than the case study.

Why evaluate?

Evaluation uses resources that could otherwise be used to provide services. Given that services are always in demand, there needs to be a strong rationale for devoting resources to evaluation rather than service provision. New or pilot interventions warrant a rigorous evaluation because, without evidence of their effectiveness or efficiency, it is difficult to argue that they should become established work practices. Other criteria that can be used to determine if evaluation is worth the effort relate to how well it can be done. If it will be impossible to obtain cooperation from the different groups involved in the activity, it is probably not worthwhile trying to evaluate. If evaluation has not been considered at the outset but is tacked on as an afterthought, the chances are that it will be so partial and biased as to be not worth the effort.

Evaluation is only worthwhile if it will make a difference. This means that the results of the evaluation need to be interpreted and fed back to the relevant audiences in an accessible form. All too often, evaluations are buried in inappropriate formats. Work reports may go no further than the manager, or academic studies full of jargon may be published in little-known journals.

Results of evaluation studies will be relevant to many different groups and it may be necessary to reproduce findings in different ways in order to reach all these groups.

What to evaluate?

Health promotion objectives may be about individual changes, service use or changes in the environment. Example 20.7 shows the range of possible objectives associated with smoking reduction interventions, each of which would need evaluation.

Although all these factors relate to health, they are quite separate, and there is no necessary connection between, say, increased knowledge and behaviour change. It is therefore inappropriate to evaluate a given objective (e.g. increased physical activity) by measuring other aspects of an intervention (e.g. number of leaflets taken at a health fair or number of people reporting that they would like to exercise more). It is important to choose appropriate indicators for the stated objectives. This issue is discussed further in Chapter 19, where the log-frame model and the use of logic to select appropriate indicators are considered.

How to evaluate: the process of evaluation

In order to evaluate, decisions need to be taken about what information is needed and how it will be gathered. This needs to be done at the outset of an intervention, in order to ensure that relevant data are gathered at the appropriate time. Rootman et al (2001) propose an eight-stage framework for the evaluation of health promotion interventions:

Many commentators have argued that the evaluation process should adhere to health-promoting principles (Thorogood & Coombes 2004; Morgan 2006). Evaluation should involve the participation of all stakeholders and be an empowering experience. The evaluation of community health promotion interventions is particularly challenging as these are complex, context-specific programmes focusing on socioeconomic and environmental determinants of well-being.

Evaluation therefore involves several key aspects that need to be considered. These key aspects may be summarized as: what to measure? who evaluates? how to evaluate, including how to gather and analyse data? and what to do with the results? or putting the findings into practice. Each of these key aspects will now be considered.

What to measure?

Deciding what to measure to assess the effects of health promotion is not easy. In theoretical terms, the many meanings and definitions of the concept of health result in a lack of consensus about how best to evaluate it. For those who subscribe to the medical model of health, data concerning morbidity, disability and mortality are appropriate measures to use for evaluation purposes. For those who adopt a more social model of health, a much broader range of measures (including, for example, measures of socioeconomic status or the quality of the environment) will be appropriate. For people who prioritize the educational model, measures of knowledge and attitude change will be paramount.

The golden rule must be to measure the objectives set during the planning process. (For more details on programme objectives, see Chapter 19.) Although this sounds straightforward, in practice it can be difficult, and a surprising number of evaluation studies violate this principle. Different stakeholders might have different objectives and the evaluation needs to take this into account. The objectives set may concern areas where there is a lack of consensus over appropriate measurement. For example, process objectives such as increased multiagency collaboration or increased community involvement are difficult to measure. To collect relevant data would require a special effort because they are not measured routinely. Change in people’s attitudes or beliefs is particularly problematic to measure.

The success of a health promotion intervention is not solely about achieving behavioural changes or reductions in disease rates. For example, a needle exchange scheme should not be judged solely by a reduction in the rate of human immunodeficiency virus (HIV) infection among drug users. Other markers of success, such as the take-up rate, are also important. In many cases, expecting a clear change in morbidity from a behaviour change would be unrealistic. Although there is a link between needle-sharing and HIV infection, there are other risk factors, and expecting a preventive outcome from this initiative might be unwise.

A programme may have several different objectives, some of which are easier to measure than others. It then becomes tempting to measure the easiest objectives and extrapolate from these findings. But if the measurements are of different classes of events (e.g. combining behavioural, environmental and attitudinal objectives), it is not legitimate to do this.

Who evaluates?

Success means different things to different groups of people, or stakeholders, who each have their own agendas and interests. Different stakeholders have unequal power to impose their evaluation agendas on others. Different groups of people engaged in health promotion interventions will each have invested something but may well be looking for different results. For example, funders of a project may be looking for efficiency or results which can be interpreted as cost-effective. Practitioners may be looking for evidence that their way of working is acceptable to clients and achieves the objectives set. Managers may be looking for evidence of increased productivity, measured by performance indicators. Clients may be looking for opportunities to take control over some health-related aspects of their lives.

It is therefore important to be clear at the outset about whose perspectives are being addressed in any evaluation. A starting point is simply to acknowledge that different vested interests are involved and try to identify them. The ideal is then to go on to represent the views of the different stakeholders by collecting data from each group. This process is called pluralistic evaluation (Smith & Cantley 1985). Using the process of methodological triangulation, which employs a wide range of data sources, an overall picture may be built up. Pluralistic evaluation which takes into account different stakeholders’ views is more complete, although the findings may be complex and lack clarity. Pluralistic evaluation is a means of building capacity and empowering clients and service users as well as practitioners.

In practice, pluralistic evaluation may appear too complex and costly, and evaluation is often carried out by external researchers or by practitioners. The former tend to be larger-scale and more ambitious in their remit. There are advantages and drawbacks to each of these options, as Box 20.12 demonstrates.

How to evaluate: gathering and analysing data

The process of evaluation involves making decisions about what methods to use to gather and analyse relevant data. Each of these stages is next discussed separately.

What to do with the evaluation: putting the findings into practice

Dissemination of findings is important in order to publicise good practice and also to flag up interventions that were not as successful as had been anticipated. Knowing what doesn’t work is as valuable as knowing what does work, but there is a great emphasis on producing and publicising positive results. As Hawe et al (1994) state:

Putting findings into practice can take many forms. The results of evaluation should ideally feed into an ongoing cycle of action and reflection, allowing more knowledgeable and reasoned interventions to take place. Evaluation may also enable stakeholders to progress activities and gain more support to do so. Evaluation helps to establish the cost-effectiveness of health promotion and contributes towards its evidence base.

Cost-effectiveness

Part of the reason for evaluation is to determine whether desired results were achieved in the most economical way and whether allocating resources to health promotion can be justified. There are many different ways of calculating the economic pluses and minuses of health promotion. Cost–benefit analysis is a way of calculating whether, and to what extent, something is worth doing. Cost–benefit analysis relies on pricing both the inputs and the benefits of a health promotion programme. An attempt is then made to calculate the cost of each benefit. This is known as a cost–benefit ratio. Putting a price on health outcomes or benefits is a very difficult exercise. One approach to this problem is to compare the cost–benefit ratio for a health promotion intervention with the cost–benefit ratio for some other health intervention. It is often assumed that prevention is cheaper than cure and that health promotion saves money, but this is not necessarily the case.

Once a decision has been made to implement an intervention, economic analysis can help to determine the most efficient way of resourcing it. Efficiency refers to the maximum benefit that can be derived from the least cost. Cost-effectiveness is a comparison in monetary terms of different methods used to achieve the same outcomes. ‘Cost-effectiveness analysis addresses technical efficiency in the sense that it can tell us the best way to do something but not whether or not that something is worth doing’ (author’s italics; Cohen 2008, p. 337). Opportunity costs refer to what is sacrificed or forgone when resources are allocated to something, e.g. a health promotion project.

Economic appraisal is an important element in evaluation because there are always competing claims for limited resources. Using economics to make health-related decisions might seem a distasteful idea and people may shy away from attempts to put a value on people’s health, well-being or life. But the reality is that people, societies and governments are constantly making choices and decisions that are influenced by economic considerations. It is therefore important to make the decision-making process transparent and include economic principles and concepts in evaluation studies.

Using evaluation to build an evidence base for health promotion

Evaluation helps build a basis of research to demonstrate which health promotion interventions succeed in meeting objectives. Evaluation therefore identifies effective health promotion practice which others can adopt. Evidence-based practice is firmly established in medicine and nursing, where RCTs of alternative treatment protocols are used to establish which form of treatment is most effective for most people. In health promotion, creating evidence-based practice is more problematic.

There are several reasons why proving an evidence base exists for health promotion is problematic. These include knowing when to evaluate, knowing what constitutes success, being able to attribute results to interventions and the inappropriateness of using RCTs.

Knowing when to evaluate is a challenge, and the timing of an evaluation can affect its results. If an evaluation is seeking to determine the outcomes of an intervention, a longer timescale is desirable. However this has problems. Health promotion is a long-term process and contexts and settings are constantly changing, so it can be difficult to be sure that any changes detected are due to the health promotion input, and not to any other factor. Health-related knowledge, attitudes and behaviour are constantly changing, regardless of health promotion programmes. Societies and environments are also changing in response to many different factors. One response to this problem might be to evaluate sooner and use a shorter timescale. However to do so might mean that longer-term sustained outcomes are missed. The best solution is to evaluate over different time periods, but this requires more resources.

Knowing the threshold for success is another challenge, and is illustrated in Example 20.17. A balance needs to be struck between setting the threshold too high, leading to interventions that are unjustly deemed to be ineffective, and setting the threshold too low, leading to a judgement that health promotion is not worth the effort. Striking the correct balance involves knowing what changes are likely to take place in the absence of the intervention, and then setting a realistic goal of what additional change is feasible and represents an efficient use of resources.

The third problem is being able to attribute any changes that occur to the health promotion input. The many different individual, organizational, social, economic and environmental factors affecting health are in a constant state of flux and it is very difficult to pin any changes down to specific causes. A health promotion intervention may trigger a variety of changes, some immediate, some intermediate, and some longer-term. It is challenging to record and capture changes happening at different times.

The most solid evidence that is used to prove cause and effect is derived from quantitative methodologies such as RCTs. However there are several reasons why the RCT is often inappropriate for health promotion interventions, and this is the fourth problem. Many interventions are complex and multicomponent and are designed to trigger effects and spread into other contexts and groups. So attempts to isolate the intervention are at odds with health promotion principles and practice.

There are also ethical problems with adopting the RCT model of evidence. When and how does the practitioner or agency decide there is sufficient proof to roll out a programme? Is it ethical to deny what is very likely to be an effective intervention in order to obtain more scientific evidence of its efficacy? These methodological and ethical problems mean the RCT is often inappropriate for health promotion interventions.

Although the RCT is often inappropriate, this does not mean that there is no evidence on which to base health promotion work. Meta-analyses or systematic reviews of research studies pool together findings from different studies in effectiveness reviews. Effectiveness reviews are a means of building up a knowledge base which can tell us what are reasonable expectations of success in health promotion. Success in health promotion is complicated because the aim is not just to change knowledge or behaviour, but to change the social determinants of health, and this requires qualitative as well as quantitative evaluation.

There is now an independent national body, the National Institute for Health and Clinical Excellence (NICE), devoted to providing evidence-based guidance on the promotion of good health and the prevention and treatment of ill health. NICE publications include guides on providing environments to encourage physical activity (NICE 2008a) and smoking cessation (NICE 2008b). In conclusion, the evidence base for health promotion is developing and includes a variety of approaches and methodologies. Syntheses of evaluations are being produced, enabling practitioners to start to practise evidence-based health promotion.