What are randomised controlled trials?

In the hierarchy of generating scientific evidence, randomised controlled trials (RCTs) are considered the gold standard.¹ A randomised clinical trial involves at least one test treatment (such as a herbal medicine) and one control treatment (a placebo or standard treatment, such as a drug). Patients are allocated to the treatment groups by a random process (use of a random-numbers table) to eliminate selection bias and confounding variables. (Note: the toss of a coin, patient social security numbers, days of the week and medical record numbers are considered unacceptable methods for randomisation). For randomisation to be successful, the allocation must remain concealed. In RCTs, some form of blinding or masking is also usually applied. Although the definitions vary, a trial can be considered double blind when the patient, investigators and outcome assessors are unaware of a patient’s assigned treatment throughout the duration of the trial. Blinding helps eliminate biases other than selection bias, such as differences in the care provided to the two groups, withdrawals from the trial and how the outcome is assessed and analysed. In some cases, double blinding cannot be implemented and a single blind design may be used.

There are numerous reasons suggested as to why phytotherapy should be subjected to the same degree of scientific scrutiny as conventional medicine. Such reasons include the need to become accepted by governments and conventional medicine, to help us determine if the therapy is safe and effective and to give patients a reasonable idea of the potential success rate.^1,2,3

This is not to say RCTs are without criticism. There are circumstances in which such a trial would be unethical. RCTs do not address why a treatment works, how participants are experiencing the treatment and/or how they give meaning to these experiences. They may not illustrate benefits other than the specific effects measured, explain the effects of the patient-practitioner relationship or take into consideration the patients’ beliefs and expectations. Statistically significant results may be produced that have no clinical significance or importance to patients or their caregivers. Outcome measures may focus on the physical while ignoring issues related to meaning, purpose and spirituality (for example, quality of life measures in terminally ill patients). This is also not to say that RCTs are the ‘only game in town’. Epidemiological and social research methods with appropriate designs also provide valuable information.^1,2,4–7

Despite its gold standard status, there is also no guarantee that a RCT will be well conducted, or will produce meaningful results (see later) and often once published results are open to distortion and misinterpretation.⁸

CAM versus conventional medicine in the published literature

Before looking at the structure of clinical trials and how they should be read, it is worth examining the published literature in perspective: what can be said about the quality of information that is published and is there a difference between how complementary and alternative medicine (CAM) is handled compared with conventional/orthodox/mainstream medicine?

An analysis of 207 randomised trials found that trials of complementary therapies often have relevant methodological weaknesses. Reporting and handling of drop-outs and withdrawals was a major problem in all therapies reviewed. (The data analysed here was originally collected for previously published systematic reviews on homeopathy (published 1997), herbal medicine (St John’s wort for depression (1998), Echinacea for common cold (1999), acupuncture (for asthma (1996) and recurrent headaches (1999).) Larger trials published more recently in journals listed in Medline and in English exhibited better methodological quality, but stringent quality restrictions would result in the exclusion of a majority of trials. However, the average methodological quality of the complementary medicine trials reviewed was not necessarily worse than trials in conventional medicine.⁹ (It would be interesting to have this analysis repeated using more recent systematic reviews containing more rigorous trials of herbal medicines such as for Ginkgo and St John’s wort.)

A randomised, controlled, double blind study investigated the hypothesis that the process of peer review favours an orthodox form of treatment over an unconventional therapy. A short report describing a randomised, placebo-controlled trial of appetite suppressants (‘orthodox’=hydroxycitrate, ‘alternative’=homeopathic sulphur) was randomly sent to 398 reviewers. Reviewers showed a wide range of responses to both versions of the paper, with a significant bias in favour of the orthodox version. Authors of technically good unconventional papers may therefore be at a disadvantage when subject to the peer review process. However, the effect is probably too small to preclude publication of their work in peer-reviewed orthodox journals.¹⁰

Publication bias is a recognised phenomenon in mainstream medicine (MM), so does it exist in complementary and alternative medicine (CAM)? An analysis of controlled trials of CAM found that more positive outcome trials than negative outcome trials are published. (The analysis looked at journals aimed specifically at a CAM readership and journals that specialised primarily in mainstream medical topics.) The only exception to this was in the highest impact factor MM-journals (this means that the publication bias in favour of a positive outcome for CAM clinical trials did not occur in those mainstream medicine journals with the highest citation rate). In non-impact factor CAM-journals, positive studies were of poorer methodological quality than the corresponding negative studies. (CAM therapies in this investigation included herbal medicine, homeopathy, chiropractic, osteopathy, spinal manipulation and acupuncture. The analysis used systematic reviews and meta-analyses from 1990 to October 1997.) Considerable bias may also be caused by using Medline as a single source of information, although this is not unique to CAM (an investigation of mainstream medicine publications found that European journals may be underrepresented in Medline).¹¹ An investigation of trials published in four prominent Medline-indexed CAM journals from 1995 to 2000 found that the bias towards the publication of a positive outcome persisted in 2000, although it was less strong than in 1995.¹²

What to look for when reading a clinical trial paper

A trial often seeks to question, rather than confirm, its own hypothesis. The authors of a study set out to demonstrate a difference between the two arms of their study (e.g. a herbal medicine compared with placebo, herbal medicine compared with a drug or a comparison of two different dosages of herb) and do it in the following way ‘Let’s assume there’s no difference; now let’s try to disprove that theory’.¹³ This is why the results are presented as ‘there was/was not a statistically significant difference between [the herb] and placebo’.

The standard (and best) presentation for papers published in medical journals includes:¹⁴

The design of the methods section is important in estimating the quality of the trial. Having worked through the methods section, you should be able to tell yourself:^14,15

When scanning clinical trials (abstracts and full papers!) it is best to start with the conclusion section: was the outcome positive or negative? What herb was trialled? Then go to the method: what objective parameter was measured? What was the form and dosage of the herb? In the methods section it really helps when it is spelled out as ‘the aim of this randomised study was to [assess the effectiveness of …], the secondary objectives were to assess …’.

Some characteristics of poorly conducted/written clinical trials:

• Abstract/summary not clearly written, conclusion not easy to find

• Method not clearly outlined

• Who received what and when; was there a washout period (crossover trials) and when did it occur? Was there a follow-up and when did it occur?

• Dosage:

 Not clearly outlined; e.g. dried herb equivalent not outlined – only the extract or concentrate or active constituent dosage given; ‘patients received one capsule of the plant extract’

 Placebo not clearly outlined or inappropriate:

 misleading information, e.g. ‘ginger extract is a preparation (250 mg of Zingiberis Rhizoma per capsule)’ when further investigation of the product (e.g. via company/product website) indicates ‘capsules contain 250 mg of a concentrated pure ginger material’

 Duration of treatment not specified

 Raw data presented but not summarised or the numbers do not add up (e.g. numbers conflicting between results and discussion/conclusion sections)

• Very small number of trial participants

• Side effect rate given but not explained (e.g. were the side effects mild or serious and were they related to the treatment?)

• Drop-out numbers not given (and where given, explanation for drop-outs not given, i.e. was it related to the treatment?)

• Abbreviations not explained; information, figures or tables missing.

Checklist for reading RCTs

The following points illustrate the type of questions to ask from a clinical trial paper. Examples are outlined below.

(a) What clinical question did the study address?

• Treatment of depression, prevention of atherosclerosis

(b) Was the study conducted with patients with pre-existing disease or healthy volunteers?

• A memory test in elderly patients with age-associated memory impairment assessing the efficacy of Ginkgo provides different information from that obtained from young healthy volunteers

(c) Were the participants studied in the trial relevant to your practice?

• Results involving Inuit (native Canadians) may not be relevant to western societies

• Results from participants of a social group that abstains from alcohol, or women not taking the contraceptive pill

• An assessment of St John’s wort involving severely depressed patients (which might be more appropriately treated with intensive psychotherapy)

(d) What treatment was considered?

• Herbal treatment, diet modification, use of compression stockings

(e) What parameters were measured and what is the clinical relevance?

(The best quality papers outline why that parameter is relevant, so you do not have to go searching to understand the connection).

In a clinical trial assessing the efficacy of a standardised willow bark extract, the primary outcome measure was the pain dimension of the WOMAC Osteoarthritis Index, a reduction in this score indicates an analgesic effect. It would be useful to compare the reduction observed from willow bark (14%) with that obtained from a standard drug (diclofenac, 19%).

‘Softer’ outcome measures include patient and physician assessments, although it is better if this information is also objectively evaluated.

• In the above willow bark trial, patient and physician overall assessments used the visual analogue scale (which assesses pain and physical function).

Here are some examples where the measured parameter may not be clinically relevant:

• Use of inappropriate endpoints (i.e. not a direct measure of clinical benefit):

 only measuring serum PSA (prostate-specific antigen) in a trial investigating saw palmetto liposterolic extract for benign prostatic hyperplasia would not be as useful as measuring urinary symptoms, pain and quality of life

 in HIV, CD4 count alone has been found not to correlate with survival – a combination of several markers gives a better indication.¹⁶

• Trials assessing pharmacological/physiological effects may increase the understanding of the mode of action of the treatment but may not provide immediately useful clinical information:

 measurement of biological markers (TNF alpha and IL-1beta) in patients treated with saw palmetto liposterolic extract to test the hypothesis that infiltrating cells are associated with progression of benign prostatic hyperplasia

 after 6 months of saw palmetto treatment a difference occurred in nuclear morphometric descriptors, suggesting the herbal blend alters the DNA chromatin structure and organisation in prostate epithelial cells→a possible molecular basis for tissue changes and therapeutic effect of the compound is suggested

 serum of women given phyto-oestrogens to ingest stimulated prostacyclin release in human endothelial cells (ex vivo)→phyto-oestrogens might produce a beneficial cardiovascular effect

 in vivo gene expression within peripheral leukocytes evaluated in healthy nonsmoking subjects who consumed Echinacea→pattern showing an anti-inflammatory and antiviral response

 Panax ginseng, at doses of 200 mg of the extract daily, increases the QTc interval (an electrocardiographic (ECG) parameter, not immediately clinically significant) and decreases diastolic blood pressure 2 h after ingestion in healthy adults on the first day of therapy→increases in ECG parameters may have clinical implications as they increase the risk of ventricular tachyarrhythmias.

• Mechanism of a potential herb-drug interaction:

 high-dose treatment with St John’s wort extract induced CYP3A activity in healthy volunteers as evidenced by increased 6-beta-hydroxycortisol excretion. Induction of this enzyme most likely contributes to the decreased bioavailability observed upon co-administration of various drugs with St John’s wort extract. The D-glucuronic acid pathway appeared unaffected by St John’s wort. (Note: Although this trial did not tell us anything immediately clinically relevant, some trials investigating cytochrome P450 enzymes are relevant, such as those that demonstrate an interaction between St John’s wort and specific drugs such as omeprazole, irinotecan, verapamil).

• For another purpose such as using a herb as a test agent:

 oral treatment with a Ginkgo biloba extract (Gibidyl Forte(R)) dilated forearm blood vessels causing increments in regional blood flow without changing blood pressure levels in healthy participants→the increments in blood flow may be used as a biological signal for pharmacokinetic studies.

(f) Was the route of administration relevant to herbal practice?

• Injection of a polysaccharide is not relevant in terms of results or practice, but injection of an active constituent might be relevant in terms of results (e.g. blood levels of the active may relate back to oral intake of a herb, providing the active from the herb gets into the bloodstream in that form (and is not altered by passage through the liver or gut) although it is not relevant to practice (injection of the active is not an option for herbal practitioners)).

(g) Was the preparation relevant to the products you use?

• The levels of active constituent may be too high to be achieved by a herbal product – a trial investigating soy protein will not be relevant to a soy product standardised for isoflavone content, as the protein level is likely to be very low in the latter

• To use the results of a trial administering triterpene fraction of gotu kola it would be necessary to know the content of these constituents in the herb/extract

• Efficacy of Pygeum africanum in benign prostatic hyperplasia is interesting but Pygeum is difficult to obtain as it is an endangered species

• Freshly expressed juice of Echinacea aerial parts will give different results than liquid extracts prepared from dried Echinacea root

• A supercritical carbon dioxide extract of chamomile will have a different phytochemical profile and hence pharmacological activity than an aqueous ethanol extract

• Products containing red clover isoflavones will not be translatable to low ethanolic liquid extracts as these constituents are present in low quantities.

(h) Was the dosage relevant to herbal practice?

• The daily dose of herbs prescribed was well below the normal therapeutic limit (hence more like a homeopathic protocol)

• Exceedingly high dosages require a safety judgment on the part of the practitioner, e.g. a 2002 trial of hawthorn used 1,800 mg/day of 5:1 dry extract=9 g/day dried herb (the usual dosage is around 3 to 6 g/day); a standardised ginger extract used a dose equivalent to 40 mg/day of gingerol, a good quality 1:2 liquid extract contains approx. 5 mg/mL=8 mL/day which is too high to prescribe for safety and compliance reasons!).

Note: A clinical trial does not necessarily determine the optimum dose, but provides information that for a given dose of a given herbal preparation a certain percentage of patients will be likely to respond i.e. [herbal preparation] at xx mg/day given for a period of yy weeks improved [the disease/disorder] in ZZ% of patients.

(i) Was the choice of placebo suitable?

• In assessing the effect of phyto-oestrogens, wheat may not be an inactive control for comparison with soy grits

• Olive oil may not be suitable to compare with evening primrose oil for the treatment of rheumatoid arthritis, as it too may have benefit.

(j) Was the blinding successful?

A well-designed trial will report the methods used to produce the double blinding. An analysis of the methodological quality of controlled trials published in 1995 found that trials for which no double blinding was reported produced larger estimates of effects.¹⁷ It is not enough for the trial to describe itself as double blind, the success of the blinding should be evaluated and reported. (One way would be to ask the patients and clinicians at the end of trial which treatment they thought they had received.) Evaluation of blinding is rarely done in trials of conventional medicine (a recent analysis found only 7% for general medicine trials)¹⁸ let alone in CAM trials. The evaluation of blinding has also been questioned. Trial participants asked to guess their treatment might be influenced by outcome (e.g. marked therapeutic or adverse effects). Assessments of blinding success would be more reliable if carried out before the clinical outcome has been determined. But in the case of a comparative treatment (no placebo) clinicians’ predictions may be a measure of pre-trial hunches and inseparable from the blinding effectiveness.^19,20

An example of evaluation of blinding in a randomised, comparative trial between St John’s wort (Hypericum perforatum) and sertraline for major depressive disorder is outlined below.²¹

At the end of 8 weeks, the proportion of patients guessing their treatment correctly was 55% for sertraline, 29% for Hypericum, and 31% for placebo (p=0.02 for differences between treatment groups). Correct guesses for clinicians totalled 66% for sertraline, 29% for Hypericum, and 36% for placebo (p=0.001 for differences between treatment groups). The change (mean) in HAM-D total score from baseline to week 8 did not differ for patients who were in the sertraline group and either had guessed the correct treatment (−11.6; 95% CI, −13.1 to −10.1) or had not (−11.9; 95% CI, −13.9 to −9.9).

This assessment was criticised, with the conclusion that the blinding was compromised. Clinicians who believed in the strength of sertraline treatment may have inadvertently let this bias affect their ratings. HAM-D is regarded as a clinician-rated measure (it would have been better to measure this against the clinician’s guesses rather than the patient’s guesses) – better yet, assess the other score (CGI-I, Clinical Global Impressions) which is a better measure of treatment effect).^22,23

(k) Was the treatment beneficial?

This will be a comparison of the results of the two groups, but most well-designed clinical trials now also report the difference between the two treatments using statistics, which is often expressed with a p value. The p value is the probability that any particular outcome would have arisen by chance. Standard practice usually regards a p value less than one in twenty (expressed as p<0.05) as ‘statistically significant’ and a p value less than one in 100 (p<0.01) as ‘statistically highly significant’, and p<0.001 more highly significant again. So if you read 20 clinical papers that reported a significant outcome of p<0.05 then it is likely that the outcome for one trial was not significant but due to chance:²⁴

• Nonstatistical results: ‘complete healing of duodenal ulcer occurred in 35% of those receiving deglycyrrhizinised licorice compared with 12% of those receiving placebo’

• Statistical: ‘the occurrence of radiation-induced acute dermatitis of grade 2 or higher was significantly lower (41% vs 63%; p<0.001) with the use of topical Calendula than trolamine’

(A cautionary note about statistics. A recent analysis found that statistical errors occurred in two leading scientific and medical journals. In 12% of cases, the significance level could have changed by one or more orders of magnitude. The conclusion would change from significant to nonsignificant in about 4% of the errors.²⁵)

(l) What was the side effect rate, side effect severity and drop-out rate?

Did it differ between the two groups?

• In a trial comparing chaste tree with fluoxetine for premenstrual dysphoric disorder, although the number of patients experiencing side effects was similar, sexual dysfunction was reported by two patients in the fluoxetine group while none of the chaste tree group experienced this side effect).

More information

The above checklist helps the reader to obtain information from the clinical trial paper, but critical appraisal (such as assessing the scientific validity and practical relevance) of the trial involves a whole lot more and is beyond the scope of this discussion. More information on critical appraisal can be found from well written books devoted to the topic,^26,27 and some examples include:

Published research of other designs have not been discussed here (e.g. controlled (nonrandomised) trials, postmarketing surveillance trials, uncontrolled trials, cohort and case-controlled trials, surveys, case reports, pharmacokinetic studies and herb-drug interactions) but many of the same principles apply. A higher level of rigour (and regarded as a higher level of evidence) can be found in systematic reviews and meta-analyses. (A systematic review summarises the available evidence on a specific clinical question with attention to methodological quality. A meta-analysis is a quantitative systematic review involving the application of statistics to the numerical results of several trials which all addressed the same question.) Systematic reviews and meta-analyses are useful in that they summarise the available clinical trial results. Meta-analyses are particularly valuable because by pooling the trial results they provide greater statistical power to measure a difference between the treatment and the control group.²⁸ Narrative (nonsystematic) reviews, as employed for the monographs in this book, may provide a useful overview of the literature but do not employ the same rigour as systematic reviews (they are not standardised or as objective). But even these more rigorous studies (systematic reviews and meta-analyses) are not beyond criticism and subjectivity.²⁹

Clinical trial papers: how to calculate dosage

Rules to consider before calculating dosage

When working out extraction and concentration factors, first consider the dosage form (starting with dried herb, liquid extract or concentrated extract).

If the amount of concentrated extract is known, first work out the dried herb equivalent and then calculate a relevant liquid extract or tablet dosage. Multiply the weight of concentrate by the first number in the ratio:

Once the dried herb equivalent is known this can be converted to a liquid extract by multiplying by the second number of the ratio:

Examples:

Product	Extraction/Dilution Ratio	Concentration Ratio
1:1 liquid extract	1	–
1:2 liquid extract	2	–
1:3 liquid extract	3	–
1:5 liquid extract	5	–
2:1 concentrated extract	–	2
4.5:1 concentrated extract	–	4.5
5:1 concentrated extract	–	5

The table below illustrates how to calculate doses cited in clinical trials and convert them to products, such as standardised liquid extracts and tablets. The first column contains a statement from the methods section of the cited clinical trial and the subsequent columns show how to convert the dosage first to the dried herb equivalent (or active constituent level) and then to liquid extract or tablet dosages where relevant. Notes explaining the calculations are provided in italics.