How do you tell if a drug treatment really works? The role of anecdotal reports & clinical trial results

One issue that has been thrown into the spotlight recently, is how we can tell if an intervention, such as a drug treatment, really works. We’ve all heard about clinical trials, but is anecdotal evidence enough during a pandemic? What do we have to lose by trying a new treatment if there aren't any existing ones to use?

Information about how effective an intervention is can be gathered in a number of ways including:

• Through casual advice from people who have used it

• By observing people who use it and documenting what happens

• By actively allocating people to receive certain interventions and then measuring the outcome

Knowing how much we should trust the evidence in front of us is vital when making important decisions, especially when your health may be at stake.

Here we take a look at the different types of evidence and touch on what changes to the evidence gathering process may be appropriate during a pandemic. The types of evidence covered include:

• Anecdotal evidence

• Evidence from observational (epidemiological) studies - including cohort, case-control and cross-sectional studies

• Evidence from interventional (experimental) studies - including community and controlled clinical trials

What to know about anecdotal evidence

Many times people rely on anecdotal evidence. This is the casual or informal advice you gather from family, friends, or nowadays even strangers on the Internet. A common example of this type of evidence is when we try a new vitamin or supplement because a friend told us it worked for them.

Anecdotal evidence is often accompanied by emotional information that our brains like to respond to. The person sharing it likely had a particularly good or bad experience, which makes the information more memorable for us.

In some situations we can benefit from a wealth of anecdotal evidence, such as in the case of an experienced healthcare professional who is able to draw on years of clinical practice treating many patients. The problem with this type of evidence, however, is that it is unreliable compared with clinical study data and we may place more importance on it than we should. This won’t necessarily do us any harm if we’re trying to pick a new leaf blower or pair of shoes, but it can if we’re trying to treat a harmful new disease.

That said, anecdotal evidence can play an important role during a pandemic. It may form the basis of a new hypothesis - point us in the direction of a potential new treatment - which can be investigated further.

What to know about evidence from studies

Clinical studies are part of evidence-based medicine. They help us to answer specific questions we have about drugs, diagnostics and diseases for example. There are a number of different types of study designs, which fall into one of two broad categories:

• Observational studies (epidemiological studies), including cohort, case-control and cross-sectional studies among others

• Interventional studies (experimental studies), including community and controlled clinical trials

Different study designs work better, or are more appropriate, in different circumstances.

Observational studies

As the name suggests, observational studies are those where researchers observe people to understand the effects of what it is they are studying, which could be a diagnostic test or risk factor for example. The researchers don’t manipulate or intervene in the situation. These types of studies are also called epidemiological studies.

Observational studies may have a retrospective design - that is to say they look back into the past via patient notes or by interviewing patients - but they can also be prospective and follow the patients going forward in time.

Observational studies can be used to look at disease prevalence, or the number of people with a disease, either at a set point in time or over a period of time. They can also be used to look at the incidence, or the frequency or rate at which a disease occurs, telling us how many new cases accumulate over a period of time.

Observational studies often measure the association or links between two factors and report the results in the form of an odds ratio. For example, a study might be used to measure the relationship between eating a particular food and having a certain disease.

Cohort, case-control and cross-sectional studies

Common observational study types include cohort studies, case control studies and cross sectional studies.

• Cohort studies: This type of study looks at a cohort, or group of participants with a shared characteristic, and follows them over a period of time (longitudinally). When the study begins, none of the participants should have the outcome being measured, which might be heart disease for example. The presence or absence of risk factors (such as high body weight or blood pressure) is documented in each participant at the beginning and then they are all typically followed going forward (prospective). Cohort studies can be expensive and time consuming, but can also be useful in situations where it would be unethical to enroll people in a trial and tell them to do something harmful like smoking.

• Case-control studies: Case control studies are different to cohort studies in that two groups of participants are recruited - one with a disease (or outcome) and one without. Researchers match the two groups so that they are the same in all other respects, such as age, sex and other factors, except for the disease being studied. These studies look backwards in time (retrospective) and can also be useful when looking for risk factors, but their results may be less reliable than other study types. This is because it can be difficult to establish whether it was the risk factor or the disease that occurred first (temporality), or because of other factors such as bias in the selection of the participants or extra variables that hadn't been accounted for (confounding factors). Case-control studies are quick and cheap to conduct and are useful when studying rare diseases.

• Cross-sectional studies: Cross sectional studies are carried out over a short time period or at a set point in time. They look for links between risk factors and the development of a disease, by analyzing information from participants all at one point in time. They are also quick and cheap to conduct, but their results may not be as useful as those obtained from other observational study types.

Interventional studies

Interventional studies, which are sometimes called experimental studies, are ones in which the researchers intervene, assigning study participants to a particular intervention and then watching for certain outcomes. Interventional studies have a prospective design, following the participants forward through time, collecting information as they go. This sort of study is similar to a cohort study with a prospective design, but differs in that the researchers actively assign participants to an intervention rather than just observing people.

Interventional studies often allocate participants randomly (randomize) to receive the intervention or to be part of a comparison or control group. Some interventional studies, however, are non-randomized and don’t carry out this process. Interventions can include all sorts of things, such as a drug therapy, vaccine, diagnostic test or educational program. The intervention can be used to prevent disease (preventative) or provide treatment (therapeutic).

Community and clinical trials

There are two broad groups of interventional studies, including community or field trials and clinical trials. Community trials involve assigning whole communities to receive different interventions, while clinical trials assign individuals to one of or more groups.

Clinical trials are further categorized by phase. For example, drug therapies start out in small phase I trials designed to determine the best dose, before moving through to phase II and larger phase III trials to confirm the efficacy and safety of the drug, before seeking marketing approval from the US Food and Drug Administration (FDA). Phase IV trials might also be conducted after the drug is marketed to continue to monitor the safety of the drug.

A well-designed trial can provide proof that an intervention really works and also inform us about its downsides. It can provide information that enables us to make informed decisions about the medications we take and decide if the benefits are worth the risks. For example, an experimental drug with unknown side effects might be an appropriate choice for a terminally or critically ill patient, but the same drug might be considered too big of a risk for someone with mild disease.

Clinical trials allow researchers to control the environment in which an intervention is being studied and minimize variation between the different intervention groups. This gives researchers more certainty that any differences observed between the groups are due to the intervention itself, rather than some bias, confounding factors or just plain old chance. They do all this with the help of statistics, a form of mathematical analysis.

When interventional studies measure risk or association they often report a hazard ratio, which is a measure of relative risk. A hazard ratio involves looking at the risk of an outcome in one group compared with the risk of an outcome in another group at a particular point in time. A clinical trial, for example, might report a hazard ratio that tells us that patients treated with a new drug were ‘x’ amount more likely to have recovered compared with those treated with an older drug.

Randomized controlled trials

Here we cover some basic features of randomized controlled trials (RCTs), which provide us with ‘gold standard’ evidence concerning the efficacy and safety of an intervention, and are commonly used to investigate new drug treatments or therapies.

RTCs typically enroll a select group of patients and randomly assign them to two different treatment groups to study how effective and safe a new drug treatment is. One group, called the control group, usually receives the current standard treatment or placebo and the other receives the drug being studied. As mentioned above, the two groups are intended to be comparable overall, with the only difference between them being the study drug. This is done to help ensure that any difference in the endpoint - the event or outcome being measured - is due to the effects of the drug and not chance or some other factor.

Basic Features of Randomized Controlled Trials

Despite the pandemic, we still need clear proof that drugs work

Without unbiased assessment of the outcomes - without unbiased evidence from clinical trials - we really don’t know whether a drug treatment will work for a particular disease or not. While anecdotal evidence has an important role to play, relying on it too much may mean we’re not only wasting money on an ineffective treatment, but it could also mean that we suffer serious side effects unnecessarily. We potentially have a lot to lose.

In a podcast focused on clinical trials during a pandemic, Dr. Janet Wittes, president and founder of Statistics Collaborative, mentions that it’s really hard to conduct a clinical trial in this sort of situation and stresses the importance of protecting patients and healthcare workers. While some niceties, such as blinding, might need be be dropped, she highlights the importance of thinking about what evidence you need to reach a conclusion and the need to keep the fundamentals of clinical trial design including:

• To randomize patients to the control and experimental groups

• Enabling unbiased collection of outcome or endpoint data

• To be clear on who is enrolled in the trial

Employing an adaptive trial design, which can be modified as the trial progresses, may also prove to be a more efficient, informative and ethical approach at this time.