Reading a study
Randomised trials, observational studies and everything in between: what makes a study strong, and the warning signs in how it's reported.
THE GOLD STANDARD
The randomised controlled trial
In a randomised controlled trial (RCT), volunteers are split by chance into two groups. One gets the treatment. The other, the control group, gets nothing or a placebo, a dummy pill that looks the same. Because chance decides who goes where, other factors, the confounders from lesson 6 like age, diet or how ill people are, end up spread about evenly between the groups. So if the groups end up different, the treatment is the most likely reason.
If people chose their own group, the most motivated might all pick the new pill, and their motivation, not the pill, could explain a better result. A coin toss doesn't care who's motivated.
Check yourself
Why do trials assign people to groups at random instead of letting them choose?
- Because it's cheaper than asking people
- To spread other factors evenly, so the treatment is the only systematic difference between groups
- To make sure the treatment group is bigger
- So the researchers can pick the healthiest people for the treatment
Show the answer
To spread other factors evenly, so the treatment is the only systematic difference between groups
Right. Chance spreads age, health, habits and everything else about evenly, including factors nobody thought to measure. Then a difference in outcomes can be credited to the treatment.
Step through it
200 volunteers A cloud of 200 grey dots on the left, labelled n = 200: people with headaches who volunteer for a trial. Beside them waits a small lilac box marked R, for randomisation. Nobody has been assigned yet.
Chance splits them into two groups Dots stream out of the cloud, through the R box, and off to one of two groups of 100: Rx at the top, outlined in orange, gets the new pill; placebo at the bottom, outlined in grey, gets a dummy one. Chance, not choice, decides who goes where. That's randomisation.
60 improve on the pill, 45 on the dummy A week later, the dots of people who improved fill in blue: 60 of 100 in the Rx group, and 45 of 100 in the placebo group. Plenty of people get better on a dummy pill, because headaches often pass on their own.
The real effect: +15 points The filled dots in each group are highlighted as a bar, labelled 60% for Rx and 45% for placebo, and a bracket linking them reads +15 pt. The pill's real effect is the difference: 15 percentage points. Without the control group, you'd have credited it with all 60.
Check yourself
In the trial, 60 of 100 people improved on the new pill and 45 of 100 on the placebo. What is the pill's effect?
- 60%: that's how many got better on the pill
- 15 percentage points: the difference between the groups
- 45%: the placebo effect
- 105 people helped in total
Show the answer
15 percentage points: the difference between the groups
Right. 45 of the 100 would probably have improved anyway, as the placebo group shows. The pill added 15 more per 100: an effect of 15 percentage points.
Blinding
Keeping people from knowing who got the real treatment. In a single-blind trial the participants don't know which group they're in. In a double-blind trial the researchers who measure the results don't know either. This limits the placebo effect and stops anyone's hopes from nudging the measurements.
If a doctor knows a patient got the new pill, she might, without meaning to, rate their pain as a little lower. Double-blinding removes that chance.
Two ways to study people
Randomised trial
Researchers assign the treatment by chance.
Can show cause and effect. Expensive, often small and short, and impossible for many questions.
Observational study
Researchers watch people who chose their own behaviour (cohorts, case-control studies, surveys).
Often the only ethical or practical option, since you can't assign people to smoke. Shows associations; confounding is the main risk.
Check yourself
Observational studies are worthless, because only randomised trials count as evidence.
Show the answer
False
False. Much of what we know about smoking and cancer came from observational studies, because a trial that assigned people to smoke would be unethical. They show associations, so confounders need careful handling, but large, well-run observational studies can be very valuable.
A rough ladder of evidence, weakest first
- Anecdote and expert opinion
"It worked for my cousin." Worth a question, not a conclusion.
- Studies in cells or animals
"In mice" or "in the lab". Useful early clues, but many results don't carry over to people.
- Observational studies
Real people and often large numbers, but they show associations that confounders may explain.
- Randomised controlled trials
Chance balances the groups, so a difference can be credited to the treatment.
- Systematic reviews and meta-analyses
Combine many studies on the same question. Higher isn't automatically right, but claims from lower rungs deserve more caution.
Check yourself
A headline says "Chocolate improves memory". The study behind it was done on mice. What's the main issue?
- Mice don't like chocolate
- Results in mice often don't carry over to people
- The study must have been too large
- Memory can't be measured
Show the answer
Results in mice often don't carry over to people
Right. Animal studies are useful early clues, but many results in mice never show up in people. "In mice" is a rung low on the ladder, and the headline dropped it.
Questions to ask of any study
- How many people? A study of 20 is fragile. Who were they? Could the results apply to you?
- Compared to what? No control group means no way to separate the treatment from recovery that would happen anyway.
- How big was the effect, in absolute terms (lesson 8), and how long did it last?
- Peer-reviewed or a preprint? Who funded it? Has anyone repeated it? In large replication projects in psychology, roughly a third to a half of published findings did not hold up.
- Press releases and headlines often overstate: "associated with" becomes "causes", mice become people. When you can, read the abstract.
Check yourself
Which is stronger evidence that a treatment works?
- One observational study of 50 people
- A meta-analysis combining 12 randomised trials
- A doctor's strong personal opinion
- A lab study on human cells
Show the answer
A meta-analysis combining 12 randomised trials
Right. Randomised trials balance the groups by chance, and combining 12 of them smooths out the luck of any single one. That puts it at the top of the ladder.
Lesson recap
- A randomised controlled trial splits people by chance into a treatment group and a control group, so confounders are spread evenly.
- In the example, 60 of 100 improved on the pill and 45 of 100 on a placebo: the effect is 15 percentage points, not 60%.
- Blinding stops participants' and researchers' expectations from shaping the results.
- Observational studies are often the only option and can be valuable, but they show associations, not proof of cause.
- Ask how many people, compared to what, how big the effect, who funded it and whether it's been repeated.