Scientific Evidence
Like clothes and cars, scientific evidence varies in quality. When you read a claim that a treatment works it is a good idea to try to work
out how good the evidence really is.
Randomised controlled trials (RCTs): the best evidence
The randomised controlled trial is the Rolls Royce of scientific evidence. In an RCT, the people who volunteer to test out the treatment are randomly placed either in a treatment group (eg, given antidepressants) or a no treatment group (eg, given a sugar pill). A systematic review is a special unbiassed method of identifying all relevant trials of a treatment and combining the results. The best possible evidence comes from a systematic review of all RCTs of a treatment.
Controlled trial, not randomised: the next best evidence
Sometimes scientists use controlled trials where volunteers are not randomly placed in groups. Suppose we give all the patients from a depression clinic in Melbourne a secret depression buster formula. At the same time we give all the patients from a depression clinic in Sydney sugar pills. We find that the Melbourne patients recover more quickly than the Sydney patients. We might conclude that the depression buster formula works. We could well be right. However, we can't be sure. The difference between the two groups might reflect a difference in the clinics, a difference in the type of people who attend the clinics, or something different about the two cities. The non-randomised controlled trial is good evidence but not as good as the RCT.
Before and after group study
Another type of evidence involves measuring health before and after treatment. If there is an improvement, we might conclude that a treatment works. The problem with this type of study is that we can't be sure that an improvement is due to the treatment. The volunteers might have improved anyway. This type of study is not as good as a study with a control group.
Little or no evidence
Sometimes people claim that a treatment works on the basis of their personal or professional experience. For example, Mary Downtheroad tells her friends that pulling her ears three times each morning has changed her life. Now life is wonderful and she no longer becomes depressed. Mary believes that ear pulling has helped her but she cannot provide any scientific evidence to support her belief. Maybe trials in the future will prove her correct and perhaps they won't. This anecdotal information is the Roller Skates of scientific evidence - you can't tell if and when you will crash.
What else is important?
Studies should involve enough people that we can be confident of the findings
The larger a study, the more likely we are to find an effect of treatment if it exists.
The best studies are 'blind'
A blind study means that the people involved in the study do not know who is receiving the treatment and who is not. (In a single blind study, the patients do not know if they have been given the active treatment or the placebo. In a double blind study, neither the volunteers nor the people treating or assessing them know who is receiving the actual treatment). The advantage of a blind study is that the volunteers and researchers cannot consciously or unconsciously bias the results of the study.
Findings should be tested for statistical significance
Sometimes differences occur by chance. There are special statistical methods for deciding if a difference between two groups (eg, one that receives a treatment and one that doesn't) is real. All good studies should report whether a finding is statistically significant.
Findings should be meaningful
Sometimes a treatment can produce a real (statistical) effect but the effect is not very large. All other things being equal, a treatment that makes a large difference is better than a treatment that makes a small difference.