Imagine asking the world’s leading intellectuals-What scientifc concept would improve everybody’s cognitive toolkit? Well, it was done and Richard Dawkin’s response was: the double blind study.
Before we get into his direct response, I’d like to share some information I learned about double blind studies and hopefully this will provide insight into why Mr. Dawkin’s believes it to be so important.
First what is a double blind study? From wiki:
Double-blind describes an especially stringent way of conducting an experiment, usually on human subjects, in an attempt to eliminate subjective bias on the part of both experimental subjects and the experimenters. In most cases, double-blind experiments are held to achieve a higher standard of scientific rigor.
In a double-blind experiment, neither the individuals nor the researchers know who belongs to the control group and who belongs to the experimental group. Only after all the data have been recorded (and in some cases, analysed) do the researchers learn which individuals are which. Performing an experiment in double-blind fashion is a way to lessen the influence of the prejudices and unintentional physical cues on the results (the placebo effect, observer bias, and experimenter’s bias). Random assignment of the subject to the experimental or control group is a critical part of double-blind research design. The key that identifies the subjects and which group they belonged to is kept by a third party and not given to the researchers until the study is over.
Double-blind methods can be applied to any experimental situation where there is the possibility that the results will be affected by conscious or unconscious bias on the part of the experimenter.
For a sampling of cognitive biases that double blind experiments are meant to minimize I turned to Steven Bratman’s article.
From Mendosa online:
The placebo effect is the process by which the power of suggestion actually causes symptoms to improve.
The placebo effect almost always comes as a surprise to those who experience it. Both doctors and patients are fooled. For example, surgeons used to think that arthroscopic surgery for knee arthritis really worked, and hundreds of thousands of such surgeries were performed every year. Then a study came out showing that fake surgery produces just as satisfactory and long-lasting benefits as the real thing.
People generally get angry if you tell them their benefits might be due to placebo. However, examples abound to show just how possible this really is. I’ll give a few here.
In a double-blind, placebo-controlled study of 30 people with carpal tunnel syndrome, use of a static magnet produced dramatic and enduring benefits, but so did use of fake magnets.34
In a study of 321 people with low back pain, chiropractic manipulation was quite helpful, but no more helpful than giving patients an educational booklet on low back pain.35
In a randomized, controlled trial of 67 people with hip pain, acupuncture produced significant benefits, but no greater benefits than placing needles in random locations.33
And in a randomized, controlled trial of 177 people with neck pain, fake laser acupuncture proved to be more effective than massage.32
Note that these studies do not actually disprove the tested therapies. The study sizes might have simply been too small to detect a modest benefit. What they do show, however, is that comparison to placebo treatment is essential: without such comparison, any random form of treatment, no matter how worthless in itself, is likely to appear to be effective.
Beyond the Placebo Effect
At least the placebo effect produces a real benefit. Many, many other illusions can create the impression of benefit although no benefit has occurred at all. In this section I discuss a few of these more insidious confounders.
Even when a fake treatment doesn’t actually improve symptoms, people may re-interpret their symptoms and experience them as less severe. For example, if I give you a drug that I say will make you cough less frequently, you will very likely experience yourself as coughing less frequently, even if your actual rate of coughing doesn’t change. In other words, you will re-interpret your symptoms to perceive them as less severe. (This effect seems to have been the primary reason why people use over-the-counter cough syrups — surprising as it may seem, current evidence suggests that they are not effective, even though people have relied upon them for decades.10)
Observer bias is a similar phenomenon, but it affects doctors rather than patients. If doctors believe that they are giving a patient an effective drug, and they interview that patient, they will observe improvements, even if there are no improvements. F
The term selection bias indicates that if researchers are allowed to choose who gets a real treatment and who doesn’t, rather than assigning them randomly, it is very likely that they will unconsciously pick people in such a way that the treatment will look better. For reasons that aren’t clear, this effect is so huge that it can multiply the apparent benefit of a treatment by seven times, and turn a useless treatment into an apparently useful one.3,4 This is why double-blind studies must be “randomized.’
Many diseases will get better on their own, as part of their natural course. Any treatment given at the beginning of such an illness will seem to work, and a doctor using such a treatment will experience what is called the illusion of agency, the sense of having helped even though the outcome would have been the same regardless. A good example is neck or back pain: most episodes of these conditions go away with time, regardless of treatment, and so any treatment at all will seem to be effective.
Regression to the mean is like natural course, but a bit trickier. It’s based on the fact that even for conditions that do not go away on their own, the severity of the condition tends to fluctuate. Blood pressure is a good example. For many people, blood pressure levels wax and wane throughout the day, and from week to week. Suppose a person’s average blood pressure is 140/90, but occasionally gets as high as 170/110. If such a person gets tested and found at the moment to have high blood pressure, he may be seen as needing treatment. However, if he happens to be more near his average blood pressure, or even lower, he won’t be seen as needing treatment. In other words, doctors will tend to treat people when they are at their worst, not when they are at their best. By the laws of statistics, after a while, a person is more likely to be near his average blood pressure than his worst blood pressure, regardless of what treatment (if any) is used. This will appear to be an improvement, though in fact it’s only natural fluctuation.
The study effect refers to the fact that people enrolled in a study tend to take better care of themselves, and may improve for this reason, rather than any specifics of the treatment under study. This is a surprisingly powerful influence. If you enroll someone in a trial of a new drug for reducing cholesterol, and then you give them a placebo, their cholesterol levels are likely to fall significantly. Why? Presumably, they begin to take better care of themselves, by eating better, exercising more, etc. Again, double-blinding and a placebo group are necessary, because otherwise this confounding factor can cause the illusion of specific benefit where none exists.
Suppose you’ve invented a truly lousy treatment that fails almost all the time, but helps one in a hundred people. If you give such a nearly worthless treatment to 100,000 people, you’ll get a thousand testimonials, and the treatment will sound great.
Suppose you give someone a treatment said to enhance their mental function, and then you use twenty different methods of testing mental function. By the law of averages, improvements will be seen on some of these measurements, even if the treatment doesn’t actually work. If you’re a supplement manufacturer, you can use these results to support the sales of your product, even though in fact the results are merely due to the way statistics work, and not any mind-stimulating effect of your product. (In order to validly test the mind-enhancing power of a supplement, you have to restrict yourself to at most a couple of ways of testing benefit).
Suppose you give 1000 people a treatment to see if it prevents heart disease, and you don’t find any benefit. This frustrates you, so you begin to study the data closely. Lo and behold, you discover that there is less lung cancer among people receiving the treatment. Have you made a new discovery? Possibly, but probably not. Again by the law of averages, if you allow yourself to dredge the data you are guaranteed to find improvements in some condition or other, simply by statistical accident.
Perhaps the trickiest statistical illusion of all relates to what are called observational studies. This is such an important topic, that again I’ll break for a new heading.
In observational studies, researchers don’t actually give people any treatment. Instead, they simply observe a vast number of people. For example, in the Nurse’s Health Study, almost 100,000 nurses have been extensively surveyed for many years, in an attempt to find connections between various lifestyle habits and illnesses. Researchers have found, for example, that nurses who consume more fruits and vegetables have less cancer. Such a finding is often taken to indicate that fruits and vegetables prevent cancer, but this would not be a correct inference. Here’s why:
All we know from such a study is that high intake of fruits and vegetables is associated with less cancer, not that it causes less cancer. People who eat more fruits and vegetables may have other healthy habits as well, even ones we don’t know anything about, and they could be the cause of the benefit, not the fruits and vegetables.
This may sound like a purely academic issue, but it’s not. Researchers looking at observational studies noticed that menopausal women who take hormone replacement therapy (HRT) have as much as 50 percent less heart disease than women who do not use HRT. This finding, along with a number of very logical arguments tending to show that estrogen should prevent heart disease, led doctors to recommend that all menopausal women take estrogen. Even as late as 2001, many doctors used to say that taking estrogen was the single most important way an older woman could protect her heart.
However, this was a terrible mistake. Observational studies don’t show cause and effect, and it was possible that women who happened to use HRT were healthier in other ways and that it was those unknown other factors that led to lower heart disease rates, and not the HRT. Doctors pooh-poohed this objection (showing that even doctors often fail to understand the need for double-blind studies) and said that it was perfectly obvious HRT helped. However, when a double-blind, placebo-controlled study was done to verify what everyone “knew” was true, it turned out that that HRT actually causes heart disease, rather than prevents it.6 It also increases risk of breast cancer. In other words, placing trust in observational studies led to the deaths of many, many women. This is not, as I say, an academic issue.
In hindsight, it appears that women who happen to use HRT are healthier because they tend to be in a higher socioeconomic class, and have better access to healthcare and also take care of themselves. However, it is also possible that the real cause of the spurious association between HRT use and reduced heart disease is due to some other factor that we have not even identified. The bottom line is that observational studies don’t prove anything, and they can lead to conclusions that are exactly backwards.
This is a lesson that the news media seem unable to understand. It constantly reports the results of observational studies as proof of cause and effect. For example, it has been observed that people who consume a moderate amount of alcohol have less heart disease than those who consume either no alcohol or too much alcohol. But, contrary to what you may have heard, this doesn’t mean that alcohol prevents heart disease! It is very likely that people who are moderate in their alcohol consumption are different in a variety of ways from people who are either teetotalers or abusers, and it is those differences, and not the alcohol per se, that causes the benefit. Maybe, for example, they are moderate in general, and that makes them healthier. The fact is, we don’t know.
Similarly, it has been observed that people who consume a diet high in antioxidants have less cancer and heart disease. However, once more this does NOT mean that antioxidants prevent heart disease and cancer. In fact, when the antioxidants vitamin E and beta-carotene were studied in gigantic double-blind studies as possible cancer- or heart-disease-preventive treatments, vitamin E didn’t work (except, possibly, for prostate cancer) and beta-carotene actually made things worse!17-28 (One can pick holes in these studies, and proponents of antioxidants frequently do, but the fact is that we still lack direct double-blind evidence to indicate that antioxidants truly provide any of the benefits claimed for them. The only evidence that does exist is directly analogous to that which falsely “proved” that HRT prevents heart disease!)
And with that, I leave you with Mr. Dawkins.
“Not all concepts wielded by professional scientists would improve everybody’s cognitive toolkit. We are here not looking for tools with which research scientists might benefit their science. We are looking for tools to help non-scientists understand science better, and equip them to make better judgments throughout their lives.
Why do half of all Americans believe in ghosts, three quarters believe in angels, a third believe in astrology, three quarters believe in Hell? Why do a quarter of all Americans and believe that the President of the United States was born outside the country and is therefore ineligible to be President? Why do more than 40 percent of Americans think the universe began after the domestication of the dog?
Let’s not give the defeatist answer and blame it all on stupidity. That’s probably part of the story, but let’s be optimistic and concentrate on something remediable: lack of training in how to think critically, and how to discount personal opinion, prejudice and anecdote, in favour of evidence. I believe that the double-blind control experiment does double duty. It is more than just an excellent research tool. It also has educational, didactic value in teaching people how to think critically. My thesis is that you needn’t actually do double-blind control experiments in order to experience an improvement in your cognitive toolkit. You only need to understand the principle, grasp why it is necessary, and revel in its elegance.
If all schools taught their pupils how to do a double-blind control experiment, our cognitive toolkits would be improved in the following ways:
1. We would learn not to generalise from anecdotes.
2. We would learn how to assess the likelihood that an apparently important effect might have happened by chance alone.
3. We would learn how extremely difficult it is to eliminate subjective bias, and that subjective bias does not imply dishonesty or venality of any kind. This lesson goes deeper. It has the salutary effect of undermining respect for authority, and respect for personal opinion.
4. We would learn not to be seduced by homeopaths and other quacks and charlatans, who would consequently be put out of business.
5. We would learn critical and sceptical habits of thought more generally, which not only would improve our cognitive toolkit but might save the world.”