Critical Thinking

Common Defenses

Sometimes you accepted that treatments were placebos, but argued that that was OK:1. Sometimes there are no other treatments.2. If it works, it doesn’t matter how.3. There are no side-effects, so taking a placebo

can be better than taking “real” medicine.4. The placebo effect is a mental effect. But

sometimes the problem is mental too.

Important Concepts

Two important concepts we discussed while considering these defenses were:• The Nocebo Effect: treatments that contain no

medicine can cause harm if you expect them to.• “Medicalizing” Problems: Treating non-medical

problems with medicine can lead people to think that their problems are medical and can be solved with pills.

Defenses of TCM

Common defenses of TCM that you provided were:1. It has a long history of many thousand years.2. It’s holistic and flexible, treating the person,

not the symptoms.3. It’s herbal or natural rather than chemical.4. There are Chinese explanations for why it

works.

Important Concepts

We discussed some other important concepts when we considered these defenses:

The Forer Effect: People are likely to think that descriptions apply to them and are highly accurate if they believe those descriptions are based on detailed information about them and if the descriptions are vague enough.

Important Concepts

We discussed some other important concepts when we considered these defenses:

The Naturalistic Fallacy: it’s wrong to assume that just because something is natural, it is good, and just because something is not natural it is not good. Some natural things are bad, some non-natural things are good.

Defenses of TCM

There were some defenses we didn’t get to:5. TCM has spiritual effects beyond the physical ones.6. TCM gets taught in schools, and published in journals– even in the West! (It is recognized by authorities.)7. It’s culturally important to use it.

5. Spiritual Healing

Another argument that some treatments are worth continuing, even if they are no better than a placebo in RCTs, is that they may be better than a placebo in spiritual effects that are not measurable to science.

Caricature: “This treatment makes your aura brighter, even though it doesn’t make your cough go away.”

How Do You Know?

Maybe. If it’s something that can’t be observed, then I suppose there can’t be any scientific evidence against it.

But there can’t be any scientific evidence for it either. How do you know it actually makes your aura brighter or whatever?

Aura Reading

If someone claims that they can tell, but science can’t… science can test that!Suppose science can’t measure auras. We can still test to see whether other people can measure auras.

Science Testing Non-science

For example, we can randomly give either the treatment or a placebo to study participants.

Then we ask the aura reader– without telling her which group each patient belongs to– to tell us whose aura has gotten better.

If the treatment works, she should be better than chance at telling who is a control or not.

6. Appeal to Authority

I also got this defense of TCM:

“Westerners teach TCM in medical schools. Journals of medical research publish studies of TCM. Lots of doctors use it, not just Chinese ones. How can it be no better than a placebo?”

Imagine!

One thing to say is that this isn’t really appropriate to the assignment. I wanted you to imagine that some treatment that was important to you was shown to be no better than a placebo. That’s consistent with all of Chinese medicine being better than a placebo. You can imagine things that are not true.

Still, let’s consider the question:

Why do medical schools, doctors, and medical journals promote fake medicine like homeopathy, reiki, naturopathy, etc.?

Why have so many Westerners turned to non-science based medicine, when the Western medical establishment is mostly science-based?

Certainty

In science, nothing is ever certain. Smoking raises the probability that you’ll get cancer; undergoing radiation therapy raises the probability that you’ll beat cancer; you could get cancer if you didn’t smoke; you can die from cancer even if we do our best to treat you. Sorry, but that’s life.

Certainty and Helplessness

People don’t like uncertainty. They don’t like feeling that things are outside their control, that their disease is caused by something they can’t see, and can’t do anything about. Uncertainty makes you feel helpless.

This is from a “Naturopathic Oncologist”

When you’re frustrated with throwing up from the chemo treatments, losing your hair, and feeling incredibly tired all the time…maybe its time to look at a different option.The advantages of using Alternative Medicine:• guaranteed safe NO side effects – no harm done• guaranteed immune boosters• guaranteed easy to use – comfort of your own home, no doc

waits• guaranteed more control of your health – can talk to

practitioners longer than 10 min• guaranteed less invasive

What about Doctors?

And why do doctors latch on to non-science-based medicine?

That can be a long story, but consider the case of antidepressants again. They (or at least SSRI’s) don’t work any better than a placebo. Yet, Western doctors prescribe them all the time.

Scientific Fraud

The answer is more-or-less: fraud. There are lots of ways to cheat in science. If you want your study to show that antidepressants do better than placebos, you can not double blind your studies, or use improper randomization techniques (this is obvious to real scientists, though).

You can also:• Only correct the baseline when it suits you.• Ignore dropouts.• Remove outliers when it suits you.• Choose a statistical test that gets the best

results.• Find your hypothesis in the data.• Publish only positive findings.

The Baseline

Often, studies don’t have the power we would ideally desire. Remember that for a 95% confidence interval of 6%, we estimated that we’d need 1,000 subjects in our study.

But if you’re studying a new drug, how do you find 1,000 people who need it in your area who are willing to sign up for your trial?

The Baseline

Scientists often test much smaller groups, and then aggregate (put together) all the data later. This is called metaanalysis, and we’ll be discussing it later.When you have a small group of people– for example 20 or 30, there is a high probability that by random chance either the control group or the experimental group will be doing better.

The Baseline

This is called “the baseline.”

If you’re testing a pain medication, for example, the control group might– merely as a matter of chance– have a higher degree of average pain than the experimental group.

They have a higher “baseline” degree of pain.

Controlling for the Baseline

You can “control for the baseline” by testing how much people’s pain improved over the course of the trial, instead of just testing how much pain they’re in at the end of the trial.

Controlling for the Baseline

If you don’t control for the baseline, then you can get results like this:The average pain score in the control group was 65 when the experiment started, and 52 for the experimental group. Nobody improved, so it was also 65 and 52 at the end. But if you report just the end scores, it looks like your treatment worked: the experimental group had 12 less average pain points!

Controlling for the Baseline

It’s best to control for the baseline, but it’s OK if you don’t.

What’s bad is when you control for the baseline when the control group is doing better, but don’t control for it when the experimental group is doing better. That’s cheating

Ignoring Dropouts

Sometimes a treatment won’t work, or will cause harmful side-effects. The people experiencing the worst of these side-effects might drop out of the trial.

If you collect data only on people who finished the trial, it will seem like your treatment has fewer side-effects than it actually does.

Outliers

Outliers

An outlier is a data point that is far away from all of your other data points– it doesn’t fit a pattern that is clearly there.

For example, in a trial for a pain medication, you might have some people get a little better, some people get a little worse, and one person who dies. Dying is an outlier, in this situation.

Controlling for Outliers

Outliers are often due to just random chance. Through no fault of your treatment, sometimes people die. It can’t be helped.

It’s accepted practice to control for outliers (which have specific definitions in statistics) by removing them from your data. You can also choose to leave all your data intact.

Controlling for Outliers

Nothing is wrong with removing outliers– except when you do it only when it suits you.

If you choose to keep negative outliers in the control group and keep positive outliers in the experimental group, but choose to eliminate positive outliers in the control group or negative outliers in the experimental group, you’re cheating!

Finding the Hypothesis in the Data

There’s a difficult-to-understand fallacy in science that goes by different names– two are “finding hypotheses in the data” and “the problem of multiple comparisons.”

Here’s the idea: suppose I gather a lot of data about you guys: shoe size, weight, what you ate for breakfast, astrological sign, mother’s date of birth, favorite TV show…

Random Correlations Everywhere

If I gather enough data, then there are bound to be accidental correlations in the data. I might find that everyone born in March either likes playing chess or has an even numbered shoe size.

That’s, of course, entirely an accident.

Use New Data!

It’s not unusual to find accidental correlations in data.

So in science we require that hypotheses be tested by new data. So I might take my March-birthdate-chess-playing-shoe-size correlation and test it against the students in a different classroom that I haven’t looked at.

Why New Data Is Important

It would be really unlikely if (i) I propose a correlation(ii) I test it against some new data(iii) The new data confirm the correlation(iv) All of that was just an accident

Compare this to the fact that it is really likely to find random correlations in the data.

fMRI

fMRI = functional Magnetic Resonance Imaging. It’s a way of measuring change in blood flow in the brain, that allows us to get an understanding of change in brain activity.

fMRI Neuroscience

Common neuroscience involving fMRI might go something like this: I put a bunch of people in fMRI machines, and have them look at various pictures.

Information Processing

When they look at pictures of happy things, like smiling babies, double rainbows, cute puppies, or whatever, I might notice that certain parts of their brains are active (and not active when I’m not showing them these pictures).I might then conclude that these parts of the brain (the active ones) are responsible for processing information about happy things.

Multiple Comparisons

But this methodology is ripe for the problem of multiple comparisons.

There are lots of areas of the brain and lots of different aspects of any picture. If I look at all the areas of the brain and all the aspects of the pictures, I will find many correlations totally by random chance.

The Proof

Dead Fish

Craig Bennett is a neuroscience graduate student. He wanted to test out his fMRI machine, so he bought a whole dead salmon.

He put the dead salmon in the machine and showed it “a series of photographs depicting human individuals in social situations.”

Experimental Design

The salmon “was asked to determine what emotion the individual in the photo must have been experiencing.”

Then Bennett looked to see whether there were correlations between changes in the blood flow in the salmon’s brain, and the pictures.

Correlations!

Unsurprisingly, there were.

16 out of 8,064 voxels (volumetric pixels) were correlated with picture-viewing.

The important thing is that lots of neuroscientists use these same methods for humans. The risk of error is great.

Publication Bias

Suppose I think that reading causes foot cancer. That’s a pretty crazy belief. But suppose I then conduct a rigorous, scientific test of this belief and show (high statistical significance, large effect size) that it is true! That’s big news, and not only will I get published in the best science journals, like Science and Nature, I’ll probably get in the newspapers too.

But suppose I instead believe that reading does not cause foot cancer. That sounds pretty reasonable! And suppose I go out and conduct a rigorous, double-blind placebo-controlled randomized trial for my belief, with a large sample of a representative set of the population, and discover, with a high degree of statistical significance, that I’m right.

Well who cares? Not Science or Nature!

We all knew that reading didn’t cause foot cancer. That’s silly.

Negative results are inherently boring and uninteresting. Positive results are exciting and informative.

Testing ESP

http://www.colbertnation.com/the-colbert-report-videos/372474/january-27-2011/time-traveling-porn---daryl-bem

Testing ESP

Dr. Daryl Bem conducted experiments where the task was for subjects to select which of two curtains had an image behind it.

The curtain with the picture was determined randomly by a computer. So we expect that people will get the answer right about 50% of the time… random guessing.

Porn from the Future

What Bem found was that when the picture was normal and not pornographic, people did guess randomly: 49.8% of the time they guessed which curtain hid the picture.

But if the picture was pornographic, subjects guessed right 53.1% of the time. This was statistically significant.

Replicability

This is not unusual. Positive results will happen frequently through mere chance. We saw this in the salmon example.

This is why science relies on replicability. Other experimenters should be able to repeat your experiment and get the same results. If they can’t, then it looks like your result was lucky.

Publication Bias

So does ESP exist? Three separate teams of scientists at the University of Edinburgh, the University of Hertfordshire, and Goldsmiths, the University of London decided to test it. They performed the same experiment that Bem did, but got negative results: even porn pictures were guessed at random.

“No Replications”

The original journal that published Bem’s results, the Journal of Personality and Social Psychology, refused to publish these new ones.

“We don’t publish replications” they said.

Two other psychological journals said the same thing. They wouldn’t even look at the paper.

Publication Bias

Just consider this for a moment.

Suppose you want to know whether ESP works. There are 4 studies and 1 is positive and 3 are negative. But only the positive one is published, no one will publish the negative ones. So you look at all the available evidence and conclude that it works. You’re wrong!

7. Cultural Importance