If you use that logic, then is a sugar pill really a placebo? Maybe the effect of a pill is actually induced by the sensation of placing it in your mouth, swallowing it and then feeling it travel down your throat.

]]>Speaking of his extended media team, they’ve been spending the past week trying to cook up a fake gaffe from Obama to take heat off Mitt. Take a look at this sad endeavor: http://www.slate.com/blogs/weigel/2012/09/24/today_on_the_great_pointless_crippling_candidate_gaffe_beat.html

]]>Note that increasing the size of your study does not protect you from this, as Nick alluded to. You increase the ability to notice a distance, but your failure rate will always be 5%.

Note also that this does not tell you the probability that there is a difference. The probability that two treatments are different given that you have a significant result is given by bayes theorem, and is a bit long winded to give here, but is critically dependent on the probability of the treatments being different, which you don’t know apriori. With the identical sugar pills the probability is of course 0, as we know that they aren’t different, so no manner of tests will persuade us otherwise.

Note also that in practice, as we don’t deal with identical pills, we simply wouldn’t expect things to be identical, so a large enough sample size will find a significant difference. Most drug trials do focus on a clinically significant difference, which is meant to be based on some meaningful amount.

Note also that this is based on the mean difference between two pills, so that itself can be misleading if the data has interesting properties.

On a related note, the significance test assumes that the data is normal (usually), which is never actually true in practice (although often true enough to get away with it)

Note finally that, following from that note earlier that we will spot a difference where there is none 5% of the time, we can discover differences in data by looking enough. This is often what happens in studies where we are told that food x causes y, as given a large enough data set I can always find some signficant relationship.

So a few things to note the next time you hear about a significant result. Relying on statistical results alone to demonstrate efficacy is dangerous without lots of repeated studies.

]]>It is impressive how much of a difference tone of a person providing arguments makes. During the placebo conversation the only thing that I was able to concentrate on, was the fact that Nick sound like a freaking dalek when he is agitated, and John has an impressive control over his emotions.

]]>