tag:blogger.com,1999:blog-5427587583898590293.post8110444793280798203..comments2023-10-31T03:41:34.562-07:00Comments on Brad Buchsbaum's Blog: Simulating Voodoo Correlations: How much voodoo, exactly, are we dealing with?Brad Buchsbaumhttp://www.blogger.com/profile/10757537675625801119noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-5427587583898590293.post-88501682227067317312009-03-09T03:10:00.000-07:002009-03-09T03:10:00.000-07:00It doesn't necessarily involve two hypothesis test...It doesn't necessarily involve two hypothesis tests but it does involve two different "steps" - albeit only the first step is a formal statistical one.<BR/><BR/>In fact the whole Vul controversy is an excellent example of the fact that it's often the informal aspects of statistics that are the most problematic. You can get all your sums formally correct and still mislead, even unintentionally.Neuroskeptichttps://www.blogger.com/profile/06647064768789308157noreply@blogger.comtag:blogger.com,1999:blog-5427587583898590293.post-39611669822583831392009-03-05T18:49:00.000-08:002009-03-05T18:49:00.000-08:00thanks for the comment, neuroskeptic!Yes, you're a...thanks for the comment, neuroskeptic!<BR/><BR/>Yes, you're absolutely right about Vul's hypothetical response to these simulations. They do not actually get to the heart of the matter. More sophisticated simulations are needed for that. As I mentioned in the second post, I thought the "non-independence error" referred to cases involving two hypothesis tests when in fact it does not require that at all.<BR/><BR/>On the other hand, if reporting correlation effect sizes in whole-brain analyses is at all justified (and many argue it is) then it will be helpful to know how these estimates vary as a function of the summary measure used and cluster size. At the very least, it seems that reporting the mean of median correlation is a better option than the peak.Brad Buchsbaumhttps://www.blogger.com/profile/10757537675625801119noreply@blogger.comtag:blogger.com,1999:blog-5427587583898590293.post-2884782807426511122009-03-04T01:29:00.000-08:002009-03-04T01:29:00.000-08:00This is fascinating stuff. I also suspected that t...This is fascinating stuff. I also suspected that the magnitude of the inflation (the "voodoo factor" if you like) would be related to the number of voxels selected, but I didn't realize that it also depends upon the size of the cluster itself. And obviously I was too lazy to actually do any simulations. Nice work.<BR/><BR/>I know what Vul's response to your arguments would be, though. <BR/><BR/>He'd say that while the degree of inflation over and above the median correlation in the selected cluster may be minor, the more serious source of inflation is the selection of the cluster itself.<BR/><BR/>Assume that the truth is that activity in the whole anterior cingulate is correlated with behaviour with median r=0.2. You search for a cluster where median r=0.6. You find one, somewhere within the anterior cingulate, and report it. But that cluster represents a chance finding.<BR/><BR/>Of course, correcting for multiple comparisons is meant to guard against such chance findings. But it's designed to guard against chance findings under the null hypothesis of no true effect (noise normally distributed with a mean of zero).<BR/><BR/>If there is a true effect, albeit a small one, then the likelihood of a chance finding of a large effect is much higher than under the null.<BR/><BR/>In other words, if you find a cluster after correcting for multiple comparisons, you can be 95% confident that there is some true effect there <I>but not that confident about the size of the effect</I>.<BR/><BR/>Such, at least, is my understanding.Neuroskeptichttps://www.blogger.com/profile/06647064768789308157noreply@blogger.com