8 — Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates

If you're into neuroimaging, check out "Cluster failure.." Eklund et al. 2016. ... caused quite the stir...

— Michael Clark (@mikeclarkmgc) August 16, 2017

fMRI is a brain-imaging modality that uses oxygen-movement as a proxy for water-movement in the brain as a proxy for blood-movement as a proxy for which brain areas are undergoing heightened levels of activity as a proxy for which brain areas are active during a given time.

I have a hard time trusting fMRI.

And since 2016 (following this paper in particular), the neuroscience community has, to its immense credit, been far more aware of the risks of clusterwise inference than previously.

To wit: There’s a lot of literature that demonstrates fMRI’s usefulness for a lot of really important things, and so you shouldn’t discount it right off the bat… But then again, this article mentions a case in which they found a 70% false-positive rate in a study, under very common data-science practices. This is right in the first paragraph and serves — at least in my opinion — as a pretty cautionary tale when dealing with fMRI. But I digress.

This paper investigates the ways in which cluster-based inference — that is, collecting a bunch of voxels and evaluating them as a brain “area” — can condemn you to statistical failure. This is in contrast with voxelwise inference, in which each voxel of the brain is treated as an independent variable.

The takeaway from this paper — which is enjoyable to read even if you’re afraid of statistics — is that clusterwise inference in fMRI studies is both highly pervasive in the field, and also very likely to be un-trustable, with wildly inflated p-values.

And furthermore, the paper notes how rarely fMRI datasets are archived and stored for future re-study; it’s very likely, according to the authors, that much (40% or more!) of the existing fMRI literature is not quite as statistically significant as we’d all like to believe, and will never be updated or edited.