For a little while there having been questions about the statistical processing of fMRI scans. These play along side less sensible questions. We have:
-
It is no clear exactly what causes the changes measured with fMRI and so the results due not have meaning.
-
The way scan results are reported in the press as ‘areas for x’ is misleading.
-
There are problems with the significance levels for large numbers of statistical tests.
-
There are problems with the independence of statistical tests.
The first two are red herrings. We do not have to know exactly how an effect is caused as long as we have a general idea and correlation with other effects. The scans do have meaning changes in the amount of neural activity. What way the popular press report research has nothing to do with the validity of the work. Calling new things ‘miracle drugs’ does not have any effect on how well or badly they may treat a disease.
The statistical problems were reviewed recently by C. Chatham at Developing Intelligence (here). He reviews the Dead Fish experiment of Bennett and the Voodoo Correlation paper of Vul. Here is his conclusion:
To summarize, the dead fish study is a point about first-pass analysis, which almost every paper I’ve ever seen does correctly. The papers that don’t always note that the result failed to pass multiple comparisons or cluster correction, and typically discuss those results with caution. On the other hand, “voodoo correlations” is a point about nonindependence in statistical tests. This has not always been done correctly, and has not always been reported clearly. Moreover it primarily affects only a subset of correlations between brain and behavior - and not the vast majority of work in fMRI, which has to do with task-brain relationships.
I believe that a good protection against being taken in by reports of research that turn out to be shoddy is to be very, very slow to accept or reject any single result. What convinces me is not any particular chain of logic or single result but a fabric of results by various people, with various methods and various view points. When such a fabric builds up, then it can be accepted.