Not exactly mind-reading

28/09/2011 by admin.

I don’t pretend to understand the computations that have been used in this study, only the general idea. The results are both a lot more and also a lot less than they appear. This is a group that have been able to fairly accurately identify a black and white photo that a subject in a fMRI scanner is looking at. They now attempt to identify a short movie clip that is being watched. There is an enormous problem here because the fMRI signal is associated with blood flow and is too slow to keep up with moving images. They manage to overcome this problem of speed with some mathematical cleverness which I don’t understand. This gives a coded output of brain activity for a particular short movie clip (not the original scan but a derived coded one).

Having this coding method, they used it in a big way. For each subject, many thousands of such short clips were viewed and the coded fMRI scan for each clip-subject combination was stored. The subjects than view a target clip, not used in the previous scans. This scan is coded and then compared with the enormous bank of coded scans from the library of clip-subject-coded scans triplets. The 100 clips with the most similar coded scans are averaged to give a composite movie clip. This tends to be fuzzy but with a resemblance to the target clip. (link to clips)

The videos of the composite movie clips are somewhat misleading if one is unaware of how they were constructed. They are colored because the clips are coloured, but no colour information was included in the coding process. The colour is an artifact of combining clips. The colour has the effect of enhancing the impression that actual qualia are being extracted from the scanner - a very, very misleading impression that is hard to shake.

Of the video comparisons published, the most successful are people that are not moving much. This may also be an artifact of the clips that were used to produce the library. It may also be a result of the process. I have the notion that movement is not being captured very accurately unless it is slow and involving large rather than small objects/elements.

Don’t miss a look at the clips if you have not done so already. (there is a link above) This is not mind-reading but it is an definite achievement. Those interested in the mathematical nitty gritty should read xcorr’s posting.

Nishimoto, S., Vu, A., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. (2011). Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies Current Biology DOI: 10.1016/j.cub.2011.08.031

Posted in Uncategorized | 4 Comments »