A "mind-reading" AI reconstructs what you are looking at from a brain scan

A team at the Weizmann Institute of Science in Rehovot, Israel, led by computational scientist Michal Irani, has developed an artificial intelligence tool capable of reconstructing, in remarkable detail, the image a person is looking at using only their brain scan from functional magnetic resonance imaging (fMRI). The system also works in reverse: given an image, it can predict what a person's brain activity would look like while viewing it. The results were presented last month at the Cognitive Computational Neuroscience conference held in New York.

fMRI works by using a powerful magnet to track the flow of oxygenated blood in the brain: the areas that "light up" on the scan are considered the most active at any given moment. Typically, each "voxel" of activity detected by an fMRI scanner covers about three cubic millimeters, which encompasses around 16,000 neurons — a resolution too coarse to capture fine detail. Irani and her team instead turned to more recent datasets, obtained with higher-resolution scanners, in which each voxel covers roughly one cubic millimeter of neurons.

Earlier attempts to reconstruct images from brain scans have been under way for years, but according to Irani the previous results left much to be desired: if a person was looking at a banana, the model might generate an image of a banana, but without the same structure or the same position as the original. To solve this, the team trained a "brain decoder" with two distinct branches: one predicts the structure of the image (for example, where the colors sit) and the other predicts its content (for example, a bunch of bananas on a plate). Both predictions then feed into a diffusion model — the type of AI known for generating images and video by progressively cleaning up a mass of noisy pixels — which produces a far more faithful representation of what the person actually saw.

The initial training used data from eight people who had each been shown some 9,000 images while inside a high-resolution fMRI scanner. Since that amount of data was still insufficient, the researchers additionally trained a second model, an "encoder" capable of predicting what brain activity would look like given any image. Encoder and decoder were then trained jointly: starting from a new image — a leopard, for instance — the encoder predicts what a person's scan would look like while viewing it, and the decoder tries to reconstruct the original image from that prediction. At first the result barely resembles a leopard, but repeatedly training this loop eventually produces notable improvements. Thanks to this method, the team was able to train its models on far more images than were originally available: according to Irani, around 70% of the training data comes from images that were never actually paired with an fMRI scan. By combining data from several different studies, the researchers also identified brain regions that appear to share a function across different people: one region would respond, for example, to images of food, and another to images of sport.

One of the most significant practical consequences is that the resulting "universal brain encoder" can be applied to a new person with minimal calibration: while other previous attempts required some 40 hours of fMRI data from a subject before they could predict what that person was seeing, Irani's decoder needs only one hour. Neuroscientist Tommy Sprague, of the University of California, Santa Barbara (not involved in the study), stresses the practical value of that saving: no research project can afford 40 hours of imaging per subject, at a cost of between $600 and $1,000 an hour, so a tool like this could considerably accelerate neuroscience research. In comparative tests, the system outperformed previously described tools "by a significant margin," according to Irani, although the researcher herself acknowledges failures: she showed, for example, an image of a cake that her tool reconstructed as a stack of three sandwiches, and an image of a dog in a bathtub that became a similarly colored goat in a bathtub. Irani adds that "mind reading" is, in her own words, "a flashy, catchy name" for what they are really doing.

Neuroethicist Judy Illes, a professor of neurology at the University of British Columbia in Canada and unconnected to the study, called the work "magnificent" and said she was especially excited by the idea of applying this approach therapeutically in people with neurological conditions. Irani now plans to go beyond still images to reconstruct video and audio as well, with the goal of representing what people think or imagine, and even the content of their dreams — something she admits they have not yet achieved, but are actively pursuing. Among the applications she is considering are helping people with locked-in syndrome, who are completely paralyzed, to communicate using only their brain activity, and helping science unravel persistent mysteries about how the mind works, such as what post-traumatic stress disorder flashbacks look like.

These advances inevitably reopen the question of mental privacy. Sprague admits that if this possibility had been put to him ten years ago, he would have laughed: getting a willing person to stay still in a scanner and actively cooperate with a research question is hard enough, so imagining someone doing it involuntarily seemed like science fiction. But Irani and other researchers are working on similar approaches to decode brain activity from electroencephalography (EEG) — measurements of electrical activity collected via a cap of electrodes, or even through headphones — and, as the models improve, it will become increasingly easy to analyze brain activity gathered this way, which is cheaper and more accessible than fMRI. That is why Sprague argues that "the ethical considerations need to be taken a bit more seriously," and he believes Irani's approach would probably work quite well at predicting images a person is simply thinking about, without needing to be looking at them.

Neuroscientist and philosopher Marcello Ienca, of the Technical University of Munich in Germany, considers the leap to EEG a "game changer": once an EEG device is calibrated to a particular user's brain, it could be relatively straightforward for companies to extract additional information from that person, potentially without their consent. Ienca also imagines that some courts could come to admit reconstructions of mental images as legal evidence, and warns that, while he does not doubt this is well-intentioned research, it is clearly something that could lend itself to ethically and socially problematic commercial uses. Irani, for her part, acknowledges the potential for misuse that EEG entails, but says she is not worried for now: "I try to think only about the good things," she says.

Taken together, the work represents a clear technical advance over previous attempts to "read" images from brain scans, both in the fidelity of the reconstructions and in the drastic reduction in the calibration time required per person. But the team itself and the experts consulted agree that this technical progress is advancing in parallel with unresolved questions about consent, mental privacy and the limits of what should be extractable from someone's brain activity, especially if the technology migrates from fMRI — expensive and impractical outside the lab — toward far more accessible EEG devices.