The New AI That Reads Your Brain Activity and Generates It
For years, decoding brain activity meant picking one direction: you could either read what someone was seeing from their fMRI scan, or you could predict brain activity from an image. Never both in the same system. A team of researchers from Tianjin University, Shanghai Artificial Intelligence Laboratory, and the Chinese University of Hong Kong just tore down that wall. Their model, BrainJanus, reads brain signals and generates them, all within a single framework.
Background & Context
The relationship between AI and neuroscience has always been a two-way street. The brain inspired neural networks, and now neural networks are giving back. For the last decade, researchers have gotten good at one specific trick: decoding what a person is seeing from their fMRI brain scan. Show someone a picture of a giraffe, record their brain activity, and an AI can reconstruct a rough version of that image or describe it in text.
But there’s been a fundamental limitation. Existing systems treat brain reading (decoding) and brain writing (encoding) as completely separate problems with separate models. You would train one model to go from fMRI to an image, and another entirely different model to go from an image to fMRI. That is not how biology works. The human brain is a single multimodal system. When you see a giraffe, your visual cortex fires, but so do semantic and language regions. A picture triggers concepts, words, memories, all at once. Previous AI models missed that interplay entirely.
They also relied heavily on external crutches. Models leaned on frozen CLIP embeddings or pretrained diffusion models to fill in semantic gaps, because the brain signals themselves were not being modeled richly enough. Think of it as translating a language by only catching every third word and guessing the rest.
What the Researchers Did
BrainJanus tackles this with an approach borrowed from language modeling. The core idea: treat brain activity as just another language.
The team built what they call a Unified Brain Tokenizer, a VQ-VAE model that converts continuous fMRI signals into discrete tokens, similar to how text tokenizers break sentences into words or how image tokenizers break pictures into patches. These brain tokens live in the same “Omni space” as visual tokens and text tokens. Once everything speaks the same token language, a single autoregressive Transformer, based on the Janus-7B architecture, can handle all four directions of communication.
The architecture works in two stages. First, the brain tokenizer is trained from scratch to compress more than 15,000 voxels of fMRI data into a compact sequence of 128 discrete codes, using a codebook of just 128 possible tokens. Then, the Transformer backbone is fine-tuned on a mix of four task types simultaneously (image-to-brain, text-to-brain, brain-to-image, and brain-to-text) so the model learns shared representations rather than task-specific shortcuts.
| Task Direction | BrainJanus | Best Previous | Metric |
|---|---|---|---|
| Brain→Text (COCO captions) | 63.20 BLEU-1 | 61.75 (MindLLM) | Higher is better |
| Brain→Text (detailed captions) | 40.21 BLEU-1 | 9.53 (MindLLM) | Higher is better |
| Brain→Image | 93.5% CLIP score | 92.5% (UMBRAE) | Higher is better |
| Brain→Image | 88.2% AlexNet(2) | ~86% (prior best) | Higher is better |
What They Found
The results are striking across the board. On brain-to-text decoding, BrainJanus generates captions that are not just statistically better; they are qualitatively richer. Where prior models wrote “a giraffe standing in a field,” BrainJanus wrote “a tall giraffe with brown spots stands on grass near a tree trunk, with a wooden fence and green trees in the background.” The model captures fine-grained visual details (colors, spatial relationships, background context) that previous systems missed entirely.
The improvement is especially dramatic when evaluating against detailed captions generated by Qwen3-VL. Previous models collapsed: MindLLM scored just 9.53 BLEU-1, barely better than random. BrainJanus scored 40.21, a 4x improvement. In human-readable terms, older models mostly described the single most prominent object. BrainJanus describes the scene.
For brain-to-image decoding, the model produces reconstructions that preserve global structure and semantic content. CLIP similarity reached 93.5%, competitive with specialized image-reconstruction models. The generated images correctly reflect object categories and spatial layouts, though fine-grained details like exact texture or lighting still lag behind the original stimuli.
On the encoding side (predicting brain activity from images or text), BrainJanus generates fMRI signals that preserve interpretable cortical topography. The synthesized brain activity maps look structurally consistent with real fMRI data, and images reconstructed from those predicted signals retain recognizable content. This is the first time a single model has demonstrated this bidirectional capability.
Why It Matters
Brain-computer interfaces have always faced a chicken-and-egg problem. To decode brain signals accurately, you need good encoding models that tell you what brain activity looks like for a given stimulus. To build good encoding models, you need to decode brain signals accurately. BrainJanus breaks this cycle by learning both directions together, in a shared representation space. The model’s ability to do zero-shot generalization (performing well on subjects and tasks it was not explicitly trained for) suggests the tokenized brain representation captures something fundamental about how neural activity encodes meaning.
There is also a practical win here. Previous brain decoding systems required hours of subject-specific fMRI data. BrainJanus’s unified tokenizer reduces this barrier. The tokenizer learns cross-subject patterns, so a model trained on six subjects can generalize to two new ones with minimal additional calibration.
How It Could Change Our Lives
Right now, BrainJanus works on 7T fMRI machines (million-dollar scanners in research hospitals). But the trend in brain-computer interfaces is toward portability. If this tokenization approach can be adapted to cheaper modalities like EEG headsets or functional near-infrared spectroscopy, the implications are direct. A stroke patient unable to speak could generate text from their brain activity. A paralyzed person could describe what they want to see on a screen and an AI could reconstruct it.
The bidirectional capability also opens a stranger possibility: targeted brain stimulation. If you can generate fMRI patterns from text descriptions, you could theoretically design visual stimuli that produce desired brain responses. Imagine calibrating a visual prosthetic by first generating the brain signature you want, then working backward to create the image that produces it.
The Bigger Picture
BrainJanus sits at the intersection of two accelerating trends. One is the unification of multimodal AI (models like Chameleon, Emu3, and Janus-Pro that handle text, images, and audio in a single architecture). The other is the maturation of neural decoding, driven by large-scale fMRI datasets like NSD (9,000 images across 8 subjects). By bringing these trends together, BrainJanus starts to look less like a specialized neuroscience tool and more like a prototype for a general-purpose brain interface.
Limitations & What’s Next
The paper is honest about what it cannot do. The fMRI data comes from the Natural Scenes Dataset, which only covers visual perception. BrainJanus cannot decode thoughts, memories, or imagination, only responses to images people are actually looking at. The brain tokenizer uses a fixed codebook of 128 entries, which may limit expressiveness for more complex neural patterns. And the model currently works best on subjects it trained on; cross-subject zero-shot performance, while present, still lags behind within-subject results.
The authors also uncovered an uncomfortable finding about evaluation protocols. Existing metrics for evaluating brain encoding (measuring how well a model predicts fMRI activity from images) are vulnerable to a “hacking” strategy where models achieve perfect scores by essentially storing the visual embedding in the voxel space rather than learning real neural mappings. They propose stricter evaluation standards that future work will need to adopt.
📄 Source: Haitao Wu et al., “BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language.” ICML 2026. arXiv:2606.30319