Stanford's Memorilla gives a frozen LLM memory in 16 tokens

Researchers at Stanford University, with colleagues at the University of Toronto, the University of Pennsylvania and MIT, report a memory system called Memorilla that lets a frozen language model draw on a whole collection of documents through 16 learned 'memory tokens' instead of pasted text. The preprint (arXiv 2610.21375, version 3 dated 3 November 2026) says Memorilla beats the memory and retrieval baselines it was compared with on average, while giving the model far less to read.

How it works

Each document, such as a user's past chat or a book chapter, is encoded once by a frozen embedding model into a persistent store. For each question, a small trainable module reads every stored embedding and compresses what matters into 16 vectors placed in the model's input. Per the paper, the documents never enter the prompt as text, and the language model itself is not retrained. With a Qwen3-8B decoder, the trainable part has about 194 million parameters. New documents can be added without updating the language model.

What the authors measured

The authors tested eight benchmarks covering personalization, long-document question answering and knowledge-intensive questions. They report the best result on five and second place on two more, and a mean score of 49.68 against 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0. On average it used 394 times fewer document tokens than full context and 113 times fewer than retrieving 50 chunks.

On FactKG, a fact-checking benchmark, Memorilla scored 84.33% against 80.43% for full context and 80.38% for MemoRAG. In a 512-document test on one H100 GPU, the authors report 0.74 seconds end to end against 19.9 seconds for answering with full context.

In an agent test in the TextWorld game, a frozen Qwen3-4B agent with the memory trained by reinforcement learning raised its average success rate from 18.6% to 24.2%. The rate at which the agent exhausted its 4,096-token budget fell from 71% to 58%. An untrained memory module did worse than no memory (15.3%), so the gain came from training.

What it leaves out

The authors list their own limits. Memorilla can lose fine-grained facts, which they attribute to an encoder not built for compression. They tested only a Qwen encoder with a Qwen decoder, and say transfer across model families is untested. They also write that it may not carry over directly to domains such as coding. All results are the authors' own, on public benchmarks and a game, with no production workload. The memory module also needs training for each setup, which plain retrieval does not.

What a team can do with it

Nothing here ships as a product. The useful part is the comparison: if your agents or assistants resend long histories on every call, you are paying for those tokens and that latency each time. The paper is a reason to measure how much of your history the model actually needs, and to keep retrieval versus compressed memory as a testable choice rather than a default.