MemoryATHENA: Adaptive Routing over Latent and Generated Memories

Abstract

Learned-memory methods store information in an explicit table and consume it through a separate reader, allowing addressing, storage, and reading to be modified independently. Prior work on cross-model memory transfer exploits this separation to reuse learned memory across frozen backbones through an adapted reader, while the representation consumed by the model still originates from stored memory. This raises a natural question: must useful memory always be retrieved from storage, or can it also be generated? We investigate this question with MemoryAthena, a memory interface with three pathways: direct Engram retrieval (E), generation from retrieved Engram cues (GE), and generation from causal backbone states without consulting the memory table (GH). Generated memory is not uniformly better than direct retrieval: it can complement E in one context but interfere with it in another. MemoryAthena therefore treats E as an explicit anchor and learns when a generated representation should intervene. With the backbone, memory, generators, and readers frozen, a lightweight causal routing head is trained from counterfactual future-token likelihood advantages of GE and GH relative to E. At inference time, an admitted candidate modifies the E residual through bounded interpolation, while rejection recovers the direct pathway exactly. On question answering, MemoryAthena raises the five-task average from 37.65 to 39.28 over the direct pathway of the same checkpoint, while the six-task general-NLP average increases from 76.73 to 79.13. The complete memory-side system contains approximately 201M parameters, excluding the frozen backbone. Further analyses show that the utility of E, GE, and GH varies across tasks and inputs, while gold-label oracles reveal additional complementarity among the three pathways. These results support generated memory as a selective correction to direct retrieval rather than a universal replacement, and highlight routing when, which, and how strongly to intervene as the central challenge.

Methods

Direct and generated memory pathways

MemoryATHENA keeps direct Engram retrieval as the reference while adding two generated-memory pathways. The generators construct target-side memory proposals from different causal sources, and the router learns when either proposal should modify the E pathway.

• AddressingStable memory lookup

  The Engram memory remains frozen and addressable, preserving a shared external artifact across target-side reader experiments.

• ReadersTarget-aligned interfaces

  Direct, Engram-conditioned, and hidden-state-conditioned readers expose complementary memory views to the target backbone.

• GenerationConstructed memory candidates

  GE generates memory from retrieved Engram cues, while GH generates memory from clean causal backbone states without consulting the memory table. Both are candidate corrections to E, not unconditional replacements.

• RouterCounterfactual supervision

  The router is trained from future-token likelihood advantages of GE and GH relative to E, not from downstream task labels.

Research questions

Experiments organized around three questions

• RQ1When does generated memory improve a strong direct-memory pathway?

  We compare E, GE, GH, alternative routing rules, and cross-backbone transfer to test when generated memory provides a useful correction to direct retrieval rather than a universal replacement.

• RQ2What drives the gains from the memory interface?

  Ablations separate the contributions of the learned memory interface, its initialization, and the learned routing policy, including architectural controls, from-scratch memory, and a random-router control.

• RQ3How does MemoryAthena use complementary memory pathways?

  Downstream routing analysis and a label-informed source oracle study how E, GE, and GH contribute, and how much complementary headroom remains beyond the deployed router.

Main results

Results across QA and general NLP

Headline aggregates from the current paper-facing audit. Values are percentages and use one training seed (42). The tables keep protocol caveats visible instead of collapsing unlike baselines into one claim.

Five-task open-domain QA

Open-domain QA cells are EM/F1; TruthfulQA cells are MC1/MC2/MC3/mean.

The standalone Engram-only row and joint-checkpoint rows have different training histories; the transfer row is reported separately.

Six-task general NLP

Accuracy (%). E/router rows use the next-token synonym-sum dCPMI protocol; the historical Vanilla row uses a different full-choice scorer.

Yahoo τ=1.0 is a test-set threshold sweep; the selected threshold is shown explicitly for transparency.

Key findings

What the experiments show

• 1. Generated memory is useful, not universal

  GE and GH can complement direct retrieval, but neither uniformly dominates E. The router keeps the strong path available on inputs where generation would interfere.

• 2. The reader is part of the transfer

  A frozen memory becomes useful on a new backbone through a target-side reader aligned to that backbone, rather than through the table alone.

• 3. Selective routing is the central mechanism

  The gains come from bounded intervention around a strong direct-memory reference, with exact fallback when generated proposals are not sufficiently useful.

Resources

Read, reproduce, inspect

The public release separates code, model artifacts, and compact experiment metadata. Large scratch paths and raw task data are intentionally not part of the web page.

BibTeX

@misc{li2026memoryathenaadaptiveroutinglatent,
  title = {MemoryAthena: Adaptive Routing over Latent and Generated Memories},
  author = {Mingyuan Li and Guangsheng Yu and Juyuan Zhang and Xu Wang and Zhibo Man and Haonan Zhang and Shaoxiong Ji},
  year = {2026},
  eprint = {2609.25853},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2609.25853}
}