Long-term memory

What was covered

• Why memory matters. An LLM keeps no state between sessions. Without memory, intelligence resets to zero each time, and real agents need continuity of preferences, past actions and goals.
• Four types of memory. In-context (the prompt window: fast and accurate, but ephemeral), in-weights (baked in by training), in-cache (reusing KV computation in a session) and external retrieval, which is the main long-term memory in production.
• The context window problem. Attention cost grows quadratically. Content in the middle of a long context is retrieved worse (“lost in the middle”). Million-token models complement retrieval and do not replace it.
• RAG. Encode, store, retrieve, generate. The design decisions that matter are the chunking strategy, the embedding model, and hybrid search with reranking.
• Memory in agents. Episodic (what happened), semantic (facts) and procedural (how to do things) memory, wired into an observe, store, retrieve, think and act loop.
• Letta. An operating-system-inspired design: the context window is RAM, external storage is disk, and the model decides what to page in and out through function calls.
• Fine-tuning versus retrieval. Fine-tuning changes behaviour deeply but risks catastrophic forgetting and is static. Retrieval is dynamic and auditable but adds latency and depends on chunking quality.
• Challenges. Silent retrieval failures, conflicts between retrieved and parametric knowledge, consistency across agents sharing memory, and privacy and data liability.

Takeaway

Memory is architecture, not a feature you bolt on. RAG dominates today, and the open problem is deciding automatically what is worth remembering.

No meeting notes were captured for this session, so this page is built from the slides.