Filesystem Persistence for Long-Running AI Agents
Decoupling storage from compute keeps agent work alive, but files alone need upkeep.

That is the failure this piece is about, and the fix is a storage layer that does not share a lifecycle with the compute running on top of it. Ephemeral sandboxes bind the file layer to the container or function that spun it up, so a crash, a timeout, or a routine scale-down event wipes out hours of accumulated work with no warning and no way to get it back. Decoupling storage from the compute lifecycle solves this directly: a persistent filesystem lets a second runtime, whether it is the same agent resuming or a different one picking up the thread, reopen the same workspace exactly where the last one stopped. An agent can start a task in the morning, and anyone or anything can resume it later with full access to every file it produced; a stateless design has no real counterpart to that, and it is the starting condition for everything that follows in this piece.
Persistent state across an agent's full tenure
Treating persistence as "don't lose the chat log" badly undercounts what an always-on agent actually carries. The operative system behind a long-running agent includes retrievable memories, but also task ledgers tracking where a multi-step job currently stands, permissions describing what the agent may do and on whose behalf, tool and credential state such as live API keys, OAuth tokens, and service configs, commitments, provenance and audit records, shared state across collaborators, trigger conditions, and the record of effects the agent has already committed to the outside world. Each of these has its own persistence requirements, and they do not collapse into a single storage pattern: a credential needs rotation and scoped access, a task ledger needs to be queryable mid-execution, and a record of an external commitment, an email already sent, a payment already issued, cannot be undone simply by resetting the agent's memory. Developers who plan for "memory" as one undifferentiated bucket tend to build systems that handle conversational recall reasonably well and then fail quietly on everything else, because credentials expire without renewal, task ledgers go stale without anyone noticing, and permission state drifts out of sync with what the agent is actually allowed to do. The scale of the problem is not static, either. Measured task horizons for AI agents have been doubling roughly every six months overall, accelerating to roughly every four months since 2023, and at the far end of that curve, one continuous effort, one codebase, months of elapsed time, session boundaries beyond counting, the failures that occur are not reasoning failures but memory failures. If an agent can sustain a task for months, its state model has to be built for months, not for a single sitting and then stretched past its design.
The filesystem as the default long-term memory medium
Given that broader state requirement, the medium that deployed agents have converged on is unglamorous: a directory tree of markdown files that the agent itself reads, writes, and reorganizes through ordinary file tools. That convergence happened in practice before research caught up to it, and the working assumptions behind it remain largely untested. The case for the filesystem rests mostly on interoperability. Files are the one format every tool in a technology stack already understands: a CSV an agent writes can be opened straight away in a spreadsheet, picked up by a separate automation, or handed to a human collaborator, while a vector embedding stays opaque to anything outside the system that generated it. Five operations define what an agent actually needs from a storage backend day to day: reading and ingesting files such as PDFs, DOCX, MP4, and JSON pulled from remote sources, writing and creating new files at a persistent path, listing and traversing a directory to see what exists, organizing and moving content to keep a project's structure sane, and sharing or transferring work by generating links or adjusting permissions. The filesystem also does something prior memory architectures struggled to do cleanly: it holds declarative memory and procedural skill in the same store, with one agent integrating and organizing incoming content, a search agent answering questions with cited sources pulled from that store, and an execution agent contributing task trajectories that get distilled into reusable skills over time. None of this makes the filesystem a finished solution. In growth studies, store organization degrades for all but the strongest management agents, and no measured agent has shown that better organization on its own produces better answers. Most teams chose the filesystem by default; no one has fully solved it as a problem.
What organized filesystem memory buys and costs
The clearest payoff from keeping a filesystem well-organized is search economy: in studies with large stores, an organized filesystem roughly halves retrieval cost compared to a disorganized one. That benefit is about efficiency, not accuracy, and the distinction carries a sharper implication than it first appears to: a disorganized store does not simply get more expensive to search as it grows, it can actively make the agent perform worse, because the agent still has to find the right file before it can use it. Left unmanaged, a filesystem accumulates specific, identifiable failure modes as an agent's tenure lengthens. Context compaction summarizes away the reasoning behind past decisions, so the agent is left holding a conclusion it can no longer justify or safely revise. Documentation inside the store goes stale with no error message attached to the process, so the agent keeps acting on outdated facts with the same confidence it would have in current ones. And as the store grows, all but the most capable management agents lose the battle to keep its structure in a state that makes retrieval cheap. The implication is straightforward: persistent agent memory functions as infrastructure, and infrastructure needs the ordinary machinery of reliability engineering, health checks, heartbeats, freshness targets, and a real cost model for alerting, not just a write path that assumes everything saved stays trustworthy forever. One more detail belongs here, because it is easy to overlook: the tool harness an agent uses shapes how its store ends up organized just as strongly as the underlying model choice does, which makes tool design a lever worth taking seriously rather than treating it as a secondary concern beneath model selection.
Provenance on stored memories
Retrieval and durability solve half the problem. An agent also needs to be able to audit, update, and erase what it has stored, and systems that flatten source evidence, extracted facts, retrieved context, and answer policy into a single opaque prompt path make that kind of inspection effectively impossible. The fix is a specific ordering discipline: raw conversational evidence gets stored before any fact derived from it becomes canonical memory, that immutable evidence is preserved first, extracted facts are checked against it before they are promoted to canonical status, and canonical facts stay linked back to their source so they can later be inspected, corrected, or deleted. Without that chain, a wrong answer from a memory-backed agent cannot be diagnosed, since the error could sit in the original evidence, in the extraction step, in retrieval ranking, or in the answer model itself, and a single end-to-end accuracy score gives no way to tell which one. You can see what happens without this discipline most clearly in heartbeat-driven background execution, where an agent checks external sources on its own schedule, independent of direct user conversation. Misinformation picked up during one of those background checks can enter the agent's short-term session context, get written into long-term memory through nothing more unusual than the agent's ordinary save behavior, and then shape what the agent does in a completely separate, later session. In a study of Claw agents, routine memory-saving behavior pushed short-term pollution into durable long-term memory at rates up to 91%, and this behavioral influence showed up across session boundaries. No prompt injection or adversarial attacker is required for this to happen. If background execution and foreground conversation share a session with no attribution attached to what gets written into the persistent store, ordinary, non-malicious misinformation is enough to quietly reshape an agent's memory and future behavior. Provenance is the design response to that fact: it guards not against a hypothetical attacker but against the agent's own routine operation.
Matching the compute layer to the storage model

Persistent storage changes the security calculus of the whole system, because an agent whose filesystem persists across sessions but runs inside a shared compute boundary creates a path for one agent's compromise to reach another's data. Persistence and isolation have to be designed as one decision, not two separate ones made by separate teams. The heartbeat vulnerability described above is itself a direct product of a shared-session design: background checks run in the same session as the user-facing conversation, so anything ingested from a monitored external source lands in the same memory context the user's own conversation writes to. The fix for that is not a patch to the memory-saving logic. It requires a harder boundary between the execution context that runs background checks and the one that runs foreground conversation. Isolation options sit on a real spectrum. Docker containers share the host kernel, so a container escape exposes the entire host along with every agent's persistent disk sitting on it. gVisor interposes a user-space kernel that blocks direct host syscalls, which closes off that escape path but also blocks GPU passthrough. Firecracker microVMs give each workload its own guest kernel behind a hardware virtualization boundary, a categorically stronger guarantee than either container-based approach offers. What makes Firecracker usable at agent scale is that this guarantee comes without the cost people associate with traditional VMs: it boots in under 125 milliseconds, adds minimal memory overhead per microVM, and a single host can launch many of them per second, which is what makes per-agent isolation economically workable rather than a theoretical nicety. GPU access survives the tradeoff, too. Firecracker's hardware virtualization path supports VFIO device passthrough directly to the microVM, giving agents real GPU access with near-native performance, something gVisor's user-space kernel cannot offer. Persistent state makes the agent valuable enough to protect; provenance protects it from corrupting itself; isolation protects it from a breach originating somewhere else. All three have to hold at once.
Snapshot-restore and the economics of persistent VM storage
The natural objection to this is cost: running a dedicated virtual machine per agent sounds like it multiplies compute spend by however many agents are deployed. That objection assumes agents are busy most of the time, and most are not. Most agents sit idle for the large majority of their existence, waiting on the next task, the next heartbeat, or the next message from a user, so billing them as if they were continuously running compute charges for work that is not happening. You should charge for storage while an agent is idle and for compute only while it is actively executing. Snapshot-restore is the mechanism that makes this practical: a VM boots once, its full memory and device state, including the complete persistent filesystem, gets frozen to disk, and that frozen image can then be thawed on demand instead of paying the cost of a fresh kernel boot every single time the agent wakes up. Making that restore fast enough to be unnoticeable relies on a specific piece of engineering: userfaultfd, or UFFD, memory streaming lets a userspace handler service page faults against a remote snapshot, so a VM can start executing before its entire memory image has finished loading locally. One limit holds regardless of how good the snapshot engineering gets: no snapshot technique restores running processes, open network sockets, or other purely in-memory state, so whatever reached disk is what survives, and the persistent filesystem, not the memory snapshot, remains the actual recovery point an agent depends on. The economic payoff of getting this right is direct: a per-user sandbox that sits idle for most of its life costs storage rather than CPU, and that arithmetic is what turns deploying one isolated agent per user into a financially sound default instead of a cost multiplier the business has to absorb.
Storage requirements for per-user agent fleets
Everything above describes one agent's lifecycle. A real deployment runs many of them, one per end user, each with its own isolated instance and its own persistent workspace, and that per-user model is shaping up to be the right default for the next generation of agent-driven products. But it also introduces coordination problems that a single shared filesystem cannot safely absorb on its own. Two storage patterns dominate how production platforms handle this. The silo pattern gives each tenant dedicated storage with no shared infrastructure underneath it, so it suits regulated industries and enterprise customers for whom any commingling of data between tenants is a non-starter. The pool pattern keeps storage shared across tenants and separates them through metadata-based filtering; this is far more cost-efficient for a platform serving many small-to-medium tenants, but it only works if query-level enforcement is applied with real rigor, since a single missed filter can leak one tenant's data into another's results. A related failure occurs once a single tenant runs multiple sub-agents concurrently: they end up competing for the same shared quota, a documented problem at fleet scale, and the fix that has emerged is a shared ledger per tenant, where each agent makes a soft reservation against the ledger at the start of a task and idle agents return unused capacity to the pool for others to draw on. You can already see where these fleets are heading in the open-source tooling circulating around this problem, including agent harnesses built for exactly this kind of per-tenant, per-agent deployment pattern. The storage layer underneath a fleet like this has to do everything described earlier for a single agent, decoupled lifecycle, rich state taxonomy, organized retrieval, provenance, and matched isolation, and do it again correctly across every tenant sharing the platform at once.
Sources
- Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents
- Mind Your HEARTBEAT! Claw Background Execution Inherently Enables Silent Memory Pollution
- Eywa: Provenance-Grounded Long-Term Memory for AI Agents
- Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platform
- Memory as Infrastructure: Reliability Engineering for Persistent Agent Memory in Months-Long LLM-Assisted Development


