Networking Architecture for Isolated Agent VMs

Firecracker micro-VM isolation gives each of the AI agents its own guest kernel, while agents in a container use a shared kernel. Its networking contract ends at the host TAP device. Networking Architecture for Isolated Agent VMs explains why per-VM namespaces, attributable IP assignments, controlled egress, and credential boundaries must form one stack to make that isolation effective in production.

Why Firecracker's isolation guarantee stops at the TAP device

Agent-generated code is untrusted before it reaches the runtime. A model produces it at inference time, without review before execution, and adversarial input can steer its behavior. Containers constrain that code with controls including seccomp, AppArmor, and SELinux, but every tenant still depends on the same kernel. A kernel-level exploit can defeat those shared-kernel protections. Firecracker changes that boundary by booting a separate guest kernel for each microVM.

The separation is enforced below the guest operating system. Intel VT-x places the host and hypervisor in VMX root mode and guest code in VMX non-root mode. Compromising a guest kernel therefore does not itself grant access to the host kernel or another guest. Reaching the host requires defeating the hardware-enforced boundary. This is a different class of problem from bypassing software policies around a shared kernel.

Firecracker also narrows the device surface available to guest code. Each microVM has one VMM process, with API, VMM, and vCPU threads, including one vCPU thread per guest CPU core. Its minimal device model omits BIOS, PCI, and legacy hardware emulation, while startup takes approximately 125ms from VMM launch to guest userspace. Networking remains an explicit handoff. The machine configuration binds a guest virtio-net device to a host TAP device but supplies no IP allocation, NAT, inter-VM isolation, or egress policy. Connecting those TAP devices to a flat Linux bridge places guest traffic on the same layer-2 segment.

How a Linux network namespace per VM turns theoretical isolation into real isolation

When each VM’s TAP device sits inside its own Linux network namespace, its traffic has a separate host networking context. A namespace has its own interfaces, routing table, ARP cache, firewall rules, and socket table. Traffic cannot cross between their interfaces without an operator-controlled connection, so namespace placement establishes the foundation for deliberate connectivity.

The canonical arrangement connects that namespace to the root namespace through a veth pair. An iptables MASQUERADE rule in the root namespace supplies NAT for outbound traffic. Guest-to-guest connectivity requires an explicitly permitted route, while root-namespace policy governs the outbound path. NAT provides address translation alongside that policy.

Virtio-vsock addresses endpoints by context-id and port, allowing logs, health pings, and snapshot metadata to move between guest and host without a guest IP address. Building its namespace, veth pair, and iptables rules cold takes on the order of 100ms per VM. Pre-allocated pools remove that construction work from the VM creation path by keeping network slots ready for assignment.

IP allocation, subnet segmentation, and keeping per-user agent traffic attributable at fleet scale

Attribution begins with address allocation. A fleet needs to link a network event to the agent and user behind it, even when there is no need or reason for payload inspection. A per-VM subnet provides one way to establish that identity. An illustrative /30 allocation contains four addresses, which are the network address, host TAP end, guest end, and broadcast address. Each assignment must be unique and non-overlapping, but the CIDR scheme can vary.

A pre-allocated pool of NATID slots connects that assignment to fleet identity. Each slot maps packet attribution back to a specific VM, and logs preserve the relationship to its user. NAT alone does not provide this record. On a flat bridge with shared outbound NAT, traffic uses the same translated source IP, leaving ephemeral source ports as the distinguisher. Without those records, finding out which agent made a call can cost a lot or be impossible.

AWS’s reference architecture for self-hosted agent sandboxes shows the tenant boundary in developer sessions that work with live database credentials, generated code, and filesystem writes. If credentials or state pass between those sessions, data is exposed. The per-user design gives each session an attributable network identity as well as an isolated execution environment. Restore timing makes that assignment operationally significant. Because a restored guest resumes from frozen memory rather than booting afresh, its namespace and subnet must be staged before it starts using the NIC.

Egress control as the line between an isolated agent and a liability

A guest can be separated from neighboring VMs while retaining an unrestricted path to an attacker. Prompt injection can direct an agent to read credentials and send them through HTTP or DNS. Generated code can also open a reverse shell or contact an attacker-controlled endpoint. All of these actions can take place without a hypervisor escape. Outbound policy must therefore set limits on what the isolated agent can reach.

BeyondTrust Phantom Labs and, separately, Palo Alto Unit 42 demonstrated DNS tunneling against AgentCore’s Code Interpreter sandbox network isolation mode. External network access was blocked, but DNS queries left an exfiltration channel. DNS requires its own policy. Blocking ordinary outbound connections without governing resolution does not close that channel, so an agent egress design needs a controlled resolver or inspection of query content for tunneling patterns.

Vercel Sandbox provides a complementary reference through SNI-based filtering, CIDR blocks, and policy that can change while a sandbox runs. SNI inspection identifies HTTPS destinations without decrypting traffic, while CIDR controls address IP-based targets. Vercel’s recommended AI Gateway BYOK configuration uses OIDC tokens to keep LLM provider keys out of the agent function. These are separate controls with different jobs.

Default-deny outbound traffic in root-namespace iptables or nftables rules, permitting only named destinations and ports. When a tool needs a new endpoint, a control-plane API should change policy while preserving the default. Firecracker’s virtio-net rate limiter adds a volume constraint on permitted traffic. It can blunt exfiltration volume but does not decide whether a destination is authorized.

What the CoreBreak disclosure reveals about the gap between VM isolation and credential security

Aviyam Ivgi and Hedi Ingber disclosed CoreBreak at Black Hat 2026, describing a structural flaw across AWS Bedrock AgentCore with the Strands SDK, Google’s Agent Development Kit, and Vercel’s AI SDK harness. Injected instructions steered managed browser and code-interpreter tools into reading execution-role cloud credentials and exfiltrating them. The attack used the agent’s own permissions. Its significance lies in the absence of any need to cross the VM boundary.

Those credentials were in the execution environment for use in legitimate work. An instruction to read and transmit them could therefore use authorized operations. Kernel isolation cannot tell when that authority is being misused. The architectural implication is to keep credentials out of ambient VM environment variables and fetch them from a credential proxy or secrets manager per call. That shortens exposure from the VM’s lifetime to the tool invocation.

Credential access also needs explicit network treatment. Calls to metadata services and secrets managers should be logged as high-sensitivity events, with more care than routine outbound traffic. AWS set MMDSv2 as the default for new agents on February 14, 2026, provided an API to disable MMDSv1 on older agents, and followed with mandatory enforcement for all runtimes on June 30, 2026. MMDSv2 requires session-token-authenticated metadata access. That prevents unauthenticated probing, but token gating does not replace limits on an agent’s authorized credential access.

Inter-agent traffic and why tool calls should cross a network boundary, not a function call

The same reasoning applies when the destination is another agent or a shared tool. An in-process function call does not pass through the firewall, network rate limiter, or network audit path. Moving the capability behind a network endpoint creates an enforceable boundary. The choice therefore shapes both architecture and code organization.

MCP defines lightweight JSON-RPC 2.0 communication with HTTP and stdio transports. For this design, HTTP is the relevant choice because stdio does not create a network boundary. Exposing tools as HTTP MCP endpoints lets policy govern which namespaces can reach each capability. Each invocation becomes observable on that path. Separate tenant VMs provide kernel and filesystem separation, while their namespaces control access to the tool endpoints.

The Scheduler communicates with the Sandbox pool through a defined API, the Memory service remains separate from the Sandbox, and the Gateway serves as the only external ingress point. Even if a Sandbox is compromised, it must call the Memory service to reach its data store. Memory scope then refines the permitted access. The scopes include user_id for cross-session user memory, agent_id for an agent instance, run_id or session_id for a conversation, and app_id or org_id for shared context. NATID-slot routing policy should restrict a VM to the Memory endpoints for its user, complementing application access control.

How snapshot-restore scheduling changes the networking layer's operational requirements

Snapshot-restore replaces idle compute with storage, but it also changes when networking must be ready. An agent may run code in under a second and then wait many seconds for output from the model. Keeping the VM running charges that waiting time as compute. Suspension preserves disk and memory instead. Copy-on-write guest memory and a reflink-cloned rootfs make restoration cheap enough to support this lifecycle without rebuilding the environment.

A restored guest already has its network configuration. Unlike a cold-booted guest, it expects the TAP device, subnet, and MAC address in its saved state. A mismatched namespace can prevent clean network initialization. The control plane must either pin a namespace to the snapshot or use a network-environment-agnostic snapshot that receives configuration through MMDS after resume. Pinning reduces pool flexibility, while configuration injection permits reuse across prepared slots.

The supplied end-to-end create figure is around 179ms at p50, covering netns pre-allocation, rootfs reflink, VMM fork, snapshot load, resume, and readiness probing. The namespace must be assigned and the configuration sent in step with the restored machine’s lifecycle.

AWS’s self-hosted sandbox architecture recommends webhook-triggered sessions. An inbound event launches a VM, the VM runs for the session, and idle policy suspends then terminates it. The network pool follows those transitions. The control plane assigns a slot at the webhook, restores into it, and returns the slot on suspension. To reuse slots instead of holding them through idle periods, the per-slot configuration must match what the snapshot expects.

Putting the stack together: a coherent networking architecture for a per-user agent fleet

One Firecracker VMM process gives each agent a hardware-isolated guest kernel through KVM. Its virtio-net interface reaches a TAP device inside a dedicated namespace, with a veth pair connecting to root-namespace NAT and policy. Virtio-vsock provides the separate host-agent channel. The assigned subnet slot binds this execution environment to an agent and user in network records.

That identity becomes the input to the policy layers. Root-namespace rules constrain destinations, ports, and DNS behavior, while device-level rate limiting bounds traffic volume. Credential proxies keep long-lived secrets out of the guest environment. HTTP MCP endpoints make tool access subject to network controls, and the Gateway limits external ingress to one service boundary. These controls reinforce one another without depending on a compromised guest to enforce them.

The lifecycle layer preserves those relationships across suspension and reuse. A webhook assigns a prepared namespace slot, MMDS supplies its configuration after resume, and suspension releases the slot for another assignment. Snapshot and namespace-pool versions must stay coordinated. Otherwise, an economical restore mechanism can resume a guest into a network environment it does not understand.

This design needs access to host networking primitives. Bare-metal hosting with direct KVM access exposes TAP placement, namespace lifecycle management, MMDS configuration, and per-VM rate limiting. A managed abstraction that hides those controls cannot run the same stack. Infrastructure selection therefore depends on explicit control over those boundaries. For a multi-tenant agent product, the useful unit is one user’s VM, attributable network identity, and constrained credential scope, managed together throughout its lifecycle.