NewCompare CPU & GPU pricing across AWS, Azure & GCP
Self-hosted sandboxes

Self-hosted sandboxes

Disposable, isolated sandboxes that run as pods inside your own Kubernetes cluster, on the gVisor node pools DevZero already manages.

Self-hosted sandboxes

A DevZero sandbox is a short-lived pod in your cluster that runs untrusted code: code an AI agent wrote, a CI job from a fork, a customer's script. Agents outside the cluster create and use sandboxes through the DevZero API, the Python SDK, curl or MCP. Applications inside the cluster can create them directly with Kubernetes objects and their own Kubernetes identity (a service account or a user), without going through DevZero at all.

Sandboxes run on the gVisor node pool a DevZero node policy provisions. The controls you already have for that cluster (RBAC, NetworkPolicy, admission, registries, quotas) apply to sandboxes, and one object, the SandboxPolicy, decides which image, resources, lifetime, network access and callers a class of sandboxes gets. See Policies and guardrails.

Isolation: what gVisor is and is not

Sandboxes are isolated with gVisor. gVisor (runsc) is a userspace kernel: it services the sandboxed process's system calls in software, so the process never issues a system call to the host kernel directly. Google uses the same runtime for GKE Sandbox, built to run untrusted code from many tenants on shared nodes.

gVisor is not a virtual machine. It runs as an ordinary process on the host, so a bug in how gVisor talks to the host kernel is in scope for an attacker, and resource accounting happens per pod. If your threat model requires a hardware-virtualized boundary, you want a sandbox that is VM-isolated with Kata Containers. Kata uses the same policy model through a different RuntimeClass:

  • a Kata runtime on DevZero node policies is on the roadmap, after general availability;
  • from Phase 2, a policy can name a Kata RuntimeClass your cluster already has (for example AKS Pod Sandboxing).

Every policy states the RuntimeClass its sandboxes use, and the dashboard shows it, so you always know which runtime a sandbox has.

Two gVisor behaviours affect how you size sandboxes:

  • The root filesystem overlay is held in memory. Anything a sandbox writes outside /workspace (and the sized /tmp) counts against the pod's memory limit. Put large files in /workspace, which is a persistent volume.
  • Process count is bounded per node. gVisor relies on the kubelet's podPidsLimit to stop a fork bomb. Set it on the sandbox node policy: the node policy form defaults it to 4096 on AWS, the recommended value.

gVisor also slows down system-call-heavy work, file and network I/O especially, and each sandbox pod reserves about 128Mi of memory overhead. Use sandboxes to isolate untrusted code, not as general-purpose compute.

Runtime selection in Phase 1

In Phase 1 a sandbox policy selects its runtime by naming a DevZero node policy that has Sandbox runtime: gVisor turned on. DevZero resolves that node policy to its RuntimeClass (devzero-gvisor) for you.

Naming a RuntimeClass directly, for example GKE Sandbox's gvisor or an AKS Kata class, arrives in Phase 2. Until then a policy that sets runtime.runtimeClassName is rejected with an error saying the form is not supported yet. Your policies will not need to change when it arrives.

Sandbox nodes are shared capacity

The gVisor pool is ordinary capacity with a sandbox runtime installed. Other workloads in your cluster can be scheduled onto sandbox nodes when there is room; the pool carries no taint that keeps them off. Sandboxes keep their gVisor isolation either way, because the runtime is chosen per pod, but:

  • a sandbox can share a node with your non-sandboxed pods, which matters if your threat model treats node co-residency as exposure;
  • cold-start time and sandbox density depend on what else is running on the pool.

To give sandboxes first claim on the pool, use the priority class described in Keeping sandbox capacity for sandboxed workloads.

Cluster requirements

The sandbox operator checks each of these and reports the result as a condition on the policy, so a missing requirement shows up in the dashboard rather than as a silent gap.

RequirementWhy it mattersHow it is checked
The sandbox RuntimeClass the policy resolves to existsisolationpolicy condition RuntimeReady (it confirms the class exists; it does not check that the node pool has room)
RuntimeClass overhead declaredthe per-pod gVisor overhead is counted in quotaspolicy condition RuntimeOverheadDeclared; the node policy declares 128Mi for devzero-gvisor
Your CNI enforces NetworkPolicyevery network control on a sandbox (no cloud metadata endpoint, no cluster services, gateway-only ingress) is a NetworkPolicya canary in each sandbox namespace sets NetworkPolicyEnforced; sandboxes are blocked while it is false
podPidsLimit set on sandbox nodesfork-bomb bound under gVisora node policy setting (the form defaults it to 4096 on AWS); DevZero does not refuse a sandbox pool without it, so check it is set
A StorageClass with a working CSI driver: the cluster default, or the one the policy names in storage.workspace.storageClassNameeach sandbox's /workspace is a PersistentVolumeClaim (10Gi by default); without a class that can provision it, the claim and the sandbox stay Pendingpolicy condition StorageClassReady checks that a named class exists, or, with no name, that a class is marked default (NoDefaultStorageClass, from dz-sandbox 0.1.2), and that the class's cloud CSI driver is installed (CSIDriverNotInstalled). It does not gate Ready, so check it (on EKS, install the EBS CSI driver and mark a class default; see Install)
IMDSv2 required, hop limit 1 (where the cloud supports it)a second layer of protection for the cloud metadata endpointa node policy setting on the sandbox pool
No service-mesh injection in sandbox namespacessidecar init containers need NET_ADMIN, which gVisor and Pod Security restricted refuse, and mesh mTLS breaks the gateway's connection to the sandboxopt-out labels on sandbox namespaces and opt-out annotations on sandbox pods
Policy engines do not rewrite sandbox podsKyverno or Gatekeeper mutations would race the sandbox admission webhookthe webhook re-runs after other mutations and rejects a pod another webhook made unsafe

NetworkPolicy enforcement per cloud

Many clusters accept NetworkPolicy objects without enforcing them. Sandboxes depend on enforcement, so DevZero tests it: a canary pod pair in each sandbox namespace tries connections the policy must block (a sibling pod, the cloud metadata endpoint on ports 80 and 443, the Azure wire server, the Kubernetes API server, an excepted address range on port 443) and connections it must allow (a public address on port 443, cluster DNS). The canary runs at install, when a sandbox namespace is created, and every 6 hours.

When the canary fails, the policy reports NetworkPolicyEnforced=False and new sandboxes are refused until you fix the CNI. A team Admin can accept the risk and allow sandboxes anyway with the guardrail network.allowUnenforcedNetworkPolicy: true; the dashboard keeps showing the warning.

Enable network policy enforcement in the Amazon VPC CNI (its network policy agent), or run Calico or Cilium. The VPC CNI does not enforce NetworkPolicy unless you turn enforcement on.

The canary checks the cases above. Passing it is necessary, not proof that every rule is enforced for every address range. Some CNIs ignore ipBlock.except; the canary detects that case and reports NetworkPolicyEnforced=False with reason ExceptNotHonoured.

Service meshes

Sandbox namespaces and sandbox pods are created with mesh opt-outs, so a mesh that injects by namespace label or pod annotation leaves them alone. Each sandbox namespace carries these labels:

metadata:
  labels:
    sandbox.devzero.io/managed: "true"
    istio.io/dataplane-mode: none     # Istio ambient
    sidecar.istio.io/inject: "false"  # Istio sidecar
    linkerd.io/inject: disabled       # Linkerd

and each sandbox pod carries the annotations sidecar.istio.io/inject: "false" and linkerd.io/inject: disabled.

If your mesh injects cluster-wide regardless of these labels, exclude the sandbox namespaces (devzero-sandbox by default) in the mesh's own configuration. Do not put sandbox namespaces into the mesh: the sandbox gateway connects to sandboxes over plaintext HTTP/2 inside the namespace's NetworkPolicy, and mesh mTLS breaks that connection.

When a node goes away

A running sandbox's processes are its state, so DevZero keeps sandboxes in use off voluntary node disruption. Every sandbox that a caller has claimed carries karpenter.sh/do-not-disrupt: "true" and cluster-autoscaler.kubernetes.io/safe-to-evict: "false", so Karpenter and cluster-autoscaler do not consolidate its node while it runs. The node scales in once the sandbox ends (its lifetime or idle timeout passes, or someone deletes it). Unclaimed warm-pool pods carry neither annotation, so they do not block consolidation. The pool is not always empty with no sandboxes, though: unclaimed warm pods run there, and so does each policy's NetworkPolicy canary. The canary runs on the sandbox pool, because it uses the policy's RuntimeClass, when the policy is created or changed and every 6 hours after that; its target pod serves for up to 30 minutes per run, so the pool scales a node up for each run even when no sandbox exists. A canary pod that fails is retried every 10 minutes.

A node can still disappear for reasons no annotation stops: a spot interruption, a hardware failure, a manual drain, someone deleting the pod. The sandbox then reports Recreating until a replacement pod is ready, and the replacement mounts the same /workspace volume, so files in /workspace survive. Running processes, terminals and anything outside /workspace (including /tmp) do not: start long-running work again once the sandbox is Ready.

Known issues

  • A slow reader can lose the end of a command's output. If a client is reading a command's output slowly (over a slow or congested connection, for example) at the moment the command exits, up to the last 64 KiB of output can be dropped with no error to show that anything is missing. This is a bug in the upstream sandbox daemon (kubernetes-sigs/agent-sandbox#2347); a fix is in review. Until it ships, for commands with large output, write the output to a file in /workspace and read the file.

What keeps working without DevZero

Running sandboxes and sessions continue until their shutdownTime without DevZero; new external sandboxes, extensions, and tokens need DevZero; in-cluster creation does not.

Sandbox traffic goes from the caller to a gateway in your cluster, not through DevZero, and tokens are checked by that gateway offline. If the connection between your cluster and DevZero drops, open sessions are not interrupted. For what DevZero can see and do, read Trust and data disclosure.

Next steps

On this page