Install self-hosted sandboxes
Prerequisites, the two sandbox Helm charts, turning on the DevZero operator's sandbox relay, setting the endpoint, and checking that every part is ready.
Install self-hosted sandboxes
This page installs sandboxes on a cluster that is already connected to DevZero. When you finish, a sandbox policy on the cluster is Ready, the sandbox gateway is serving, and the quickstart can create a sandbox.
The order matters, because each part waits on the one before it:
- the two sandbox Helm charts, so the sandbox CRDs exist in the cluster;
- the DevZero operator's sandbox relay, so DevZero can deliver sandbox objects to the cluster. When it connects, DevZero sends the cluster its signed trust bundle (the gateway's JWT keys) and the gateway turns ready;
- guardrails and a sandbox policy in DevZero, which the relay delivers;
- the endpoint: the hostname and certificate external callers connect to.
Prerequisites
| Requirement | Details |
|---|---|
| Kubernetes 1.30 or later | Both charts refuse an older cluster before Helm writes anything. See Compatibility. |
| The DevZero operator, 0.2.36 or later | The dakr-operator chart, installed with the dashboard's command (release dakr in devzero-system). 0.2.36 is the first release with the sandbox relay. You turn the relay on in step 3. |
| Sandboxes turned on for your team | The Sandbox section of the dashboard (policies, guardrails, sandboxes) appears only for teams with the Sandboxes feature. Ask your DevZero contact to turn it on. |
| A gVisor node policy targeting the cluster | A DevZero node policy with Sandbox runtime: gVisor, with the cluster as a target. See gVisor node pools. On AWS, set podPidsLimit (the form defaults it to 4096) and keep IMDSv2 required with hop limit 1. |
| Nodes for the gateway | The gateway runs at least two replicas, one per node, never on the sandbox pool. Each requests 1.5 CPU by default, so it needs nodes with 1.5 CPU free after their DaemonSets: 4-vCPU nodes have room, while a 2-vCPU node has about 1.45 to 1.7 CPU free depending on its DaemonSets. A rollout needs one more such node than there are replicas. If the default does not fit your 2-vCPU nodes, install with gateway.resources.requests.cpu=1200m. See Gateway sizing. |
| A default StorageClass with a CSI driver | Each sandbox's /workspace is a PersistentVolumeClaim (10Gi by default) on the cluster's default StorageClass, or the one the policy names. Without one, the claim stays Pending and so does the sandbox. See Storage. |
| NetworkPolicy enforcement | Every network control on a sandbox is a NetworkPolicy, and sandboxes are refused while the CNI does not enforce them. See NetworkPolicy. |
| For a load balancer DevZero creates on AWS: the AWS Load Balancer Controller | The sandbox load balancer is an NLB with IP targets, which only the AWS Load Balancer Controller provisions. Without it the load balancer never gets an address; from 0.1.2 the endpoint reports LoadBalancerCapable=False with reason LoadBalancerControllerMissing straight away. Not needed if you route to the gateway with your own ingress. |
kubectl, Helm 3 or later, and a DevZero token | A team Admin token sets the guardrails, policies and endpoint (or use the dashboard). A Developer token is enough to create sandboxes. |
Storage
Install the Amazon EBS CSI driver (the aws-ebs-csi-driver EKS add-on, with an IAM role for its service account), then mark a StorageClass that uses it as the default. On recent EKS versions the gp2 class EKS creates is not the default, and it uses the old in-tree provisioner name.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
type: gp3Check that exactly one class is marked (default):
kubectl get storageclassFrom dz-sandbox 0.1.2, a policy reports this check as its StorageClassReady condition: False with reason NoDefaultStorageClass when the policy relies on the cluster default and no class is annotated storageclass.kubernetes.io/is-default-class=true, or CSIDriverNotInstalled when the class's provisioner needs a cloud CSI driver (for example ebs.csi.aws.com, also behind the in-tree kubernetes.io/aws-ebs name) that is not installed. It is re-checked every minute, so annotating a class or installing the driver clears it with no change to the policy. StorageClassReady does not gate Ready, so check it even when the policy is Ready.
NetworkPolicy
A canary in each sandbox namespace tests that the CNI blocks what the policy must block (the cloud metadata endpoint, sibling pods, the Kubernetes API server) and reports the result as the policy condition NetworkPolicyEnforced. While it is False, new sandboxes are refused. NetworkPolicy enforcement per cloud covers AKS and GKE.
On EKS with the default Amazon VPC CNI, enforcement is off until you turn it on. For the VPC CNI managed add-on (version 1.14 or later):
aws eks update-addon --cluster-name <cluster> --addon-name vpc-cni \
--configuration-values '{"enableNetworkPolicy":"true"}'This restarts the aws-node DaemonSet, so do it in a maintenance window. Calico or Cilium in policy mode also work.
If you cannot turn on enforcement, a team Admin can let sandboxes run anyway with the guardrail network.allowUnenforcedNetworkPolicy: true (in the dashboard, Sandbox → Guardrails). Understand what that gives up. With no enforcement, none of the sandbox network rules apply: code in a sandbox can reach the cloud metadata endpoint (169.254.169.254), the other pods in its namespace, including other people's sandboxes, and the Kubernetes API server, as well as anything else the node can reach. On a real install the canary found all three reachable. On AWS, IMDSv2 required with a hop limit of 1 on the sandbox node policy still keeps pods from getting a metadata token, but that is the only layer left. Use this setting for evaluation on a cluster with nothing else in it, not for untrusted code.
Install
Find the cluster ID
The gateway accepts only sandbox tokens issued for this cluster, so it needs the cluster's DevZero ID. In the dashboard, open the cluster and use Copy cluster ID next to its name. It is a UUID, and it is also the <id> in the cluster page's URL, /dashboard/clusters/<id>/overview.
export CLUSTER_ID=<cluster id>Install the two charts
Install dz-sandbox-upstream first. It installs the pinned upstream agent-sandbox controller into agent-sandbox-system. Keep its Helm release record outside that namespace, because the chart owns the namespace. devzero-system works.
helm upgrade --install dz-sandbox-upstream \
oci://registry-1.docker.io/devzeroinc/dz-sandbox-upstream \
--version 0.1.2 --namespace devzero-system --waitThen install dz-sandbox, the sandbox operator, the gateway and the DevZero CRDs. It must go into devzero-system and refuses any other namespace:
helm upgrade --install dz-sandbox \
oci://registry-1.docker.io/devzeroinc/dz-sandbox \
--version 0.1.2 --namespace devzero-system \
--set gateway.jwt.clusterID="$CLUSTER_ID" \
--set operator.cloud=aws # or azure, gcp| Value | Set it to |
|---|---|
gateway.jwt.clusterID | Required. The cluster ID from step 1. While it is empty the gateway accepts no sandbox token and never becomes ready. |
gateway.jwt.environment | Leave at the default, production. It selects which DevZero signing root the gateway trusts, and must match the DevZero environment the cluster is connected to (production for dakr.devzero.io). A mismatch keeps the gateway from ever turning ready; see Check the install. |
operator.cloud | aws, azure or gcp. The operator never detects the cloud; it uses this to annotate a load balancer it creates for the endpoint. With the default, none, it refuses an internal load balancer. |
gateway.resources.requests.cpu | 1200m if your non-sandbox nodes have 2 vCPU. |
Do not pass --wait to the dz-sandbox install. The gateway pods start, but they stay 0/1 Ready until DevZero sends the cluster its trust bundle, which happens only once the relay is on (step 3). With --wait, Helm waits for them, times out and marks the release failed, although everything it installed is in place. If that already happened, nothing needs undoing: continue with the next step. The operator pods become ready straight away.
From 0.1.2 the chart's install notes also warn about --wait (they still say the bundle waits for a policy or guardrails; it no longer does), and each gateway pod logs which readiness checks are failing whenever that changes, so you can see what it is waiting for:
kubectl -n devzero-system logs deploy/devzero-sandbox-gateway | grep readinessUntil the trust bundle arrives, the line reads gateway not ready and its failing list names SandboxTrustBundle/default. Its reason text also says DevZero sends the bundle once the cluster has a policy or guardrails; since DevZero now sends it when the relay connects, read that as step 3.
Both charts check the Kubernetes version, and dz-sandbox also checks that the upstream controller runs at the version it was built against. A refusal fails the install before Helm writes anything and prints the compatibility URL.
Turn on the sandbox relay
The DevZero operator delivers sandbox objects to the cluster and reports their status back. Turn on its v2 delivery and the relay together; the chart refuses the relay without v2 delivery.
helm upgrade dakr oci://registry-1.docker.io/devzeroinc/dakr-operator \
--version 0.2.37 --namespace devzero-system --reuse-values \
--set operator.deliveryV2=true \
--set operator.sandboxRelay=true
kubectl -n devzero-system rollout status deploy/dakr-dakr-operator-operatorKeep --version in the command: without it, helm upgrade moves the operator to the latest chart release as well. The deployment name assumes the release name dakr, which the dashboard's install command uses. The relay grants the operator read and write on the sandbox.devzero.io API group only, plus read of namespaces and of SandboxClaims in sandbox namespaces.
Staying on dakr-operator 0.2.36? There, --reuse-values can fail the chart's values schema check on operator.serviceAccount.annotations when the stored release has null there. Use --version 0.2.36 and add --set-json 'operator.serviceAccount.annotations={}'; it replaces the stored null with an empty map and changes nothing else. 0.2.37 does not need it.
Within about a minute of the relay connecting, DevZero sends the cluster its trust bundle and an empty token denylist, and the gateway pods turn ready. You do not need guardrails or a policy for this:
kubectl get sandboxtrustbundle default
kubectl -n devzero-system get pods -l app.kubernetes.io/name=devzero-sandbox-gatewaySave guardrails and create a policy
A team Admin sets the cluster's guardrails and creates the first policy:
- Guardrails: in the dashboard, Sandbox → Guardrails, pick the cluster, review the defaults and save. This is where you set
network.allowUnenforcedNetworkPolicyif you need it (see NetworkPolicy). - A policy: Sandbox → Policies → New, naming the gVisor node policy. The quickstart shows the same with the API.
Both are refused with cluster_unreachable while the relay from step 3 is not connected.
Set up the endpoint
Callers reach sandboxes through the gateway, on a hostname you choose (bring-your-own, the Phase 1 mode). Every sandbox DevZero creates for a DevZero token gets a URL on this hostname, so the hostname must resolve wherever the SDK runs, including in pods of the same cluster. Only a pod that connects with its Kubernetes service account (Sandbox.in_cluster(...), see In-cluster callers) uses the gateway's internal Service name directly.
- Get a certificate for the hostname and its wildcard, for example
sbx.acme.internaland*.sbx.acme.internal, and store it as the labelled Secretsandbox-gateway-tlsindevzero-system. See Store it in your cluster. - Decide how traffic arrives: DevZero creates a load balancer (on AWS this needs the AWS Load Balancer Controller), or you route to the internal Service with TLS passthrough.
- A team Admin sets the endpoint with
SetSandboxEndpoint:
curl -sS https://dakr.devzero.io/sandbox.v1.SandboxService/SetSandboxEndpoint \
-H "Authorization: Bearer $DEVZERO_TOKEN" -H 'Content-Type: application/json' \
-d '{
"clusterId": "'"$CLUSTER_ID"'",
"spec": {
"mode": "ENDPOINT_MODE_BYO",
"hostname": "sbx.acme.internal",
"visibility": "ENDPOINT_VISIBILITY_INTERNAL",
"loadBalancer": { "managedByDevzero": true }
}
}'- Point DNS for the hostname and its wildcard at the load balancer's address, or at your ingress.
Leave out loadBalancer when you route to the gateway yourself.
If the hostname does not resolve inside the cluster (an internal endpoint on the ClusterIP Service with no DNS record, for example), point it at the gateway's Service for pods that call sandboxes with a DevZero token: a CoreDNS rewrite of the hostname and its wildcard to devzero-sandbox-gateway.devzero-system.svc.cluster.local, or hostAliases on the calling pod with the Service's cluster IP (kubectl -n devzero-system get svc devzero-sandbox-gateway). The gateway serves the endpoint's certificate on that Service, so TLS checks out against the hostname. Exposure modes covers every field, visibility and the load balancer pre-flight in detail.
Check the install
# Upstream controller, sandbox operator and gateway: all Available
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller
kubectl -n devzero-system get deploy devzero-sandbox-operator devzero-sandbox-gateway
# DevZero has delivered the trust bundle (the gateway's JWT keys)
kubectl get sandboxtrustbundle default
# Policies and their conditions; READY must be True
kubectl get sandboxpolicies
kubectl get sandboxpolicy <name> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
# The endpoint: TLSReady, LoadBalancerCapable and LoadBalancerReady (when DevZero creates the load balancer) and Ready
kubectl get sandboxendpoint default -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'| Symptom | Cause | Fix |
|---|---|---|
Gateway pods 0/1 Ready, no sandboxtrustbundle | DevZero has not sent the trust bundle | turn on the relay (step 3) and check that it connected (the dakr-dakr-operator-operator pod logs). From 0.1.2, kubectl -n devzero-system logs deploy/devzero-sandbox-gateway | grep readiness lists the failing checks |
| Gateway pods stay not ready with a trust bundle present | gateway.jwt.clusterID is empty or names another cluster (from 0.1.2 the gateway's readiness log line names the failing check) | helm upgrade dz-sandbox ... --reuse-values --set gateway.jwt.clusterID=<id> |
Gateway pods stay not ready with a trust bundle present; the readiness line keeps reporting jwks_missing, and the gateway logs ignoring SandboxTrustBundle; keeping the last verified key set with signed by root kid "X", want "Y" | gateway.jwt.environment names a different DevZero environment from the one that signed the bundle, so the gateway rejects every bundle | helm upgrade dz-sandbox ... --reuse-values --set gateway.jwt.environment=production (the valid values are production, production-in and preproduction; use the one for the DevZero environment the cluster is connected to) |
Gateway pod Pending | no node with 1.5 CPU free outside the sandbox pool, or nodes without a topology.kubernetes.io/zone label | larger nodes, gateway.resources.requests.cpu=1200m, or gateway.topologySpread.whenUnsatisfiable=ScheduleAnyway for nodes without zone labels |
Policy NetworkPolicyEnforced=False | the CNI does not enforce NetworkPolicy; the message lists what the canary reached | NetworkPolicy |
Policy RuntimeReady=False, NodePolicyNoSandboxRuntime | the named node policy has no sandbox runtime | turn on Sandbox runtime: gVisor on it, or name another |
Policy StorageClassReady=False, NoDefaultStorageClass or CSIDriverNotInstalled (0.1.2 and later) | no StorageClass is marked default, or the class's CSI driver is not installed | Storage |
Sandbox stays Pending, its PVC Pending | no default StorageClass, or no CSI driver (on 0.1.1 and earlier the policy does not report it) | Storage |
Policy or guardrails save returns cluster_unreachable | the relay is off or disconnected | step 3; check the dakr-dakr-operator-operator pod logs |
Endpoint LoadBalancerCapable=False, LoadBalancerControllerMissing (0.1.2 and later) | operator.cloud=aws but the AWS Load Balancer Controller is not installed | install the controller (re-checked every minute, no other change needed), or route to the internal Service yourself |
Endpoint LoadBalancerCapable=False | the cloud could not create a load balancer; the message carries the cloud's error | the pre-flight check |
Endpoint Ready=True but LoadBalancerCapable=False, after switching to routing the gateway yourself | the endpoint has no loadBalancer, so the load balancer pre-flight no longer applies; a condition left from the earlier setting can remain | nothing to fix: with no loadBalancer, Ready is the condition to check |
Next: create your first sandbox.
Uninstall
Delete your sandbox policies in DevZero first and let them drain, then:
helm -n devzero-system uninstall dz-sandbox
helm -n devzero-system uninstall dz-sandbox-upstreamBy design, the sandbox CRDs (and so every sandbox object), the devzero-sandbox namespace and the objects DevZero applied to the cluster stay after helm uninstall. Uninstalling dz-sandbox-upstream deletes the agent-sandbox-system namespace. Turn the relay back off with operator.sandboxRelay=false on the dakr release (keep the same --version), so DevZero stops delivering sandbox objects.
To remove everything else, run this after both helm uninstalls:
# 1. With the operator gone, nothing removes its finalizers: SandboxRequests keep
# sandbox.devzero.io/request-tombstone and an undrained SandboxPolicy keeps
# sandbox.devzero.io/drain. Strip them first, or deleting the CRDs hangs.
for r in $(kubectl -n devzero-system get sandboxrequests -o name); do
kubectl -n devzero-system patch "$r" --type=merge -p '{"metadata":{"finalizers":null}}'
done
for r in $(kubectl get sandboxpolicies -o name); do
kubectl patch "$r" --type=merge -p '{"metadata":{"finalizers":null}}'
done
# 2. The sandbox CRDs (this deletes every object of them) and the sandbox namespace
kubectl get crd -o name | grep -E '\.sandbox\.devzero\.io$|\.agents\.x-k8s\.io$' | xargs kubectl delete
kubectl delete namespace devzero-sandbox
# 3. What the operator created in devzero-system: its CA, webhook and gateway
# certificates, the audit key, the sandbox token root key, the CA bundle
# ConfigMap and its leader-election Lease. A reinstall reuses these silently if they stay.
kubectl -n devzero-system delete secret devzero-sandbox-ca devzero-sandbox-operator-webhook-tls \
sandbox-audit-hmac sandbox-gateway-internal-tls sandbox-token-root --ignore-not-found
kubectl -n devzero-system delete configmap devzero-sandbox-ca --ignore-not-found
kubectl -n devzero-system delete lease devzero-sandbox-operator-leader --ignore-not-found
# 4. Copies of the CA bundle in the namespaces of in-cluster callers, if any
kubectl delete configmap -A -l app.kubernetes.io/managed-by=devzero-sandbox-operator,sandbox.devzero.io/ca-bundle=true
# 5. Your endpoint certificate, if you no longer need it
kubectl -n devzero-system delete secret sandbox-gateway-tls --ignore-not-foundThe gateway Services, devzero-sandbox-gateway and devzero-sandbox-gateway-headless, belong to the SandboxEndpoint, so Kubernetes deletes them with it in step 2.
Uninstalling removes nothing in DevZero. DevZero keeps the cluster's endpoint and guardrails, and pushes them to the cluster again when you reinstall and turn the relay back on. If you do not want them back, change the guardrails in the dashboard (Sandbox → Guardrails) and the endpoint with SetSandboxEndpoint; the dashboard has no endpoint settings page yet.
Overview
Disposable, isolated sandboxes that run as pods inside your own Kubernetes cluster, on the gVisor node pools DevZero already manages.
Quickstart
Create a sandbox policy, then create a sandbox with the Python SDK, run a command, write and read a file, and delete it. Also connects an MCP client.