Exposure modes
How agents outside your cluster reach sandboxes, what each exposure mode sends through DevZero, who holds the DNS name, certificate and key, and how to fix a failed setup.
Exposure modes
Agents outside your cluster create sandboxes through the DevZero API, then talk to them through the sandbox gateway, a proxy that runs in your cluster. The exposure mode decides how traffic from outside reaches that gateway. You choose one mode per cluster.
In every mode, TLS ends at the gateway in your cluster, the gateway's private key never leaves your cluster, and DevZero cannot read sandbox traffic. In-cluster callers always use the gateway's internal Service, devzero-sandbox-gateway.devzero-system.svc, whatever the mode.
Gateway sizing
The gateway runs at least two replicas, one per node, and never on a sandbox node pool. Each replica requests 1.5 CPU and 1 GiB of memory, with limits of 2 CPU and 2 GiB:
gateway:
resources:
requests: { cpu: 1500m, memory: 1Gi }
limits: { cpu: "2", memory: 2Gi }So each replica needs a node with 1.5 CPU free after its DaemonSets. A 2-vCPU node (for example an m6i.large) has about 1.45 to 1.7 CPU left, depending on which DaemonSets run on it, so the default fits on some 2-vCPU nodes and not on others; nodes of 4 vCPU or more have room with typical DaemonSets. A rollout replaces one replica at a time and starts the new one first, so it needs one more eligible node than there are replicas, or a cluster autoscaler that adds one.
If your cluster has only 2-vCPU nodes, install with gateway.resources.requests.cpu: 1200m. The replicas then fit, but under full load the gateway adds more latency and throughput overhead than at the default.
Autoscaling
The chart installs a HorizontalPodAutoscaler for the gateway, on by default:
gateway:
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 20
targetCPUUtilizationPercentage: 60 # of the CPU request: 900m at the default
activeStreams:
enabled: false
averageValue: 600At the default request it scales out at about 750 to 830 open streams per replica, before the per-replica limit of 1,000. The CPU target needs the Kubernetes metrics API (metrics-server, which EKS does not install by default); without it the HPA cannot read CPU and the gateway stays at minReplicas. Because replicas run one per node, every replica the HPA adds needs another node with room, so set maxReplicas no higher than the nodes you have or your autoscaler can add. activeStreams also scales on open streams, but needs a custom-metrics adapter (such as prometheus-adapter) serving the gateway's stream metric; leave it off until you have one. With autoscaling.enabled: false, the chart runs gateway.replicas (2) instead.
Comparing the modes
| BYO endpoint | Managed endpoint | Relay | |
|---|---|---|---|
| Available | Phase 1 | Phase 2 | Phase 3 |
| Hostname | yours (for example sbx.acme.internal) | <cluster>.sbx.devzero.io | <cluster>.sbx.devzero.io |
| Load balancer | DevZero creates an L4 load balancer, or you route to an internal Service yourself | DevZero creates an L4 load balancer | none needed; the gateway dials out to DevZero's edge |
| DNS records | you create them | DevZero creates them | DevZero creates them |
| TLS certificate | yours, from any CA | issued by DevZero through ACME (Let's Encrypt) | yours or DevZero's |
| TLS private key | yours, in a Secret in your cluster | generated in your cluster; DevZero sees only a certificate signing request | generated in your cluster |
| Sandbox bytes through DevZero | none | none | yes, encrypted end to end; DevZero's edge forwards TLS bytes without decrypting them, and stores nothing |
| Metadata to DevZero | sandbox status, image reference, environment variable names, byte counts, IDs, command hashes (same in every mode) | same | same, plus connection metadata at the edge |
| Can DevZero issue a certificate for your hostname? | no | yes: DevZero controls the DNS name, so a coerced DevZero could obtain a new certificate for it (visible in Certificate Transparency logs) | with a BYO certificate, no; with a managed name, as for managed mode |
| Requires | a hostname, a wildcard certificate, and either load balancer permissions or your own ingress | load balancer permissions | the guardrail endpoint.allowRelay: true |
Once all three exist, managed is the default, then BYO, then relay. If you need DevZero unable to either decrypt or issue certificates for your traffic, use a BYO certificate, with relay where you need it.
The full disclosure table is on Trust and data disclosure.
Set up a BYO endpoint (Phase 1)
Choose a hostname and get a certificate
Pick a hostname, for example sbx.acme.internal. Get a certificate whose subject alternative names include both the hostname and its wildcard (sbx.acme.internal and *.sbx.acme.internal). A corporate CA is fine: the SDKs use the system trust store and accept a custom CA bundle. Without the wildcard name the endpoint reports TLSReady=False/MissingWildcardSAN.
Store it in your cluster
Create the TLS Secret sandbox-gateway-tls in devzero-system, once, as a cluster admin. It must carry the label sandbox.devzero.io/gateway-tls=true: the gateway serves the certificate only while the label is there. DevZero never modifies, copies or deletes this Secret, and the key in it never leaves your cluster.
kubectl -n devzero-system create secret tls sandbox-gateway-tls \
--cert=fullchain.pem --key=privkey.pem
kubectl -n devzero-system label secret sandbox-gateway-tls \
sandbox.devzero.io/gateway-tls=trueTo rotate the certificate later without cluster-admin rights, list the people or service accounts (cert-manager, for example) in the chart value byoTLS.writers. The installer binds them to the Role devzero-sandbox-byo-tls-writer, which allows get, update and patch on this one Secret and nothing else. It cannot allow create, because Kubernetes RBAC cannot limit create to one name, so the first copy is always created by an admin.
Choose how traffic arrives
Decide three things:
- the hostname, for example
sbx.acme.internal(lower-case, at least two DNS labels, without the wildcard); - public or internal visibility (internal is the default; public is refused if your cluster's guardrails disallow it);
- whether DevZero creates the load balancer, or you route to the gateway yourself.
The two ways traffic can arrive:
- DevZero creates a load balancer. The sandbox operator creates
Service/devzero-sandbox-gateway-lb-public(or-internal), a layer-4 TCP load balancer, and reports its address on the endpoint. Point DNS for the hostname and its wildcard at that address. Before you choose this:- On AWS, install the AWS Load Balancer Controller and tag your subnets (
kubernetes.io/role/elbfor public,kubernetes.io/role/internal-elbfor internal). Sandbox load balancers are NLBs with IP targets, which only that controller provisions; without it the load balancer never gets an address and the pre-flight check fails after 5 minutes. The controller and the cloud need the permissions below. To keep a rollout from sending new connections to a gateway pod that is going away, also labeldevzero-systemwithelbv2.k8s.aws/pod-readiness-gate-inject=enabledand installdz-sandboxwithgateway.loadBalancerReadinessGate=aws. - Set the chart value
operator.cloud(aws,azureorgcp) when you install, so the operator renders the right load balancer annotations. The operator does not detect your cloud. With the default,none, the cloud's default scheme applies and may be internet-facing, so the operator creates a load balancer only for public visibility and refuses internal visibility withLoadBalancerReady=Falseand reasonCloudNotConfigured.
- On AWS, install the AWS Load Balancer Controller and tag your subnets (
- You route to it. Only the internal
ClusterIPService,devzero-sandbox-gatewayon port 443, exists. Send traffic to it from your own ingress with TLS passthrough, so TLS still ends at the gateway. See Routing with your own ingress.
Set the endpoint in DevZero
A team Admin sets the endpoint with the SetSandboxEndpoint API call. There is no endpoint settings page in the dashboard yet. If you prefer, send the same settings to your DevZero contact and DevZero sets it for you.
curl -sS https://dakr.devzero.io/sandbox.v1.SandboxService/SetSandboxEndpoint \
-H "Authorization: Bearer $DEVZERO_TOKEN" -H 'Content-Type: application/json' \
-d '{
"clusterId": "<cluster id>",
"spec": {
"mode": "ENDPOINT_MODE_BYO",
"hostname": "sbx.acme.internal",
"visibility": "ENDPOINT_VISIBILITY_INTERNAL",
"loadBalancer": {
"managedByDevzero": true,
"sourceRanges": ["10.0.0.0/8"]
}
}
}'| Field | Values |
|---|---|
mode | ENDPOINT_MODE_BYO, the only mode Phase 1 accepts |
hostname | the base hostname; the certificate must cover it and *. of it |
visibility | ENDPOINT_VISIBILITY_INTERNAL (default) or ENDPOINT_VISIBILITY_PUBLIC |
loadBalancer.managedByDevzero | true makes the operator create the load balancer; leave loadBalancer out to route to the ClusterIP Service yourself |
loadBalancer.sourceRanges | CIDRs allowed to reach the load balancer (at most 64) |
loadBalancer.extraAnnotations | extra annotations for the load balancer Service (at most 32; the keys that set the scheme are refused, because visibility decides it) |
loadBalancer.preserveClientIp | preserve client IPs on Azure or GCP; see Permissions per cloud |
The loadBalancer settings other than managedByDevzero need managedByDevzero: true. The call replaces the whole endpoint, so send every field each time. GetSandboxEndpoint with {"clusterId": "<cluster id>"} returns the current settings and whether the cluster has applied them (apply.state). The call is refused with cluster_unreachable while the DevZero operator's sandbox relay is not connected.
DevZero applies the settings to the cluster as SandboxEndpoint/default. Your part stays in your cluster and your DNS: the certificate in sandbox-gateway-tls (step 2) and the DNS records for the hostname and its wildcard. Until the endpoint is set, sandboxes you create have no URL to connect to.
Check it
The endpoint's status in your cluster reports LoadBalancerReady (when DevZero creates the load balancer), TLSReady and Ready. Read it, resolve the hostname and test the connection as in Check an endpoint by hand.
Routing with your own ingress (TLS passthrough)
The controller must run with --enable-ssl-passthrough.
ingress-nginx picks a passthrough backend by an exact match on the TLS server name. A wildcard rule such as *.sbx.acme.internal never matches a real name such as 8080-abc.sbx.acme.internal; those connections reach nginx's own TLS server, which terminates them instead of passing them to the gateway (kubernetes/ingress-nginx#11982, closed as not planned). So the example routes only the bare hostname, with no wildcard rule.
Phase 1 sandbox URLs all use the bare hostname (https://sbx.acme.internal/t/<team>/s/<id>/…), so this route serves everything Phase 1 offers. Port previews (Phase 2) use per-sandbox names under *.sbx.acme.internal, which ingress-nginx cannot pass through; for those, use a Gateway API TLSRoute or let DevZero create the load balancer.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devzero-sandbox
namespace: devzero-system
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
ingressClassName: nginx
rules:
- host: sbx.acme.internal
http: { paths: [{ path: /, pathType: Prefix, backend: { service: { name: devzero-sandbox-gateway, port: { number: 443 } } } }] }An ingress that terminates TLS and forwards HTTP is not supported. It would break the gateway's certificate model and, later, native gRPC. Use passthrough.
Public and internal visibility
Public suits agents outside your network. Internal puts the load balancer on private addresses, for callers on your VPC or VPN. Internal endpoints have two caveats:
- DNS rebinding protection can block them. In BYO and managed modes the public DNS name resolves to a private (RFC 1918) address. Resolvers with rebinding protection refuse such answers: dnsmasq with
stop-dns-rebind, Pi-hole, many home and office routers, and some zero-trust network agents. A caller behind one sees the name fail to resolve even though the load balancer is healthy. - Public DNS reveals private addresses. Publishing the record exposes a private IP and so a little of your VPC layout. If that matters, serve the record from a private DNS zone instead (Route 53 private hosted zone, Azure Private DNS, Cloud DNS private zone). In BYO mode you control DNS, so create the record only in your private zone. For managed mode, a private-zone variant where DevZero hands you the records and publishes nothing publicly arrives in Phase 2.
Changing visibility creates a second load balancer Service of the new kind, because AWS and GCP fix a load balancer's scheme at creation. The operator removes the old one as soon as the new one has an address. In BYO mode DNS is yours, so point the hostname and its wildcard at the new address straight away; until you do, callers resolving the old address cannot connect.
When the load balancer fails: the pre-flight check
Before DevZero creates a load balancer, the sandbox operator tests whether it can.
On AWS (operator.cloud=aws), from dz-sandbox 0.1.2 it first checks that the AWS Load Balancer Controller is installed (the cluster serves its elbv2.k8s.aws API group). If it is not, the endpoint reports LoadBalancerCapable=False with reason LoadBalancerControllerMissing within one reconcile, its message names the controller with its install link and the internal-Service alternative, and no throwaway Service is created. The check repeats every minute, so installing the controller clears it with no change to the endpoint.
Otherwise it creates a throwaway Service, devzero-sandbox-preflight-lb, and waits up to 5 minutes for an address. While it waits, the endpoint reports LoadBalancerCapable=Unknown with reason LoadBalancerPending, and the condition's message carries the cloud's latest error as soon as the cloud reports one (usually within seconds), so a missing permission or untagged subnet shows up without waiting out the 5 minutes:
kubectl get sandboxendpoint default -o jsonpath='{.status.conditions[?(@.type=="LoadBalancerCapable")].message}'The cloud's error is copied from a Kubernetes Event on the Service. When the cloud reports the same failure repeatedly, Kubernetes merges the events, and the message then starts with (combined from similar events): before the cloud's own text. The error after the prefix is the one to act on.
- Success: the Service is deleted and the endpoint reports
LoadBalancerCapable=True. - Failure: the endpoint reports
LoadBalancerCapable=Falsewith the cloud's or the admission controller's error word for word: reasonLoadBalancerForbiddenwhen the API server or an admission policy refused the Service,LoadBalancerPendingwhen no address arrived in time. The operator creates no load balancer rather than leave a Service pending forever; route to the internal Service yourself instead. A refused check is retried periodically, so lifting the policy is enough; a timed-out check, which holds a cloud load balancer for up to 5 minutes, is not retried until the endpoint settings change.
Common causes:
| Error looks like | Cause | Fix |
|---|---|---|
| admission webhook denied the request (Kyverno, Gatekeeper) | a cluster policy forbids type: LoadBalancer | grant an exception for devzero-system, or route to the internal Service with your own ingress |
reason LoadBalancerControllerMissing, on EKS (0.1.2 and later) | the AWS Load Balancer Controller is not installed | install it (see Choose how traffic arrives); sandbox load balancers are NLBs with IP targets. The check reruns every minute on its own |
no controller has recorded an Event on it, on EKS (0.1.2 and later) | the controller's CRDs are installed but nothing is acting: it is scaled to zero or crash-looping | get the controller running again (usually Deployment kube-system/aws-load-balancer-controller) |
got no load-balancer address within 5m0s and the cloud reported no error, on EKS (0.1.1 and earlier) | the AWS Load Balancer Controller is not installed, so nothing acts on the Service | install it (see Choose how traffic arrives), or upgrade to 0.1.2, which reports this at once. A timed-out check reruns only when the endpoint settings change; setting identical settings again changes nothing |
| access denied / not authorized from the cloud API | the cloud controller lacks permissions | grant the permissions below |
| no subnets found, on EKS | subnets lack the load balancer role tags | tag them (kubernetes.io/role/elb for public, kubernetes.io/role/internal-elb for internal) |
LoadBalancerReady=False, reason CloudNotConfigured | internal visibility, but the operator was installed with operator.cloud: none | set operator.cloud to aws, azure or gcp and upgrade the chart, use public visibility, or route to the internal Service yourself |
| quota exceeded | load balancer or IP address quota reached in the account | raise the quota or free addresses |
Permissions per cloud
The AWS Load Balancer Controller needs elasticloadbalancing:Create* and elasticloadbalancing:Describe* (as in its standard IAM policy), and the subnets need the role tags above. With IP targets the NLB health-checks and reaches the gateway pods directly.
The operator turns on client IP preservation on the NLB's target group (service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true), so the gateway sees each caller's real address. It needs that address to rate-limit failed authentication per client, and without it loadBalancerSourceRanges may not be enforced on an NLB that has no security group. Target group attributes you set yourself in loadBalancer.extraAnnotations are kept, and the operator's setting is appended; if you set preserve_client_ip.enabled yourself, your value wins.
With preservation on, the NLB does not support NAT loopback: a connection to an internal NLB from a pod on the same node as a gateway replica can be routed back to that node and time out. In-cluster callers should use the devzero-sandbox-gateway Service (devzero-sandbox-gateway.devzero-system.svc), not the NLB address. If something in the cluster must go through the NLB, set preserve_client_ip.enabled=false in extraAnnotations; the gateway then sees the NLB's address for every caller and can no longer rate-limit per client.
Check an endpoint by hand
dz sandbox doctor arrives with the DevZero CLI in Phase 2. It will print every endpoint and policy condition with its remediation, resolve the hostname, test TCP on port 443, and exit non-zero when something is wrong. Until then, the same checks by hand:
# Endpoint and pre-flight conditions, including the verbatim cloud error
kubectl get sandboxendpoint default -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason} {.message}{"\n"}{end}'
# Does the name resolve, and to what? (an internal endpoint should give a private address)
dig +short sbx.acme.internal
dig +short test.sbx.acme.internal
# Is port 443 reachable, and does the certificate cover the wildcard?
openssl s_client -connect sbx.acme.internal:443 -servername test.sbx.acme.internal </dev/null 2>/dev/null \
| openssl x509 -noout -subject -ext subjectAltNameIf dig returns nothing from your laptop but works from a VPC host, suspect DNS rebinding protection (above).
Policies and guardrails
How a SandboxPolicy governs a class of sandboxes, how cluster guardrails cap every policy, and why the sandbox namespace is the security boundary.
Images
What a sandbox image needs, which DevZero base images exist, how registry allowlists and guardrails work, cloud registry auth, mirroring the sandboxd injector, and why an image override is a cold start.