NewCompare CPU & GPU pricing across AWS, Azure & GCP
Self-hosted sandboxes

Exposure modes

How agents outside your cluster reach sandboxes, what each exposure mode sends through DevZero, who holds the DNS name, certificate and key, and how to fix a failed setup.

Exposure modes

Agents outside your cluster create sandboxes through the DevZero API, then talk to them through the sandbox gateway, a proxy that runs in your cluster. The exposure mode decides how traffic from outside reaches that gateway. You choose one mode per cluster.

In every mode, TLS ends at the gateway in your cluster, the gateway's private key never leaves your cluster, and DevZero cannot read sandbox traffic. In-cluster callers always use the gateway's internal Service, devzero-sandbox-gateway.devzero-system.svc, whatever the mode.

Gateway sizing

The gateway runs at least two replicas, one per node, and never on a sandbox node pool. Each replica requests 1.5 CPU and 1 GiB of memory, with limits of 2 CPU and 2 GiB:

gateway:
  resources:
    requests: { cpu: 1500m, memory: 1Gi }
    limits: { cpu: "2", memory: 2Gi }

So each replica needs a node with 1.5 CPU free after its DaemonSets. A 2-vCPU node (for example an m6i.large) has about 1.45 to 1.7 CPU left, depending on which DaemonSets run on it, so the default fits on some 2-vCPU nodes and not on others; nodes of 4 vCPU or more have room with typical DaemonSets. A rollout replaces one replica at a time and starts the new one first, so it needs one more eligible node than there are replicas, or a cluster autoscaler that adds one.

If your cluster has only 2-vCPU nodes, install with gateway.resources.requests.cpu: 1200m. The replicas then fit, but under full load the gateway adds more latency and throughput overhead than at the default.

Autoscaling

The chart installs a HorizontalPodAutoscaler for the gateway, on by default:

gateway:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 20
    targetCPUUtilizationPercentage: 60   # of the CPU request: 900m at the default
    activeStreams:
      enabled: false
      averageValue: 600

At the default request it scales out at about 750 to 830 open streams per replica, before the per-replica limit of 1,000. The CPU target needs the Kubernetes metrics API (metrics-server, which EKS does not install by default); without it the HPA cannot read CPU and the gateway stays at minReplicas. Because replicas run one per node, every replica the HPA adds needs another node with room, so set maxReplicas no higher than the nodes you have or your autoscaler can add. activeStreams also scales on open streams, but needs a custom-metrics adapter (such as prometheus-adapter) serving the gateway's stream metric; leave it off until you have one. With autoscaling.enabled: false, the chart runs gateway.replicas (2) instead.

Comparing the modes

BYO endpointManaged endpointRelay
AvailablePhase 1Phase 2Phase 3
Hostnameyours (for example sbx.acme.internal)<cluster>.sbx.devzero.io<cluster>.sbx.devzero.io
Load balancerDevZero creates an L4 load balancer, or you route to an internal Service yourselfDevZero creates an L4 load balancernone needed; the gateway dials out to DevZero's edge
DNS recordsyou create themDevZero creates themDevZero creates them
TLS certificateyours, from any CAissued by DevZero through ACME (Let's Encrypt)yours or DevZero's
TLS private keyyours, in a Secret in your clustergenerated in your cluster; DevZero sees only a certificate signing requestgenerated in your cluster
Sandbox bytes through DevZerononenoneyes, encrypted end to end; DevZero's edge forwards TLS bytes without decrypting them, and stores nothing
Metadata to DevZerosandbox status, image reference, environment variable names, byte counts, IDs, command hashes (same in every mode)samesame, plus connection metadata at the edge
Can DevZero issue a certificate for your hostname?noyes: DevZero controls the DNS name, so a coerced DevZero could obtain a new certificate for it (visible in Certificate Transparency logs)with a BYO certificate, no; with a managed name, as for managed mode
Requiresa hostname, a wildcard certificate, and either load balancer permissions or your own ingressload balancer permissionsthe guardrail endpoint.allowRelay: true

Once all three exist, managed is the default, then BYO, then relay. If you need DevZero unable to either decrypt or issue certificates for your traffic, use a BYO certificate, with relay where you need it.

The full disclosure table is on Trust and data disclosure.

Set up a BYO endpoint (Phase 1)

Choose a hostname and get a certificate

Pick a hostname, for example sbx.acme.internal. Get a certificate whose subject alternative names include both the hostname and its wildcard (sbx.acme.internal and *.sbx.acme.internal). A corporate CA is fine: the SDKs use the system trust store and accept a custom CA bundle. Without the wildcard name the endpoint reports TLSReady=False/MissingWildcardSAN.

Store it in your cluster

Create the TLS Secret sandbox-gateway-tls in devzero-system, once, as a cluster admin. It must carry the label sandbox.devzero.io/gateway-tls=true: the gateway serves the certificate only while the label is there. DevZero never modifies, copies or deletes this Secret, and the key in it never leaves your cluster.

kubectl -n devzero-system create secret tls sandbox-gateway-tls \
  --cert=fullchain.pem --key=privkey.pem
kubectl -n devzero-system label secret sandbox-gateway-tls \
  sandbox.devzero.io/gateway-tls=true

To rotate the certificate later without cluster-admin rights, list the people or service accounts (cert-manager, for example) in the chart value byoTLS.writers. The installer binds them to the Role devzero-sandbox-byo-tls-writer, which allows get, update and patch on this one Secret and nothing else. It cannot allow create, because Kubernetes RBAC cannot limit create to one name, so the first copy is always created by an admin.

Choose how traffic arrives

Decide three things:

  • the hostname, for example sbx.acme.internal (lower-case, at least two DNS labels, without the wildcard);
  • public or internal visibility (internal is the default; public is refused if your cluster's guardrails disallow it);
  • whether DevZero creates the load balancer, or you route to the gateway yourself.

The two ways traffic can arrive:

  • DevZero creates a load balancer. The sandbox operator creates Service/devzero-sandbox-gateway-lb-public (or -internal), a layer-4 TCP load balancer, and reports its address on the endpoint. Point DNS for the hostname and its wildcard at that address. Before you choose this:
    • On AWS, install the AWS Load Balancer Controller and tag your subnets (kubernetes.io/role/elb for public, kubernetes.io/role/internal-elb for internal). Sandbox load balancers are NLBs with IP targets, which only that controller provisions; without it the load balancer never gets an address and the pre-flight check fails after 5 minutes. The controller and the cloud need the permissions below. To keep a rollout from sending new connections to a gateway pod that is going away, also label devzero-system with elbv2.k8s.aws/pod-readiness-gate-inject=enabled and install dz-sandbox with gateway.loadBalancerReadinessGate=aws.
    • Set the chart value operator.cloud (aws, azure or gcp) when you install, so the operator renders the right load balancer annotations. The operator does not detect your cloud. With the default, none, the cloud's default scheme applies and may be internet-facing, so the operator creates a load balancer only for public visibility and refuses internal visibility with LoadBalancerReady=False and reason CloudNotConfigured.
  • You route to it. Only the internal ClusterIP Service, devzero-sandbox-gateway on port 443, exists. Send traffic to it from your own ingress with TLS passthrough, so TLS still ends at the gateway. See Routing with your own ingress.

Set the endpoint in DevZero

A team Admin sets the endpoint with the SetSandboxEndpoint API call. There is no endpoint settings page in the dashboard yet. If you prefer, send the same settings to your DevZero contact and DevZero sets it for you.

curl -sS https://dakr.devzero.io/sandbox.v1.SandboxService/SetSandboxEndpoint \
  -H "Authorization: Bearer $DEVZERO_TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "clusterId": "<cluster id>",
    "spec": {
      "mode": "ENDPOINT_MODE_BYO",
      "hostname": "sbx.acme.internal",
      "visibility": "ENDPOINT_VISIBILITY_INTERNAL",
      "loadBalancer": {
        "managedByDevzero": true,
        "sourceRanges": ["10.0.0.0/8"]
      }
    }
  }'
FieldValues
modeENDPOINT_MODE_BYO, the only mode Phase 1 accepts
hostnamethe base hostname; the certificate must cover it and *. of it
visibilityENDPOINT_VISIBILITY_INTERNAL (default) or ENDPOINT_VISIBILITY_PUBLIC
loadBalancer.managedByDevzerotrue makes the operator create the load balancer; leave loadBalancer out to route to the ClusterIP Service yourself
loadBalancer.sourceRangesCIDRs allowed to reach the load balancer (at most 64)
loadBalancer.extraAnnotationsextra annotations for the load balancer Service (at most 32; the keys that set the scheme are refused, because visibility decides it)
loadBalancer.preserveClientIppreserve client IPs on Azure or GCP; see Permissions per cloud

The loadBalancer settings other than managedByDevzero need managedByDevzero: true. The call replaces the whole endpoint, so send every field each time. GetSandboxEndpoint with {"clusterId": "<cluster id>"} returns the current settings and whether the cluster has applied them (apply.state). The call is refused with cluster_unreachable while the DevZero operator's sandbox relay is not connected.

DevZero applies the settings to the cluster as SandboxEndpoint/default. Your part stays in your cluster and your DNS: the certificate in sandbox-gateway-tls (step 2) and the DNS records for the hostname and its wildcard. Until the endpoint is set, sandboxes you create have no URL to connect to.

Check it

The endpoint's status in your cluster reports LoadBalancerReady (when DevZero creates the load balancer), TLSReady and Ready. Read it, resolve the hostname and test the connection as in Check an endpoint by hand.

Routing with your own ingress (TLS passthrough)

The controller must run with --enable-ssl-passthrough.

ingress-nginx picks a passthrough backend by an exact match on the TLS server name. A wildcard rule such as *.sbx.acme.internal never matches a real name such as 8080-abc.sbx.acme.internal; those connections reach nginx's own TLS server, which terminates them instead of passing them to the gateway (kubernetes/ingress-nginx#11982, closed as not planned). So the example routes only the bare hostname, with no wildcard rule.

Phase 1 sandbox URLs all use the bare hostname (https://sbx.acme.internal/t/<team>/s/<id>/…), so this route serves everything Phase 1 offers. Port previews (Phase 2) use per-sandbox names under *.sbx.acme.internal, which ingress-nginx cannot pass through; for those, use a Gateway API TLSRoute or let DevZero create the load balancer.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: devzero-sandbox
  namespace: devzero-system
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
    - host: sbx.acme.internal
      http: { paths: [{ path: /, pathType: Prefix, backend: { service: { name: devzero-sandbox-gateway, port: { number: 443 } } } }] }

An ingress that terminates TLS and forwards HTTP is not supported. It would break the gateway's certificate model and, later, native gRPC. Use passthrough.

Public and internal visibility

Public suits agents outside your network. Internal puts the load balancer on private addresses, for callers on your VPC or VPN. Internal endpoints have two caveats:

  • DNS rebinding protection can block them. In BYO and managed modes the public DNS name resolves to a private (RFC 1918) address. Resolvers with rebinding protection refuse such answers: dnsmasq with stop-dns-rebind, Pi-hole, many home and office routers, and some zero-trust network agents. A caller behind one sees the name fail to resolve even though the load balancer is healthy.
  • Public DNS reveals private addresses. Publishing the record exposes a private IP and so a little of your VPC layout. If that matters, serve the record from a private DNS zone instead (Route 53 private hosted zone, Azure Private DNS, Cloud DNS private zone). In BYO mode you control DNS, so create the record only in your private zone. For managed mode, a private-zone variant where DevZero hands you the records and publishes nothing publicly arrives in Phase 2.

Changing visibility creates a second load balancer Service of the new kind, because AWS and GCP fix a load balancer's scheme at creation. The operator removes the old one as soon as the new one has an address. In BYO mode DNS is yours, so point the hostname and its wildcard at the new address straight away; until you do, callers resolving the old address cannot connect.

When the load balancer fails: the pre-flight check

Before DevZero creates a load balancer, the sandbox operator tests whether it can.

On AWS (operator.cloud=aws), from dz-sandbox 0.1.2 it first checks that the AWS Load Balancer Controller is installed (the cluster serves its elbv2.k8s.aws API group). If it is not, the endpoint reports LoadBalancerCapable=False with reason LoadBalancerControllerMissing within one reconcile, its message names the controller with its install link and the internal-Service alternative, and no throwaway Service is created. The check repeats every minute, so installing the controller clears it with no change to the endpoint.

Otherwise it creates a throwaway Service, devzero-sandbox-preflight-lb, and waits up to 5 minutes for an address. While it waits, the endpoint reports LoadBalancerCapable=Unknown with reason LoadBalancerPending, and the condition's message carries the cloud's latest error as soon as the cloud reports one (usually within seconds), so a missing permission or untagged subnet shows up without waiting out the 5 minutes:

kubectl get sandboxendpoint default -o jsonpath='{.status.conditions[?(@.type=="LoadBalancerCapable")].message}'

The cloud's error is copied from a Kubernetes Event on the Service. When the cloud reports the same failure repeatedly, Kubernetes merges the events, and the message then starts with (combined from similar events): before the cloud's own text. The error after the prefix is the one to act on.

  • Success: the Service is deleted and the endpoint reports LoadBalancerCapable=True.
  • Failure: the endpoint reports LoadBalancerCapable=False with the cloud's or the admission controller's error word for word: reason LoadBalancerForbidden when the API server or an admission policy refused the Service, LoadBalancerPending when no address arrived in time. The operator creates no load balancer rather than leave a Service pending forever; route to the internal Service yourself instead. A refused check is retried periodically, so lifting the policy is enough; a timed-out check, which holds a cloud load balancer for up to 5 minutes, is not retried until the endpoint settings change.

Common causes:

Error looks likeCauseFix
admission webhook denied the request (Kyverno, Gatekeeper)a cluster policy forbids type: LoadBalancergrant an exception for devzero-system, or route to the internal Service with your own ingress
reason LoadBalancerControllerMissing, on EKS (0.1.2 and later)the AWS Load Balancer Controller is not installedinstall it (see Choose how traffic arrives); sandbox load balancers are NLBs with IP targets. The check reruns every minute on its own
no controller has recorded an Event on it, on EKS (0.1.2 and later)the controller's CRDs are installed but nothing is acting: it is scaled to zero or crash-loopingget the controller running again (usually Deployment kube-system/aws-load-balancer-controller)
got no load-balancer address within 5m0s and the cloud reported no error, on EKS (0.1.1 and earlier)the AWS Load Balancer Controller is not installed, so nothing acts on the Serviceinstall it (see Choose how traffic arrives), or upgrade to 0.1.2, which reports this at once. A timed-out check reruns only when the endpoint settings change; setting identical settings again changes nothing
access denied / not authorized from the cloud APIthe cloud controller lacks permissionsgrant the permissions below
no subnets found, on EKSsubnets lack the load balancer role tagstag them (kubernetes.io/role/elb for public, kubernetes.io/role/internal-elb for internal)
LoadBalancerReady=False, reason CloudNotConfiguredinternal visibility, but the operator was installed with operator.cloud: noneset operator.cloud to aws, azure or gcp and upgrade the chart, use public visibility, or route to the internal Service yourself
quota exceededload balancer or IP address quota reached in the accountraise the quota or free addresses

Permissions per cloud

The AWS Load Balancer Controller needs elasticloadbalancing:Create* and elasticloadbalancing:Describe* (as in its standard IAM policy), and the subnets need the role tags above. With IP targets the NLB health-checks and reaches the gateway pods directly.

The operator turns on client IP preservation on the NLB's target group (service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true), so the gateway sees each caller's real address. It needs that address to rate-limit failed authentication per client, and without it loadBalancerSourceRanges may not be enforced on an NLB that has no security group. Target group attributes you set yourself in loadBalancer.extraAnnotations are kept, and the operator's setting is appended; if you set preserve_client_ip.enabled yourself, your value wins.

With preservation on, the NLB does not support NAT loopback: a connection to an internal NLB from a pod on the same node as a gateway replica can be routed back to that node and time out. In-cluster callers should use the devzero-sandbox-gateway Service (devzero-sandbox-gateway.devzero-system.svc), not the NLB address. If something in the cluster must go through the NLB, set preserve_client_ip.enabled=false in extraAnnotations; the gateway then sees the NLB's address for every caller and can no longer rate-limit per client.

Check an endpoint by hand

dz sandbox doctor arrives with the DevZero CLI in Phase 2. It will print every endpoint and policy condition with its remediation, resolve the hostname, test TCP on port 443, and exit non-zero when something is wrong. Until then, the same checks by hand:

# Endpoint and pre-flight conditions, including the verbatim cloud error
kubectl get sandboxendpoint default -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason} {.message}{"\n"}{end}'

# Does the name resolve, and to what? (an internal endpoint should give a private address)
dig +short sbx.acme.internal
dig +short test.sbx.acme.internal

# Is port 443 reachable, and does the certificate cover the wildcard?
openssl s_client -connect sbx.acme.internal:443 -servername test.sbx.acme.internal </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -ext subjectAltName

If dig returns nothing from your laptop but works from a VPC host, suspect DNS rebinding protection (above).

On this page