GCP GKE
Set up the DevZero Node Operator on Google Cloud Platform Google Kubernetes Engine (GCP GKE).

This guide will help you onboard onto DevZero Karp in an GKE cluster, we make the following assumptions:
- You will use an existing GKE cluster
- The cluster has dakr-operator installed
- You have
gcloud,helm,kubectlinstalled
Set Environment Variables
export PROJECT_ID=<your-google-project-id>
export GSA_NAME=karpenter-gsa
export CLUSTER_NAME=<gke-cluster-name>
# region or zone depending on if you are running a zonal or regional cluster
export REGION=<gke-region-or-zone-name>
export DZKARP_VERSION="1.4.0"Enable Required APIs
dzKarp prices attached GPUs from the Cloud Billing Catalog and exits at startup if the Cloud Billing API is disabled on the project. Enabling it is safe to repeat and is also required when upgrading from a version before 1.4.0.
gcloud services enable cloudbilling.googleapis.com --project=$PROJECT_IDCreate GCP Service Account
# Create Google Service Account
gcloud iam service-accounts create $GSA_NAME --project=$PROJECT_ID
# Add required IAM roles
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/compute.admin"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/container.admin"
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/iam.serviceAccountUser"Ensure Cluster Supports Workload Identity ```bash gcloud container clusters
update REGION
--workload-pool=$PROJECT_ID.svc.id.goog ```
Create Dedicated Nodepool for DzKarp
gcloud container node-pools create dzkarp \
--cluster=$CLUSTER_NAME \
--location=$REGION \
--workload-metadata=GKE_METADATA \
--machine-type=e2-medium \
--num-nodes=2Configure Workload Identity Binding
gcloud iam service-accounts add-iam-policy-binding $GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT_ID.svc.id.goog[devzero-system/karpenter]"Deploy dzKarp
Install dzKarp via Helm
# Logout of helm registry to perform an unauthenticated pull against the public ECR
helm registry logout public.ecr.aws
# CRDs first: the karpenter-crd release is what delivers CRD updates on later
# upgrades (CRDs bundled in the controller chart are never upgraded by Helm).
helm upgrade --install karpenter-crd oci://public.ecr.aws/devzeroinc/dzkarp-gcp/karpenter-crd \
--version "${DZKARP_VERSION}" \
--namespace kube-system
helm upgrade --install karpenter oci://public.ecr.aws/devzeroinc/dzkarp-gcp/karpenter \
--version "${DZKARP_VERSION}" \
--namespace devzero-system --create-namespace \
--set "controller.settings.projectID=${PROJECT_ID}" \
--set "controller.settings.location=${REGION}" \
--set "controller.settings.clusterName=${CLUSTER_NAME}" \
--set controller.featureGates.spotToSpotConsolidation=true \
--set controller.settings.implicitPDBMinAvailable="20%" \
--set "credentials.enabled=false" \
--set "serviceAccount.annotations.iam\.gke\.io/gcp-service-account=${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
--waitimplicitPDBMinAvailable specifies the minimal available subset of pods in any deployment or statefulset dzkarp must maintain during node disruption for which an explicit pdb is not already defined.
This value can be a percentage string or a natural number (for a nominal pod count). Set this value to zero if you do not want a implicit fallback pdb.
Verify dzKarp
kubectl get pods -n devzero-system
kubectl logs -n devzero-system deployment/karpentersee that no unexpected errors are produced
Upgrade dzKarp
To move to a newer release, set DZKARP_VERSION to the new version and re-run the two helm upgrade --install commands from the Deploy step, CRDs first. Check the project prerequisites first: upgrading from a version before 1.4.0 needs the Cloud Billing API enabled (see Enable Required APIs), or the controller exits at startup.
If you originally installed without the karpenter-crd release (chart 1.0.x and earlier installed the CRDs from the controller chart), let Helm adopt the existing CRDs once before installing it. Helm refuses to take over resources without this ownership metadata.
for crd in $(kubectl get crd -o name | grep -E '\.(karpenter\.sh|karpenter\.k8s\.gcp|devzero\.karpenter\.sh)$'); do
kubectl label "$crd" app.kubernetes.io/managed-by=Helm --overwrite
kubectl annotate "$crd" meta.helm.sh/release-name=karpenter-crd meta.helm.sh/release-namespace=kube-system --overwrite
doneIf you installed through the DevZero dashboard's setup script instead, re-run that script: it enables any newly required APIs, reconciles IAM, and reinstalls at the current version.
Set nodeAffinity for critical workloads (optional)
Autoscaled nodes can be prone to churn and result in workload disturbance.
You may want to set a nodeAffinity critical cluster workloads to mitigate this.
Some examples are
- coredns
- metric-server
add the following to your cluster critical workload deployments
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExistCreate Node Policy
We need to create a Node Policy in DevZero, and have it target the cluster on which dzKarp was just installed.
Head over to the optimization dashboard, click on "Create Node Policy" and follow the form to create a policy suitable for your needs.
After the Policy is created, click on it in the menu and point it at the cluster you just created via "Create Target".
In about a minute this should create nodepool and nodeclass objects in your kubernetes cluster.
Check them out:
kubectl describe gcenodeclass
kubectl describe nodepoolsMigrate workloads onto autoscaled nodes
If your workloads do not have pod disruption budgets set, the following commands will cause periods of workload unavailability
If you have cluster-autoscaler installed, it must be disabled first, scale its deployment down to zero before you proceed.
The most cost ideal state of running dzKarp is to have as few manually configured nodes as possible. The below command will drain all nodepools not called dzkarp
for POOL_NAME in $(gcloud container node-pools list --cluster=$CLUSTER_NAME --location=$REGION --format="value(name)" | grep -v '^dzkarp$'); do
echo "Scaling down nodepool: $POOL_NAME to 0 nodes (and disabling autoscaling)..."
gcloud container clusters update $CLUSTER_NAME \
--no-enable-autoscaling \
--node-pool=$POOL_NAME \
--location=$REGION \
--quiet
gcloud container clusters resize $CLUSTER_NAME \
--node-pool=$POOL_NAME \
--num-nodes=0 \
--location=$REGION \
--quiet
doneIf you have a lot of nodes or workloads you may want to slowly scale down your node groups by a few instances at a time. It is recommended to watch the transition carefully for workloads that may not have enough replicas running or disruption budgets configured.
As nodepools are drained you can verify that dzKarp is creating nodes for your workloads.
kubectl logs -n devzero-system deployment/karpenterYou should also see new nodes created in your cluster as the old nodes are removed.
kubectl get nodes