Zero-Downtime Migration from Cluster Autoscaler to Karpenter

Background: CAS and Karpenter

The Challenge

The Official Migration Guide

1. Deploy Karpenter alongside your existing cluster with a default NodePool
2. Set node affinity so Karpenter itself runs on CAS-managed nodes
3. Scale CAS to zero replicas: kubectl scale deploy/cluster-autoscaler --replicas=0
4. Shrink ASG desired counts to force nodes out: aws eks update-nodegroup-config --scaling-config "minSize=2,maxSize=2,desiredSize=2"
5. Watch Karpenter logs and verify new nodes appear as old ones are removed

• Rollback. The guide doesn't describe a rollback path. Once CAS is at zero replicas and ASGs are shrunk, reversing course means manually scaling everything back up.
• Safety net during migration. Stopping CAS removes the autoscaler before the migration is complete. If Karpenter hits a capacity issue, there is nothing to fall back on.
• Automation. The guide recommends scaling down "a few instances at a time" and watching carefully. For a fleet of clusters, this requires significant manual effort.
• Non-disruptive migration. Shrinking ASGs terminates EC2 instances directly — workloads are evicted, and Pod Disruption Budgets (PDBs) are the only protection. The guide itself notes: "If your workloads do not have pod disruption budgets set, the following command will cause workloads to be unavailable."

• Hands-off — change a config value, merge the PR, CI does the rest
• Observable — clear signal on progress at any point
• Reversible — hands-off, non-disruptive rollback

The Approach: Cordon and Wait

Migration Modes

graph LR
    A[cas-only] --> B[disabled]
    B --> C[rollout]
    C --> D[done]
    C --> E[rollback]
    E --> B

• disabled deploys Karpenter with bounded validation capacity (32 CPU, 256Gi) while CAS remains primary. Scheduling constraints keep ordinary workloads away while NodePools and EC2NodeClasses are validated.
• rollout lowers CAS --max-nodes-total to an environment-specific global ceiling, leaving controlled fallback capacity while Karpenter takes over.
• done removes all CAS resources (NodeGroups, autoscaler deployment, IAM roles) automatically through IaC.

The CronJob: Rollout Script

1. Pre-flight Checks

# Controller pods running?
kubectl get pods -n karpenter -l app.kubernetes.io/name=karpenter

# All NodePools ready?
kubectl get nodepools -o jsonpath='{range .items[*]}{.metadata.name}={.status.conditions[?(@.type=="Ready")].status} {end}'

# All EC2NodeClasses ready?
kubectl get ec2nodeclasses -o jsonpath='{range .items[*]}{.metadata.name}={.status.conditions[?(@.type=="Ready")].status} {end}'

2. Cordon CAS Nodes

LABEL_KEY="example.com/cordoned-by"
LABEL_VALUE="karpenter-migration"

for node in $(kubectl get nodes -l 'eks.amazonaws.com/nodegroup' \
  --field-selector spec.unschedulable!=true -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node "$node" "$LABEL_KEY=$LABEL_VALUE" --overwrite
  kubectl cordon "$node"
done

3. Restart One Workload (After Delay)

• StatefulSet with RollingUpdate → kubectl rollout restart, where its update strategy and availability checks permit it
• StatefulSet with OnDelete → excluded from the unattended path and handled by an approved runbook
• Deployment → kubectl rollout restart (via the pod's ReplicaSet owner), where its update strategy and availability checks permit it

The CronJob: Rollback Script

# Uncordon nodes we cordoned
for node in $(kubectl get nodes -l "eks.amazonaws.com/nodegroup,$MIGRATION_LABEL" \
  --field-selector spec.unschedulable=true -o jsonpath='{.items[*].metadata.name}'); do
  kubectl uncordon "$node"
  kubectl label node "$node" "$LABEL_KEY-"
done

# Retire one Karpenter node after CAS capacity and health are verified
nc=$(kubectl get nodeclaims -o jsonpath='{.items[0].metadata.name}')
if [ -n "$nc" ]; then
  kubectl delete nodeclaim "$nc" --wait=false
fi

IaC Integration

NodePool Limits Change Per Mode

disabled / rollback → { cpu: "32", memory: "256Gi" }    # bounded validation/handback capacity
rollout / done      → limits from config                # Karpenter is primary

CAS Gets Capped During Rollout

done Mode Removes CAS Entirely

• Skip creating EKS Managed NodeGroups and Launch Templates
• Skip deploying the Cluster Autoscaler Helm chart
• Skip creating CAS IAM roles
• Skip creating the migration CronJob

Monitoring

1. Node counts — CAS nodes should trend to zero, Karpenter nodes should grow:

echo "CAS: $(kubectl get nodes -l eks.amazonaws.com/nodegroup --no-headers | wc -l)"
echo "Karpenter: $(kubectl get nodes -l karpenter.sh/nodepool --no-headers | wc -l)"

2. Pod placement — find non-DaemonSet pods still on CAS nodes:

CAS_NODES=$(kubectl get nodes -l eks.amazonaws.com/nodegroup \
  -o jsonpath='{.items[*].metadata.name}' | tr ' ' '|')
kubectl get pods -A -o json | jq -r --arg nodes "$CAS_NODES" \
  '.items[] | select(.spec.nodeName | test($nodes)) |
   select(.metadata.ownerReferences[0].kind != "DaemonSet") |
   "\(.metadata.namespace)/\(.metadata.name)"'

# Pods on CAS nodes
count(
  kube_pod_info
  * on(pod, namespace) group_left() (max by(pod, namespace) (kube_pod_owner{owner_kind!="DaemonSet"}))
  * on(node) group_left() (kube_node_labels{label_eks_amazonaws_com_nodegroup!=""})
) or vector(0)

# Pods on Karpenter nodes
count(
  kube_pod_info
  * on(pod, namespace) group_left() (max by(pod, namespace) (kube_pod_owner{owner_kind!="DaemonSet"}))
  * on(node) group_left() (kube_node_labels{label_karpenter_sh_nodepool!=""})
) or vector(0)

# CAS nodes
count(kube_node_labels{label_eks_amazonaws_com_nodegroup!=""}) or vector(0)

# Karpenter nodes
count(kube_node_labels{label_karpenter_sh_nodepool!=""}) or vector(0)

Real-World Results

Lessons Learned

1. It is faster than expected. Natural churn does most of the work. Deployments, CronJobs, scaling events, and CAS draining underutilised nodes — all of this is already happening in any active cluster. Cordoning simply redirects it to Karpenter. The forced restart mechanism is only needed for the long tail.
2. One restart at a time contains failure. The CronJob selects one eligible controller per run, then waits for its rollout and application-health gates. Because PDBs do not govern controller-driven restarts, we enforce availability through update strategy, replica count, workload-specific safeguards, and health gates. Singleton and other unsupported workloads stay outside the unattended path. We did not experience a capacity incident or application downtime during these migrations.
3. Cap CAS; do not disable it. During rollout, CAS is still running under a global node ceiling. It can preserve a rollback path and may scale its existing ASGs while below that ceiling, so the chosen value must retain the capacity needed for system workloads and handback.
4. Label what is cordoned. The migration label (example.com/cordoned-by=karpenter-migration) is what makes rollback safe. Without it, the rollback script would uncordon nodes that were cordoned for maintenance or by other automation.
5. Keep the Descheduler. After migration completes, Karpenter consolidates underutilised nodes. The Descheduler helps rebalance pods that end up unevenly distributed after consolidation events.

Adapting This for Your Environment

1. A mode flag in your cluster config that controls NodePool limits, CAS settings, and CronJob deployment
2. Two shell scripts — rollout (cordon + delayed restart) and rollback (uncordon + delete NodeClaims)
3. A serialized CronJob with concurrencyPolicy: Forbid, a Lease, fail-closed health gates, and RBAC to read/patch nodes, read Karpenter CRDs, and restart allowlisted workloads
4. A tracking label on cordoned nodes to distinguish migration cordons from manual ones

Conclusion

This article is provided as a general guide for general information purposes only. It does not constitute advice. CECG disclaims liability for actions taken based on the materials.