Zero-downtime migration to new Kubernetes clusters
Moving a whole platform onto new clusters is the kind of change that goes wrong at 2 a.m. Here is how I did it with no downtime: a blue-green cutover at the cluster level, an internal DNS switch, and scale-to-zero as a one-command rollback.
Prepare application and data migration before DNS cutover, then validate the destination.
Contribution at a glance
- My responsibility
- Staged workload/data movement, destination validation and DNS cutover with rollback.
- Scope
- Cluster-level migration with legacy workloads retained during the transition.
- Collaboration & boundaries
- I coordinated the infrastructure move around workload and data dependencies; each service requires its own cutover acceptance.
- Outcome
- A repeatable phased migration strategy with a rollback route. No universal recovery-time or availability guarantee is inferred.
Rebuilding clusters is only half the job. Once the new clusters exist, every workload still has to move onto them, and "move production to a different cluster" is exactly the sentence that makes people schedule a midnight maintenance window. I wanted no window and no downtime, so I treated the whole thing as a blue-green deployment, just at the level of the cluster instead of a single service.
The strategy: blue-green at the cluster level
The new clusters ran in parallel with the legacy ones. Nothing was deleted until the new side was proven. The sequence:
- Stand up the workloads on the new clusters while the legacy clusters keep serving all live traffic.
- Cut over by pointing internal DNS for each migrated app at the new clusters. Traffic shifts at the DNS layer, not by redeploying callers.
- Scale the legacy workloads to zero rather than deleting them. They stay as a warm rollback: if anything looks wrong, scaling the old side back up and flipping DNS back is a one-command revert.
- Decommission only after soak, once the new side has run clean long enough to trust.
What actually had to move
A platform is more than its stateless pods. The migration covered the whole surface:
- Applications — the stateless services, redeployed onto the new clusters.
- Cloud Functions, schedulers and task queues — the event-driven and cron-driven glue that is easy to forget until something stops firing.
- Firestore data — the stateful part, migrated to the new project so the apps on the new clusters read from the right place.
- Scale schedules — the up/down scaling windows for non-production, recreated so the new clusters cost the same to idle as the old ones.
Scale to zero, do not delete
The single most valuable move in a cluster migration is keeping the old workloads at zero replicas instead of deleting them. It costs almost nothing to park an idle deployment, and it turns rollback from a rebuild into a one-line scale command plus a DNS flip. Delete only after the new side has earned your trust.
The parts people forget
The migration is not done when traffic flows. Two cleanups mattered:
- Monitoring that still probes the old hosts. After the cutover, blackbox probes were checking the legacy internal hostnames and showing a wall of false "down" alerts. Repointing them at the migrated hosts was what actually made the migration "finished".
- Tooling left behind on the legacy clusters. Cost agents and other cluster-scoped add-ons had to be uninstalled cleanly (Helm releases plus leftover CRDs and namespaces), so the empty clusters really were empty before teardown.
What I would tell someone starting this
- Treat a cluster move as blue-green. Run both in parallel, cut over at the DNS layer, and never delete the old side until the new one is proven.
- Scale the old workloads to zero, do not delete them. That is your rollback, and it is nearly free to keep.
- Inventory the whole surface first: functions, schedulers, task queues and stateful data, not just the obvious pods.
- Finish the job. Repoint monitoring at the new hosts and clean up the old clusters, or you will chase phantom alerts for weeks.
Decisions and next steps
A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.
Alternative and rationale
An in-place move reduces parallel capacity; parallel clusters enable destination validation before shifting traffic.
Trade-off
Parallel operation adds temporary cost, DNS/cache behavior and state-synchronization risks.
What I would improve next
Next improvement: record per-service acceptance, rollback time and the conditions for safely removing legacy capacity.