Upgrade to NKP 2.18, Part 4: The Kubernetes Node Roll (and the PDB That Froze It)

Platform at v2.18.0 everywhere (Part 2), Loki migrated (Part 3). Now the actual Kubernetes upgrade: the management cluster's nodes move from 1.34.3 to 1.35.2, new OS image included. This is the phase where VMs get replaced, which means your workloads' PodDisruptionBudgets now matter.

What this phase does#

nkp upgrade cluster nutanix upgrades the CAPI stack itself (this is where the v1beta2 migration happens), updates the ClusterClass, runs 2.18's new preflight checks, then replaces every node: a new VM provisions from the 1.35.2 image, joins, the old node drains and is deleted. Control plane first, one at a time with etcd quorum held, then workers. Around 12 minutes per node on AHV in our run; 9 nodes took about 3 hours including a 67-minute stall we planted ourselves.

What each node gets: Kubernetes 1.35.2, Rocky Linux 9.7, containerd v2 (2.1.6, up from 1.7.29), and a control plane with a TLS hardening pass nobody documented (the release article has the details).

Step 1: find your real cluster name (the docs get this wrong)#

The guide says to read the management cluster's name from the NKPCluster host label. On clusters upgraded from 2.17 that returns the adopted object's name (host-cluster), and the CLI then fails:

$ nkp upgrade cluster nutanix --cluster-name host-cluster ...
error getting cluster: clusters.cluster.x-k8s.io "host-cluster" not found

Adoption names the NKPCluster after the KommanderCluster; the CAPI cluster keeps its real name. Use the CAPI name and namespace:

kubectl get cluster.cluster.x-k8s.io -A     # find the real name + namespace

Step 2: run the upgrade#

nkp upgrade cluster nutanix \
  --cluster-name itcs-nkp-mgmt-prod -n default \
  --vm-image "nkp-rocky-9.7-release-cis-1.35.2-20260626061237.qcow2"

Output from our run:

 ✓ Installing CAPI Stack operators
 ✓ Creating CAPI components            <- CAPI v1beta2 migration happens here
 ✓ Updating ClusterClass resources
 ✓ Waiting for ClusterClasses to be ready
 ✓ Running preflight checks            <- new in 2.18, quiet when it passes
 ✗ Upgrading the cluster
error while waiting for cluster to be ready: context deadline exceeded

That final timeout is normal on any real cluster: the CLI's wait window cannot cover 9 nodes at 10 minutes each. The roll is controller-driven and continues without the CLI. Two immediate signs the deep changes landed: every kubectl command starts warning cluster.x-k8s.io/v1beta1 Machine is deprecated; use v1beta2, and the first replacement machine appears within minutes.

Step 3: watch the roll#

From here, kubectl is the progress bar:

# Version distribution shifting old -> new:
watch -n15 'kubectl get nodes --no-headers | awk "{print \$5}" | sort | uniq -c'

# Machine churn:
kubectl get machines -A -w

# A node stuck in SchedulingDisabled? Its Machine object says exactly why:
kubectl get machine <machine-name> -n <ns> \
  -o jsonpath='{range .status.conditions[?(@.status!="True")]}{.type}: {.message}{"\n"}{end}'

Our watch-node-upgrade.sh script (published at github.com/fen0l/nkp-quickstart) shows all of this in one refreshing pane: version distribution, machines in transition, cordoned nodes with their drain blocker, and the PDB list.

Step 4: if the roll freezes, it is almost certainly a PDB#

We planted this failure on purpose because it is the classic upgrade killer: a perfectly healthy application whose PodDisruptionBudget allows zero disruptions (minAvailable equal to replica count). The roll reached the node hosting one of its pods and froze. Completely, and invisibly: no failing pods, no red HelmReleases, no events in the app's namespace. Just one cordoned node, indefinitely. We let it sit to measure the impact: 67 minutes frozen, and it also consumed the CLI's wait deadline.

The only object that tells the truth is the Machine:

DrainingSucceeded: Drain not completed yet (started at 2026-07-07T16:43:53Z):
* Pod upgrade-showcase/pdb-blocked-app-...: cannot evict pod as it would violate
  the pod's disruption budget. The disruption budget pdb-blocked-app needs 2
  healthy pods and has 2 currently

Remediation - one line, instant recovery:

kubectl patch pdb pdb-blocked-app -n upgrade-showcase --type merge -p '{"spec":{"minAvailable":1}}'

The pod evicted within seconds, rescheduled onto an already-upgraded node, the drain completed, the roll resumed on its own. For contrast: our planted CrashLoopBackOff pods were evicted mid-roll without any issue. Broken apps do not block drains; healthy, over-protected apps do. Hence the Part 1 pre-check: kubectl get pdb -A | awk '$4=="0"' - fix every hit or schedule the intervention, because the upgrade will do neither for you.

Step 5: verify#

$ kubectl get nodes --no-headers | awk '{print $5}' | sort | uniq -c
      9 v1.35.2
$ kubectl get nodes -o jsonpath='{range .items[*]}{.status.nodeInfo.osImage} | {.status.nodeInfo.containerRuntimeVersion}{"\n"}{end}' | sort | uniq -c
      9 Rocky Linux 9.7 (Blue Onyx) | containerd://2.1.6-d2iq.1
$ kubectl get hr -A --no-headers | awk '$4!="True"' | wc -l
0

Plus the 2.18-specific checks: upgraded management clusters keep their node pools in the default namespace (only fresh 2.18 installs use kommander, NCN-113871), gatekeeper-audit is on the watch list for OOM (NCN-115572, clean on ours), and the Insights credentials-rotate CronJob is still broken (the release article has the check).

Step 6: repeat for the workload clusters#

Same command per CLI-managed cluster, same monitoring, same PDB caution:

nkp upgrade cluster nutanix --cluster-name <name> -n <namespace> \
  --vm-image "nkp-rocky-9.7-release-cis-1.35.2-20260626061237.qcow2"

GitOps-managed clusters are different: 2.18 wants the repository migrated to NKPCluster manifests, and the migration commit carries the Kubernetes upgrade with it. That is Part 5, including the webhook that rejects your old manifests and the RBAC your Flux service account is missing.