This part upgrades two workload clusters with an UpgradePlan object: one from kubectl, one from the UI. Prerequisites: management cluster on 2.19 (Part 3), and the Cilium reference from Part 4 if your node subnet is Network Controller managed. The first cluster took 27 minutes. The second stalled for a day on a documented PodDisruptionBudget issue.
CLI dry run
First cluster: 1 control plane and 3 workers, Rocky Linux 9.7, containerd 2.1.6, NDK 2.3.0, 18 HelmReleases Ready. The Cilium SNAT reference is set before the upgrade, to test that custom add-on values survive a roll.
Dry run of the documented command:
nkp upgrade cluster nutanix --cluster-name demo-wkl-02 --namespace debug-test-xhj92 \
--vm-image nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2 --dry-run -o yaml
✓ Waiting for ClusterClasses to be ready
✓ Running preflight checks
The output is the rewritten NKPCluster. These fields change: ClusterClass reference (nkp-nutanix-v2.18.0 to nkp-nutanix-v2.19.0), Kubernetes version, control plane and worker image names, KommanderCluster platform version. We create the UpgradePlan object directly.
Step 1: the Cilium reference, before anything else
Apply the ConfigMap of Cilium values from Part 4 in the cluster's namespace, then reference it on the NKPCluster:
kubectl apply -f demo-wkl-02-cilium-cni-helm-values-template.yaml
kubectl get nkpcluster demo-wkl-02 -n debug-test-xhj92 -o json \
| jq '(.spec.capiCluster.topology.variables[] | select(.name=="clusterConfig") | .value.addons.cni.values)
= {sourceRef: {kind: "ConfigMap", name: "demo-wkl-02-cilium-cni-helm-values-template"}}' \
| kubectl replace -f -
kubectl get cluster demo-wkl-02 -n debug-test-xhj92 \
-o jsonpath='{.spec.topology.variables[?(@.name=="clusterConfig")].value.addons.cni}{"\n"}'
No live edit. The cluster stays in DSR until the upgrade applies the change.
Step 2: the object
apiVersion: upgrade.nkp.nutanix.com/v1alpha1
kind: UpgradePlan
metadata:
name: demo-wkl-02-v1.36.2-v2.19.0
namespace: debug-test-xhj92
labels:
upgrade.nkp.nutanix.com/nkpcluster-name: demo-wkl-02
upgrade.nkp.nutanix.com/target-version: v2.19.0
spec:
nkpClusterRef:
name: demo-wkl-02
namespace: debug-test-xhj92
version: v2.19.0
kubernetesVersion: v1.36.2
controlPlaneImage: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2
workerImages:
md-0: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2
A server-side dry run (kubectl apply --server-side --dry-run=server -f) runs the UpgradePlan webhook: Prism Central version checked, both images found. The UI adds the annotation preflight.cluster.caren.nutanix.com/skip: "" (skip nothing). Omitting it is equivalent.
Step 3: what happened, minute by minute
kubectl get upgradeplan -n debug-test-xhj92 -w
kubectl get upgradeplan demo-wkl-02-v1.36.2-v2.19.0 -n debug-test-xhj92 \
-o jsonpath='{range .status.steps[*]}{.name}: {.state} {.startTime} -> {.endTime}{"\n"}{end}'
validating-inputs: Completed 2026-10-20T13:57:09Z -> 2026-10-20T13:57:15Z
upgrading-platform-applications: Completed 2026-10-20T13:57:15Z -> 2026-10-20T14:09:22Z
upgrading-kubernetes: Completed 2026-10-20T14:09:22Z -> 2026-10-20T14:24:04Z
A workload cluster's plan has three steps (no management-plane step):
- Validating inputs, 6 seconds: the webhook's Prism Central checks.
- Upgrading platform applications, 12 minutes. The HelmReleases on the cluster were already at their 2.19 charts since the Kommander upgrade. The step waits for the AppDeployment status on the management side. The UI tracker shows it for a plan created with kubectl too: "10 out of 15 platform applications are ready", waiting for cert-manager, external-secrets, gatekeeper, kube-prometheus-stack and opencost.
- Upgrading Kubernetes and cluster infrastructure, 15 minutes.
![]()
Sequence of the Kubernetes step:
- 16:09: the runtime hook fetches the chart information for every add-on (Cilium, Multus, cluster autoscaler, NFD, CSI, snapshot controller, COSI, MetalLB, cloud provider, Konnector agent) and applies the Cilium preflight release.
- 16:09: the hook rewrites the Cilium
HelmChartProxyfrom our ConfigMap (loadBalancer.mode: snat, Helm revision 3).cilium-confighasbpf-lb-mode: snatbefore the first new node exists. - 16:09 to 16:12: new control plane created, old one drained.
- 16:12 to 16:23: three workers replaced one at a time.
- 16:24: plan
Upgraded.
27 minutes from kubectl apply to Upgraded for four VMs. No stall.
Step 4: verify
$ kubectl --kubeconfig demo-wkl-02.conf get nodes -o custom-columns=NAME:.metadata.name,VER:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage,CRI:.status.nodeInfo.containerRuntimeVersion
NAME VER OS CRI
demo-wkl-02-md-0-9h798-sb45h-fdshs v1.36.2 Rocky Linux 9.8 (Blue Onyx) containerd://2.3.1-d2iq.1
demo-wkl-02-md-0-9h798-sb45h-ptrxx v1.36.2 Rocky Linux 9.8 (Blue Onyx) containerd://2.3.1-d2iq.1
demo-wkl-02-md-0-9h798-sb45h-t274x v1.36.2 Rocky Linux 9.8 (Blue Onyx) containerd://2.3.1-d2iq.1
demo-wkl-02-rfzj2-zszv2 v1.36.2 Rocky Linux 9.8 (Blue Onyx) containerd://2.3.1-d2iq.1
$ kubectl --kubeconfig demo-wkl-02.conf -n kube-system get cm cilium-config -o jsonpath='{.data.bpf-lb-mode}{"\n"}'
snat
$ kubectl get helmreleaseproxies -n debug-test-xhj92 -o custom-columns=NAME:.metadata.name,CHART:.spec.chartName,VER:.spec.version,REV:.status.revision
NAME CHART VER REV
cilium-demo-wkl-02-slfft cilium 1.19.4 3
konnector-agent-demo-wkl-02-dhhp2 konnector-agent 1.4.2 3
metallb-demo-wkl-02-rkvbd metallb 0.15.3 2
nkp-cloud-provider-nutanix-demo-wkl-02-4fhqr nkp-cloud-provider-nutanix 0.8.12 1
node-feature-discovery-demo-wkl-02-q458w node-feature-discovery 0.19.0 3
nutanix-csi-storage-demo-wkl-02-dpbth nutanix-csi-storage 3.9.0 3
(...)
Add-ons after the roll:
| Add-on | Before | After |
|---|---|---|
| Cilium | 1.19.4, DSR | 1.19.4, SNAT (new Helm revision) |
| Nutanix CSI | 3.7.1 | 3.9.0 |
| Konnector agent | 1.4.1 | 1.4.2 |
| Node Feature Discovery | 0.18.3 | 0.19.0 |
| Nutanix cloud provider | 0.6.2 | 0.8.12 (new release name) |
| MetalLB, snapshot controller, NDK 2.3.0 | unchanged |
All HelmReleases Ready, no pod not Running. From the bastion, the LoadBalancer address and a NodePort on each of the four nodes answer 3 of 3, with the MetalLB announcer on a node without a Traefik pod. A Cilium values ConfigMap referenced from addons.cni.values.sourceRef is applied by the upgrade, before the roll.
The second cluster: a documented PodDisruptionBudget stall
Second cluster: Nutanix Enterprise AI 2.8, 3 control-plane nodes, an 8-node pool for the NAI services, a 2-node inference pool.
The 2.19 release notes list a known issue for this setup. The upgrade can stall when NAI runs a single-instance gateway, NAI API and PostgreSQL with the PodDisruptionBudget enabled. The pods cannot be evicted, the old worker stays Draining, and later node pools do not start. Documented workaround (release notes): run several instances of the gateway and the NAI API, and disable the PostgreSQL PodDisruptionBudget before the upgrade.
On our cluster the gateway and the API already ran two replicas. The database PDB was still enabled when the upgrade was started from the UI at 16:05. This is what the documented stall looks like.

Timeline: inputs validated in 6 seconds, platform applications in 8 minutes, control plane rolled 16:14 to 16:22, four new pool nodes 16:24 to 16:30. At 16:31 the drain of the first old pool node stops:
$ kubectl get machine nai-demo-nai-pool-demo-c56m2-cn4gl-c6v6j -n nai-demo-jk9kl-zxbpw \
-o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' | grep -v ': $'
Ready: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode,
delay likely due to PodDisruptionBudgets
Deleting: Drain not completed yet (started at 2026-10-20T14:31:01Z):
* Pod nai-system/nai-db-iep-1: cannot evict pod as it would violate the pod's disruption budget.
The disruption budget nai-db-iep-primary needs 1 healthy pods (...)
$ kubectl --kubeconfig nai-demo.conf get pdb -A | awk '$5=="0"'
nai-system nai-db-iep-primary 1 N/A 0 15d
The PDB belongs to the CloudNativePG cluster that NAI deploys (cnpg.io/cluster: nai-db-iep, one instance, minAvailable: 1). CloudNativePG recreates it if you delete it. The inference pool does not start. The plan stays in UpgradingKubernetes. On our cluster the drain stayed blocked for 24 hours and 10 minutes.
The fix is the documented one, applied through the NAI operators Helm release rather than on the PDB itself, so that the operator does not put it back:
helm upgrade nai-operators <chart> -n nai-system --reuse-values --set naiDatabase.enablePDB=false
Result:
- After the Helm upgrade, the CloudNativePG operator removes the PDB.
- 16:43: Cluster API evicts the database pod.
- 16:46: the old machine is gone, the database pod runs on a new node, CloudNativePG reports "Cluster in healthy state". Database unavailable for about three minutes.
- 16:54: the inference pool starts rolling.
- 16:56: plan
Upgraded.
Do not use a MachineDrainRule or a nodeDrainTimeout here: both evict a single-instance database ungracefully. Do not delete the PDB by hand: its operator recreates it.
Run the PDB audit on every workload cluster, and read the known issues of the applications you run:
kubectl --kubeconfig <workload>.conf get pdb -A | awk '$5=="0"'
State after the upgrades
$ kubectl get nkpclusters -A
NAMESPACE NAME NKP VERSION KUBERNETES VERSION PHASE
debug-test-xhj92 demo-wkl-02 v2.19.0 v1.36.2 Reconciled
kommander host-cluster v2.19.0 v1.36.2 Reconciled
nai-demo-jk9kl-zxbpw nai-demo v2.19.0 v1.36.2 Reconciled
$ kubectl get upgradeplans -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase
NS NAME PHASE
debug-test-xhj92 demo-wkl-02-v1.36.2-v2.19.0 Upgraded
kommander host-cluster-v1.36.2-v2.19.0 Upgraded
nai-demo-jk9kl-zxbpw nai-demo-v1.36.2-v2.19.0 Upgraded
All three LoadBalancer addresses answer from the bastion. Every HelmRelease and Kustomization on the management cluster is Ready.
To watch after a roll: controllers that hit their memory limit. On ours, the NDK controller manager at 512Mi (15 restarts), the Kommander controller manager at 512Mi (2) and a Kyverno reports controller at 128Mi (3). Raise limits through the apps' override ConfigMaps, not on the Deployments.
The same clusters to 2.19.1
After the 2.19.1 Kommander upgrade the management cluster reports v2.19.1. The workload clusters still report v2.19.0. Kubernetes does not change, so the CLI does nothing:
$ nkp upgrade cluster nutanix --cluster-name demo-wkl-02 --namespace debug-test-xhj92 --vm-image <same image> --dry-run -o yaml
Cluster "demo-wkl-02" is already using the latest supported Kubernetes version "1.36.2"
An UpgradePlan moves them. Same object with version: v2.19.1, the same kubernetesVersion: v1.36.2 and the same image names:
| Cluster | validating-inputs | upgrading-platform-applications | upgrading-kubernetes | Nodes replaced |
|---|---|---|---|---|
| 4 nodes | 1 s | 33 s | 0 s | 0 |
| NAI cluster | 5 s | 8 s | 0 s | 0 |
Result on both clusters: v2.19.1, every HelmRelease Ready, same nodes, Cilium in SNAT mode, LoadBalancer addresses answering. The ClusterClass reference stays nkp-nutanix-v2.19.0. The v2.19.1 classes exist and differ only in template names.
Known issue in the 2.19.1 release notes: with a public registry mirror and no mirror credentials, the plan can stall in the platform-applications step. Workaround: create an empty <cluster-name>-oci-creds secret in the workspace namespace on the managed cluster. Our clusters have mirror credentials and were not affected.
The three ways, compared
CLI nkp upgrade cluster nutanix |
UpgradePlan you write |
GitOps commit of the NKPCluster |
|
|---|---|---|---|
| What it changes | the NKPCluster (12 lines on our dry run) and, per the guide, an UpgradePlan | the NKPCluster, written by the controller | the NKPCluster, written by Flux |
| Preflight | the CLI's preflight webhook (passed on our dry run) | the UpgradePlan webhook: Prism Central version and image existence | none before the apply; the runtime hook's checks at upgrade time |
| Progress | the CLI's spinner, then the objects | status.steps and status.phase on the plan, and the UI |
the NKPCluster conditions and the machines; the guide notes the UI does not track it |
| Fits in git | no | yes, one small file per upgrade | yes, it is the cluster definition |
| What we used | dry run only | written by hand on the first cluster; created by the UI button on the second (same object, plus the preflight annotation) | not this time |
All three give the same end state: ClusterClass nkp-nutanix-v2.19.0, Kubernetes 1.36.2, the 2.19 add-ons, new VMs from the 1.36.2 image.
Summary
- A workload cluster upgrade is one
UpgradePlanobject. - 4 nodes: 27 minutes. Custom Cilium values from the cluster variable are applied before the roll.
- A PodDisruptionBudget with zero allowed disruptions stalls the roll until it is removed. Audit PDBs on every cluster first.
- To 2.19.1: a second UpgradePlan, under a minute, no node replaced.


