This part upgrades two workload clusters with an UpgradePlan object: one from kubectl, one from the UI. Prerequisites: management cluster on 2.19 (Part 3), and the Cilium reference from Part 4 if your node subnet is Network Controller managed. The first cluster took 27 minutes. The second stalled for a day on a documented PodDisruptionBudget issue.

CLI dry run

First cluster: 1 control plane and 3 workers, Rocky Linux 9.7, containerd 2.1.6, NDK 2.3.0, 18 HelmReleases Ready. The Cilium SNAT reference is set before the upgrade, to test that custom add-on values survive a roll.

Dry run of the documented command:

bash
nkp upgrade cluster nutanix --cluster-name demo-wkl-02 --namespace debug-test-xhj92 \
  --vm-image nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2 --dry-run -o yaml
console
 ✓ Waiting for ClusterClasses to be ready
 ✓ Running preflight checks

The output is the rewritten NKPCluster. These fields change: ClusterClass reference (nkp-nutanix-v2.18.0 to nkp-nutanix-v2.19.0), Kubernetes version, control plane and worker image names, KommanderCluster platform version. We create the UpgradePlan object directly.

Step 1: the Cilium reference, before anything else

Apply the ConfigMap of Cilium values from Part 4 in the cluster's namespace, then reference it on the NKPCluster:

bash
kubectl apply -f demo-wkl-02-cilium-cni-helm-values-template.yaml
kubectl get nkpcluster demo-wkl-02 -n debug-test-xhj92 -o json \
  | jq '(.spec.capiCluster.topology.variables[] | select(.name=="clusterConfig") | .value.addons.cni.values)
         = {sourceRef: {kind: "ConfigMap", name: "demo-wkl-02-cilium-cni-helm-values-template"}}' \
  | kubectl replace -f -
kubectl get cluster demo-wkl-02 -n debug-test-xhj92 \
  -o jsonpath='{.spec.topology.variables[?(@.name=="clusterConfig")].value.addons.cni}{"\n"}'

No live edit. The cluster stays in DSR until the upgrade applies the change.

Step 2: the object

yaml
apiVersion: upgrade.nkp.nutanix.com/v1alpha1
kind: UpgradePlan
metadata:
  name: demo-wkl-02-v1.36.2-v2.19.0
  namespace: debug-test-xhj92
  labels:
    upgrade.nkp.nutanix.com/nkpcluster-name: demo-wkl-02
    upgrade.nkp.nutanix.com/target-version: v2.19.0
spec:
  nkpClusterRef:
    name: demo-wkl-02
    namespace: debug-test-xhj92
  version: v2.19.0
  kubernetesVersion: v1.36.2
  controlPlaneImage: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2
  workerImages:
    md-0: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2

A server-side dry run (kubectl apply --server-side --dry-run=server -f) runs the UpgradePlan webhook: Prism Central version checked, both images found. The UI adds the annotation preflight.cluster.caren.nutanix.com/skip: "" (skip nothing). Omitting it is equivalent.

Step 3: what happened, minute by minute

bash
kubectl get upgradeplan -n debug-test-xhj92 -w
kubectl get upgradeplan demo-wkl-02-v1.36.2-v2.19.0 -n debug-test-xhj92 \
  -o jsonpath='{range .status.steps[*]}{.name}: {.state} {.startTime} -> {.endTime}{"\n"}{end}'
console
validating-inputs: Completed 2026-10-20T13:57:09Z -> 2026-10-20T13:57:15Z
upgrading-platform-applications: Completed 2026-10-20T13:57:15Z -> 2026-10-20T14:09:22Z
upgrading-kubernetes: Completed 2026-10-20T14:09:22Z -> 2026-10-20T14:24:04Z

A workload cluster's plan has three steps (no management-plane step):

  1. Validating inputs, 6 seconds: the webhook's Prism Central checks.
  2. Upgrading platform applications, 12 minutes. The HelmReleases on the cluster were already at their 2.19 charts since the Kommander upgrade. The step waits for the AppDeployment status on the management side. The UI tracker shows it for a plan created with kubectl too: "10 out of 15 platform applications are ready", waiting for cert-manager, external-secrets, gatekeeper, kube-prometheus-stack and opencost.
  3. Upgrading Kubernetes and cluster infrastructure, 15 minutes.

Cluster Upgrade Tracker for workload cluster demo-wkl-02, 1 of 3 steps completed: Upgrading Platform Applications running, 10 out of 15 platform applications are ready, waiting for cert-manager, external-secrets, gatekeeper, kube-prometheus-stack, opencost

Sequence of the Kubernetes step:

  • 16:09: the runtime hook fetches the chart information for every add-on (Cilium, Multus, cluster autoscaler, NFD, CSI, snapshot controller, COSI, MetalLB, cloud provider, Konnector agent) and applies the Cilium preflight release.
  • 16:09: the hook rewrites the Cilium HelmChartProxy from our ConfigMap (loadBalancer.mode: snat, Helm revision 3). cilium-config has bpf-lb-mode: snat before the first new node exists.
  • 16:09 to 16:12: new control plane created, old one drained.
  • 16:12 to 16:23: three workers replaced one at a time.
  • 16:24: plan Upgraded.

27 minutes from kubectl apply to Upgraded for four VMs. No stall.

Step 4: verify

console
$ kubectl --kubeconfig demo-wkl-02.conf get nodes -o custom-columns=NAME:.metadata.name,VER:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage,CRI:.status.nodeInfo.containerRuntimeVersion
NAME                                 VER       OS                            CRI
demo-wkl-02-md-0-9h798-sb45h-fdshs   v1.36.2   Rocky Linux 9.8 (Blue Onyx)   containerd://2.3.1-d2iq.1
demo-wkl-02-md-0-9h798-sb45h-ptrxx   v1.36.2   Rocky Linux 9.8 (Blue Onyx)   containerd://2.3.1-d2iq.1
demo-wkl-02-md-0-9h798-sb45h-t274x   v1.36.2   Rocky Linux 9.8 (Blue Onyx)   containerd://2.3.1-d2iq.1
demo-wkl-02-rfzj2-zszv2              v1.36.2   Rocky Linux 9.8 (Blue Onyx)   containerd://2.3.1-d2iq.1
$ kubectl --kubeconfig demo-wkl-02.conf -n kube-system get cm cilium-config -o jsonpath='{.data.bpf-lb-mode}{"\n"}'
snat
$ kubectl get helmreleaseproxies -n debug-test-xhj92 -o custom-columns=NAME:.metadata.name,CHART:.spec.chartName,VER:.spec.version,REV:.status.revision
NAME                                            CHART                        VER      REV
cilium-demo-wkl-02-slfft                        cilium                       1.19.4   3
konnector-agent-demo-wkl-02-dhhp2               konnector-agent              1.4.2    3
metallb-demo-wkl-02-rkvbd                       metallb                      0.15.3   2
nkp-cloud-provider-nutanix-demo-wkl-02-4fhqr    nkp-cloud-provider-nutanix   0.8.12   1
node-feature-discovery-demo-wkl-02-q458w        node-feature-discovery       0.19.0   3
nutanix-csi-storage-demo-wkl-02-dpbth           nutanix-csi-storage          3.9.0    3
(...)

Add-ons after the roll:

Add-on Before After
Cilium 1.19.4, DSR 1.19.4, SNAT (new Helm revision)
Nutanix CSI 3.7.1 3.9.0
Konnector agent 1.4.1 1.4.2
Node Feature Discovery 0.18.3 0.19.0
Nutanix cloud provider 0.6.2 0.8.12 (new release name)
MetalLB, snapshot controller, NDK 2.3.0 unchanged

All HelmReleases Ready, no pod not Running. From the bastion, the LoadBalancer address and a NodePort on each of the four nodes answer 3 of 3, with the MetalLB announcer on a node without a Traefik pod. A Cilium values ConfigMap referenced from addons.cni.values.sourceRef is applied by the upgrade, before the roll.

The second cluster: a documented PodDisruptionBudget stall

Second cluster: Nutanix Enterprise AI 2.8, 3 control-plane nodes, an 8-node pool for the NAI services, a 2-node inference pool.

The 2.19 release notes list a known issue for this setup. The upgrade can stall when NAI runs a single-instance gateway, NAI API and PostgreSQL with the PodDisruptionBudget enabled. The pods cannot be evicted, the old worker stays Draining, and later node pools do not start. Documented workaround (release notes): run several instances of the gateway and the NAI API, and disable the PostgreSQL PodDisruptionBudget before the upgrade.

On our cluster the gateway and the API already ran two replicas. The database PDB was still enabled when the upgrade was started from the UI at 16:05. This is what the documented stall looks like.

NKP 2.19 UI, Upgrades, workload clusters: demo-wkl-02 and nai-demo both Upgrading, Kubernetes v1.35.2 to v1.36.2

Timeline: inputs validated in 6 seconds, platform applications in 8 minutes, control plane rolled 16:14 to 16:22, four new pool nodes 16:24 to 16:30. At 16:31 the drain of the first old pool node stops:

console
$ kubectl get machine nai-demo-nai-pool-demo-c56m2-cn4gl-c6v6j -n nai-demo-jk9kl-zxbpw \
    -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' | grep -v ': $'
Ready: * Deleting: Machine deletion in progress since more than 15m, stage: DrainingNode,
  delay likely due to PodDisruptionBudgets
Deleting: Drain not completed yet (started at 2026-10-20T14:31:01Z):
* Pod nai-system/nai-db-iep-1: cannot evict pod as it would violate the pod's disruption budget.
  The disruption budget nai-db-iep-primary needs 1 healthy pods (...)
$ kubectl --kubeconfig nai-demo.conf get pdb -A | awk '$5=="0"'
nai-system   nai-db-iep-primary   1   N/A   0   15d

The PDB belongs to the CloudNativePG cluster that NAI deploys (cnpg.io/cluster: nai-db-iep, one instance, minAvailable: 1). CloudNativePG recreates it if you delete it. The inference pool does not start. The plan stays in UpgradingKubernetes. On our cluster the drain stayed blocked for 24 hours and 10 minutes.

The fix is the documented one, applied through the NAI operators Helm release rather than on the PDB itself, so that the operator does not put it back:

bash
helm upgrade nai-operators <chart> -n nai-system --reuse-values --set naiDatabase.enablePDB=false

Result:

  • After the Helm upgrade, the CloudNativePG operator removes the PDB.
  • 16:43: Cluster API evicts the database pod.
  • 16:46: the old machine is gone, the database pod runs on a new node, CloudNativePG reports "Cluster in healthy state". Database unavailable for about three minutes.
  • 16:54: the inference pool starts rolling.
  • 16:56: plan Upgraded.

Do not use a MachineDrainRule or a nodeDrainTimeout here: both evict a single-instance database ungracefully. Do not delete the PDB by hand: its operator recreates it.

Run the PDB audit on every workload cluster, and read the known issues of the applications you run:

bash
kubectl --kubeconfig <workload>.conf get pdb -A | awk '$5=="0"'

State after the upgrades

console
$ kubectl get nkpclusters -A
NAMESPACE              NAME           NKP VERSION   KUBERNETES VERSION   PHASE
debug-test-xhj92       demo-wkl-02    v2.19.0       v1.36.2              Reconciled
kommander              host-cluster   v2.19.0       v1.36.2              Reconciled
nai-demo-jk9kl-zxbpw   nai-demo       v2.19.0       v1.36.2              Reconciled
$ kubectl get upgradeplans -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase
NS                     NAME                            PHASE
debug-test-xhj92       demo-wkl-02-v1.36.2-v2.19.0     Upgraded
kommander              host-cluster-v1.36.2-v2.19.0    Upgraded
nai-demo-jk9kl-zxbpw   nai-demo-v1.36.2-v2.19.0        Upgraded

All three LoadBalancer addresses answer from the bastion. Every HelmRelease and Kustomization on the management cluster is Ready.

To watch after a roll: controllers that hit their memory limit. On ours, the NDK controller manager at 512Mi (15 restarts), the Kommander controller manager at 512Mi (2) and a Kyverno reports controller at 128Mi (3). Raise limits through the apps' override ConfigMaps, not on the Deployments.

The same clusters to 2.19.1

After the 2.19.1 Kommander upgrade the management cluster reports v2.19.1. The workload clusters still report v2.19.0. Kubernetes does not change, so the CLI does nothing:

console
$ nkp upgrade cluster nutanix --cluster-name demo-wkl-02 --namespace debug-test-xhj92 --vm-image <same image> --dry-run -o yaml
Cluster "demo-wkl-02" is already using the latest supported Kubernetes version "1.36.2"

An UpgradePlan moves them. Same object with version: v2.19.1, the same kubernetesVersion: v1.36.2 and the same image names:

Cluster validating-inputs upgrading-platform-applications upgrading-kubernetes Nodes replaced
4 nodes 1 s 33 s 0 s 0
NAI cluster 5 s 8 s 0 s 0

Result on both clusters: v2.19.1, every HelmRelease Ready, same nodes, Cilium in SNAT mode, LoadBalancer addresses answering. The ClusterClass reference stays nkp-nutanix-v2.19.0. The v2.19.1 classes exist and differ only in template names.

Known issue in the 2.19.1 release notes: with a public registry mirror and no mirror credentials, the plan can stall in the platform-applications step. Workaround: create an empty <cluster-name>-oci-creds secret in the workspace namespace on the managed cluster. Our clusters have mirror credentials and were not affected.

The three ways, compared

CLI nkp upgrade cluster nutanix UpgradePlan you write GitOps commit of the NKPCluster
What it changes the NKPCluster (12 lines on our dry run) and, per the guide, an UpgradePlan the NKPCluster, written by the controller the NKPCluster, written by Flux
Preflight the CLI's preflight webhook (passed on our dry run) the UpgradePlan webhook: Prism Central version and image existence none before the apply; the runtime hook's checks at upgrade time
Progress the CLI's spinner, then the objects status.steps and status.phase on the plan, and the UI the NKPCluster conditions and the machines; the guide notes the UI does not track it
Fits in git no yes, one small file per upgrade yes, it is the cluster definition
What we used dry run only written by hand on the first cluster; created by the UI button on the second (same object, plus the preflight annotation) not this time

All three give the same end state: ClusterClass nkp-nutanix-v2.19.0, Kubernetes 1.36.2, the 2.19 add-ons, new VMs from the 1.36.2 image.

Summary

  • A workload cluster upgrade is one UpgradePlan object.
  • 4 nodes: 27 minutes. Custom Cilium values from the cluster variable are applied before the roll.
  • A PodDisruptionBudget with zero allowed disruptions stalls the roll until it is removed. Audit PDBs on every cluster first.
  • To 2.19.1: a second UpgradePlan, under a minute, no node replaced.