This part upgrades the management cluster's nodes from Kubernetes 1.35.2 to 1.36.2 on the Rocky Linux 9.8 image. Every VM is replaced. Prerequisite: Part 2. On 2.19 the upgrade is a Kubernetes object, UpgradePlan, with a controller, a webhook and a readable status.

Note

The guide says to upgrade the management cluster with the CLI and to use the UI for workload clusters. After the Kommander upgrade the UI also offers the Upgrade button for the management cluster. We used it. The CLI is the documented path for this step.

NKP 2.19 UI, Administration, Upgrades: the management cluster host-cluster shows Ready, v2.19.0, Kubernetes v1.35.2 to v1.36.2 with an Upgrade button; the two workload clusters are listed as v2.18.0 to v2.19.0

What this phase does

  1. The upgrade creates an UpgradePlan.
  2. Its controller validates the inputs. The management plane and platform app steps are already done and complete in seconds.
  3. The controller rewrites the NKPCluster: new ClusterClass, Kubernetes version and image names.
  4. The NKPCluster controller patches the Cluster API Cluster.
  5. A runtime hook re-applies the cluster add-ons for the new class.
  6. Cluster API replaces the nodes: control plane first (scale to four, remove one old machine at a time), then workers one at a time.

Each node gets Kubernetes 1.36.2, Rocky Linux 9.8 (kernel 5.14.0-687) and containerd 2.3.1. Control-plane nodes get etcd 3.6.8, up from 3.5.

Step 1: the object the UI creates

The wizard has two screens: the version, then the OS image for the control plane and each worker pool. Preflight checks can only be skipped from the CLI.

Upgrade Management Cluster wizard, step 1 Select Version: current NKP v2.19.0 and Kubernetes v1.35.2, new version NKP v2.19.0 and Kubernetes v1.36.2 recommended, with the 2.19.0 release notes

Upgrade Management Cluster wizard, step 2 Upgrade Configuration: current and new OS image for the control plane and worker node pool, and the Pre-Checks note

The wizard creates one object in the kommander namespace:

yaml
apiVersion: upgrade.nkp.nutanix.com/v1alpha1
kind: UpgradePlan
metadata:
  name: host-cluster-v1.36.2-v2.19.0
  namespace: kommander
  annotations:
    preflight.cluster.caren.nutanix.com/skip: ""      # empty = skip nothing
  labels:
    upgrade.nkp.nutanix.com/nkpcluster-name: host-cluster
    upgrade.nkp.nutanix.com/target-version: v2.19.0
spec:
  nkpClusterRef:
    name: host-cluster
    namespace: kommander
  version: v2.19.0
  kubernetesVersion: v1.36.2
  controlPlaneImage: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2
  workerImages:
    md-0: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2

The plan references the NKPCluster, not the CAPI cluster. The NKPCluster controller knows which CAPI Cluster it owns. The 2.18 problem with the host-cluster name on clusters upgraded from 2.17 does not apply here.

Step 2: the admission webhook

The validating webhook of the UpgradePlan calls Prism Central. From its log:

output
"msg":"validating pc version for management cluster"
"msg":"checking Prism Central version compatibility"
OPTIONS https://<pc>:9440/api/prism/unversioned/info       -> Negotiated Version with server : v4.4
GET https://<pc>:9440/api/prism/v4.4/config/domain-managers -> HTTP/1.1 200 OK
"msg":"validating images for upgrade plan"
"msg":"checking image existence in Prism Central"
GET https://<pc>:9440/api/vmm/v4.3/content/images?$filter=name eq 'nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2'
"msg":"control plane image found in Prism Central"
"msg":"worker image found in Prism Central"

These checks run before anything is written to the cluster. On the UI path the CLI's preflight webhook ("Running preflight checks") is not called. The checks above and the runtime hook in Step 3 replace it.

Step 3: the controller

bash
kubectl get upgradeplan -n kommander -w
kubectl get upgradeplan host-cluster-v1.36.2-v2.19.0 -n kommander \
  -o jsonpath='{range .status.steps[*]}{.name}: {.state} {.startTime} -> {.endTime}{"\n"}{end}'
console
validating-inputs: Completed 2026-10-20T11:48:50Z -> 2026-10-20T11:48:50Z
upgrading-management-plane: Completed 2026-10-20T11:48:50Z -> 2026-10-20T11:48:50Z
upgrading-platform-applications: Completed 2026-10-20T11:48:54Z -> 2026-10-20T11:48:54Z
upgrading-kubernetes: Progressing 2026-10-20T11:48:59Z ->

The UI tracker shows the same four steps. The first three complete within seconds.

Cluster Upgrade Tracker, management cluster, 2 of 4 steps completed after 8 seconds: Validating Inputs and Upgrading Management Plane and Kommander done, Upgrading Platform Applications running

In order, within seconds of the click:

  1. The plan controller rewrites the NKPCluster: topology.classRef.name from nkp-nutanix-v2.18.0 to nkp-nutanix-v2.19.0, topology.version to v1.36.2, both image names, and the KommanderCluster platform version.
  2. The NKPCluster controller patches the CAPI Cluster (Patching Cluster (v1beta2), Updated CAPI Cluster topology). The Cluster now carries runtime.cluster.x-k8s.io/pending-hooks: AfterClusterUpgrade,AfterControlPlaneUpgrade,AfterWorkersUpgrade,BeforeWorkersUpgrade.
  3. The CAPI runtime extension's BeforeClusterUpgrade hook runs and re-applies the add-ons for the new class: the Cilium preflight release, MetalLB, the CSI provider, the cloud controller manager. This is where custom add-on values are re-rendered (Part 4 depends on it).
  4. The KubeadmControlPlane goes to four replicas. The first 1.36.2 control-plane machine is created 31 seconds after the click and is Ready two minutes later.

Watch the roll with kubectl:

bash
watch -n15 'kubectl get nodes --no-headers | awk "{print \$5}" | sort | uniq -c'
kubectl get machines -A -w
kubectl get kubeadmcontrolplane,machinedeployments -A

Cluster Upgrade Tracker, 3 of 4 steps completed, Upgrading Kubernetes and Cluster Infrastructure running for 10 minutes: KubeadmControlPlane upgrading to v1.36.2, MachineDeployment pending upgrade

Our run: control plane 13:49 to 14:01, workers 14:00 to 14:49, Upgraded at 14:49. 60 minutes for 3 control-plane and 6 worker VMs. No stall. Two PDBs with zero allowed disruptions did not block a drain.

Step 4: a Failed status while the roll continues

At 14:29 the UI shows Failed on the step "Upgrading Kubernetes and Cluster Infrastructure":

output
Failed to update NKPCluster: Internal error occurred: failed calling webhook
"webhook.nkpcluster.kommander.mesosphere.io": failed to call webhook:
Post "https://nkp-cluster-webhook.kommander.svc:443/defaults?timeout=30s":
dial tcp 10.98.155.95:443: connect: no route to host

Cluster Upgrade Tracker at 43 minutes: Upgrading Kubernetes and Cluster Infrastructure with the orange error banner Failed to update NKPCluster, failed calling webhook, no route to host

At that moment 7 of 10 nodes are on 1.36.2 and a worker is in Deleting. The roll is running. Cause, from the events and the controller log:

  1. 14:29: a worker drain evicts the only pod of nkp-cluster-webhook (one replica, no PodDisruptionBudget). Its replacement is still starting on a new worker.
  2. The UpgradePlan controller re-applies the plan to the NKPCluster, as it does on every reconcile. The write goes through the NKPCluster mutating webhook /defaults, which has failurePolicy: Fail. With no endpoint, the API server rejects the write. The controller sets the plan to Failed and emits NKPClusterUpgradeFailure.
  3. The controller retries with backoff. Cluster API is not affected: the NKPCluster spec already holds the target.
  4. 14:40: the retry succeeds and the phase returns to UpgradingKubernetes. 14:49: Upgraded, all conditions True.

The UI showed Failed for 11 minutes while the upgrade was running. The plan recovered without any action. Before you act on that status, check the machines.

A catalog app caused the same effect. Our Kyverno admission controller has one replica and a Fail validating webhook. While it moved off a drained node, pod creations were refused. List the webhooks that can do this:

output
failed calling webhook "vpol.validate.kyverno.svc-fail"

A single-replica admission webhook with failurePolicy: Fail blocks the cluster each time its node is drained. List yours before the roll:

bash
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o json \
  | jq -r '.items[] | .metadata.name as $cfg | .webhooks[]? | select(.failurePolicy=="Fail")
           | "\($cfg)\t\(.clientConfig.service.namespace)/\(.clientConfig.service.name)"' | sort -u

Then check the replica count and the PDB of each Deployment behind them. For your own apps, run two replicas with a PDB. NKP's webhook Deployments are managed by its operators.

Step 5: verify

console
$ kubectl get upgradeplan -n kommander
NAME                           VERSION   KUBERNETES VERSION   PHASE      AGE
host-cluster-v1.36.2-v2.19.0   v2.19.0   v1.36.2              Upgraded   72m
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,VER:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage,CRI:.status.nodeInfo.containerRuntimeVersion
NAME                                        VER       OS                            CRI
itcs-nkp-mgmt-prod-md-0-xbscj-22kk9-2fff6   v1.36.2   Rocky Linux 9.8 (Blue Onyx)   containerd://2.3.1-d2iq.1
(... 9 nodes, all identical)
$ kubectl -n kube-system exec etcd-<cp-node> -- etcdctl --cacert ... endpoint status --cluster -w table
| VERSION 3.6.8 | STORAGE VERSION 3.6.0 | ... | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED false |

Cluster Upgrade Tracker: the management cluster has been successfully upgraded to NKP 2.19.0, 4 of 4 steps completed, duration 59 minutes 49 seconds

Then check HelmReleases, Kustomizations and pods not Running. After our roll, the NKP UI's LoadBalancer address stopped answering. The cause is Cilium's DSR mode on our subnet, not the upgrade: see Part 4.

2.19.1: no node roll

2.19.1 keeps Kubernetes 1.36.2 and the same node image. After its Kommander upgrade the management cluster reports v2.19.1. The cluster upgrade command has nothing to do:

console
$ nkp upgrade cluster nutanix --cluster-name host-cluster --namespace kommander --vm-image <same image> --dry-run -o yaml
Cluster "itcs-nkp-mgmt-prod" is already using the latest supported Kubernetes version "1.36.2"

The cause of the Failed status is unchanged on 2.19.1: nkp-cluster-webhook runs one replica with no PodDisruptionBudget, and its defaulting webhook has failurePolicy: Fail. Expect the same status at the next upgrade that drains nodes.

The CLI path

The documented command for this step:

bash
nkp upgrade cluster nutanix --cluster-name host-cluster --namespace kommander \
  --vm-image nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2

Its dry run outputs the rewritten NKPCluster only, with the same fields the UpgradePlan controller writes in Step 3.

Three ways lead to the same NKPCluster change: the CLI, an UpgradePlan you write, or a GitOps commit of the NKPCluster. The last one creates no UpgradePlan, so the UI does not track it. Part 5 uses the second one.