This part upgrades the management plane and the platform apps on the management cluster. Kubernetes does not change here; that is Part 3. Prerequisite: Part 1.

What this phase does

nkp upgrade kommander upgrades:

  • the operators that own NKPCluster, ManagementPlane, Release and UpgradePlan (the last two are new in 2.19)
  • the Cluster API stack and all its providers
  • the ClusterClasses
  • every platform app on the management cluster

It also rewrites the AppDeployment objects of every workspace to the 2.19 app versions.

The guide says attached-cluster tokens are invalidated. Certificate-based admin kubeconfigs stay valid. Dex-issued ones must be fetched again.

Step 1: run it

The command is the same for connected and air-gapped clusters. There is no applications tarball and no flag (see Part 1).

bash
export KUBECONFIG=~/<management-cluster>.conf
nkp upgrade kommander

The run prints six steps:

console
2026-10-20 12:31:01 INF  • Checking for deprecated GitRepository-based catalog apps...
 ✓ Checking for deprecated GitRepository-based catalog apps
2026-10-20 12:31:02 INF  • Ensuring that new app dependencies introduced by the upgrade are enabled...
2026-10-20 12:31:02 INF Checking that CloudNativePG is enabled on clusters that run NKP AI Navigator or NKP Insights...
 ✓ Ensuring that new app dependencies introduced by the upgrade are enabled
2026-10-20 12:31:03 INF  • Ensuring upgrading conditions are met...
 ✓ Ensuring upgrading conditions are met
2026-10-20 12:31:03 INF  • Checking docker.io/nutanix credentials are present for the fleet...
 ✓ Checking docker.io/nutanix credentials are present for the fleet
2026-10-20 12:31:03 INF  • Upgrading ManagementPlane nkp-platform...
 ✓ Upgrading ManagementPlane nkp-platform
2026-10-20 12:58:07 INF  • Ensuring platform Kommander applications are upgraded...
 ✓ Ensuring platform Kommander applications are upgraded

Duration on our lab VMs: 43 minutes (27 for the management plane, 16 for the platform apps). No timeout.

  • The Docker Hub credentials check passes on an air-gapped cluster with no Docker Hub token.
  • The GitRepository catalog check fails if you skipped the cleanup in Part 1.

Step 2: watch the management plane

During "Upgrading ManagementPlane", the object's conditions go False in turn as each operator is replaced. Order on our run: ManagementPlaneOperatorReady, CAPIStackReady, NKPClusterOperatorReady, UpgradePlanOperatorReady, ReleaseOperatorReady, KommanderCoreReady.

bash
kubectl get managementplane nkp-platform \
  -o jsonpath='{.spec.version} {.status.version}{"\n"}{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,addonproviders -A

When the step completes, every provider is Ready at its 2.19 version: Cluster API 1.12.11, CAPX 1.10.15, pre-provisioned 0.42.1, Helm addon 0.6.2, and the new host-agent provider 0.9.4. Of the 15 AppDeployments that change, 7 still reference the 2.18 versions at this point. The next step moves them.

Step 3: watch the platform apps

These two commands list what is still reconciling. On our run nkp-insights was the last HelmRelease to become Ready, about eight minutes after the others.

bash
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'

When the CLI returns:

console
$ kubectl get nkpclusters -A
NAMESPACE              NAME           NKP VERSION   KUBERNETES VERSION   PHASE        AGE
debug-test-xhj92       demo-wkl-02    v2.18.0       v1.35.2              Reconciled   105d
kommander              host-cluster   v2.19.0       v1.35.2              Reconciled   105d
nai-demo-jk9kl-zxbpw   nai-demo       v2.18.0       v1.35.2              Reconciled   15d
$ kubectl get kommandercore -A
NAME             VERSION
kommander-core   v2.19.0

Expected state: management cluster at v2.19.0, Kubernetes unchanged, workload clusters still at v2.18.0.

Step 4: no workspace step

nkp upgrade workspace is deprecated. For ClusterClass clusters, nkp upgrade cluster upgrades platform apps and infrastructure in one command. Check the workload clusters right after the Kommander upgrade:

bash
# AppDeployments in a workspace, as rewritten by the Kommander upgrade:
kubectl get appdeployments -n <workspace-ns> -o custom-columns=NAME:.metadata.name,APP:.spec.appRef.name
# HelmReleases on the attached cluster itself:
kubectl --kubeconfig <workload>.conf get hr -A

Result on our clusters:

  • Every workspace AppDeployment points at the 2.19 versions: cert-manager 1.20.3, Gatekeeper 3.23.0, Traefik 39.0.7, kube-prometheus-stack 87.16.0, OpenCost 2.5.28.
  • Both attached clusters, still on NKP 2.18 and Kubernetes 1.35.2, have reconciled them: 18 of 18 and 19 of 19 HelmReleases Ready.

The release notes describe this as a display issue: the UI shows the new versions while the clusters still run the old ones. On our clusters the apps were really upgraded, through each cluster's own Flux and OCI proxy.

Consequences:

  • Workload clusters need scheduling headroom for the new app versions on the day of the Kommander upgrade.
  • The "platform applications" step of the later cluster upgrade has little left to do.

The same step on 2.19.1

Same command with the 2.19.1 CLI on a cluster at 2.19.0:

  • Same four pre-checks, same two steps.
  • 22 minutes: 15 for the management plane, 7 for the platform apps.
  • Five HelmReleases are not Ready for a few minutes, then recover: Kommander, its app management, the UI, the Dex authenticator, the forward-auth proxy.
  • Platform app versions do not change, except Kommander (0.19.1), its app management and its UI.

A HelmRelease check does not show everything. Before our 2.19.1 run the NKP Insights backend was crash-looping while its HelmRelease was Ready. It did not block the upgrade. Use the pod list from the pre-flight, not only the HelmRelease list.

Troubleshooting

A full Prometheus volume

Symptom. kube-prometheus-stack moves from 82.13.6 to 87.16.0 and restarts Prometheus. The pod goes into CrashLoopBackOff:

output
panic: write /prometheus/chunks_head/003759: no space left on device

Cause. The 100Gi volume was full (97.9 GB used, 30-day retention, no size limit). The restart replays the WAL and the replay needs free space. This is not a 2.19 defect. The chart bump restarts Prometheus.

Fix. Raising the volume size in the kube-prometheus-stack values does nothing. The chart keeps the storage block of an existing Prometheus object (a lookup in the template). Grow the PVC directly:

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <prometheus-pvc>        # kubectl get pvc -n kommander | grep prometheus
  namespace: kommander
spec:
  resources:
    requests:
      storage: 150Gi
bash
kubectl apply --server-side --force-conflicts -f prometheus-pvc.yaml
kubectl get pvc -n kommander <pvc> -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}'

Then:

  1. The Nutanix CSI expands the volume online. The PVC reports FileSystemResizePending.
  2. Delete the Prometheus pod once. The filesystem is resized at mount.
  3. Prometheus repairs its WAL and scrapes again.
  4. Set retentionSize in your override.

Check the PVC usage before the upgrade.

Gatekeeper audit OOMKilled

The chart default for gatekeeper-audit is 512Mi, on Gatekeeper 3.23.0 too. On our cluster 1Gi was still OOMKilled (three restarts in nine minutes). 2Gi is stable. The override is a ConfigMap of Helm values in the app's namespace, referenced by the HelmRelease:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gatekeeper-overrides
  namespace: kommander
data:
  values.yaml: |
    audit:
      resources:
        limits:
          memory: 2Gi

Related: the NKP Insights backend was OOMKilled at its 1Gi default on 2.18. 2.19 raises the default to 2Gi.

Monitoring commands

bash
kubectl get managementplane nkp-platform -o jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get nkpcluster host-cluster -n kommander -o jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'
kubectl get pods -A | grep -vE 'Running|Completed'

The object's conditions name the blocker. The HelmRelease status gives the reason.

Next: Part 3, the node roll.