This part upgrades the management plane and the platform apps on the management cluster. Kubernetes does not change here; that is Part 3. Prerequisite: Part 1.
What this phase does
nkp upgrade kommander upgrades:
- the operators that own NKPCluster, ManagementPlane, Release and UpgradePlan (the last two are new in 2.19)
- the Cluster API stack and all its providers
- the ClusterClasses
- every platform app on the management cluster
It also rewrites the AppDeployment objects of every workspace to the 2.19 app versions.
The guide says attached-cluster tokens are invalidated. Certificate-based admin kubeconfigs stay valid. Dex-issued ones must be fetched again.
Step 1: run it
The command is the same for connected and air-gapped clusters. There is no applications tarball and no flag (see Part 1).
export KUBECONFIG=~/<management-cluster>.conf
nkp upgrade kommander
The run prints six steps:
2026-10-20 12:31:01 INF • Checking for deprecated GitRepository-based catalog apps...
✓ Checking for deprecated GitRepository-based catalog apps
2026-10-20 12:31:02 INF • Ensuring that new app dependencies introduced by the upgrade are enabled...
2026-10-20 12:31:02 INF Checking that CloudNativePG is enabled on clusters that run NKP AI Navigator or NKP Insights...
✓ Ensuring that new app dependencies introduced by the upgrade are enabled
2026-10-20 12:31:03 INF • Ensuring upgrading conditions are met...
✓ Ensuring upgrading conditions are met
2026-10-20 12:31:03 INF • Checking docker.io/nutanix credentials are present for the fleet...
✓ Checking docker.io/nutanix credentials are present for the fleet
2026-10-20 12:31:03 INF • Upgrading ManagementPlane nkp-platform...
✓ Upgrading ManagementPlane nkp-platform
2026-10-20 12:58:07 INF • Ensuring platform Kommander applications are upgraded...
✓ Ensuring platform Kommander applications are upgraded
Duration on our lab VMs: 43 minutes (27 for the management plane, 16 for the platform apps). No timeout.
- The Docker Hub credentials check passes on an air-gapped cluster with no Docker Hub token.
- The GitRepository catalog check fails if you skipped the cleanup in Part 1.
Step 2: watch the management plane
During "Upgrading ManagementPlane", the object's conditions go False in turn as each operator is replaced. Order on our run: ManagementPlaneOperatorReady, CAPIStackReady, NKPClusterOperatorReady, UpgradePlanOperatorReady, ReleaseOperatorReady, KommanderCoreReady.
kubectl get managementplane nkp-platform \
-o jsonpath='{.spec.version} {.status.version}{"\n"}{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get coreproviders,bootstrapproviders,controlplaneproviders,infrastructureproviders,addonproviders -A
When the step completes, every provider is Ready at its 2.19 version: Cluster API 1.12.11, CAPX 1.10.15, pre-provisioned 0.42.1, Helm addon 0.6.2, and the new host-agent provider 0.9.4. Of the 15 AppDeployments that change, 7 still reference the 2.18 versions at this point. The next step moves them.
Step 3: watch the platform apps
These two commands list what is still reconciling. On our run nkp-insights was the last HelmRelease to become Ready, about eight minutes after the others.
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'
When the CLI returns:
$ kubectl get nkpclusters -A
NAMESPACE NAME NKP VERSION KUBERNETES VERSION PHASE AGE
debug-test-xhj92 demo-wkl-02 v2.18.0 v1.35.2 Reconciled 105d
kommander host-cluster v2.19.0 v1.35.2 Reconciled 105d
nai-demo-jk9kl-zxbpw nai-demo v2.18.0 v1.35.2 Reconciled 15d
$ kubectl get kommandercore -A
NAME VERSION
kommander-core v2.19.0
Expected state: management cluster at v2.19.0, Kubernetes unchanged, workload clusters still at v2.18.0.
Step 4: no workspace step
nkp upgrade workspace is deprecated. For ClusterClass clusters, nkp upgrade cluster upgrades platform apps and infrastructure in one command. Check the workload clusters right after the Kommander upgrade:
# AppDeployments in a workspace, as rewritten by the Kommander upgrade:
kubectl get appdeployments -n <workspace-ns> -o custom-columns=NAME:.metadata.name,APP:.spec.appRef.name
# HelmReleases on the attached cluster itself:
kubectl --kubeconfig <workload>.conf get hr -A
Result on our clusters:
- Every workspace AppDeployment points at the 2.19 versions: cert-manager 1.20.3, Gatekeeper 3.23.0, Traefik 39.0.7, kube-prometheus-stack 87.16.0, OpenCost 2.5.28.
- Both attached clusters, still on NKP 2.18 and Kubernetes 1.35.2, have reconciled them: 18 of 18 and 19 of 19 HelmReleases Ready.
The release notes describe this as a display issue: the UI shows the new versions while the clusters still run the old ones. On our clusters the apps were really upgraded, through each cluster's own Flux and OCI proxy.
Consequences:
- Workload clusters need scheduling headroom for the new app versions on the day of the Kommander upgrade.
- The "platform applications" step of the later cluster upgrade has little left to do.
The same step on 2.19.1
Same command with the 2.19.1 CLI on a cluster at 2.19.0:
- Same four pre-checks, same two steps.
- 22 minutes: 15 for the management plane, 7 for the platform apps.
- Five HelmReleases are not Ready for a few minutes, then recover: Kommander, its app management, the UI, the Dex authenticator, the forward-auth proxy.
- Platform app versions do not change, except Kommander (0.19.1), its app management and its UI.
A HelmRelease check does not show everything. Before our 2.19.1 run the NKP Insights backend was crash-looping while its HelmRelease was Ready. It did not block the upgrade. Use the pod list from the pre-flight, not only the HelmRelease list.
Troubleshooting
A full Prometheus volume
Symptom. kube-prometheus-stack moves from 82.13.6 to 87.16.0 and restarts Prometheus. The pod goes into CrashLoopBackOff:
panic: write /prometheus/chunks_head/003759: no space left on device
Cause. The 100Gi volume was full (97.9 GB used, 30-day retention, no size limit). The restart replays the WAL and the replay needs free space. This is not a 2.19 defect. The chart bump restarts Prometheus.
Fix. Raising the volume size in the kube-prometheus-stack values does nothing. The chart keeps the storage block of an existing Prometheus object (a lookup in the template). Grow the PVC directly:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: <prometheus-pvc> # kubectl get pvc -n kommander | grep prometheus
namespace: kommander
spec:
resources:
requests:
storage: 150Gi
kubectl apply --server-side --force-conflicts -f prometheus-pvc.yaml
kubectl get pvc -n kommander <pvc> -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}'
Then:
- The Nutanix CSI expands the volume online. The PVC reports
FileSystemResizePending. - Delete the Prometheus pod once. The filesystem is resized at mount.
- Prometheus repairs its WAL and scrapes again.
- Set
retentionSizein your override.
Check the PVC usage before the upgrade.
Gatekeeper audit OOMKilled
The chart default for gatekeeper-audit is 512Mi, on Gatekeeper 3.23.0 too. On our cluster 1Gi was still OOMKilled (three restarts in nine minutes). 2Gi is stable. The override is a ConfigMap of Helm values in the app's namespace, referenced by the HelmRelease:
apiVersion: v1
kind: ConfigMap
metadata:
name: gatekeeper-overrides
namespace: kommander
data:
values.yaml: |
audit:
resources:
limits:
memory: 2Gi
Related: the NKP Insights backend was OOMKilled at its 1Gi default on 2.18. 2.19 raises the default to 2Gi.
Monitoring commands
kubectl get managementplane nkp-platform -o jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get nkpcluster host-cluster -n kommander -o jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'
kubectl get pods -A | grep -vE 'Running|Completed'
The object's conditions name the blocker. The HelmRelease status gives the reason.
Next: Part 3, the node roll.


