Upgrade to NKP 2.18, Part 2: Kommander, Platform Apps and the Workspaces
Artifacts staged and cluster verified (Part 1), the actual upgrade starts here: first the management cluster's platform (Kommander plus every platform app), then the same platform apps on every workspace. Kubernetes versions do not move yet - that is Part 4.
What this phase does
This phase installs 2.18's new architecture. Beyond upgrading every platform app, nkp upgrade kommander installs the new management-plane operators, creates the NKPCluster CRD, and adopts every existing cluster into an NKPCluster object - the new authoritative API. It also removes the deprecated AI Navigator apps (including their folders in the management git repo) and quietly extends the kommander root CA to 10 years. Expect the NKP UI to be flaky during the run, and re-fetch workload cluster kubeconfigs afterwards: auth tokens are invalidated.
Step 1: upgrade Kommander and the platform apps
From the extracted bundle root (the CLI resolves chart paths relative to it):
export KUBECONFIG=~/itcs-nkp-mgmt-prod.conf
nkp upgrade kommander \
--kommander-applications-repository ./application-repositories/kommander-applications-v2.18.0.tar.gz
The run works through a long list of steps; this is the actual sequence from our terminal, condensed:
✓ Ensuring upgrading conditions are met
✓ Checking Loki migration requirements
✓ Checking for deprecated GitRepository-based catalog apps
✓ Checking for potential conflict with non-NKP CloudNativePG operator
✓ Deploying new management plane operators
✓ Creating ManagementPlane nkp-platform
✓ Ensuring CRDs are migrated
✓ Removing deprecated AI Navigator RAG and Cluster Info Agent apps
✓ Ensuring root CA duration is upgraded
✓ Ensuring NKPCluster is created for the management cluster
✓ Ensuring NKPClusters are created for all KommanderClusters
✓ Ensuring core Kommander application [kommander] is upgraded
✓ Ensuring core Kommander application [kommander-appmanagement] is upgraded
✓ Ensuring core Kommander application [kommander-flux] is upgraded
✓ Ensuring KommanderCore is upgraded
✓ Ensuring new default Kommander applications are enabled
✗ Ensuring platform Kommander applications are upgraded [====> 27/28] (time elapsed 60m00s)
A banner warns about the Loki 2.x to 3.x migration early in the run. Take it seriously if you customized log storage (Part 3 is that story).
The run ends in one of two ways: a clean pass, or this:
✗ Ensuring platform Kommander applications are upgraded [====> 27/28] (time elapsed 60m00s)
failed to ensure "Ensuring platform Kommander applications are upgraded":
platform upgrade did not complete successfully: timed out waiting for
platform apps upgrade to complete: context deadline exceeded
Step 2: if the CLI times out, find the blocker by name
A timeout here is not a rollback and usually not a failure. The CLI has a 60-minute wait window; the controllers keep reconciling after it gives up. The frustrating part is the counter: 27/28 of what? The CLI never says. These do:
# Name the blocker:
kubectl get nkpcluster host-cluster -n kommander \
-o jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
# Or simply:
kubectl get hr -A --no-headers | awk '$4!="True"'
On our cluster, at the moment of the timeout, that second command returned exactly two lines - the 27/28 counter's missing app had a name all along:
kommander grafana-logging 70d False could not resolve ConfigMap chart values reference
'kommander/grafana-logging-config-overrides' with key 'values.yaml': error unmarshaling
JSON: cannot unmarshal string into Go value of type common.Values
kommander velero 70d False Helm upgrade failed for release kommander/velero
with chart velero@12.0.0: client rate limiter Wait returned an error: would exceed
context deadline
Fix what they name, then re-run the same command - it is resumable and skips completed work. Repeat until the run completes. Important non-finding from our run: we watched for 13 minutes to see if the platform version would flip on its own after we fixed the blocker. It does not - the final promotion needs the CLI run to complete.
Step 3: verify the platform version
$ kubectl get kommandercluster host-cluster -n kommander -o jsonpath='{.status.platform.version}'
v2.18.0
Step 4: upgrade every workspace
Same platform apps, now per workspace, all of them before any Kubernetes upgrade:
nkp get workspaces
nkp upgrade workspace <workspace-name> # repeat for each
The run starts by listing every AppDeployment it bumps, with the exact version pairs - useful as a record of what changed on that workspace:
$ nkp upgrade workspace debug-test
Starting port forwarding to service git-operator-system/git-operator-git, port 443
the port-forward has signaled ready
AppDeployment "cert-manager" has been upgraded to version "cert-manager-1.19.3" from "cert-manager-1.18.2"
AppDeployment "gatekeeper" has been upgraded to version "gatekeeper-3.22.0" from "gatekeeper-3.20.1"
AppDeployment "kommander-flux" has been upgraded to version "kommander-flux-2.8.5" from "kommander-flux-2.7.2"
AppDeployment "kube-prometheus-stack" has been upgraded to version "kube-prometheus-stack-82.13.6" from "kube-prometheus-stack-78.4.0"
AppDeployment "kubernetes-dashboard" has been upgraded to version "kubernetes-dashboard-7.14.1" from "kubernetes-dashboard-7.14.0"
AppDeployment "opencost" has been upgraded to version "opencost-2.5.14" from "opencost-2.4.0"
AppDeployment "prometheus-adapter" has been upgraded to version "prometheus-adapter-5.3.0" from "prometheus-adapter-5.0.0"
(...)
If the workspace runs Loki, the migration banner appears here too. It lists both scopes, workspace and project, and each one is a separate migration you will deal with (Part 3):
$ nkp upgrade workspace customera-dev
================================================================================
Loki Migration Information
================================================================================
Loki AppDeployments detected in:
- workspace namespace customera-dev (AppDeployment: grafana-loki)
- project: website-dev-x65zs | namespace: website-dev-x65zs (AppDeployment: project-grafana-loki)
This upgrade will migrate Loki from version 2.x to 3.x.
================================================================================
Then the CLI waits for the workload cluster to reconcile everything. A blocked workspace ends like the Kommander phase does:
✗ Ensuring applications are upgraded on clusters [...] (time elapsed 30m00s)
platform upgrade did not complete successfully: platform upgrade failed on 1/1 clusters
Empty workspaces complete in seconds. When a workspace hangs, the pending message on its NKPCluster names the exact objects it is waiting for - real examples from our runs:
waiting for: [Deployment/customera-dev/opencost status: 'InProgress']
waiting for: [HelmRelease/customera-dev/grafana-loki-v3 status: 'InProgress']
waiting for: [Job/kommander/grafana-loki-v3-pre-install status: 'InProgress']
Each name is a thread to pull: the object's own status or its controller's logs give the reason.
Step 5: verify the intermediate state
When all workspaces are done:
$ kubectl get nkpclusters -A
NAMESPACE NAME VERSION K8S VERSION STATUS
debug-test-xhj92 demo-wkl-02 v2.18.0 v1.34.3 Reconciled
customera-dev customera-wkl-02 v2.18.0 v1.34.3 Reconciled
flux-demo-github nkp-wrkld-demo-flux-github01 v2.18.0 v1.34.3 Reconciled
kommander host-cluster v2.18.0 v1.34.3 Reconciled
Platform everywhere at v2.18.0, Kubernetes untouched. Correct.
When it goes wrong: four failures from our run, with fixes
Bug 1: a broken app blocks the final phase, invisibly
Symptom: the 27/28 timeout above. The two failing HelmReleases (shown in Step 2) had been broken for 68 days - a garbage string saved through the 2.17 UI's override editor (that editor bug is fixed in 2.18, NCN-112624), and a velero that was never configured for this cluster.
Diagnosis: kubectl get hr -A --no-headers | awk '$4!="True"' and read each STATUS message.
Fix, one decision per app:
# grafana-logging: the override was garbage - delete it and let Flux retry
kubectl delete cm grafana-logging-config-overrides -n kommander
kubectl annotate hr grafana-logging -n kommander reconcile.fluxcd.io/requestedAt=$(date +%s) --overwrite
# velero: never used, chart defaults expect Rook Ceph which Nutanix deployments do not run - remove it
kubectl delete appdeployment velero -n kommander
Both answers are legitimate; decide per app before you start, not at minute 59 of the run.
Bug 2: capacity deadlock on a small cluster
Symptom: a 2-worker workload cluster could not schedule the new kube-prometheus-stack and the workspace upgrade timed out. From our capture:
Warning FailedScheduling 0/3 nodes are available: 1 node(s) had untolerated taint(s),
2 Insufficient cpu.
Normal NotTriggerScaleUp pod didn't trigger scale-up: 1 max node group size reached
$ kubectl describe nodes | grep -A1 "cpu.*%"
cpu 3560m (89%)
cpu 3420m (85%)
Diagnosis: the obvious fix - add a worker - does not work here: 2.18 freezes all topology changes while a platform upgrade reconciles. The webhook rejects direct CAPI edits, and the NKPCluster controller accepts your change but requeues it silently (Platform version not yet reconciled, requeueing - INFO level, only visible in kubectl logs -n kommander deploy/nkp-cluster-controller-manager).
Fix: free capacity on the demand side instead. We scaled down non-essential workloads (a Traefik replica, Grafana); Prometheus scheduled within seconds and the upgrade completed. The frozen worker change then applied itself - the controller log shows Patching Cluster two seconds after the workspace reported Reconciled, and the new machine provisioned with no further action:
13:49:17 INFO controllers.NKPClusterTopology Patching Cluster {"NKPCluster": "demo-wkl-02"}
13:49:19 demo-wkl-02-md-0-...-jrzqs phase=Provisioned <- the worker we asked for an hour earlier
Check capacity before upgrading small clusters; the new app versions need scheduling headroom.
Bug 3: a dead cluster cannot be upgraded, or easily deleted
Symptom: one cluster had been NotReady for 71 days (cni plugin not initialized - a CNI experiment that never worked). Its workspace upgrade would hang forever, so deletion was the right call - and nkp delete cluster wedged:
✓ Deleting NKPCluster
✗ Waiting for NKPCluster to be fully deleted: could not delete: ...
Diagnosis: a 2.18 ordering bug. The deletion removes the KommanderCluster first, after which NKP's own reconcilers crash on the object they just deleted (KommanderCluster not found) and the CAPI cluster is never touched.
Fix: delete the CAPI object directly; CAPX tears down the VMs and every finalizer drains:
kubectl delete cluster.cluster.x-k8s.io <name> -n <namespace>
Related known issue: the 2.18 CLI cannot delete pre-2.18 clusters at all (NCN-113770) - another reason the nkp217 binary stays installed.
Bug 4: the Loki migration
Symptom: the platform upgrade waits on 0 out of 1 logging stacks are ready, on the management cluster and again on every workspace with logging enabled.
Diagnosis and fix: big enough for its own article, Part 3 - including the converter's undocumented input contract, field by field, and the quick-fix procedure if you are stuck right now.
The monitoring kit
Everything above was diagnosed with five commands:
kubectl get nkpcluster <name> -n <ns> \
-o jsonpath='{range .status.conditions[?(@.status=="False")]}{.type}: {.message}{"\n"}{end}'
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'
kubectl logs -n kommander deploy/nkp-cluster-controller-manager -f
kubectl logs -n kommander deploy/loggingstack-controller-manager -f
The pattern held every single time: the NKPCluster condition names the blocker, the owning controller's log names the reason. The CLI spinner names nothing.
We wrapped these into two watch scripts (watch-upgrade.sh for the platform phase, watch-node-upgrade.sh for the node roll) that we are publishing at github.com/fen0l/nkp-quickstart.
Next: Part 4, the Kubernetes node roll - preflights, the cluster-name gotcha, and the PDB that froze our roll for 67 minutes.
