This part covers everything before the first nkp upgrade command: gates, cleanups, artifacts and the registry push. Read the release post first. Tested on an air-gapped management cluster (3 control plane + 6 workers, Ultimate, Nutanix AHV) upgraded from 2.18.0.
What you are about to do
Three phases:
- Management plane and platform apps:
nkp upgrade kommander. - Management cluster: Kubernetes and node roll.
- Each workload cluster.
The per-workspace step of 2.18 is gone. End on 2.19.1: from 2.18 the path is 2.18.0, 2.19.0, 2.19.1.
Step 1: check the compatibility gates
| Gate | Requirement | Check |
|---|---|---|
| Current NKP | 2.18.0 exactly. The Release object that ships with 2.19 declares a single upgrade path, fromVersion: v2.18.0 |
kubectl get nkpclusters -A |
| Prism Central / AOS | pc.7.5 or pc.7.6 (the Release object lists 7.5, 7.5.1 and 7.6) |
PC UI, or the UpgradePlan webhook will tell you (Part 3) |
| Kubernetes on every cluster | 1.35.x. Cluster API refuses to skip a minor version, so a cluster still on 1.34 must be rolled to 1.35.2 first | kubectl get nkpclusters -A (the KUBERNETES VERSION column) |
| OS image | the 1.36.2 pre-built image uploaded to Prism Central | PC image service |
| Prism Central role | 2.19 adds permissions to the NKP role; the guide says to update it to the union of both versions before upgrading. | PC Roles |
| NDK / NAI | NDK 2.3 and NAI 2.8.x are the supported catalog versions for 2.19 | kubectl get pods -n ntnx-system -o jsonpath='{..image}' | tr ' ' '\n' | grep ndk |
| Cluster names | 58 characters or fewer for workload clusters (a documented known issue breaks platform app installation above that) | kubectl get clusters.cluster.x-k8s.io -A |
| Prism Central projects | one project per cluster, control plane and worker pools alike (Projects 2.0) | the PC project of every VM of the cluster |
| Registry | space for a new image set | your registry's storage dashboard |
A cluster still on 1.34 must be rolled to 1.35.2 with the 2.18 CLI first, or decommissioned.
Step 2: the cleanups 2.19 asks for
GitRepository-based catalogs are removed. nkp upgrade kommander checks for them first. Deleting a source also deletes the catalog App objects it owns: on our cluster, 22 Apps were removed with it. Check that no AppDeployment uses them before you delete:
# catalog Apps that come from a GitRepository, and whether any AppDeployment uses them
kubectl get apps -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,KIND:.spec.sourceRef.kind,SRC:.spec.sourceRef.name | grep GitRepository
kubectl get appdeployments -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,APP:.spec.appRef.name,KIND:.spec.appRef.kind
kubectl delete gitrepository <name> -n <workspace-ns>
Headlamp becomes a platform app. A Headlamp installed from a catalog on 2.18 uses the same AppDeployment name, headlamp. Remove the catalog one before the upgrade. Its HelmRelease goes away about a minute later. The chart's headlamp namespace stays, empty: delete it by hand.
The Kubernetes Dashboard is not touched. It keeps running through the upgrade.
Decommissioning a cluster has an order.
- Delete the Flux Kustomizations that own the cluster manifest. With
prune: falsethey would re-apply it. - Delete the
NKPCluster. Its finalizers remove the CAPI Cluster, the KommanderCluster and the VMs. - Wait until the KommanderCluster is gone, then delete the workspace. The workspace webhook refuses earlier.
- Delete the leftover
KubeFedClusterof that cluster. It stays Offline with no owner.
Step 3: verify the cluster is ready
# 1. Everything green?
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'
# 2. Pods that are not Running or Completed: each one is a question for the platform phase
kubectl get pods -A | grep -vE 'Running|Completed'
# 3. PDB audit: zero allowed disruptions = a drain that may never finish (Part 3)
kubectl get pdb -A | awk '$5=="0"'
# 4. Can Cluster API still create a VM? Control plane at full count, no machine stuck, no failed VM creation in the provider log
kubectl get kubeadmcontrolplane -A
kubectl get machines -A | grep -v Running
kubectl logs -n capx-system deploy/capx-controller-manager --since=24h | grep -c 'Failed to create VM'
# 5. Backup: an etcd snapshot, copied off the node (the file is root-only under /var/lib/etcd, hence the copy to a home directory first)
kubectl -n kube-system exec etcd-<cp-node> -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
snapshot save /var/lib/etcd/pre-219-upgrade.db
kubectl -n kube-system exec etcd-<cp-node> -- etcdutl snapshot status /var/lib/etcd/pre-219-upgrade.db -w table
ssh <user>@<cp-node> 'sudo install -o $USER -m 600 /var/lib/etcd/pre-219-upgrade.db ~/pre-219-upgrade.db'
scp <user>@<cp-node>:pre-219-upgrade.db .
sha256sum pre-219-upgrade.db
Notes:
- Item 2: fix the pods that never become Ready before you start, so the platform phase is not waiting on them. Typical case:
gatekeeper-audit, OOMKilled at the chart default of 512Mi (6,125 restarts on our cluster). Raise the limit before the upgrade with agatekeeper-overridesConfigMap. We needed 2Gi. Part 2 has the YAML. - Item 5: the roll replaces the node that holds the snapshot. Copy the file off the node, then verify size and hash. Ours: 884 MB, 53,763 keys.
Step 4: stage the artifacts
1. The air-gapped bundle nkp-air-gapped-bundle_v2.19.0_linux_amd64.tar.gz: 34.6 GiB to download, 36 GiB extracted. New layout:
nkp-v2.19.0/
cli/nkp # the 2.19 CLI
kubectl # 1.36.2 client
container-images/
kommander-image-bundle-v2.19.0.tar # 17.7 GB, 263 image refs
konvoy-image-bundle-v2.19.0.tar # 6.2 GB, 122 image refs
nkp-nutanix-product-catalog-image-bundle-2.19.tar # 671 MB, NDK 2.3.0 (renamed)
nkp-release-specs-v2.19.0.tar # 70 KiB, NEW: the Release object
nkp-ai-applications-catalog-image-bundle-2.19.tar # 4.3 GB, NEW, optional: AI catalog
kubernetes-os-ubuntu-24.04-nkp-v2.19.0-v1.36.2_linux_amd64.tar # 1.8 GB, NEW: NKP Metal only
nkp-ubuntu-24.04-release-amd64.iso # NEW: NKP Metal node OS, not for AHV
image-artifacts/ # containerd and Kubernetes tarballs for image builds
konvoy-bootstrap-image-v2.19.0.tar
nkp-image-builder-image-v2.19.0.tar
Changes from 2.18:
- No
application-repositories/folder. The applications collection ships as an image in the kommander bundle. --kommander-applications-repositoryis no longer supported. The CLI help still lists it. Do not pass it.- The catalog tar is named
nkp-nutanix-product-catalog-image-bundle-2.19.tar. The guide's push command uses the old name,nutanix-product-catalog-<version>-airgapped.tar.
2. The 2.19 CLI, installed alongside the old one:
sudo cp /usr/local/bin/nkp /usr/local/bin/nkp218 # keep 2.18
sudo install -m 755 ./cli/nkp /usr/local/bin/nkp
nkp version
catalog: v0.11.1
diagnose: v0.14.0
imagebuilder: v2.19.0
kommander: v2.19.0
konvoy: v2.19.0
konvoy-cli: v2.19.0
konvoybundlepusher: v2.19.0
mindthegap: v1.26.1
nkp: v2.19.0
Keep the 2.18 binary until the last cluster is upgraded.
3. The OS image, uploaded to Prism Central. Ours: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2. An Ubuntu 24.04 CIS image for 1.36.2 is also published. You need the exact image name later.
Step 5: push the bundles to your registry
Push four tars from the extracted bundle root. Add the AI catalog tar if you want it. Skip the Metal tar and the ISO: they are for bare-metal hosts.
nkp push bundle \
--bundle ./container-images/kommander-image-bundle-v2.19.0.tar,./container-images/konvoy-image-bundle-v2.19.0.tar,./container-images/nkp-nutanix-product-catalog-image-bundle-2.19.tar,./container-images/nkp-release-specs-v2.19.0.tar \
--to-registry=https://<registry>/<project> \
--to-registry-username=<user> --to-registry-password='***' \
--to-registry-ca-cert-file <your-registry-ca.crt>
Measured on our push (five tars, AI catalog included):
- 456 images pushed: 1 release specs, 9 product catalog, 131 konvoy, 263 kommander, 52 AI catalog. No errors.
- Harbor: 433 to 501 repositories, +20.3 GB. Layers shared with 2.18 are deduplicated.
- The 2.19 images go in the same project as the 2.18 ones. Remove nothing: clusters on 2.18 still pull their images.
- Duration: 7 minutes with bundle and registry on the same host. Yours depends on your network.
Check a few tags before you start:
for r in mesosphere/kommander-applications/collection:v2.19.0 mesosphere/kommander-chart:v2.19.0 \
nutanix/ntnx-csi:3.9.0 kube-proxy:v1.36.2 nutanix-cloud-native/nkp-release-specs/releases:latest; do
curl -sk -o /dev/null -w "%{http_code} $r\n" -u <user>:'***' \
"https://<registry>/api/v2.0/projects/<project>/repositories/$(echo ${r%%:*} | sed 's#/#%252F#g')/artifacts/${r##*:}"
done
How an air-gapped cluster fetches 2.19 without a flag
The 2.19 guide has no air-gapped variant of the upgrade commands. None is needed. Two settings made at install time do the work:
# nodes: every image pull goes to the mirror (containerd configuration from the cluster variable)
kubectl get nkpcluster host-cluster -n kommander -o json \
| jq '.spec.capiCluster.topology.variables[] | select(.name=="clusterConfig") | .value.globalImageRegistryMirror.url'
# Flux: every chart and the applications collection are fetched through an in-cluster OCI proxy
kubectl get secret nkp-oci-proxy -n kommander -o jsonpath='{.data.address}' | base64 -d; echo
kubectl get ocirepository -n kommander -o json \
| jq '[.items[] | select(.spec.proxySecretRef.name=="nkp-oci-proxy")] | length'
On our cluster: the mirror URL, the proxy address http://flux-oci-mirror.kommander-flux.svc.cluster.local:8443, and 180 of 180 OCIRepositories.
- The upgrade adds a
v2.19.0collection reference next to thev2.18.0one. - The proxy maps
ghcr.ioanddocker.ioto the mirror. - The pre-check "Checking docker.io/nutanix credentials are present for the fleet" passes on an air-gapped cluster with no Docker Hub token.
The 2.19.1 bundle
Push the 2.19.1 bundle the same way.
| 2.19.0 | 2.19.1 | |
|---|---|---|
| Download / extracted | 34.6 GiB / 36 GiB | 33.4 GiB / 35 GiB |
| Image tars | 6 | 6, renamed -v2.19.1 (the two catalog tars keep 2.19) |
| NKP Metal ISO in the archive root | yes | no |
| Registry growth on our push | +20.3 GB | +4.9 GB, no new repository |
| Accepted source version | 2.18.0 | 2.19.0 |
| Kubernetes / node image | 1.36.2 | unchanged, same image |
Ready checklist
- All HelmReleases and Kustomizations green, every non-Running pod explained
- PDB audit list known; zero-disruption PDBs have a plan
- CAPX log has no failed VM creations; control plane at full replica count
- etcd snapshot taken, copied off the node, hash verified
- GitRepository catalogs gone, catalog Headlamp removed, 1.34 clusters rolled or gone
-
nkp versionshows the target version andnkp218 versionshows 2.18.0 - Four (or five) image bundles pushed; the collection and chart tags answer in the registry
- The 1.36.2 OS image visible in Prism Central, name copied verbatim
Next: Part 2, the Kommander upgrade.


