This part covers everything before the first nkp upgrade command: gates, cleanups, artifacts and the registry push. Read the release post first. Tested on an air-gapped management cluster (3 control plane + 6 workers, Ultimate, Nutanix AHV) upgraded from 2.18.0.

What you are about to do

Three phases:

  1. Management plane and platform apps: nkp upgrade kommander.
  2. Management cluster: Kubernetes and node roll.
  3. Each workload cluster.

The per-workspace step of 2.18 is gone. End on 2.19.1: from 2.18 the path is 2.18.0, 2.19.0, 2.19.1.

Step 1: check the compatibility gates

Gate Requirement Check
Current NKP 2.18.0 exactly. The Release object that ships with 2.19 declares a single upgrade path, fromVersion: v2.18.0 kubectl get nkpclusters -A
Prism Central / AOS pc.7.5 or pc.7.6 (the Release object lists 7.5, 7.5.1 and 7.6) PC UI, or the UpgradePlan webhook will tell you (Part 3)
Kubernetes on every cluster 1.35.x. Cluster API refuses to skip a minor version, so a cluster still on 1.34 must be rolled to 1.35.2 first kubectl get nkpclusters -A (the KUBERNETES VERSION column)
OS image the 1.36.2 pre-built image uploaded to Prism Central PC image service
Prism Central role 2.19 adds permissions to the NKP role; the guide says to update it to the union of both versions before upgrading. PC Roles
NDK / NAI NDK 2.3 and NAI 2.8.x are the supported catalog versions for 2.19 kubectl get pods -n ntnx-system -o jsonpath='{..image}' | tr ' ' '\n' | grep ndk
Cluster names 58 characters or fewer for workload clusters (a documented known issue breaks platform app installation above that) kubectl get clusters.cluster.x-k8s.io -A
Prism Central projects one project per cluster, control plane and worker pools alike (Projects 2.0) the PC project of every VM of the cluster
Registry space for a new image set your registry's storage dashboard

A cluster still on 1.34 must be rolled to 1.35.2 with the 2.18 CLI first, or decommissioned.

Step 2: the cleanups 2.19 asks for

GitRepository-based catalogs are removed. nkp upgrade kommander checks for them first. Deleting a source also deletes the catalog App objects it owns: on our cluster, 22 Apps were removed with it. Check that no AppDeployment uses them before you delete:

bash
# catalog Apps that come from a GitRepository, and whether any AppDeployment uses them
kubectl get apps -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,KIND:.spec.sourceRef.kind,SRC:.spec.sourceRef.name | grep GitRepository
kubectl get appdeployments -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,APP:.spec.appRef.name,KIND:.spec.appRef.kind
kubectl delete gitrepository <name> -n <workspace-ns>

Headlamp becomes a platform app. A Headlamp installed from a catalog on 2.18 uses the same AppDeployment name, headlamp. Remove the catalog one before the upgrade. Its HelmRelease goes away about a minute later. The chart's headlamp namespace stays, empty: delete it by hand.

The Kubernetes Dashboard is not touched. It keeps running through the upgrade.

Decommissioning a cluster has an order.

  1. Delete the Flux Kustomizations that own the cluster manifest. With prune: false they would re-apply it.
  2. Delete the NKPCluster. Its finalizers remove the CAPI Cluster, the KommanderCluster and the VMs.
  3. Wait until the KommanderCluster is gone, then delete the workspace. The workspace webhook refuses earlier.
  4. Delete the leftover KubeFedCluster of that cluster. It stays Offline with no owner.

Step 3: verify the cluster is ready

bash
# 1. Everything green?
kubectl get hr -A --no-headers | awk '$4!="True"'
kubectl get kustomizations -A --no-headers | awk '$4!="True"'

# 2. Pods that are not Running or Completed: each one is a question for the platform phase
kubectl get pods -A | grep -vE 'Running|Completed'

# 3. PDB audit: zero allowed disruptions = a drain that may never finish (Part 3)
kubectl get pdb -A | awk '$5=="0"'

# 4. Can Cluster API still create a VM? Control plane at full count, no machine stuck, no failed VM creation in the provider log
kubectl get kubeadmcontrolplane -A
kubectl get machines -A | grep -v Running
kubectl logs -n capx-system deploy/capx-controller-manager --since=24h | grep -c 'Failed to create VM'

# 5. Backup: an etcd snapshot, copied off the node (the file is root-only under /var/lib/etcd, hence the copy to a home directory first)
kubectl -n kube-system exec etcd-<cp-node> -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/pre-219-upgrade.db
kubectl -n kube-system exec etcd-<cp-node> -- etcdutl snapshot status /var/lib/etcd/pre-219-upgrade.db -w table
ssh <user>@<cp-node> 'sudo install -o $USER -m 600 /var/lib/etcd/pre-219-upgrade.db ~/pre-219-upgrade.db'
scp <user>@<cp-node>:pre-219-upgrade.db .
sha256sum pre-219-upgrade.db

Notes:

  • Item 2: fix the pods that never become Ready before you start, so the platform phase is not waiting on them. Typical case: gatekeeper-audit, OOMKilled at the chart default of 512Mi (6,125 restarts on our cluster). Raise the limit before the upgrade with a gatekeeper-overrides ConfigMap. We needed 2Gi. Part 2 has the YAML.
  • Item 5: the roll replaces the node that holds the snapshot. Copy the file off the node, then verify size and hash. Ours: 884 MB, 53,763 keys.

Step 4: stage the artifacts

1. The air-gapped bundle nkp-air-gapped-bundle_v2.19.0_linux_amd64.tar.gz: 34.6 GiB to download, 36 GiB extracted. New layout:

output
nkp-v2.19.0/
  cli/nkp                                              # the 2.19 CLI
  kubectl                                              # 1.36.2 client
  container-images/
    kommander-image-bundle-v2.19.0.tar                 # 17.7 GB, 263 image refs
    konvoy-image-bundle-v2.19.0.tar                    #  6.2 GB, 122 image refs
    nkp-nutanix-product-catalog-image-bundle-2.19.tar  #  671 MB, NDK 2.3.0 (renamed)
    nkp-release-specs-v2.19.0.tar                      #   70 KiB, NEW: the Release object
    nkp-ai-applications-catalog-image-bundle-2.19.tar  #  4.3 GB, NEW, optional: AI catalog
    kubernetes-os-ubuntu-24.04-nkp-v2.19.0-v1.36.2_linux_amd64.tar   # 1.8 GB, NEW: NKP Metal only
  nkp-ubuntu-24.04-release-amd64.iso                   # NEW: NKP Metal node OS, not for AHV
  image-artifacts/                                     # containerd and Kubernetes tarballs for image builds
  konvoy-bootstrap-image-v2.19.0.tar
  nkp-image-builder-image-v2.19.0.tar

Changes from 2.18:

  • No application-repositories/ folder. The applications collection ships as an image in the kommander bundle.
  • --kommander-applications-repository is no longer supported. The CLI help still lists it. Do not pass it.
  • The catalog tar is named nkp-nutanix-product-catalog-image-bundle-2.19.tar. The guide's push command uses the old name, nutanix-product-catalog-<version>-airgapped.tar.

2. The 2.19 CLI, installed alongside the old one:

bash
sudo cp /usr/local/bin/nkp /usr/local/bin/nkp218      # keep 2.18
sudo install -m 755 ./cli/nkp /usr/local/bin/nkp
nkp version
output
catalog: v0.11.1
diagnose: v0.14.0
imagebuilder: v2.19.0
kommander: v2.19.0
konvoy: v2.19.0
konvoy-cli: v2.19.0
konvoybundlepusher: v2.19.0
mindthegap: v1.26.1
nkp: v2.19.0

Keep the 2.18 binary until the last cluster is upgraded.

3. The OS image, uploaded to Prism Central. Ours: nkp-rocky-9.8-release-cis-1.36.2-20261014035643.qcow2. An Ubuntu 24.04 CIS image for 1.36.2 is also published. You need the exact image name later.

Step 5: push the bundles to your registry

Push four tars from the extracted bundle root. Add the AI catalog tar if you want it. Skip the Metal tar and the ISO: they are for bare-metal hosts.

bash
nkp push bundle \
  --bundle ./container-images/kommander-image-bundle-v2.19.0.tar,./container-images/konvoy-image-bundle-v2.19.0.tar,./container-images/nkp-nutanix-product-catalog-image-bundle-2.19.tar,./container-images/nkp-release-specs-v2.19.0.tar \
  --to-registry=https://<registry>/<project> \
  --to-registry-username=<user> --to-registry-password='***' \
  --to-registry-ca-cert-file <your-registry-ca.crt>

Measured on our push (five tars, AI catalog included):

  • 456 images pushed: 1 release specs, 9 product catalog, 131 konvoy, 263 kommander, 52 AI catalog. No errors.
  • Harbor: 433 to 501 repositories, +20.3 GB. Layers shared with 2.18 are deduplicated.
  • The 2.19 images go in the same project as the 2.18 ones. Remove nothing: clusters on 2.18 still pull their images.
  • Duration: 7 minutes with bundle and registry on the same host. Yours depends on your network.

Check a few tags before you start:

bash
for r in mesosphere/kommander-applications/collection:v2.19.0 mesosphere/kommander-chart:v2.19.0 \
         nutanix/ntnx-csi:3.9.0 kube-proxy:v1.36.2 nutanix-cloud-native/nkp-release-specs/releases:latest; do
  curl -sk -o /dev/null -w "%{http_code} $r\n" -u <user>:'***' \
    "https://<registry>/api/v2.0/projects/<project>/repositories/$(echo ${r%%:*} | sed 's#/#%252F#g')/artifacts/${r##*:}"
done

How an air-gapped cluster fetches 2.19 without a flag

The 2.19 guide has no air-gapped variant of the upgrade commands. None is needed. Two settings made at install time do the work:

bash
# nodes: every image pull goes to the mirror (containerd configuration from the cluster variable)
kubectl get nkpcluster host-cluster -n kommander -o json \
  | jq '.spec.capiCluster.topology.variables[] | select(.name=="clusterConfig") | .value.globalImageRegistryMirror.url'

# Flux: every chart and the applications collection are fetched through an in-cluster OCI proxy
kubectl get secret nkp-oci-proxy -n kommander -o jsonpath='{.data.address}' | base64 -d; echo
kubectl get ocirepository -n kommander -o json \
  | jq '[.items[] | select(.spec.proxySecretRef.name=="nkp-oci-proxy")] | length'

On our cluster: the mirror URL, the proxy address http://flux-oci-mirror.kommander-flux.svc.cluster.local:8443, and 180 of 180 OCIRepositories.

  • The upgrade adds a v2.19.0 collection reference next to the v2.18.0 one.
  • The proxy maps ghcr.io and docker.io to the mirror.
  • The pre-check "Checking docker.io/nutanix credentials are present for the fleet" passes on an air-gapped cluster with no Docker Hub token.

The 2.19.1 bundle

Push the 2.19.1 bundle the same way.

2.19.0 2.19.1
Download / extracted 34.6 GiB / 36 GiB 33.4 GiB / 35 GiB
Image tars 6 6, renamed -v2.19.1 (the two catalog tars keep 2.19)
NKP Metal ISO in the archive root yes no
Registry growth on our push +20.3 GB +4.9 GB, no new repository
Accepted source version 2.18.0 2.19.0
Kubernetes / node image 1.36.2 unchanged, same image

Ready checklist

  • All HelmReleases and Kustomizations green, every non-Running pod explained
  • PDB audit list known; zero-disruption PDBs have a plan
  • CAPX log has no failed VM creations; control plane at full replica count
  • etcd snapshot taken, copied off the node, hash verified
  • GitRepository catalogs gone, catalog Headlamp removed, 1.34 clusters rolled or gone
  • nkp version shows the target version and nkp218 version shows 2.18.0
  • Four (or five) image bundles pushed; the collection and chart tags answer in the registry
  • The 1.36.2 OS image visible in Prism Central, name copied verbatim

Next: Part 2, the Kommander upgrade.