Series overviewThe start of the Upgrade to NKP 2.19 series · 5 parts
Start with Part 1 →

NKP 2.19.0 was released in October 2026 and 2.19.1 in November. We upgraded our air-gapped management cluster (3 control plane + 6 workers, Ultimate license, Nutanix AHV) to 2.19.0 the day after GA, then to 2.19.1, and checked the release notes against the cluster with kubectl each time.

Go to 2.19.1, not 2.19.0. It is a patch release with no new features and a list of fixes that matter for an upgrade from 2.18, a security fix in the UI and one in the MCP server. The details are in the first section. The rest of the post describes 2.19 as a whole and applies to both versions.

This is the first post of our 2.19 series. The upgrade procedure follows in five parts.

2.19.0 vs 2.19.1

What the 2.19.1 release notes list as resolved:

Fix Who is affected on 2.19.0
Remote code execution in the UI's Node.js Brotli decoder (CVE-2026-111497); the UI moves to Node.js 24.22.1 everyone
The NKP MCP server on the management cluster could grant cluster administrator access without authenticating the caller connected Pro and Ultimate management clusters (the MCP server is not in air-gapped bundles)
nkp upgrade cluster named the Nutanix registry credentials secret after the NKPCluster instead of the CAPI cluster; rolled nodes did not get the credentials and the upgrade to 2.19 was blocked management clusters created on 2.17, where the two names differ
Preflight checks failed for a mirror registry that needs no credentials anonymous mirrors
A UI upgrade failed when the registry preflight check had to be skipped; the UI showed the task as running while the UpgradePlan had failed Harbor as global registry with a public project
Pre-provisioned cluster upgrade from 2.18 did not complete; upgraded machines did not join pre-provisioned clusters
The air-gapped bundle shipped dkp-diagnostics-node-collector 0.12.0 while nkp diagnose needs 0.14.0; support bundles had no node-level logs air-gapped clusters
Metal Host Inventory not visible with the Workspace Admin role NKP Metal

Two known issues are new in the 2.19.1 notes:

  • A ClusterClass workload cluster that uses a public registry mirror and was created without mirror credentials can stall in UpgradingPlatformApplications. The upgrade sets ociRegistry.secretRef to <cluster-name>-oci-creds and nothing creates that secret. Workaround from the notes: create an empty secret of that name in the workspace namespace on the managed cluster.
  • With Prism Central 7.6 and AOS older than 7.6, the UI offers only "None" as GPU mode when you create a cluster or a node pool.

What we measured on the bundle and the cluster:

2.19.0 2.19.1
Kubernetes / node image 1.36.2 / Rocky 9.8 unchanged, same image
Image references in the bundle 449 449
Kommander and applications collection v2.19.0 v2.19.1
Kommander UI 17.313.79 17.313.87
Diagnostics node collector v0.12.0 v0.14.0
Cluster API stack operator 0.6.29 0.6.30
Cluster API runtime extensions 0.62.7 0.62.8
Platform apps, Cilium, MetalLB, CSI, CAPI, CAPX unchanged
Supported upgrade path (Release object) from 2.18.0 from 2.19.0 only
nkp upgrade kommander on our lab VMs 43 min 22 min
Node roll yes, 9 nodes in 60 min none: same Kubernetes version, same node image

On 2.19.1 the upgrade is the Kommander step only. The cluster upgrade command returns Cluster "itcs-nkp-mgmt-prod" is already using the latest supported Kubernetes version "1.36.2" and changes nothing.

Workload clusters need one more step: an UpgradePlan with version: v2.19.1. It took under a minute per cluster and replaced no node (Part 5).

From 2.18, the Release object in each bundle declares a single source version, so the path is 2.18.0 to 2.19.0 to 2.19.1. The second hop is short. Plan both in the same change window.

Unchanged on 2.19.1: the five version rows below, the Gatekeeper audit memory default, the single-replica nkp-cluster-webhook (Part 3), and the --kommander-applications-repository flag in the CLI help.

What Nutanix announced

Three themes: Kubernetes on bare metal, AI workloads, simpler lifecycle management.

Area 2.18 2.19
Bare metal - NKP Metal: Kubernetes on physical servers through Foundation Central, with its own node OS image
AI AI Navigator AI applications catalog (KAI Scheduler, Slurm Operator, Kueue, Kubeflow Pipelines, JupyterHub, Milvus Operator), GPU scheduling and slicing through Kubernetes dynamic resource allocation for NVIDIA and AMD, CNCF AI conformance, an MCP server enabled by default on connected Pro and Ultimate management clusters (not in air-gapped bundles)
Upgrade model Kommander, then every workspace, then every cluster Kommander, then one command or one click per cluster with automated preflight checks (nkp upgrade workspace deprecated)
Kubernetes 1.35.2 1.36.2
Node OS Rocky Linux 9.7 and Ubuntu 24.04 pre-built images Rocky Linux 9.8 and Ubuntu 24.04 pre-built images; RHEL 9.8 is supported but you build that image yourself with the NKP image builder
Container runtime / etcd containerd 2.1.6 / etcd 3.5.24 containerd 2.3.1 / etcd 3.6.8
Cluster API / CAPX 1.12.4 / 1.9.2 1.12.11 / 1.10.15
Nutanix CSI 3.7.x 3.9.0
Cluster dashboard Kubernetes Dashboard Headlamp (enabled by default on new deployments, not on upgraded ones)
Catalogs Git and OCI sources OCI only (GitRepository catalogs removed), new substitution variables for catalog apps
Prism Central pc.7.3 / AOS 7.3 minimum pc.7.5 and pc.7.6; Projects 2.0 (one project per cluster); every cluster registered in Prism Central through the Konnector agent
Image registry (connected) - Nutanix registry on Docker Hub with a portal token; air-gapped clusters keep their mirror
AHV VMs - VM management through Kubernetes APIs with a VM Operator (announced in the 2.19.0 notes and no longer listed in the 2.19.1 revision; no image for it in either air-gapped bundle)
Air-gapped bundle applications tarball + 3 image tars no applications tarball, 6 image tars, a Release metadata artifact

Deprecated or removed in 2.19, per the notes: nkp upgrade workspace, the --kommander-applications-repository flag on nkp install kommander and nkp upgrade kommander, GitRepository-based catalog applications, and the Kubernetes Dashboard platform app.

The version table vs the cluster

We compared the component and application tables in the release notes with the images that run after the upgrade.

Matches: Kubernetes 1.36.2, etcd 3.6.8, containerd 2.3.1, CAPI 1.12.11, CAPX 1.10.15, Cilium 1.19.4, CSI 3.9.0, kube-vip 1.1.1, and every platform app we run on 2.19.0.

Five rows differ. In each case the cluster runs a newer build than the table. The application table is the same in the 2.19.1 revision of the notes, so on 2.19.1 three more rows differ: Kommander and its app management (0.19.1, listed as 0.19.0) and the UI (17.313.87, listed as 17.313.79).

Component Release notes Our cluster
MetalLB 0.15.2 v0.15.3 (controller and speaker)
Cluster Autoscaler 1.36.0 v1.36.1
Dex 2.37.0 v2.42.0
Thanos 0.39.2 v0.41.0
Trivy (NKP Insights table) v0.64.* 0.69.3

Also worth knowing:

  • AI Navigator 0.9.0 and NKP Pulse 0.5.1 are in the supported-applications table. The air-gapped bundle has no image for them, and an air-gapped cluster has no ClusterApp for them.
  • The Nutanix cloud provider (0.8.12) and the Konnector agent (1.4.2) run on every cluster. They are not in the component table.

If you keep a version inventory, read it from the cluster:

bash
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' \
  | sort -u | grep -E 'metallb|cluster-autoscaler|/dex:|thanos:|trivy:'

What the upgrade installs

The upgrade is a custom resource

The Upgrade button in the UI creates one object: an UpgradePlan (upgrade.nkp.nutanix.com/v1alpha1). A GitOps-driven upgrade does not.

  • Spec: target NKP version, Kubernetes version, control plane and worker image names, a reference to the NKPCluster.
  • Controller: upgrade-plan-controller-manager.
  • Phases: ValidatingInputs, UpgradingManagementPlane, UpgradingKommander, UpgradingPlatformApplications, UpgradingKubernetes, Upgraded or Failed.
  • Admission webhook: calls Prism Central before the object is accepted. It reads the PC version from /api/prism/v4.4/config/domain-managers and checks both image names with /api/vmm/v4.3/content/images.

The object is small. You can write it by hand and keep it in git.

bash
kubectl get upgradeplans -A
kubectl get upgradeplan <name> -n <ns> -o jsonpath='{range .status.steps[*]}{.name}: {.state}{"\n"}{end}'

The NKP Metal provider is installed everywhere

The Kommander upgrade installs cluster-api-provider-host-agent (0.9.4) as an InfrastructureProvider. It also adds a nkp-host-agent ClusterClass in every workspace namespace. This happens on AHV-only clusters too, as it already did for the AWS, Azure, GCP and vSphere providers. Add it to your controller and CRD inventory. The UI gets a Metal Host Inventory page, empty on AHV.

NKP 2.19 UI, Administration, Metal Host Inventory page on our AHV management cluster: No Hosts Onboarded

bash
kubectl get infrastructureproviders -A
kubectl get clusterclasses -A | grep host-agent

Platform apps on attached clusters upgrade with Kommander

The notes say that after the Kommander upgrade the UI shows the new application versions for every workspace, while clusters you did not upgrade still run the previous versions.

On our clusters the applications were really upgraded. The Kommander upgrade rewrites every workspace AppDeployment to the 2.19 versions. Both attached clusters, still on NKP 2.18 and Kubernetes 1.35, reconciled them through their own Flux: cert-manager 1.20.3, Gatekeeper 3.23.0, Traefik chart 39.0.7, kube-prometheus-stack 87.16.0.

Consequence: workload clusters need scheduling headroom on the day of the Kommander upgrade.

bash
# on a workload cluster, right after the Kommander upgrade:
kubectl get hr -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CHART:.status.lastAttemptedRevision,READY:.status.conditions[0].status'

A machine-readable compatibility matrix

The bundle has a new tar, nkp-release-specs-<version>.tar. It holds a Release custom resource (release.nkp.nutanix.com/v1alpha1) with:

  • compatible AOS and Prism Central versions: 7.5, 7.5.1, 7.6
  • supported Kubernetes versions: 1.36.2, 1.35.2, 1.34.3, 1.34.1
  • the supported upgrade path: one source version

The UI reads this object. After the Kommander upgrade, the first login shows the available version and its release notes.

bash
kubectl get $(kubectl api-resources --api-group=release.nkp.nutanix.com -o name) -A

NKP 2.19 UI after the Kommander upgrade: the Upgrade Available dialog lists NKP v2.19.0 as the latest version with Kubernetes v1.36.2 and the release notes, and the left menu has new Upgrades and Metal Host Inventory entries

Kubernetes Dashboard and Headlamp

As documented, the upgrade leaves the Kubernetes Dashboard running: AppDeployment kubernetes-dashboard 7.14.1, four kubernetesui/* pods. Headlamp 0.45.0 is available as a platform app. It is enabled by default on new deployments only.

If you installed Headlamp from a catalog on 2.18, its AppDeployment has the same name as the platform app. Remove the catalog one before you enable the platform one.

KubeFed

The 2.19 guide no longer has a KubeFed chapter. The kubefed ClusterApp (0.12.1) is still shipped and installed. It still federates workspaces, projects and RBAC.

NATS

The konvoy bundle contains three NATS images: nats:2.10-alpine, nats-box and the config reloader. No pod on our management cluster runs them, on 2.19.0 or 2.19.1. The guide and the notes do not describe a NATS component. We will look at what NKP uses it for in a later post.

The deprecated flag in the CLI help

The notes say --kommander-applications-repository is no longer supported. nkp upgrade kommander --help still lists it, with the release version as default. Do not pass it. The applications collection now ships as an image, mesosphere/kommander-applications/collection.

vSphere provider memory

On 2.18 the vSphere provider controller was OOMKilled at its 150Mi limit on our AHV cluster: 910 restarts in two days. 2.19 sets 384Mi. Zero restarts since.

Node stack changes

2.18 node 2.19 node
OS Rocky Linux 9.7, kernel 5.14.0-611 Rocky Linux 9.8, kernel 5.14.0-687
containerd 2.1.6 2.3.1
etcd 3.5.24 3.6.8
kube-proxy none (Cilium) none (Cilium)
Multus 4.2.4, unused 4.2.4, unused

etcd 3.5 to 3.6 is a major version. etcdctl endpoint status reports storage version 3.6.0. A snapshot taken before the upgrade is a 3.5 snapshot. Check that your restore procedure matches the version you restore into.

On the management cluster:

  • Two new operators, Release and UpgradePlan, run next to NKPCluster and ManagementPlane.
  • The CAPI stack operator moves from 0.4.x to 0.6.29 (0.6.30 on 2.19.1). 13 providers are Ready at their new versions afterwards.

Deprecated does not mean removed

After the upgrade:

  • The Kubernetes Dashboard still runs.
  • KubeFed still runs.
  • The CLI still accepts --kommander-applications-repository.
  • GitRepository catalogs are removed. nkp upgrade kommander checks for them first.

Before you delete a GitRepository catalog source, check what uses it. When we deleted two sources, the catalog controller removed the 22 App objects they owned.

bash
kubectl get apps -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SRC:.spec.sourceRef.kind,REF:.spec.sourceRef.name | grep GitRepository
kubectl get appdeployments -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,APP:.spec.appRef.name

Gatekeeper audit memory

gatekeeper-audit is OOMKilled at the chart default of 512Mi. The 2.18 notes list it as a known issue. The Gatekeeper 3.23.0 chart in 2.19.0 and 2.19.1 keeps the same default.

On our cluster: 6,125 restarts in 104 days at 512Mi, still OOMKilled at 1Gi, stable at 2Gi. Part 2 has the override.

bash
kubectl get pod -n kommander -l gatekeeper.sh/operation=audit \
  -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LIMIT:.spec.containers[0].resources.limits.memory'

Should you upgrade?

Yes, to 2.19.1. On our cluster the Kommander upgrade finished without a timeout and the node roll replaced 9 nodes in an hour.

Check four things first, each covered in the series:

  • Free space on the Prometheus volume. The chart bump restarts Prometheus and a full volume crash-loops on WAL replay (Part 2).
  • Admission webhooks with one replica and failurePolicy: Fail. A node drain takes them down, and the UI can show Failed while the roll continues (Part 3).
  • Whether your node subnet is Network Controller managed. Cilium's DSR default does not work there (Part 4).
  • PodDisruptionBudgets with zero allowed disruptions on every cluster, NAI's PostgreSQL in particular (Part 5).

Start with Part 1, preparation and the air-gapped push.