SeriesPart 5 of 5 in Upgrade to NKP 2.18
View the journey →

Upgrade to NKP 2.18, Part 5: GitOps Migration to the NKPCluster API

Our workload cluster nkp-wrkld-demo-flux-github01 is fully GitOps-managed: its CAPI Cluster manifest lives in a GitHub repository, Flux reconciles it, and for months "scale the cluster" has meant "push a commit".

Then we upgraded the management cluster to NKP 2.18 and pushed a routine commit: workers from 2 to 3. This article documents what happened next, and the repository migration 2.18 requires. It is part of our Upgrade to NKP 2.18 series.

The setup

The repository layout, simplified:

output
clusters/nkp-wrkld-demo-flux-github01/
  cluster/
    cluster.yaml          # CAPI Cluster (topology, workers, autoscaler bounds)
    kustomization.yaml
  ...                     # companion secrets, NDK resources
management/
  namespace-sync.yaml     # ServiceAccount flux-deployer + RBAC (hand-applied bootstrap)

On the management cluster, a Flux Kustomization reconciles the cluster/ path with a dedicated service account and, importantly, no pruning:

yaml
spec:
  path: ./clusters/nkp-wrkld-demo-flux-github01/cluster
  serviceAccountName: flux-deployer
  prune: false

One prerequisite we had already satisfied, straight from the release notes: spec.prune must be false on the Kustomization that manages the cluster. If pruning is on, the migration step later in this article - which removes the Cluster YAML from git - would make Flux delete your live cluster.

The commit that used to work

A one-line change: the worker pool's autoscaler bounds from 2 to 3. The kind of commit this repo has taken dozens of times. Push, and within Flux's 5-minute sync window:

console
$ kubectl get kustomization nkp-wrkld-demo-flux-github01 -n flux-clusters-lh5x6 \
    -o jsonpath='ready={...status} applied={...lastAppliedRevision} {...message}'
ready=False applied=main@sha1:5d98a40...   # <- the OLD revision. The new one never landed.
message: Cluster/flux-demo-github/nkp-wrkld-demo-flux-github01 dry-run failed
(NotAcceptable): admission webhook "webhook.nkpcluster.child.kommander.mesosphere.io"
denied the request: spec.topology: Forbidden: CAPI Cluster topology spec is managed by
NKPCluster flux-demo-github/nkp-wrkld-demo-flux-github01 and must not be modified
directly. Update .spec.capiCluster.topology on the NKPCluster resource instead.

Details worth noticing: the rejection happens at Flux's server-side dry-run, so nothing was partially applied - lastAppliedRevision still points at the pre-commit sha, the live topology is untouched, and the machine count never moved. Clean refusal, zero drift. The webhook even tells you the new write path. But your pipeline is red, and it will stay red on every retry until the repository speaks NKPCluster.

The kustomize-controller log shows the exact timing. This Kustomization had applied the Cluster manifest 47 times without incident (the adoption webhook tolerates no-op applies where git matches live):

output
15:30:11  server-side apply completed          # last successful reconcile, 289ms
15:30:53  Dependencies do not meet ready condition, retrying in 30s   # new commit arriving
15:31:53  Reconciliation failed after 149ms, next try in 2m0s          # first rejection
15:33:53  Reconciliation failed after 173ms, next try in 2m0s          # and every retry after

Note when this happened: not at the upgrade, but at the first commit that actually changed the topology, 42 seconds after the last successful apply. A no-drift repo can sit on 2.18 for weeks looking green, because the webhook tolerates applies that match the live state. Your GitOps pipeline breaks on the first real change after the upgrade, which might be an urgent scale-up at a bad hour.

How it decides is visible in the webhook's own logs (nkp-cluster-webhook in kommander, logger capi-topology-match, one entry per Flux retry):

output
"msg":"comparing CAPI Cluster topology to NKPCluster raw topology",
"cluster":"flux-demo-github/nkp-wrkld-demo-flux-github01"

The webhook is a drift gate: every incoming write to the CAPI Cluster's topology is diffed against the NKPCluster's stored topology. Identical (or subset) - allowed, which is how our repo's 47 no-op applies passed. Divergent - denied, with the exact mismatched fields listed:

output
workers.machineDeployments[0].metadata.annotations.cluster.x-k8s.io/
  cluster-api-autoscaler-node-group-max-size: value mismatch (typed: 3, raw: 2)
  cluster-api-autoscaler-node-group-min-size: value mismatch (typed: 3, raw: 2)

typed is what git wants; raw is what the NKPCluster holds. The error message doubles as a drift report.

The admission webhook webhook.nkpcluster.child.kommander.mesosphere.io now guards every CAPI Cluster whose lifecycle NKP manages. Since the 2.18 Kommander upgrade auto-created NKPCluster objects for all existing clusters ("adoption"), that means your clusters, including the GitOps-managed ones. The Cluster object your repo has been writing to for months is now read-only to you and to Flux.

This is not a bug. It is 2.18's core architectural change: the NKPCluster resource is the single authoritative API for cluster lifecycle, and everything else - CAPI Cluster, KommanderCluster - is derived state. But if you manage clusters through GitOps and do not read the upgrade guide's fine print, the first you hear of it is a Forbidden error in a kustomization that has worked for a year.

Why you cannot just migrate the repo immediately

The obvious fix is to put an NKPCluster manifest in git instead. The subtle trap is spec.version: it carries the platform version. If your git manifest says v2.17.1 while nkp upgrade workspace moves the live object to v2.18.0, Flux will fight the upgrade and win, reverting your platform version continuously. The guide sequences the repository migration AFTER the workspace and management cluster upgrades for exactly this reason. Order of operations:

  1. Disable/verify prune: false (before the Kommander upgrade)
  2. nkp upgrade kommander
  3. nkp upgrade workspace <ws> (platform apps to v2.18.0)
  4. Management cluster Kubernetes upgrade
  5. Repository migration (this article) + workload cluster Kubernetes upgrade

The migration, step by step

You do not write the NKPCluster manifest by hand - the CLI generates it, and the generation doubles as the workload cluster's Kubernetes upgrade:

bash
nkp upgrade cluster --cluster-name nkp-wrkld-demo-flux-github01 \
  --namespace flux-demo-github \
  --kubeconfig itcs-nkp-mgmt-prod.conf \
  --dry-run -o yaml > nkpcluster-upgrade.yaml

That was step 1 (generate). Step 2 is cleanup: the output contains the migrated resource with everything the 2.18 upgrade needs already applied: spec.capiCluster.topology.classRef.name bumped to the new ClusterClass, topology.version at the new Kubernetes version, updated OS image references. Clean out the server-side fields (resourceVersion, uid, creationTimestamp, generation, managedFields, status), keep name/namespace/labels/annotations - and add your own pending intent. In our case: the worker scale from 2 to 3 that the webhook rejected at the top of this article.

Step 3, before you push: give your GitOps service account RBAC for the new API group (see the Forbidden story below for why):

yaml
- apiGroups: ["clusters.nkp.nutanix.com"]
  resources: ["nkpclusters"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

Step 4: replace the old cluster.yaml contents with the NKPCluster (pruning is off, so removing the Cluster YAML is safe), one commit:

bash
git commit -m "Migrate nkp-wrkld-demo-flux-github01 to NKPCluster, upgrade to 2.18/K8s 1.35, scale workers to 3"

One push now carries three changes through one reconcile: the API migration, the Kubernetes upgrade (the NKPCluster controller rolls the control plane, then workers), and the worker scale that was refused as a CAPI write.

Afterwards you may re-enable prune: true if you had it on before.

Two more Forbiddens before the finish line

Our migration commit did not go green on the first try, and both failures are absent from the documentation:

RBAC. A least-privilege Flux service account has never heard of the new API group:

console
NKPCluster/flux-demo-github/nkp-wrkld-demo-flux-github01 dry-run failed (Forbidden):
nkpclusters.clusters.nkp.nutanix.com "nkp-wrkld-demo-flux-github01" is forbidden:
User "system:serviceaccount:flux-clusters-lh5x6:flux-deployer" cannot patch resource
"nkpclusters" in API group "clusters.nkp.nutanix.com"

Add the rule to whatever Role/ClusterRole your GitOps deployer uses, in the same PR as the migration:

yaml
- apiGroups: ["clusters.nkp.nutanix.com"]
  resources: ["nkpclusters"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

The bootstrap gap. We committed that RBAC fix to git and nothing happened - because our RBAC lives in the repository's bootstrap layer, which Flux does not reconcile (it cannot: the service account must exist before Flux can use it). Committing the fix is not deploying the fix; the bootstrap files need their original kubectl apply. Know which of your files are hand-applied.

Verified

console
$ kubectl get kustomization nkp-wrkld-demo-flux-github01 -n flux-clusters-lh5x6
ready=True applied=main@sha1:8ee002f...        # first green since the demo began

$ kubectl get machines -n flux-demo-github --sort-by=.metadata.creationTimestamp | tail -1
nkp-wrkld-demo-flux-github01-md-0-...-2nqpt   2026-07-07T16:05:59Z   Provisioned   # machine #4, born from a git push

The worker a CAPI commit could not create now exists, created by a commit through the NKPCluster API, with the right RBAC, with the bootstrap RBAC applied by hand where Flux cannot. Three different Forbidden errors, three one-line fixes.

Takeaways

  1. 2.18 adoption makes every NKP-managed CAPI Cluster read-only - for kubectl, for controllers, and for your GitOps repo. The webhook error tells you where to write instead: .spec.capiCluster.topology on the NKPCluster.
  2. prune: false before anything else. Removing the Cluster YAML with pruning on deletes the cluster.
  3. Do not migrate the repo before the platform upgrades finish - the NKPCluster spec.version field will fight your upgrade.
  4. Topology changes freeze during platform upgrades (see Part 2) - sequence scaling commits outside upgrade windows.
  5. After migration, GitOps works as before, one API level higher: you commit NKPCluster manifests, the controller derives the rest.
nkpkubernetesnutanixgitopsfluxupgrade
5
View the full journey →
Next part →