SeriesPart 5 of 6 in Introduction to NKP

Version note

This article was written for NKP 2.17. The current release is 2.18. See the Upgrade to NKP 2.18 series for the latest guidance.

Nutanix Kubernetes Platform (NKP) | Part 5: Workload Clusters & Applications

So far we've focused on the management cluster. In production, that cluster manages your infrastructure, it doesn't run your workloads. That's what workload clusters are for.

This guide covers creating managed workload clusters from the management plane, organizing teams with workspaces and projects, deploying applications through the NKP catalog, and setting up GitOps workflows with Flux CD. Everything here builds on the management cluster from Part 2 (or Part 3 for air-gapped) and the Day 2 configuration from Part 4.

Management vs Workload Clusters

Quick recap from Part 1:

	Management Cluster	Workload Cluster
Purpose	Runs CAPI controllers, Kommander, fleet management	Runs your application workloads
Who manages it	Self-managed (owns its own Machine objects)	Managed by the management cluster via CAPI
Platform services	Full Kommander stack (Prometheus, Loki, Dex, etc.)	Subset pushed from management via fleet management
Scaling	Rarely changes after initial deployment	Scales based on workload demand
Blast radius	Affects all clusters if it goes down	Only affects its own workloads

The separation matters. Management cluster issues affect everyone. Workload cluster issues are isolated. Different teams get different clusters, each with its own scaling, upgrade schedule, and failure domain.

Creating a Workload Cluster

Via the CLI

A workload cluster uses the same nkp create cluster nutanix command as the management cluster, but without --self-managed:

bash

nkp create cluster nutanix \
  --cluster-name team-platform \
  --endpoint "https://${NUTANIX_ENDPOINT}:9440" \
  --control-plane-prism-element-cluster ${NUTANIX_CLUSTER} \
  --control-plane-subnets ${NUTANIX_SUBNET} \
  --control-plane-endpoint-ip 10.12.52.230 \
  --control-plane-replicas 3 \
  --control-plane-vm-image "${NKP_IMAGE}" \
  --worker-prism-element-cluster ${NUTANIX_CLUSTER} \
  --worker-subnets ${NUTANIX_SUBNET} \
  --worker-replicas 3 \
  --worker-vm-image "${NKP_IMAGE}" \
  --kubernetes-service-load-balancer-ip-range "10.12.52.231-10.12.52.235" \
  --csi-storage-container ${STORAGE_CONTAINER} \
  --ssh-username ${SSH_USERNAME} \
  --ssh-public-key-file ${SSH_PUBLIC_KEY} \
  --insecure

Key differences from the management cluster:

Flag	Management	Workload
`--self-managed`	Yes	No (managed by mgmt cluster)
Kommander	Auto-installed (AHV)	Platform apps pushed via fleet management
VIP	Separate from mgmt	Must be a different IP
MetalLB range	Separate from mgmt	Must be a different range
Sizing	Sized for platform services	Sized for your workloads

For air-gapped workload clusters, add the same --airgapped and --registry-mirror-* or --bundle flags as in Part 3.

Via YAML (ClusterClass Topology)

For GitOps-managed clusters, define the workload cluster as a CAPI Cluster resource:

yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: demo-wkl-01
  namespace: demo              # workspace namespace
  labels:
    cluster.x-k8s.io/cluster-name: demo-wkl-01
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  topology:
    class: nkp-nutanix-v2.17.1
    version: v1.34.3
    controlPlane:
      replicas: 1
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 2
    variables:
      - name: clusterConfig
        value:
          controlPlane:
            nutanix:
              machineDetails:
                cluster:
                  name: PIKACHU
                  type: name
                subnets:
                  - name: pika-vlan52-workloads
                    type: name
                image:
                  name: nkp-rocky-9.7-release-cis-1.34.3-20260504011927.qcow2
                  type: name
                vcpus: 4
                memorySize: 8Gi
                systemDiskSize: 80Gi
          workers:
            nutanix:
              machineDetails:
                cluster:
                  name: PIKACHU
                  type: name
                subnets:
                  - name: pika-vlan52-workloads
                    type: name
                image:
                  name: nkp-rocky-9.7-release-cis-1.34.3-20260504011927.qcow2
                  type: name
                vcpus: 4
                memorySize: 16Gi
                systemDiskSize: 80Gi

Apply this YAML on the management cluster, and CAPI handles the rest: VM creation, kubeadm bootstrap, CNI/CSI deployment.

Workload Cluster Sizing

Workload Type	Workers	vCPU	RAM	Disk	Notes
General web apps	3-5	4	16 GiB	80 GiB	Standard stateless workloads
Data / ML pipelines	3-5	8	32 GiB	200 GiB	Larger disk for datasets
GPU workloads	2-4	8	32 GiB	200 GiB	GPU passthrough, Ubuntu OS
Dev / staging	1-3	2	8 GiB	80 GiB	Minimal footprint

Fleet Management

Workspaces

Workspaces are the top-level organizational boundary. They map to teams, environments, or business units. Each workspace:

Gets its own namespace on the management cluster (auto-generated, e.g., demo-2nc68)
Contains one or more clusters
Has its own RBAC policies
Can deploy apps to all clusters within it

Create workspaces via the CLI:

bash

nkp create workspace production
nkp create workspace demo

You can also create them from the dashboard (Workspaces, Create Workspace) or declaratively with a Workspace resource:

yaml

apiVersion: workspaces.kommander.mesosphere.io/v1alpha1
kind: Workspace
metadata:
  name: production
  namespace: kommander
spec:
  namespaceName: ""     # auto-generated

Projects

Projects provide namespace-level isolation within workload clusters. They sit inside workspaces and represent applications or services.

yaml

apiVersion: workspaces.kommander.mesosphere.io/v1alpha1
kind: Project
metadata:
  name: demo-apps
  namespace: demo              # workspace namespace
spec:
  placement:
    clusterSelector:
      matchLabels:
        kommander.d2iq.io/cluster-name: demo-wkl-01

Important

Project placement must use explicit label selectors. Using clusterSelector: {} (select all clusters) is rejected by the admission webhook with "project cannot be propagated to all member clusters". Use matchLabels for a single cluster or matchExpressions with operator: In for multiple clusters.

The auto-label kommander.d2iq.io/cluster-name: <name> is set by NKP on every KubeFedCluster object.

When a project is created, NKP:

Creates the project namespace on each targeted workload cluster
Deploys any project-scoped apps (logging, etc.)
Federates ConfigMaps, Secrets, Roles, RoleBindings, NetworkPolicies, and ResourceQuotas to the workload clusters

Push Applications to Workload Clusters

Platform apps can be deployed at the workspace level, which pushes them to all clusters in that workspace.

Foundational workspace apps (deploy these first):

yaml

# Reloader: watches ConfigMap/Secret changes, restarts dependent workloads
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: reloader
  namespace: demo              # workspace namespace
spec:
  appRef:
    kind: ClusterApp
    name: reloader-2.2.5
---
# Traefik: ingress controller for workload cluster services
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: traefik
  namespace: demo
spec:
  appRef:
    kind: ClusterApp
    name: traefik-37.1.2

Additional workspace apps for monitoring and logging:

yaml

# kube-prometheus-stack: Prometheus + Grafana + AlertManager
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: kube-prometheus-stack
  namespace: demo
spec:
  appRef:
    kind: ClusterApp
    name: kube-prometheus-stack-78.4.0
---
# cert-manager: TLS certificate management
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: cert-manager
  namespace: demo
spec:
  appRef:
    kind: ClusterApp
    name: cert-manager-1.18.2

The ClusterApp names include the version. Discover available apps:

bash

kubectl get clusterapps -n kommander

Attach Existing Clusters (Ultimate Only)

NKP can monitor clusters it didn't create: EKS, AKS, GKE, or any conformant Kubernetes cluster. Clusters that NKP creates via the CLI attach automatically. External clusters are attached from the dashboard under Clusters, Add Cluster, Attach Cluster, where you provide the target cluster's kubeconfig.

The declarative equivalent is a KommanderCluster resource in the workspace namespace that references a Secret holding the external kubeconfig. There is no nkp attach cluster CLI subcommand.

What "attached" means:

Kommander deploys its agent (kubefed + monitoring) to the cluster
The cluster appears in the dashboard
You can push apps and policies to it
CAPI does not manage its lifecycle: you're still responsible for scaling, upgrades, and infrastructure

Federated Resources

NKP uses KubeFed to distribute resources from the management cluster to workload clusters. At the project level, you can federate:

yaml

# Federated ConfigMap: distributed to all clusters in the project
apiVersion: types.kubefed.io/v1beta1
kind: FederatedConfigMap
metadata:
  name: app-config
  namespace: demo-apps-h6n8q     # project namespace on mgmt cluster
spec:
  template:
    data:
      DATABASE_URL: "postgres://db.internal:5432/myapp"
      LOG_LEVEL: "info"
  placement:
    clusterSelector:
      matchLabels: {}             # all clusters in the project
---
# Federated Secret
apiVersion: types.kubefed.io/v1beta1
kind: FederatedSecret
metadata:
  name: app-secrets
  namespace: demo-apps-h6n8q
spec:
  template:
    type: Opaque
    stringData:
      API_KEY: "your-api-key"
  placement:
    clusterSelector:
      matchLabels: {}
---
# Federated NetworkPolicy: default deny ingress on all clusters
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: demo-apps-h6n8q
spec:
  template:
    spec:
      podSelector: {}
      policyTypes:
        - Ingress
  placement:
    clusterSelector:
      matchLabels: {}

Application Catalog

Built-in Applications

NKP ships a catalog of platform applications that can be enabled per-cluster or per-workspace:

App	License	Category
Harbor	Pro/Ultimate	Container registry
Istio	Pro/Ultimate	Service mesh
Knative	Pro/Ultimate	Serverless
NKP Insights	Ultimate	Security scanning, CIS benchmarks
OpenCost	Ultimate	Cost tracking and allocation
NVIDIA GPU Operator	Pro/Ultimate	GPU workload support

Enable catalog apps in kommander.yaml:

yaml

apps:
  harbor:
    enabled: true
  nkp-insights-management:
    enabled: true

Or deploy them as AppDeployments at the workspace level for targeted deployment.

Custom Application Catalog

Add your own Helm charts to the NKP catalog. There are two approaches:

Git-Based Catalog

Create a Git repository with your Helm charts, then register it as a catalog:

yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-company-catalog
  namespace: kommander-flux
spec:
  url: https://gitlab.mycompany.com/platform/helm-charts.git
  ref:
    branch: main
  interval: 10m
  secretRef:
    name: git-credentials

OCI Registry Catalog (Air-Gapped)

For air-gapped environments, push Helm charts to Harbor as OCI artifacts:

bash

# Push a Helm chart to Harbor OCI
helm push my-app-1.0.0.tgz oci://harbor.itcs.local/my-project

Then create an OCIRepository source:

yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: my-app
  namespace: kommander-flux
spec:
  url: oci://harbor.itcs.local/my-project/my-app
  interval: 10m
  provider: generic

App CR for Direct Deployment

For apps that don't need catalog UI integration, create an App CR directly:

yaml

apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: App
metadata:
  name: podinfo
  namespace: demo              # workspace namespace
spec:
  gitOps:
    repository:
      kind: HelmRepository
      name: podinfo-repo
      namespace: kommander-flux
    chart: podinfo
    version: "6.5.0"
  configOverrides:
    name: podinfo-overrides    # ConfigMap with custom Helm values

GitOps with Flux CD

How NKP Uses Flux

NKP's entire application deployment is built on Flux CD. Every platform service is a HelmRelease reconciled by Flux controllers. This means GitOps is built into the platform from day one.

Flux resources in NKP:

Resource	Purpose
HelmRelease	Deploys a Helm chart with version pinning and automatic reconciliation
Kustomization	Applies a set of Kubernetes manifests from a Git source
GitRepository	Connects to a Git repo (SSH or HTTPS)
HelmRepository	Connects to a Helm chart repository
OCIRepository	Connects to an OCI-compatible registry (Harbor, ECR, etc.)

Set Up a GitOps Workflow

Management Cluster (Federated)

For workspace-level GitOps, create Flux resources on the management cluster. They get federated to workload clusters via KubeFed:

yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: team-platform-apps
  namespace: demo              # workspace namespace
spec:
  url: https://gitlab.mycompany.com/team-platform/k8s-apps.git
  ref:
    branch: main
  interval: 5m
  secretRef:
    name: git-credentials
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: team-platform-apps
  namespace: demo
spec:
  sourceRef:
    kind: GitRepository
    name: team-platform-apps
  path: ./k8s/overlays/production
  interval: 10m
  prune: true

Workload Cluster (Direct)

For workload cluster-level GitOps, create Flux resources directly on the workload cluster. This is useful for application-specific deployments that don't need management cluster involvement:

bash

# Get workload cluster kubeconfig
nkp get kubeconfig --cluster-name team-platform > team-platform.conf
export KUBECONFIG=team-platform.conf

# Create Flux source
kubectl apply -f - <<EOF
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-app
  namespace: flux-system
spec:
  url: https://gitlab.mycompany.com/team/my-app.git
  ref:
    branch: main
  interval: 1m
  secretRef:
    name: git-credentials
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: my-app
  namespace: default
spec:
  interval: 5m
  chart:
    spec:
      chart: ./charts/my-app
      sourceRef:
        kind: GitRepository
        name: my-app
        namespace: flux-system
  values:
    replicaCount: 3
    image:
      repository: harbor.itcs.local/my-project/my-app
      tag: "1.0.0"
EOF

Project-Level Helm App

Deploy a Helm chart at the project level using NKP's App CR with a ConfigMap for value overrides:

yaml

# ConfigMap with Helm value overrides
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-overrides
  namespace: demo-apps-h6n8q     # project namespace
data:
  values.yaml: |
    replicaCount: 3
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi
---
# App deployment
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: my-app
  namespace: demo-apps-h6n8q
spec:
  appRef:
    kind: App
    name: my-app
  configOverrides:
    name: my-app-overrides

Scaling and Lifecycle

Scale Worker Nodes

bash

# Scale up via CLI
nkp scale nodepools md-0 \
  --cluster-name team-platform \
  --replicas 6

Or set up autoscaling with MachineDeployment annotations:

yaml

metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "3"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "10"

Note

Autoscaling annotations only work if the Cluster Autoscaler platform service is enabled via Kommander.

Cluster Upgrades

Workload cluster upgrades are driven from the management cluster:

bash

nkp upgrade cluster nutanix \
  --cluster-name team-platform \
  --vm-image nkp-rocky-9.7-release-cis-1.35.0 \
  --kubeconfig mgmt-cluster.conf

--vm-image applies the same image to every node. If the control plane and worker pools use different images, use the per-role flags instead:

bash

nkp upgrade cluster nutanix \
  --cluster-name team-platform \
  --control-plane-vm-image nkp-rocky-9.7-release-cis-1.35.0 \
  --worker-vm-images md-0=nkp-rocky-9.7-release-cis-1.35.0 \
  --kubeconfig mgmt-cluster.conf

--worker-vm-images takes pool=image pairs; when you use it, every worker pool must be listed. Use the full image name as it appears in Prism Central (e.g. nkp-rocky-9.7-release-cis-1.35.0-<build-timestamp>.qcow2).

Upgrades are rolling: new nodes are provisioned with the new OS image, workloads are drained from old nodes, and old nodes are deleted. Zero downtime if your application has multiple replicas.

Important

Always upgrade the management cluster first, then workload clusters. NKP enforces an N-1 version policy: the management cluster must be at least the same version as its workload clusters.

Delete a Workload Cluster

bash

nkp delete cluster --cluster-name team-platform

CAPI handles the teardown: drain nodes, delete VMs in Prism Central, clean up CAPI resources. PersistentVolumes with reclaimPolicy: Delete are cleaned up. PVs with Retain are preserved on the storage container.

Troubleshooting

Symptom	Cause	Fix
Workload cluster stuck Provisioning	Management cluster can't reach PC	Check CAPX logs: `kubectl -n capx-system logs -l control-plane=controller-manager`
Apps not deploying to workload cluster	Cluster not in the correct workspace	Verify cluster is added to the workspace and KubeFedCluster object exists
Project namespace not created on workload	Placement selector doesn't match	Check `clusterSelector` labels vs `kommander.d2iq.io/cluster-name` on KubeFedCluster
Federated resources not syncing	KubeFed controller issue	Check kubefed controller logs in the workspace namespace
Flux reconciliation failing	Git credentials expired or wrong	Update Secret in `flux-system` or `kommander-flux` namespace
HelmRelease stuck	Chart values error	`kubectl -n <ns> describe helmrelease <name>` for the error message
Attached cluster not showing apps	Agent not running	Check kommander-agent pods on the attached cluster

What's Next

Part 6: Operations covers the operational side: monitoring and alerting with Prometheus/Grafana, centralized logging, backup with Velero, cluster upgrades, and troubleshooting patterns.

For cluster setup, see Part 2 or Part 3. For authentication and RBAC, see Part 4.

Summary

Workload clusters separate your application workloads from the management plane. NKP's fleet management model, workspaces for teams and projects for applications, gives you multi-tenant isolation with federated configuration. The application catalog and Flux CD-based GitOps mean you can define your entire stack as code, whether that's platform apps pushed from the management cluster or custom applications deployed via Git. For air-gapped environments, OCI repositories in Harbor replace Git sources seamlessly.

nkp kubernetes nutanix gitops flux

Part 5 of 6

Introduction to NKP

View the full journey →

Nutanix Kubernetes Platform (NKP) | Part 5: Workload Clusters & Applications

Nutanix Kubernetes Platform (NKP) | Part 5: Workload Clusters & Applications

Management vs Workload Clusters#

Creating a Workload Cluster#

Via the CLI#

Via YAML (ClusterClass Topology)#

Workload Cluster Sizing#

Fleet Management#

Workspaces#

Projects#

Push Applications to Workload Clusters#

Attach Existing Clusters (Ultimate Only)#

Federated Resources#

Application Catalog#

Built-in Applications#

Custom Application Catalog#

Git-Based Catalog#

OCI Registry Catalog (Air-Gapped)#

App CR for Direct Deployment#

GitOps with Flux CD#

How NKP Uses Flux#

Set Up a GitOps Workflow#

Management Cluster (Federated)#

Workload Cluster (Direct)#

Project-Level Helm App#

Scaling and Lifecycle#

Scale Worker Nodes#

Cluster Upgrades#

Delete a Workload Cluster#

Troubleshooting#

What's Next#

Summary#

Anthony Pradal

Related reading

Upgrade to NKP 2.18, Part 5: GitOps Migration to the NKPCluster API

Debugging x509 and Missing-Content Errors on NKP OCIRepositories

Debugging NKP: A Layered Approach

Management vs Workload Clusters

Creating a Workload Cluster

Via the CLI

Via YAML (ClusterClass Topology)

Workload Cluster Sizing

Fleet Management

Workspaces

Projects

Push Applications to Workload Clusters

Attach Existing Clusters (Ultimate Only)

Federated Resources

Application Catalog

Built-in Applications

Custom Application Catalog

Git-Based Catalog

OCI Registry Catalog (Air-Gapped)

App CR for Direct Deployment

GitOps with Flux CD

How NKP Uses Flux

Set Up a GitOps Workflow

Management Cluster (Federated)

Workload Cluster (Direct)

Project-Level Helm App

Scaling and Lifecycle

Scale Worker Nodes

Cluster Upgrades

Delete a Workload Cluster

Troubleshooting

What's Next

Summary