SeriesPart 6 of 6 in Introduction to NKP
View the journey →

Version note

This article was written for NKP 2.17. The current release is 2.18. See the Upgrade to NKP 2.18 series for the latest guidance.

Nutanix Kubernetes Platform (NKP) | Part 6: Operations

Deploying a cluster is day one. Keeping it running, observable, backed up, and up to date is every day after.

This guide covers the operational aspects of running NKP in production: monitoring and alerting, centralized logging, backup and disaster recovery, cluster upgrades, and the troubleshooting patterns we use across client environments. This is the final part of the series, building on everything from Part 1 through Part 5.

Monitoring

The NKP Monitoring Stack

NKP ships a complete monitoring stack (Pro/Ultimate license):

Component What It Does
kube-prometheus-stack Prometheus (metrics collection) + Grafana (visualization) + AlertManager (alerting)
prometheus-adapter Exposes custom metrics for Horizontal Pod Autoscaler (HPA)
Thanos (Ultimate) Multi-cluster metrics aggregation with long-term S3 storage
centralized-grafana (Ultimate) Single Grafana instance with dashboards from all clusters
Karma (Ultimate) AlertManager dashboard for multi-cluster alert visualization

Access Grafana

Grafana is accessible through the Kommander dashboard or directly:

bash
# Via Kommander dashboard → Monitoring → Grafana
# Or check the service URL:
kubectl -n kommander get svc centralized-grafana

Default dashboards included:

  • Cluster overview (CPU, memory, disk, network)
  • Node metrics (per-node resource utilization)
  • Pod metrics (per-pod resource usage and limits)
  • Kubernetes API server (request latency, error rates)
  • etcd (leader elections, disk fsync latency, DB size)
  • Kommander platform services (HelmRelease health)

Custom Dashboards and Alerts

Add Custom Scrape Targets

Monitor your own applications by creating a ServiceMonitor:

yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  namespace: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Custom Alerting Rules

yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  namespace: my-app
spec:
  groups:
    - name: my-app
      rules:
        - alert: HighErrorRate
          expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on {{ $labels.instance }}"
        - alert: PodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total{namespace="my-app"}[15m]) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"

AlertManager Routing

Route alerts to Slack, email, PagerDuty, or any webhook:

yaml
# Via kube-prometheus-stack values in kommander.yaml
alertmanager:
  config:
    route:
      group_by: ['alertname', 'namespace']
      group_wait: 2m
      group_interval: 5m
      repeat_interval: 1h
      receiver: "null"
      routes:
        # Silence Watchdog (heartbeat alert)
        - match:
            alertname: Watchdog
          receiver: "null"
        # Critical alerts → Slack
        - match:
            severity: critical
          receiver: slack-critical
        # Warning alerts → email
        - match:
            severity: warning
          receiver: email-team
    receivers:
      - name: "null"
      - name: slack-critical
        slack_configs:
          - channel: '#nkp-alerts'
            send_resolved: true
            title: '{{ .GroupLabels.alertname }}'
            text: >-
              {{ range .Alerts }}*{{ .Labels.severity }}*:
              {{ .Annotations.summary }}{{ end }}
      - name: email-team
        email_configs:
          - to: 'platform-team@corp.local'
            from: 'nkp-alerts@corp.local'
            smarthost: 'smtp.corp.local:587'

Multi-Cluster Monitoring (Ultimate)

With an Ultimate license, Thanos aggregates metrics from all clusters into a single query layer:

  • Thanos Sidecar runs alongside each cluster's Prometheus, uploading TSDB blocks to S3
  • Thanos StoreGateway on the management cluster reads historical data from S3
  • Thanos Compactor downsamples old data (30d raw → 90d at 5m → 365d at 1h resolution)
  • Thanos Query federates queries across all Prometheus instances and the StoreGateway
  • Centralized Grafana connects to Thanos Query for a single pane of glass

Verify Thanos is working:

bash
# Check Thanos sidecar logs
kubectl logs prometheus-kube-prometheus-stack-prometheus-0 \
  -n kommander -c thanos-sidecar --tail=20

# Check StoreGateway
kubectl get pods -n kommander | grep thanos-storegateway

# Check Compactor
kubectl get pods -n kommander | grep thanos-compactor

Metrics Retention and Storage

Tier Local Retention Long-Term Storage
Prometheus (local PVC) 15-30 days (configurable) N/A
Thanos (S3) Unlimited Downsample: 30d raw, 90d@5m, 365d@1h

Tune retention via ConfigMap override:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-prometheus-stack-overrides
  namespace: kommander
data:
  values.yaml: |
    prometheus:
      prometheusSpec:
        retention: 30d
        storageSpec:
          volumeClaimTemplate:
            spec:
              storageClassName: nutanix-volume
              resources:
                requests:
                  storage: 100Gi

Logging

The NKP Logging Stack

Component Role
Fluent Bit DaemonSet on every node. Collects container logs, systemd logs, kernel logs
Fluentd Aggregator (managed by logging-operator). Buffers and forwards to Loki
Grafana Loki Log storage and query engine. S3-backed for durability
Grafana Log visualization via the Loki data source
output
Pods → stdout/stderr → Fluent Bit (DaemonSet) → Fluentd (aggregator) → Loki → S3

Query Logs in Grafana

Access logs via Grafana → Explore → Loki data source. LogQL examples:

logql
# All logs from a namespace
{namespace="my-app"}

# Dex authentication errors
{namespace="kommander", container="dex"} |= "error"

# Structured log filtering (JSON logs)
{namespace="my-app"} | json | level="error"

# Rate of error logs over time
rate({namespace="my-app"} |= "error" [5m])

# Top 10 pods by log volume
topk(10, sum by(pod) (rate({namespace="my-app"}[1h])))

Forward Logs to External SIEM

For compliance (NIS2, DORA), logs often need to go to a central SIEM alongside NKP's internal Loki.

Configure Fluent Bit outputs via kommander.yaml or ConfigMap override:

yaml
apps:
  fluent-bit:
    values: |
      config:
        outputs: |
          # Forward to Splunk HEC
          [OUTPUT]
              Name        splunk
              Match       *
              Host        splunk.corp.local
              Port        8088
              TLS         On
              Splunk_Token ${HEC_TOKEN}

          # Forward to Elasticsearch
          [OUTPUT]
              Name        es
              Match       *
              Host        es.corp.local
              Port        9200
              Index       nkp-logs
              TLS         On

          # Keep Loki as well (dual output)
          [OUTPUT]
              Name        loki
              Match       *
              Host        grafana-loki-loki-distributed-gateway.kommander.svc
              Port        80

Log Retention

Loki retention is configured in the grafana-loki values:

yaml
loki:
  structuredConfig:
    limits_config:
      retention_period: 360h    # 15 days

For compliance requiring longer retention, forward logs to your SIEM and retain them there. Loki is optimized for recent log queries, not long-term archival.

Backup and Disaster Recovery

Velero on NKP

Velero handles backup and restore for both Kubernetes resources and persistent volumes:

  • Namespace-level backup: Kubernetes manifests + PV snapshots
  • Scheduled backups: CronJob-style scheduling with configurable retention
  • Cross-cluster restore: Restore backups to a different cluster for DR

Configure a Backup Target

Velero uses S3-compatible storage as its backup target. With Nutanix Objects:

yaml
# Already configured in kommander.yaml (see Part 4)
velero:
  values: |
    configuration:
      features: EnableCSI
      backupStorageLocation:
        - name: nutanix-objects
          bucket: nkp-velero
          provider: "aws"
          default: true
          config:
            region: us-east-1
            s3ForcePathStyle: "false"
            insecureSkipTLSVerify: "true"
            s3Url: "https://objects.yourdomain.local"
            profile: ntnx-object-nkp
          credential:
            key: aws
            name: velero-nutanix-credentials

Verify the backup location is available:

bash
kubectl get backupstoragelocation -n kommander
# Should show "Available" phase

Create and Schedule Backups

bash
# One-time backup of the management cluster
velero backup create mgmt-backup-$(date +%Y%m%d) \
  --include-namespaces kommander,kommander-flux,capx-system,caren-system,cert-manager

# Scheduled backup (daily at 2 AM, 30-day retention)
velero schedule create daily-mgmt \
  --schedule="0 2 * * *" \
  --include-namespaces kommander,kommander-flux,capx-system,caren-system \
  --ttl 720h

What to back up:

Cluster Namespaces
Management kommander, kommander-flux, capx-system, caren-system, cert-manager, ntnx-system
Workload Application namespaces, PVCs with important data

Restore

bash
# List available backups
velero backup get

# Restore to same cluster
velero restore create --from-backup mgmt-backup-20260401

# Restore specific namespaces only
velero restore create --from-backup mgmt-backup-20260401 \
  --include-namespaces kommander,capx-system

Note

PV restore requires CSI snapshot support (enabled by default with features: EnableCSI). CRDs must exist on the target cluster before restoring resources that depend on them.

etcd Backup (Management Cluster)

For full disaster recovery of the management cluster, take etcd snapshots directly:

bash
# SSH to a control plane node
ssh ${SSH_USERNAME}@<cp-node-ip>

# Take snapshot
sudo etcdctl snapshot save /tmp/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify snapshot
sudo etcdctl snapshot status /tmp/etcd-snapshot.db --write-table

This is your last resort if Velero backups are unavailable. Automate it with a CronJob:

yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 */6 * * *"    # Every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
            - effect: NoSchedule
              key: node-role.kubernetes.io/control-plane
          containers:
            - name: etcd-backup
              # Match your cluster's etcd version (3.5.24-0 on NKP 2.17.1 / K8s 1.34.3):
              # kubectl -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'
              image: registry.k8s.io/etcd:3.5.24-0
              command: ["/bin/sh", "-c"]
              args:
                - |
                  etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M).db \
                    --endpoints=https://127.0.0.1:2379 \
                    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                    --cert=/etc/kubernetes/pki/etcd/server.crt \
                    --key=/etc/kubernetes/pki/etcd/server.key
              volumeMounts:
                - name: etcd-certs
                  mountPath: /etc/kubernetes/pki/etcd
                  readOnly: true
                - name: backup
                  mountPath: /backup
          volumes:
            - name: etcd-certs
              hostPath:
                path: /etc/kubernetes/pki/etcd
            - name: backup
              persistentVolumeClaim:
                claimName: etcd-backup-pvc
          restartPolicy: OnFailure

Cluster Upgrades

Upgrade Strategy

NKP upgrades follow a strict order:

output
1. NKP CLI (on jump host)
2. Management cluster
3. Workload clusters (one at a time)

NKP enforces an N-1 version policy: the management cluster must be at least the same version as any workload cluster it manages. You cannot upgrade a workload cluster past the management cluster version.

Upgrade Type Example What Changes
Patch 2.17.0 → 2.17.1 Bug fixes, security patches, OS image update
Minor 2.16 → 2.17 New features, Kubernetes version bump (1.33 → 1.34)

Pre-Upgrade Checks

Before upgrading, validate the cluster is healthy:

bash
# Check all nodes are Ready
kubectl get nodes

# Check all HelmReleases are healthy
kubectl -n kommander get helmreleases

# Check CAPI cluster status
kubectl get clusters.cluster.x-k8s.io -A

# Check for deprecated APIs (NKP Insights or Pluto)
kubectl get insights -n kommander  # if Insights is enabled

Upgrade the Management Cluster

bash
# 1. Install new CLI version
tar -xzvf nkp_v2.18.0_linux_amd64.tar.gz
sudo mv nkp /usr/local/bin/
nkp version

# 2. Upload new OS image to Prism Central
# Download from portal, upload via PC UI

# 3. Upgrade
nkp upgrade cluster nutanix \
  --cluster-name ${CLUSTER_NAME} \
  --vm-image nkp-rocky-9.7-release-cis-1.35.0 \
  --kubeconfig ${CLUSTER_NAME}.conf

Use the full image name as it appears in Prism Central (e.g. nkp-rocky-9.7-release-cis-1.35.0-<build-timestamp>.qcow2). If the control plane and worker pools run different images, replace --vm-image with --control-plane-vm-image <image> and --worker-vm-images <pool>=<image>[,<pool>=<image>] (all worker pools must be listed).

What happens during upgrade:

  1. Control plane nodes are replaced one at a time (rolling). New node joins, old node is drained and deleted
  2. Workers are replaced in batches (configurable). Workloads get rescheduled
  3. Kommander components are updated via Flux CD
  4. Zero downtime if applications have multiple replicas and proper PodDisruptionBudgets

Upgrade Workload Clusters

Same command, different cluster name and kubeconfig:

bash
nkp upgrade cluster nutanix \
  --cluster-name team-platform \
  --vm-image nkp-rocky-9.7-release-cis-1.35.0 \
  --kubeconfig mgmt-cluster.conf

Air-Gapped Upgrades

Same seed-transfer-push cycle as the initial deployment:

  1. Download new NKP bundle and OS image on an internet-connected machine
  2. Transfer to the air-gapped bastion
  3. Push new images to the local registry: nkp push bundle --bundle <new-bundle.tar> ...
  4. Upload new OS image to Prism Central
  5. Load new bootstrap image: docker load --input konvoy-bootstrap-image-v2.18.0.tar
  6. Run the upgrade command with the new image names

Support Bundles

When you need Nutanix support, collect a diagnostic bundle:

bash
nkp diagnose --kubeconfig ${CLUSTER_NAME}.conf --bundle-name-prefix "${CLUSTER_NAME}-"

What it collects:

  • Node logs (kubelet, containerd, kernel, systemd)
  • Pod logs from all namespaces
  • Kubernetes events
  • CAPI resource state
  • Helm release status
  • Node resource usage and disk space
  • etcd health

The output is a tarball you send to Nutanix support.

NKP Insights (AI-Powered Troubleshooting)

NKP Insights (Ultimate license) provides automated security scanning and troubleshooting:

Scanner What It Does Default Schedule
Polaris Configuration best practices (resource limits, security context, probes) Every 37 minutes
Pluto Deprecated Kubernetes API detection Every 41 minutes
Kube-bench CIS Kubernetes benchmark compliance Every 35 minutes
Nova Helm chart version checking Every 34 minutes (disabled by default)
Trivy CVE vulnerability scanning Every 2 hours (disabled by default)

Enable Trivy scanning via ConfigMap override:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nkp-insights-overrides
  namespace: kommander
data:
  values.yaml: |
    trivy:
      enabled: true
      schedule: "@every 2h"

Access Insights through the Kommander dashboard → Insights tab.

Troubleshooting Patterns

Node Issues

Symptom Debug Fix
Node NotReady kubectl describe node <name>, check kubelet logs via SSH Restart kubelet, check disk/memory pressure conditions
Disk pressure df -h on node, check /var/lib/containerd Clean unused images: crictl rmi --prune
OOM kills dmesg | grep -i oom, check pod resource limits Increase worker RAM or set proper resource limits
Node unreachable Check VM status in Prism Central, ping node IP Verify VM is running, check subnet/VLAN connectivity

Platform Service Issues

Symptom Debug Fix
Dashboard unreachable Check Traefik pod + MetalLB + DNS kubectl -n kommander get svc traefik, verify EXTERNAL-IP
Auth failures Check Dex logs kubectl -n kommander logs -l app.kubernetes.io/name=dex
Metrics missing Check Prometheus targets in Grafana Grafana → Status → Targets
HelmRelease degraded Check Flux status kubectl -n kommander describe helmrelease <name>
Logs not appearing Check Fluent Bit + Loki kubectl -n kommander logs -l app.kubernetes.io/name=fluent-bit

CAPI / Cluster Lifecycle Issues

Symptom Debug Fix
Machine stuck Provisioning CAPX controller logs kubectl -n capx-system logs -l control-plane=controller-manager
Cluster stuck Deleting Finalizers blocking deletion Check for stuck finalizers on Cluster/Machine objects
Pivot failure Bootstrap cluster logs docker logs $(docker ps -q --filter name=konvoy)
VM not created in PC PC credentials or permissions Verify global-nutanix-credentials secret, check PC user role

Recovering a Stuck HelmRelease

If a HelmRelease enters exhausted or rollback in progress state:

bash
# Suspend the HelmRelease
kubectl -n kommander patch helmrelease <NAME> \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/suspend", "value": true}]'

# Unsuspend to trigger fresh reconciliation
kubectl -n kommander patch helmrelease <NAME> \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/suspend", "value": false}]'

If the entire Kommander installation is broken, reinstall is safe to retry:

bash
nkp install kommander \
  --installer-config kommander.yaml \
  --kubeconfig ${CLUSTER_NAME}.conf \
  --wait-timeout 1h

Emergency: Management Cluster Recovery (2/3 CP Lost)

If two out of three control plane nodes are permanently lost:

bash
# SSH to the surviving CP node
ssh ${SSH_USERNAME}@<surviving-cp-ip>

# Stop etcd
sudo systemctl stop etcd

# Force new single-node cluster
sudo etcdctl snapshot restore /var/lib/etcd/member/snap/db \
  --data-dir=/var/lib/etcd-restored \
  --name=$(hostname) \
  --initial-cluster="$(hostname)=https://$(hostname):2380" \
  --initial-advertise-peer-urls="https://$(hostname):2380"

# Replace data directory
sudo mv /var/lib/etcd /var/lib/etcd-broken
sudo mv /var/lib/etcd-restored /var/lib/etcd

# Restart etcd and kubelet
sudo systemctl start etcd
sudo systemctl restart kubelet

Warning

This is a last-resort procedure. It results in a single-node etcd cluster. You must immediately scale up new control plane nodes and restore HA. Always maintain regular etcd snapshots to avoid this scenario.

Operational Checklist

Daily

  • Check Grafana dashboards for anomalies (CPU, memory, disk)
  • Review AlertManager / Karma for active alerts
  • Verify HelmRelease health: kubectl -n kommander get hr

Weekly

  • Check node resource utilization trends
  • Review Velero backup status: velero backup get
  • Check for pending NKP or Kubernetes updates
  • Review NKP Insights findings (if enabled)

Monthly

  • Test backup restore procedure (restore to a test cluster)
  • Review and rotate credentials (PC user, registry, S3 keys)
  • Review Gatekeeper audit logs for policy violations
  • Check certificate expiration dates: kubectl get certificates -A

Quarterly

  • Plan NKP version upgrades
  • Review capacity planning (node count, storage utilization)
  • Security audit: RBAC bindings, network policies, CIS compliance
  • Test disaster recovery procedure (etcd restore)

What's Next

This wraps up the NKP deployment and operations series. Here's the full reading order:

  1. Part 1: Architecture & Key Concepts: what NKP is and how it works
  2. Part 2: Prerequisites & Deployment: internet-connected deployment
  3. Part 3: Air-Gapped Deployment: registry mirror and bundle methods
  4. Part 4: Day 2 Configuration: auth, RBAC, storage, certificates
  5. Part 5: Workload Clusters & Applications: fleet management, GitOps
  6. Part 6: Operations: monitoring, backup, upgrades (you are here)

Every part includes real commands and configurations from our ITCS Lab running NKP 2.17.1 on Nutanix AHV.

Summary

Operations is where NKP earns its keep. The monitoring stack (Prometheus, Grafana, Thanos) gives you visibility from pod-level metrics to multi-cluster aggregation. The logging pipeline (Fluent Bit, Loki) captures everything and can forward to external SIEMs for compliance. Velero and etcd snapshots protect against data loss. Rolling upgrades keep you current without downtime. And NKP Insights automates the security scanning that would otherwise require manual CIS audits. Pair these built-in tools with the operational checklist above and you have a production-grade Kubernetes platform that satisfies both technical and regulatory requirements.

nkpkubernetesnutanixmonitoringbackupoperations
6
View the full journey →
Next part →