After the node roll in Part 3, the NKP UI's LoadBalancer address stopped answering. The API VIP on the same subnet still worked. The cause is not an NKP 2.19 defect. It is Cilium's DSR mode on a subnet that checks source addresses. This part gives the diagnosis and a fix that survives upgrades.
Symptom
$ for i in 1 2 3; do curl -sk -o /dev/null -m 4 -w "%{http_code} " https://10.12.52.6/dkp/kommander/dashboard/; done
000 000 000
$ curl -sk -o /dev/null -m 4 -w "%{http_code}\n" https://10.12.52.5:6443/healthz # API VIP, same subnet
401
MetalLB is not the cause. The bastion's ARP cache holds the MAC of the announcing node. Restarting the MetalLB speakers moves the announcement and changes nothing. The problem is the node that answers.
Diagnosis
NKP configures Cilium with loadBalancer.mode: dsr and Geneve dispatch. In DSR mode the node that receives a packet for a LoadBalancer or NodePort does not translate it. It forwards it to the node that hosts the backend pod. That node replies directly to the client, with the service IP as source.
A curl from the bastion with cilium monitor on the nodes shows:
- Bastion.
curlto 10.12.52.6:443 times out. No SYN-ACK arrives. - Announcing node (no Traefik pod). The SYN arrives four times (client retransmits) and is forwarded over the overlay each time:
-> overlay ... 10.12.52.11:34350 -> 192.168.13.127:8443 tcp SYN. - Backend node. The pod answers. Cilium sets the service IP as source and sends the reply out of the physical interface:
-> network ... ifindex eth0 ... 10.12.52.6:443 -> 10.12.52.11:34350 tcp SYN, ACK, five times. This is the last BPF hook on the node. No firewall,rp_filteroff, no drop reported by Cilium.
The reply leaves the VM and never reaches the bastion on the same L2 segment. It is dropped between the virtual NIC and the wire.
Control tests
| Target | Result | Why |
|---|---|---|
| The LB address, announcer has no Traefik pod | 0 of 3 | reply always leaves another node with the LB IP as source |
| NodePort on a node that hosts a Traefik pod | works only when Cilium picks the local backend | a remote backend replies with the first node's IP from its own NIC |
| NodePort on a node without a pod | 0 of 3 | always a remote backend |
| The API VIP (kube-vip) | works | the VIP owner answers from the NIC that owns it |
| A workload cluster on the same VLAN, old VMs from August, same DSR config | LB works, 3 of 3 | its single Traefik pod happens to sit on the announcing node |
| NodePorts on that cluster's three other nodes | 0 of 3 each | same drop, on VMs that were not touched by any upgrade |
| A second workload cluster on another Network Controller VLAN, Cilium in SNAT mode | LB and all 13 NodePorts, 3 of 3 everywhere | replies go back through the node that received the request |
One rule explains every row: a frame whose source IP is not owned by the sending VM's NIC is dropped. kube-vip and MetalLB work because the owner of the address answers. DSR fails because another node answers.
Why it happens
Cilium did not change: chart 1.19.4 and bpf-lb-mode=dsr before and after. The subnet did. Our node VLAN was a VLAN Basic subnet and became a Network Controller managed VLAN subnet. On such a subnet Prism Central records the IP addresses learned on each VM NIC (learnedIpAddresses in the vmm v4 API). A DSR reply leaves a node with the LoadBalancer address as source, an address that node does not own. On this subnet that frame does not leave the host.
While a Traefik pod ran on the announcing node, some connections worked. After the roll the announcing node had no pod, and none worked.
Cilium's documentation states that DSR needs source/destination IP checking to be disabled where the environment implements it (AWS is its example). A Network Controller managed VLAN does such checking. The per-NIC learned addresses are read-only in the API.
This is not the upstream DSR bug cilium/cilium#41962. That bug affects a pod inside the cluster reaching a NodePort through another node's IP. Here the client is outside the cluster and the reply leaves the NIC correctly formed.
The fix: SNAT, declared in the cluster variable
In SNAT mode the receiving node translates the request and forwards it. The reply returns through the same node with that node's IP. No frame carries a foreign source address.
Cost: pods see the node IP instead of the client IP. Traefik's X-Forwarded-For loses the real client unless the service uses externalTrafficPolicy: Local.
NKP renders Cilium's values from the cluster variable addons.cni through a Cluster API runtime hook. The hook re-applies the add-ons at every upgrade. A change made only in the live cilium-config ConfigMap is overwritten by the next upgrade.
The guide documents custom Cilium values at cluster creation ("Creating a Nutanix Cluster With Custom Cilium Configuration"): a ConfigMap of Helm values referenced from the cluster variable. The same reference can be added to a running cluster through its NKPCluster.
1. The values ConfigMap, in the namespace of the CAPI Cluster (on a management cluster that is usually default, not kommander); <cluster> is the CAPI cluster name. Start from what the hook renders today (it is in the HelmChartProxy named cilium-<cluster uuid> in that namespace) and change only the load balancer mode:
kubectl get helmchartproxy -n <cluster-ns> | grep cilium # named cilium-<cluster uuid>
kubectl get helmchartproxy <cilium-hcp> -n <cluster-ns> -o jsonpath='{.spec.valuesTemplate}' > values.yaml
# excerpt; everything else is left as rendered
loadBalancer:
mode: snat # was: mode: dsr, with dsrDispatch: geneve
apiVersion: v1
kind: ConfigMap
metadata:
name: <cluster>-cilium-cni-helm-values-template
namespace: <cluster-ns>
data:
values.yaml: |
<the rendered values, with the change above>
preflight-values.yaml: |
<the rendered preflight values, unchanged>
2. The reference on the NKPCluster, under clusterConfig.addons.cni:
cni:
provider: Cilium
strategy: HelmAddon
values:
sourceRef:
kind: ConfigMap
name: <cluster>-cilium-cni-helm-values-template
On the management cluster the NKPCluster is host-cluster in kommander, while the CAPI Cluster and the ConfigMap are in default. Edit the NKPCluster, not the CAPI Cluster: since 2.18 the notes ask to update the NKPCluster directly. Check that the variable reached the CAPI Cluster:
kubectl get cluster <capi-cluster> -n <ns> \
-o jsonpath='{.spec.topology.variables[?(@.name=="clusterConfig")].value.addons.cni}{"\n"}'
3. The live part, for a cluster that is not about to be upgraded. The reference alone is read at the next upgrade. To apply it now, change the same value in the HelmChartProxy's valuesTemplate (the Helm addon provider upgrades the release; ours went to revision 3), then restart the agents:
kubectl -n kube-system get cm cilium-config -o jsonpath='bpf-lb-mode={.data.bpf-lb-mode} dsr-dispatch={.data.bpf-lb-dsr-dispatch}{"\n"}'
kubectl -n kube-system rollout restart ds/cilium deploy/cilium-operator
kubectl -n kube-system rollout status ds/cilium
Result after the restart: bpf-lb-mode=snat, no bpf-lb-dsr-dispatch key, nine agents Ready. From the bastion:
10.12.52.6:443 307 307 307 # the LB address
10.12.52.215:30230 307 307 307 # NodePort on the announcer, no pod there
10.12.52.90:30230 307 307 307 # NodePort on a node with no pod
10.12.52.97:30230 307 307 307
A 307 is a redirect from Traefik: the reply reaches the client.
Does the reference survive an upgrade?
Yes. Test on a workload cluster with the reference set and no live edit:
| Step | Result |
|---|---|
ConfigMap + sourceRef on the NKPCluster, cluster still in DSR |
no change on the cluster |
| UpgradePlan applied | the runtime hook rewrites the Cilium HelmChartProxy from the ConfigMap at the start of the Kubernetes step: mode: snat, Helm revision 3 |
| Before the first node is replaced | cilium-config has bpf-lb-mode=snat |
| After the roll | 4 new nodes in SNAT mode; LoadBalancer and NodePort answer 3 of 3 on every node |
| After the 2.19.1 Kommander upgrade | still snat on all three clusters |
Details in Part 5.
Summary
- Check whether your node subnets are Network Controller managed (per subnet in Prism Central,
learnedIpAddresseson the NICs). On those, DSR fails for any LoadBalancer or NodePort whose backend is not on the receiving node. - Test from outside the cluster: a NodePort on a node that does not host the pod. No answer means you are affected.
- Declare SNAT in the cluster variable so every upgrade renders it. The live HelmChartProxy edit only covers the time until the next upgrade.
- Pods lose the client source IP unless the service uses
externalTrafficPolicy: Local.
Next: Part 5, the workload clusters.


