After the node roll in Part 3, the NKP UI's LoadBalancer address stopped answering. The API VIP on the same subnet still worked. The cause is not an NKP 2.19 defect. It is Cilium's DSR mode on a subnet that checks source addresses. This part gives the diagnosis and a fix that survives upgrades.

Symptom

console
$ for i in 1 2 3; do curl -sk -o /dev/null -m 4 -w "%{http_code} " https://10.12.52.6/dkp/kommander/dashboard/; done
000 000 000
$ curl -sk -o /dev/null -m 4 -w "%{http_code}\n" https://10.12.52.5:6443/healthz   # API VIP, same subnet
401

MetalLB is not the cause. The bastion's ARP cache holds the MAC of the announcing node. Restarting the MetalLB speakers moves the announcement and changes nothing. The problem is the node that answers.

Diagnosis

NKP configures Cilium with loadBalancer.mode: dsr and Geneve dispatch. In DSR mode the node that receives a packet for a LoadBalancer or NodePort does not translate it. It forwards it to the node that hosts the backend pod. That node replies directly to the client, with the service IP as source.

A curl from the bastion with cilium monitor on the nodes shows:

  1. Bastion. curl to 10.12.52.6:443 times out. No SYN-ACK arrives.
  2. Announcing node (no Traefik pod). The SYN arrives four times (client retransmits) and is forwarded over the overlay each time: -> overlay ... 10.12.52.11:34350 -> 192.168.13.127:8443 tcp SYN.
  3. Backend node. The pod answers. Cilium sets the service IP as source and sends the reply out of the physical interface: -> network ... ifindex eth0 ... 10.12.52.6:443 -> 10.12.52.11:34350 tcp SYN, ACK, five times. This is the last BPF hook on the node. No firewall, rp_filter off, no drop reported by Cilium.

The reply leaves the VM and never reaches the bastion on the same L2 segment. It is dropped between the virtual NIC and the wire.

Control tests

Target Result Why
The LB address, announcer has no Traefik pod 0 of 3 reply always leaves another node with the LB IP as source
NodePort on a node that hosts a Traefik pod works only when Cilium picks the local backend a remote backend replies with the first node's IP from its own NIC
NodePort on a node without a pod 0 of 3 always a remote backend
The API VIP (kube-vip) works the VIP owner answers from the NIC that owns it
A workload cluster on the same VLAN, old VMs from August, same DSR config LB works, 3 of 3 its single Traefik pod happens to sit on the announcing node
NodePorts on that cluster's three other nodes 0 of 3 each same drop, on VMs that were not touched by any upgrade
A second workload cluster on another Network Controller VLAN, Cilium in SNAT mode LB and all 13 NodePorts, 3 of 3 everywhere replies go back through the node that received the request

One rule explains every row: a frame whose source IP is not owned by the sending VM's NIC is dropped. kube-vip and MetalLB work because the owner of the address answers. DSR fails because another node answers.

Why it happens

Cilium did not change: chart 1.19.4 and bpf-lb-mode=dsr before and after. The subnet did. Our node VLAN was a VLAN Basic subnet and became a Network Controller managed VLAN subnet. On such a subnet Prism Central records the IP addresses learned on each VM NIC (learnedIpAddresses in the vmm v4 API). A DSR reply leaves a node with the LoadBalancer address as source, an address that node does not own. On this subnet that frame does not leave the host.

While a Traefik pod ran on the announcing node, some connections worked. After the roll the announcing node had no pod, and none worked.

Cilium's documentation states that DSR needs source/destination IP checking to be disabled where the environment implements it (AWS is its example). A Network Controller managed VLAN does such checking. The per-NIC learned addresses are read-only in the API.

This is not the upstream DSR bug cilium/cilium#41962. That bug affects a pod inside the cluster reaching a NodePort through another node's IP. Here the client is outside the cluster and the reply leaves the NIC correctly formed.

The fix: SNAT, declared in the cluster variable

In SNAT mode the receiving node translates the request and forwards it. The reply returns through the same node with that node's IP. No frame carries a foreign source address.

Cost: pods see the node IP instead of the client IP. Traefik's X-Forwarded-For loses the real client unless the service uses externalTrafficPolicy: Local.

NKP renders Cilium's values from the cluster variable addons.cni through a Cluster API runtime hook. The hook re-applies the add-ons at every upgrade. A change made only in the live cilium-config ConfigMap is overwritten by the next upgrade.

The guide documents custom Cilium values at cluster creation ("Creating a Nutanix Cluster With Custom Cilium Configuration"): a ConfigMap of Helm values referenced from the cluster variable. The same reference can be added to a running cluster through its NKPCluster.

1. The values ConfigMap, in the namespace of the CAPI Cluster (on a management cluster that is usually default, not kommander); <cluster> is the CAPI cluster name. Start from what the hook renders today (it is in the HelmChartProxy named cilium-<cluster uuid> in that namespace) and change only the load balancer mode:

bash
kubectl get helmchartproxy -n <cluster-ns> | grep cilium      # named cilium-<cluster uuid>
kubectl get helmchartproxy <cilium-hcp> -n <cluster-ns> -o jsonpath='{.spec.valuesTemplate}' > values.yaml
yaml
# excerpt; everything else is left as rendered
loadBalancer:
  mode: snat          # was: mode: dsr, with dsrDispatch: geneve
yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: <cluster>-cilium-cni-helm-values-template
  namespace: <cluster-ns>
data:
  values.yaml: |
    <the rendered values, with the change above>
  preflight-values.yaml: |
    <the rendered preflight values, unchanged>

2. The reference on the NKPCluster, under clusterConfig.addons.cni:

yaml
cni:
  provider: Cilium
  strategy: HelmAddon
  values:
    sourceRef:
      kind: ConfigMap
      name: <cluster>-cilium-cni-helm-values-template

On the management cluster the NKPCluster is host-cluster in kommander, while the CAPI Cluster and the ConfigMap are in default. Edit the NKPCluster, not the CAPI Cluster: since 2.18 the notes ask to update the NKPCluster directly. Check that the variable reached the CAPI Cluster:

bash
kubectl get cluster <capi-cluster> -n <ns> \
  -o jsonpath='{.spec.topology.variables[?(@.name=="clusterConfig")].value.addons.cni}{"\n"}'

3. The live part, for a cluster that is not about to be upgraded. The reference alone is read at the next upgrade. To apply it now, change the same value in the HelmChartProxy's valuesTemplate (the Helm addon provider upgrades the release; ours went to revision 3), then restart the agents:

bash
kubectl -n kube-system get cm cilium-config -o jsonpath='bpf-lb-mode={.data.bpf-lb-mode} dsr-dispatch={.data.bpf-lb-dsr-dispatch}{"\n"}'
kubectl -n kube-system rollout restart ds/cilium deploy/cilium-operator
kubectl -n kube-system rollout status ds/cilium

Result after the restart: bpf-lb-mode=snat, no bpf-lb-dsr-dispatch key, nine agents Ready. From the bastion:

console
10.12.52.6:443        307 307 307     # the LB address
10.12.52.215:30230    307 307 307     # NodePort on the announcer, no pod there
10.12.52.90:30230     307 307 307     # NodePort on a node with no pod
10.12.52.97:30230     307 307 307

A 307 is a redirect from Traefik: the reply reaches the client.

Does the reference survive an upgrade?

Yes. Test on a workload cluster with the reference set and no live edit:

Step Result
ConfigMap + sourceRef on the NKPCluster, cluster still in DSR no change on the cluster
UpgradePlan applied the runtime hook rewrites the Cilium HelmChartProxy from the ConfigMap at the start of the Kubernetes step: mode: snat, Helm revision 3
Before the first node is replaced cilium-config has bpf-lb-mode=snat
After the roll 4 new nodes in SNAT mode; LoadBalancer and NodePort answer 3 of 3 on every node
After the 2.19.1 Kommander upgrade still snat on all three clusters

Details in Part 5.

Summary

  • Check whether your node subnets are Network Controller managed (per subnet in Prism Central, learnedIpAddresses on the NICs). On those, DSR fails for any LoadBalancer or NodePort whose backend is not on the receiving node.
  • Test from outside the cluster: a NodePort on a node that does not host the pod. No answer means you are affected.
  • Declare SNAT in the cluster variable so every upgrade renders it. The live HelmChartProxy edit only covers the time until the next upgrade.
  • Pods lose the client source IP unless the service uses externalTrafficPolicy: Local.

Next: Part 5, the workload clusters.