Istio Ambient Mode on NKP: Service Mesh Without Sidecars

Traditional Istio injects an Envoy sidecar into every pod. That doubles your pod count, eats memory across the cluster, and makes upgrades a chore. Ambient mode removes the sidecar entirely. mTLS, identity, traffic management, and observability move into shared infrastructure that runs once per node, not once per pod.

This article walks through a complete ambient-mode deployment on NKP 2.17 with Istio 1.23.6. We expose a multi-version application through HTTPS, enforce strict mTLS, write identity-based access policies, validate JWTs, inject faults, mirror traffic, and wire up Kiali and Jaeger for observability. Everything is declarative YAML. No sidecars are involved.

The Two-Layer Architecture#

Ambient mode splits the data plane in two:

Ztunnel handles every connection. The waypoint is optional and only deployed where you need L7 features. Traffic between two ambient pods that don't need L7 features stays on the lightweight L4 path.

The transport between ztunnels and the waypoint is HBONE: HTTP/2 CONNECT tunnels wrapped in mTLS on port 15008. HBONE preserves the original SPIFFE identity end-to-end, which is what makes identity-based AuthorizationPolicy work even with intermediate hops.

Why NKP Ships Helm-Based Istio (Not the Operator)#

NKP 2.17 deprecated the in-cluster Istio Operator and ships istio-helm as the AppDeployment. This mirrors upstream Istio's own deprecation of the operator in favor of Helm and revisioned installs.

Practically, this means:

• istiod runs in the istio-helm-system namespace as istiod-istio-helm.
• The control plane revision label is istio.io/rev: istio-helm.
• The ingress gateway lives in istio-helm-gateway-ns.
• Customizations go into the istio-helm-config-overrides ConfigMap in kommander, which is mounted as Helm values by the AppDeployment.

The revision name (istio-helm) becomes important later for the waypoint proxy: the GatewayClass controller only auto-provisions waypoint pods when the revision label matches.

Prerequisites#

• NKP 2.17 with the istio-helm AppDeployment enabled (istiod + ztunnel DaemonSet + ingress gateway).
• MetalLB providing LoadBalancer IPs.
• kube-prometheus-stack (bundled with NKP).
• Kiali and Jaeger operators (bundled in istio-system).

Verify the Istio control plane and data plane:

kubectl get pods -A | grep -E 'istiod|ztunnel|istio-cni|kiali|jaeger'

You should see one istiod (revision istio-helm), one ztunnel pod per node, one istio-cni-node per node, plus Kiali and Jaeger in istio-system.

Step 1: Enroll a Namespace in the Mesh#

Enrolling a namespace is a single label. No restart, no init container, no Deployment changes:

apiVersion: v1
kind: Namespace
metadata:
  name: istio-demo
  labels:
    istio.io/dataplane-mode: ambient

Ztunnel starts intercepting traffic for every pod in the namespace within seconds. Verify enrollment:

kubectl get pods -n istio-demo \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.ambient\.istio\.io/redirection}{"\n"}{end}'

helloworld-v1-7fff6b556d-sgdzn   enabled
helloworld-v2-69b6f588f6-2mcmj   enabled
sleep-6fb4bdc59d-22mkr           enabled

Step 2: Deploy the Application#

Two versions of Istio's helloworld sample with a Service that fronts both. The Service carries istio.io/use-waypoint so traffic to the service routes through a waypoint (deployed in Step 7):

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  namespace: istio-demo
  labels:
    app: helloworld
    istio.io/use-waypoint: helloworld-waypoint
spec:
  ports:
    - port: 5000
      targetPort: 5000
      name: http
  selector:
    app: helloworld

The Deployments for helloworld-v1 and helloworld-v2 are standard, each labelled app: helloworld plus version: v1 / version: v2. We also deploy two curl clients in the namespace:

• sleep: uses the namespace's default service account.
• notsleep: uses its own notsleep service account.

These two clients are otherwise identical. Different SAs give them different SPIFFE identities, which is what we'll use to demonstrate access control.

Verify in-mesh connectivity:

kubectl exec -n istio-demo deploy/sleep -- \
  curl -s helloworld.istio-demo:5000/hello

Hello version: v1, instance: helloworld-v1-7fff6b556d-sgdzn

This request was already encrypted with mTLS through ztunnel. The application code is unchanged.

Step 3: Expose via the Istio Ingress Gateway#

NKP exposes the Istio ingress gateway as a LoadBalancer. We received 10.12.52.222 from MetalLB. The Gateway resource binds to it via the istio: istio-helm-ingressgateway selector:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: helloworld-gateway
  namespace: istio-demo
spec:
  selector:
    istio: istio-helm-ingressgateway
  servers:
    - port: { number: 443, name: https, protocol: HTTPS }
      tls:
        mode: SIMPLE
        credentialName: helloworld-tls
      hosts: ["helloworld.demo.local"]
    - port: { number: 80, name: http, protocol: HTTP }
      hosts: ["helloworld.demo.local"]

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /tmp/hw.key -out /tmp/hw.crt \
  -subj "/CN=helloworld.demo.local"
kubectl create secret tls helloworld-tls -n istio-helm-gateway-ns \
  --cert=/tmp/hw.crt --key=/tmp/hw.key

Why Not the Kubernetes Gateway API?#

Istio 1.23 ships an istio GatewayClass for the upstream Gateway API. With NKP's revisioned istiod, the controller didn't auto-provision the gateway deployment (the Gateway stayed in AddressNotUsable). The classic networking.istio.io/v1 Gateway + VirtualService model attaches to the existing ingress gateway by label selector and works without per-Gateway pod provisioning. We use the classic API throughout.

Step 4: Enforce Strict mTLS#

Istio defaults to PERMISSIVE mode (accepts both plaintext and mTLS). Switch the namespace to STRICT:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-demo
spec:
  mtls:
    mode: STRICT

Now any non-mesh client that tries to reach helloworld pods directly (bypassing the gateway and ztunnel) is rejected. Verify mTLS is in use by checking ztunnel metrics:

ZT=$(kubectl get pod -n istio-helm-gateway-ns -l app=ztunnel \
       -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n istio-helm-gateway-ns $ZT -- \
  curl -s localhost:15020/metrics | grep connection_security_policy

istio_tcp_connections_opened_total{...,connection_security_policy="mutual_tls"} 36

Every connection carries connection_security_policy="mutual_tls". Kiali shows this as a lock badge on the traffic edges.

Step 5: Traffic Management#

This is where the mesh earns its keep. We configure subsets, circuit breaking, header-based routing, fault injection, traffic mirroring, retries, and timeouts. None of it touches application code.

Subsets and Circuit Breaking#

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: helloworld
  namespace: istio-demo
spec:
  host: helloworld
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
    connectionPool:
      tcp: { maxConnections: 100 }
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
  subsets:
    - name: v1
      labels: { version: v1 }
    - name: v2
      labels: { version: v2 }

Outlier detection ejects pods returning 5 consecutive 5xx errors for 30 seconds. With ambient mode, this enforcement happens in the waypoint proxy (the L7 hop) since ztunnel is L4-only and doesn't track per-pod HTTP error rates.

Routing Rules#

The VirtualService uses header-based routing to expose four behaviours through one URL. Each rule is independent so you can demonstrate them without interference:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: helloworld
  namespace: istio-demo
spec:
  hosts:
    - "helloworld.demo.local"
    - "helloworld.istio-demo.svc.cluster.local"
  gateways:
    - helloworld-gateway
    - mesh
  http:
    - match: [{ headers: { x-bug: { exact: "true" } } }]
      fault: { abort: { httpStatus: 500, percentage: { value: 100 } } }
      route: [{ destination: { host: helloworld, subset: v2, port: { number: 5000 } } }]

    - match: [{ headers: { x-mirror: { exact: "true" } } }]
      route: [{ destination: { host: helloworld, subset: v1, port: { number: 5000 } } }]
      mirror: { host: helloworld, subset: v2, port: { number: 5000 } }
      mirrorPercentage: { value: 100 }

    - match: [{ headers: { x-fault: { exact: "true" } } }]
      route:
        - { destination: { host: helloworld, subset: v1, port: { number: 5000 } }, weight: 50 }
        - { destination: { host: helloworld, subset: v2, port: { number: 5000 } }, weight: 50 }
      fault:
        delay: { fixedDelay: 3s, percentage: { value: 30 } }
        abort: { httpStatus: 503, percentage: { value: 20 } }

    - route:
        - { destination: { host: helloworld, subset: v1, port: { number: 5000 } }, weight: 50 }
        - { destination: { host: helloworld, subset: v2, port: { number: 5000 } }, weight: 50 }
      retries:
        attempts: 3
        perTryTimeout: 5s
        retryOn: 5xx,reset,connect-failure
      timeout: 20s

A few notes worth calling out:

• gateways: [helloworld-gateway, mesh] makes the rules apply to both external traffic (via the ingress gateway) and internal traffic (the special mesh keyword). Without mesh, the rules wouldn't apply when sleep calls the service directly.
• perTryTimeout: 5s is generous because the waypoint proxy adds a hop. With a tighter 2 s timeout, normal traffic would time out before the waypoint finishes its work.
• The mirror rule uses mirrorPercentage: { value: 100 } so every request is shadowed. In production you'd typically use 1-10%.

# Default route (50/50, no faults)
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s helloworld.istio-demo:5000/hello
# Hello version: v1, instance: helloworld-v1-...

# Force a 500 from v2
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s -H "x-bug: true" helloworld.istio-demo:5000/hello
# fault filter abort

# Mirror (v1 responds, v2 also processes the request silently)
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s -H "x-mirror: true" helloworld.istio-demo:5000/hello
# Hello version: v1, instance: helloworld-v1-...

Step 6: Identity-Based Access Control#

Strict mTLS gives every workload a SPIFFE identity of the form spiffe://cluster.local/ns/<namespace>/sa/<service-account>. AuthorizationPolicy lets you write rules against those identities. We deny notsleep and let everything else through:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: helloworld-deny-notsleep
  namespace: istio-demo
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: helloworld
  action: DENY
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-demo/sa/notsleep

# sleep (sa/default): allowed
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s helloworld.istio-demo:5000/hello
# Hello version: v2, instance: helloworld-v2-...

# notsleep (sa/notsleep): denied
kubectl exec -n istio-demo deploy/notsleep -- \
  curl -s helloworld.istio-demo:5000/hello
# RBAC: access denied (HTTP 403)

Same namespace, same network, same image, same Pod Security Standards. Only the service account differs. The mesh enforces identity at the cryptographic layer.

JWT Authentication#

For external requests, RequestAuthentication validates JWT tokens and AuthorizationPolicy decides when one is required:

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: helloworld-jwt
  namespace: istio-demo
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: helloworld
  jwtRules:
    - issuer: "testing@secure.istio.io"
      jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.23/security/tools/jwt/samples/jwks.json"
      forwardOriginalToken: true
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: helloworld-require-jwt
  namespace: istio-demo
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: helloworld
  action: DENY
  rules:
    - from:
        - source:
            notRequestPrincipals: ["*"]
      when:
        - key: request.headers[x-secure]
          values: ["true"]

RequestAuthentication only validates tokens that are present; it doesn't reject requests without one. The AuthorizationPolicy enforces the requirement: requests with x-secure: true must carry a valid JWT.

# No token, no x-secure header: passes
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s helloworld.istio-demo:5000/hello
# Hello version: v1

# Invalid token: rejected by RequestAuthentication
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s -H "Authorization: Bearer bad-token" helloworld.istio-demo:5000/hello
# Jwt is not in the form of Header.Payload.Signature (HTTP 401)

# x-secure: true without JWT, denied by AuthorizationPolicy
kubectl exec -n istio-demo deploy/sleep -- \
  curl -s -H "x-secure: true" helloworld.istio-demo:5000/hello
# RBAC: access denied (HTTP 403)

Step 7: Observability with Kiali#

A fresh ambient deployment shows "Empty Graph" in Kiali. Two reasons.

Scrape ztunnel metrics#

NKP's bundled envoy-stats-monitor ServiceMonitor scrapes the sidecar Envoy port (15090). Ztunnel exposes metrics on port 15020 with port name ztunnel-stats. Add a dedicated PodMonitor:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ztunnel
  namespace: istio-helm-gateway-ns
  labels:
    prometheus.kommander.d2iq.io/select: "true"
spec:
  selector:
    matchLabels:
      app: ztunnel
  podMetricsEndpoints:
    - port: ztunnel-stats
      path: /metrics
      interval: 15s

Deploy a waypoint for L7 metrics#

Ztunnel only emits istio_tcp_* metrics. Kiali's HTTP graph and most error/latency dashboards rely on istio_requests_total, which is L7. The waypoint proxy is what produces those:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: helloworld-waypoint
  namespace: istio-demo
  labels:
    istio.io/waypoint-for: service
    istio.io/rev: istio-helm
spec:
  gatewayClassName: istio-waypoint
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE

The Service already carries istio.io/use-waypoint: helloworld-waypoint (Step 2), so traffic to the service routes through the waypoint automatically. After 1-2 minutes of traffic Kiali shows the full picture: ingress gateway → waypoint → helloworld v1/v2, with error rates, latency, and mTLS lock badges.

Accessing Kiali on NKP#

NKP publishes Kiali behind Traefik Forward Auth at:

https://<ingress-lb>/dkp/kiali/

Authentication uses your Dex SSO session. Local fallback:

kubectl port-forward -n istio-system svc/kiali 20001:20001
# Open http://localhost:20001

Step 8: Distributed Tracing with Jaeger#

Kiali shows traffic flow. Jaeger shows what happens inside a request: which services were called, in what order, and how long each hop took.

NKP installs Jaeger in istio-system, but istiod's mesh config has no tracing provider configured by default. The zipkin service stub in istio-helm-system has zero endpoints because the Jaeger pods live in istio-system. Two things to fix that.

Add the tracing provider to meshConfig#

Patch the istio-helm AppDeployment values to add a Zipkin-format extensionProvider pointing at the Jaeger collector's FQDN:

kubectl patch cm istio-helm-config-overrides -n kommander \
  --type merge --patch-file day2/81-istio-helm-jaeger-patch.json

kubectl rollout restart deploy/istiod-istio-helm -n istio-helm-system

The patch file (81-istio-helm-jaeger-patch.json) merges this snippet into the existing istio-helm Helm values:

istiod:
  meshConfig:
    extensionProviders:
      - name: jaeger
        zipkin:
          service: jaeger-jaeger-operator-jaeger-collector.istio-system.svc.cluster.local
          port: 9411

Enable sampling for the demo namespace#

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: tracing
  namespace: istio-demo
spec:
  tracing:
    - providers:
        - name: jaeger
      randomSamplingPercentage: 100

Accessing Jaeger on NKP#

https://<ingress-lb>/dkp/jaeger/

Or via port-forward:

kubectl port-forward -n istio-system svc/jaeger-jaeger-operator-jaeger-query 16686:16686

Pick istio-helm-ingressgateway or helloworld-waypoint as the service to see end-to-end traces.

Validation Matrix#

After applying everything (and labelling the service for the waypoint), every behaviour can be reproduced in one line:

All eight should pass before you call the deployment good.

Operational Gotchas#

These tripped us up; you'll likely hit at least one.

MetalLB + Cilium ARP conflict#

NKP 2.17 uses Cilium as the CNI. Without explicit interfaces, MetalLB speakers may bind their gratuitous ARP responder to a Cilium lxc* veth instead of the physical NIC. When Cilium recycles that veth, ARP for new LB IPs starts failing:

"error":"writing gratuitous packet for 10.12.52.222:
  write packet: sendto: no such device or address"

Pings work (a bit confusingly), but TCP times out. Fix by restricting the L2Advertisement to the physical interface:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: metallb
  namespace: metallb-system
spec:
  ipAddressPools: [metallb]
  interfaces: [eth0]

Restart the speakers (kubectl rollout restart ds/metallb-speaker -n metallb-system) after applying. The issue typically only manifests for new LoadBalancer services after the cluster has been running for a while; existing services often happen to have bound to eth0 and keep working. Always set interfaces explicitly on Cilium-based clusters.

istio.io/rev placement#

The revision label belongs on the waypoint Gateway, not on the namespace. On the namespace it conflicts with dataplane-mode: ambient. On the Gateway it's the only thing that gets the waypoint deployed under a revisioned istiod.

targetRefs vs selector#

For AuthorizationPolicy, RequestAuthentication, and Telemetry resources that target a service behind a waypoint, prefer targetRefs to a Service over selector to pod labels. With a waypoint in the path, the destination ztunnel sees the waypoint's identity, not the original caller's. targetRefs runs the policy at the waypoint where the original SPIFFE identity is preserved.

Air-gapped JWT validation#

RequestAuthentication.jwtRules.jwksUri is fetched by the waypoint at runtime. On air-gapped clusters that fetch fails. Either mirror the JWKS into the cluster and point at an internal URL, or use the inline jwks field with the keys pasted in directly.

Jaeger requires a meshConfig provider#

The Jaeger operator that NKP ships in istio-system is not automatically wired to istiod. The zipkin service stub in istio-helm-system has no endpoints because its selector matches app: jaeger in the wrong namespace. You must add an extensionProviders entry to meshConfig pointing at the Jaeger collector's full DNS name.

Cleanup#

kubectl delete -f day2/80-istio-helloworld-demo.yaml
kubectl delete secret helloworld-tls -n istio-helm-gateway-ns

# Optional: revert the istiod tracing override
kubectl patch cm istio-helm-config-overrides -n kommander \
  --type=json -p='[{"op":"remove","path":"/data/values.yaml"}]'
kubectl rollout restart deploy/istiod-istio-helm -n istio-helm-system

Summary#

Ambient mode is what Istio always wanted to be: a service mesh that doesn't tax every workload. On NKP 2.17 it's a first-class deployment option with a few NKP-specific patterns to learn:

• The data plane is two layers. Ztunnel handles L4 mTLS and identity. The waypoint adds L7 (HTTP routing, retries, AuthZ, JWT, request metrics). Deploy waypoints only where you need them.
• targetRefs, not selector, for service-scoped policies. The waypoint preserves the original caller's SPIFFE identity through HBONE, but only when the policy is bound to the waypoint via targetRefs. Selector-based policies see the waypoint's identity instead.
• Revisioned istiod needs revision labels in the right places. The namespace gets dataplane-mode: ambient only. The waypoint Gateway gets istio.io/rev: istio-helm. Mixing them up costs hours.
• NKP wires very little of the observability stack by default. The Prometheus PodMonitor needs the Kommander selector label; the Jaeger collector needs a meshConfig provider; the MetalLB L2Advertisement needs explicit interfaces. None of this is hard once you know where to look.

The next step is multi-cluster ambient: shared root CA, cross-cluster service discovery, and HBONE tunnels between workload clusters. We'll cover that in a follow-up article.

All the YAML from this article lives in our nkp-quickstart repository under mgmt-cluster/day2/80-istio-helloworld-demo.yaml.