Debugging x509 and Missing-Content Errors on NKP OCIRepositories

This one is a goblin: it appears on a cluster, you fix it in two commands, you don't see it for a month, then it reappears on a different cluster after a CA rotation, then again after a workspace re-join. It is hard to reproduce on demand. The exact root cause varies (timing, rotation, drift, manual edits), but the symptom is always the same: an OCIRepository reports it cannot pull because a secret it references is missing or its TLS bundle is wrong, and the workspace's HelmRelease chain stalls behind it. The diagnosis path is identical every time.

This post is the playbook. We walk through what the failure looks like, how to confirm what is actually missing, and the safe rename-and-copy fix we keep in our operations repo. The goal is not to explain every possible root cause, it is to give you a 5-minute resolution path the next time it shows up.

The Setup#

In air-gapped NKP, every workload cluster pulls Helm charts from the management cluster's flux-oci-mirror. Source-controller in kommander-flux opens an HTTPS connection to the proxy, the proxy presents its TLS cert, source-controller validates it against a CA bundle, then authenticates with a registry credential. All three pieces (proxy URL, CA bundle, registry credential) are stored as Kubernetes secrets, in the workspace namespace, on every workload cluster:

If any of those is missing or stale, the OCIRepository cannot fetch. Flux is doing the right thing by failing closed. The job is to find which one is missing or wrong, and put the right content where Flux expects to find it.

The Symptom#

Two failure messages cover almost every case we have seen.

Variant A: object referencing missing content. Flux says the secret named in the OCIRepository's certSecretRef (or secretRef) is not in this namespace. The URL field of the OCIRepository points at ghcr.io because flux-oci-mirror is configured via proxySecretRef, not by URL rewriting.

Reason:   AuthenticationFailed
Message:  failed to generate transport for 'oci://ghcr.io/mesosphere/charts/cert-manager-crds':
          secret 'demo-fdjc9-g6mdq/nkp-oci-proxy-ca' not found

Variant B: x509 trust failure. The secret is present, but its content does not match what the proxy is presenting.

Reason:   AuthenticationFailed
Message:  GET "https://flux-oci-mirror.kommander-flux:8443/v2/.../manifests/...":
          x509: certificate signed by unknown authority

Both are the same problem at different stages. Variant A is "object referencing missing content": the OCIRepository spec names a secret that does not exist in the workspace namespace on this cluster. Variant B is "object referencing wrong content": the named secret exists but the CA inside it does not validate the proxy's certificate. The diagnosis path is identical for both.

Whatever variant you see, the canary is usually the same. cert-manager-crds is the first AppDeployment most workspaces apply because everything else depends on cert-manager CRDs. When it sits at Ready=False for more than a few minutes, this failure mode is the first hypothesis to test.

Following the Trail#

AppDeployments are a Kommander abstraction. Under the hood they create an OCIRepository (the source) and a HelmRelease (the renderer). Both live in the workspace namespace on the workload cluster. Always trust the lowest unhealthy object: do not start with the AppDeployment, do not start with the HelmRelease, start with the OCIRepository.

WS_NS=$(kubectl get workspace demo -o jsonpath='{.status.namespaceRef.name}')

KUBECONFIG=$PWD/workload.conf \
  kubectl get ocirepository,helmrelease -n "$WS_NS"

NAME                                              URL                                                 READY  STATUS
ocirepository.../cert-manager-1.18.2-crds         oci://ghcr.io/mesosphere/charts/cert-manager-crds   False  AuthenticationFailed
NAME                                              AGE   READY   STATUS
helmrelease.../cert-manager-crds                  5m    False   waiting on source

The HelmRelease is blocked on its source. Describe the OCIRepository and read the condition message verbatim:

KUBECONFIG=$PWD/workload.conf \
  kubectl describe ocirepository -n "$WS_NS" cert-manager-1.18.2-crds

Or read the Ready condition message directly with jsonpath:

KUBECONFIG=$PWD/workload.conf \
  kubectl get ocirepository -n "$WS_NS" cert-manager-1.18.2-crds \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'

The Message field tells you exactly which secret name is involved. That is the thread to pull. Variant A names a missing secret. Variant B names a TLS error and you have to look at which secret the OCIRepository's certSecretRef points at.

Cross-Checking the Object and the Content#

The OCIRepository spec declares three secret references. Print them, then list what actually exists in the workspace namespace:

KUBECONFIG=$PWD/workload.conf \
  kubectl get ocirepository -n "$WS_NS" cert-manager-1.18.2-crds \
  -o jsonpath='proxySecretRef={.spec.proxySecretRef.name}{"\n"}certSecretRef={.spec.certSecretRef.name}{"\n"}secretRef={.spec.secretRef.name}{"\n"}'

KUBECONFIG=$PWD/workload.conf \
  kubectl get secret -n "$WS_NS"

You are looking for one of three things:

1. A name in the spec that has no corresponding secret in the namespace (variant A).
2. A secret that exists but whose ca.crt does not match the proxy's certificate (variant B).
3. Two CA secrets in the namespace (nkp-oci-proxy-ca and <cluster>-oci-cert) for different endpoints, where the OCIRepository's certSecretRef points at one but the other is the only one populated.

The third case is the most subtle: there is a CA bundle in the namespace, just not the right one for the proxy. <cluster>-oci-cert is for a different endpoint (typically Harbor direct), not flux-oci-mirror. They are not interchangeable.

To confirm content, decode the secret and compare with what the proxy presents. The flux-oci-mirror service is in the kommander-flux namespace on port 8443, so the openssl probe must run from inside the cluster:

# What the OCIRepository expects
KUBECONFIG=$PWD/workload.conf \
  kubectl get secret -n "$WS_NS" nkp-oci-proxy-ca \
  -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject -issuer

# What the proxy actually presents (run from inside the cluster)
KUBECONFIG=$PWD/workload.conf \
  kubectl run osslp --image=alpine/openssl --rm -it --restart=Never --command -- \
    /bin/sh -c "echo | openssl s_client -showcerts \
      -connect flux-oci-mirror.kommander-flux:8443 2>/dev/null \
      | openssl x509 -noout -subject -issuer"

If subjects do not match, the nkp-oci-proxy-ca content is wrong. If the secret does not exist, it never landed. If the proxy presents a cert from a CA you do not have a secret for, you have a different bug, almost certainly a registry-direct pull rather than a proxy pull.

The Fix#

The safe path is rename-and-copy. Take the correct CA bundle (from the management cluster's kommander/nkp-oci-proxy-ca, which is the source of truth), apply it into the workspace namespace on the workload cluster under the exact name the OCIRepository expects, then force a reconcile.

# On the management cluster context
WS_NS=$(kubectl get workspace demo -o jsonpath='{.status.namespaceRef.name}')
CLUSTER_NAME=demo-wkl-02   # adjust to your workload cluster name

# Copy the CA bundle from kommander to the workspace namespace on the workload cluster.
# This name must match the OCIRepository's certSecretRef exactly.
kubectl -n kommander get secret nkp-oci-proxy-ca -o json |
  jq --arg ns "$WS_NS" '.metadata.namespace=$ns
    | del(.metadata.uid,
          .metadata.resourceVersion,
          .metadata.creationTimestamp,
          .metadata.ownerReferences,
          .metadata.labels,
          .metadata.annotations)' |
  KUBECONFIG=$PWD/workload.conf kubectl create -f -

# If the registry credential is wrong (rare), copy and rename. On the workload cluster,
# the OCIRepository's secretRef.name is cluster-prefixed (<cluster>-oci-creds),
# NOT host-cluster-oci-creds (that is the source name on the management cluster).
# Use jq to rename when copying:
#
# kubectl -n kommander get secret host-cluster-oci-creds -o json |
#   jq --arg ns "$WS_NS" --arg name "${CLUSTER_NAME}-oci-creds" \
#     '.metadata.namespace=$ns | .metadata.name=$name
#       | del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,
#             .metadata.ownerReferences,.metadata.labels,.metadata.annotations)' |
#   KUBECONFIG=$PWD/workload.conf kubectl create -f -

# Force every OCIRepository in this workspace to retry now, skipping Flux's backoff
KUBECONFIG=$PWD/workload.conf \
  kubectl -n "$WS_NS" annotate ocirepository --all \
  reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite

The jq step strips cluster-specific metadata. Apply with kubectl create rather than kubectl apply to avoid the "annotations too long" error that hits when the entire base64 payload would land in last-applied-configuration. The annotation step at the end skips Flux's exponential backoff, which can otherwise add several minutes of waiting after the secret arrives. Naming matters: secrets on workload clusters that mirror cluster-specific identity are cluster-prefixed; secrets that are environment-wide (the proxy CA) keep their canonical name.

Verify:

KUBECONFIG=$PWD/workload.conf \
  kubectl get ocirepository -n "$WS_NS"
# all should be Ready=True within seconds

The HelmRelease reconciles. On the management cluster, the AppDeployment's per-cluster condition flips:

kubectl get appdeployment -n "$WS_NS" cert-manager-crds \
  -o jsonpath='{range .status.clusters[*]}{.name}={.conditions[?(@.type=="AppDeploymentInstanceSynced")].status}{"\n"}{end}'
# demo-wkl-02=True

Dependent charts unblock.

When Else to Run This#

We see this same pattern in a few other contexts. The fix is the same.

1. CA rotation. The internal CA that signs flux-oci-mirror is rotated on the management cluster. The new bundle reaches kommander/nkp-oci-proxy-ca immediately, but workload clusters can lag depending on how the propagation is wired in your environment. Variant B (x509 trust failure) is the typical signal.

2. Registry credential rotation. The Harbor robot account is rotated. Old host-cluster-oci-creds content stops working before the new one is everywhere. Variant A on the auth path, sometimes Variant B if the rotation is paired with a CA change.

3. Workspace bootstrap race. A cluster joins a workspace and the OCIRepositories reach reconcile before the CA secret has propagated to the workspace namespace. Self-healing in most cases, but the annotate-to-retry step shortens the recovery window.

4. Manual edits or partial restores. A backup restore that covered some namespaces but not others, or a half-finished kubectl edit on a secret. Less common but the failure mode is identical.

In every case, the fix above puts the correct content under the correct name and forces a retry. Keep the script in your operations repo.

What the Symptom Looked Like at Every Layer#

Just for completeness, here is what the same fault produces if you start at the top of the stack:

Read top-down and it looks like a Flux bug. Read bottom-up and the missing or wrong content is obvious. Always read bottom-up.

Lessons#

• Trust object conditions over logs. The condition message on the OCIRepository points straight at the broken thing. Logs say the same thing, just more verbosely.
• The OCIRepository is the lowest unhealthy object, not the HelmRelease. Skipping past sources to debug Helm is a 30-minute detour.
• One secret being present is not evidence that the right one is. Always cross-check the spec's secret references against the secrets in the namespace. The cluster may have a CA bundle, just not under the name the spec asks for.
• The fix is rename-and-copy from the source of truth. kommander/nkp-oci-proxy-ca on the management cluster is the source of truth. Everything else is a propagated copy that may have drifted.
• Document this fix in your operations repo. Keep the script, keep the verification steps, keep the rotation triggers. The next time it appears, the engineer who did not debug it the first time will need it.

Summary#

The OCIRepository x509 / missing-secret failure on air-gapped NKP is a generic content-mismatch bug surfaced by Flux's strict TLS and auth handling. Causes vary: timing, rotation, drift, manual changes. The diagnosis pattern is the same every time: read the OCIRepository condition, identify the named secret, verify the content against the source on the management cluster, copy or rename, force reconcile. Five minutes once you have the playbook. Hours of log diving without it. For the broader debugging methodology that produces this kind of fast bottom-up trace, see the companion post on the layered NKP debugging approach.