Upgrade to NKP 2.18, Part 3: The Loki v2 to v3 Migration with Custom S3 Storage

NKP 2.18 migrates the logging stack from Loki 2.9 (the loki-distributed chart) to Loki 3.x (the loki chart). If you run stock configuration, an operator does the whole thing for you. If you customized Loki's storage backend, as anyone running Nutanix Objects instead of the built-in Rook Ceph has, the migration stops your entire platform upgrade until you rewrite your override in a format that is documented nowhere. This article maps that format, one error message at a time. Companion article: Part 2, the platform upgrade.

The fix, if you are mid-upgrade right now#

Four steps, details in the rest of the article:

1. Rewrite your override ConfigMap to the structuredConfig contract (working example and the full accepted-keys table below)
2. Let the operator retry (it does every 2 minutes), or force it: annotate the LoggingStack with reconcile.loggingstack.nkp.nutanix.com/requestedAt=$(date +%s)
3. Nutanix Objects only: add region to the generated grafana-loki-v3-config-overrides ConfigMap immediately after conversion (the converter drops it, and Objects rejects region-less writes)

Then re-run nkp upgrade kommander. The rest of this article is how we mapped that contract, and the four other Rook Ceph assumptions waiting at workspace and project scope.

The setup#

On NKP 2.17 the loki-distributed chart had no structured storage values. Pointing Loki at Nutanix Objects meant re-shipping the entire loki.config in an overrides ConfigMap with the S3 endpoint swapped, because the chart hard-coded the Rook Ceph service URL. Ugly, but it was the only way, and it worked for a year.

During nkp upgrade kommander to 2.18, this banner scrolls by:

================================================================================
Loki Migration Information
================================================================================
Loki AppDeployment detected in kommander namespace (AppDeployment: grafana-loki).
This upgrade will migrate Loki from version 2.x to 3.x.
Please review the release notes for important migration information and breaking changes.
================================================================================

At the CLI's 60-minute deadline:

 ✗ Ensuring platform Kommander applications are upgraded [====> 26/27] (time elapsed 60m00s)
failed to ensure "Ensuring platform Kommander applications are upgraded":
platform upgrade did not complete successfully: timed out waiting for
platform apps upgrade to complete: context deadline exceeded

The CLI does not tell you which app is number 27. The NKPCluster does:

$ kubectl get nkpcluster host-cluster -n kommander \
    -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
PlatformAppsReconciled=False PlatformAppsReconciling: 0 out of 1 logging stacks
  are ready (waiting for loggingstack-host-cluster)

And the new LoggingStack resource names the actual problem:

$ kubectl get loggingstack loggingstack-host-cluster -n kommander -o jsonpath='...conditions...'
GrafanaLokiV3Deployed=False Failed: Failed to convert Loki v2 overrides
configurations to v3 format for: workspace 'kommander-workspace'. Please update
the affected override ConfigMaps manually and annotate the LoggingStack with
reconcile.loggingstack.nkp.nutanix.com/requestedAt=<timestamp> to retry.

"Update the affected override ConfigMaps manually." To what format? The docs don't say. The operator's logs do, one rejection at a time.

Mapping the converter's contract, one error at a time#

The loggingstack-controller-manager deployment in kommander logs every conversion attempt and retries every 2 minutes on its own (the annotation just forces an immediate retry). Each rejection lists the field it refuses and the fields it accepts:

Attempt 1 - our v2 full-config override:

error: failed to build loki v3 format values: unsupported field(s) in loki:
  [config]; supported fields are: [annotations structuredConfig]

Attempt 2 - so we wrote v3-native loki.storage values (the format the new chart actually uses). Rejected too:

error: unsupported field(s) in loki: [storage]; supported fields are:
  [annotations structuredConfig]

The converter does not want v3 values. It wants v2 structuredConfig input and insists on producing the v3 values itself.

Attempt 3 - structuredConfig.common.storage.s3 (valid Loki 2.x config):

error: unsupported field(s) in loki.structuredConfig: [common];
  supported fields are: [limits_config storage_config]

Attempt 4 - the useful one. Under storage_config.aws, discrete endpoint/bucketnames fields are also out:

error: unsupported field(s) in loki.structuredConfig.storage_config.aws:
  [bucketnames endpoint]; supported fields are:
  [http_config insecure s3 s3forcepathstyle]

Four attempts, four rejections - but the accepted-keys lists add up to a complete picture: the converter wants the S3 target as a single URL with the bucket in the path, under structuredConfig.storage_config.aws.s3.

This is what works#

We use S3 on Nutanix Objects; here is the override that passed conversion on the first try:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-loki-overrides    # the SAME ConfigMap your v2 AppDeployment references
  namespace: kommander
data:
  values.yaml: |
    loki:
      structuredConfig:
        storage_config:
          aws:
            s3: http://FS-OBJECT.itcs.local/nkp-loki
            s3forcepathstyle: true
            insecure: true

The complete input contract, log-verified:

Credentials never appear here: both the v2 and v3 charts mount the dkp-loki Secret as environment variables (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) on every Loki component via extraEnvFrom, so an existing secret carries over untouched.

Within two minutes of applying the ConfigMap, the operator converted it, created the grafana-loki-v3 AppDeployment, and moved on. Now look at what the converter wrote into grafana-loki-v3-config-overrides:

loki:
  storage:
    bucketNames:
      admin: nkp-loki
      chunks: nkp-loki
      ruler: nkp-loki
    s3:
      endpoint: http://FS-OBJECT.itcs.local
      insecure: true
      s3ForcePathStyle: true
    type: s3

Character for character the v3-native values we wrote in attempt 2, which it rejected as input. The converter only accepts v2 shapes and only produces v3 shapes. There is no passthrough.

The Rook Ceph scare that wasn't#

Right after conversion, the grafana-loki-v3-pre-install job started looping:

rook-ceph-cluster Kustomization does not exist yet, retrying after 10s...

On a Nutanix-only deployment (Rook Ceph suppressed, Objects as the S3 backend) this looks like the next blocker. It isn't: the job's script waits at most 5 minutes for the Rook Kustomization, then prints rook-ceph-cluster Kustomization does not exist, exiting and exits 0. The custom-storage case was actually handled here. Wait the 5 minutes before touching anything.

The region the converter dropped#

Conversion passed, the v3 stack rolled out (a noticeably bigger topology than v2: three zone-aware ingesters, an index-gateway, a query-scheduler), and the LoggingStack conditions were progressing. Meanwhile the legacy v2 ingester restarted - it consumes the same overrides ConfigMap we had just rewritten - and started failing every chunk flush:

level=error caller=flush.go:168 msg="failed to flush"
  err="failed to flush chunks: store put chunk: IllegalLocationConstraintException:
  Attempting to access a bucket from a different region than where the bucket
  exists. status code: 400"

Our original 2.17 override had carried region: us-east-1. The converter's whitelist has no way to express a region, so the converted values (and our converter-compliant input) simply lost it. Rook Ceph does not care about regions. Nutanix Objects enforces them - every S3 write is rejected with a 400.

Two fixes, one per generation:

• Legacy v2: once the v3 AppDeployment exists, the converter logs already exists with config overrides, skipping creation and never validates your ConfigMap again - so we restored the original full-config override (with region) from the pre-upgrade nkp diagnose support bundle. Worth knowing: an nkp diagnose support bundle contains every ConfigMap as it was at capture time. That is how we restored the original byte for byte.
• v3: added region: us-east-1 to the generated grafana-loki-v3-config-overrides before the first flush cycle. Result: zero S3 errors on the v3 stack, ever.

Cost of learning this live: the v2 ingester failed flushes for about twenty minutes before being retired, and with wal.flush_on_shutdown: false those buffered chunks were lost with the pod. Acceptable on a lab. On production, this is the argument for converting your override ConfigMap before running the upgrade, not during.

The migration cleans up after itself#

We did not have to decommission anything. Once v3 reported healthy, the LoggingStack retired the legacy stack on its own and the conditions tell the whole story:

$ kubectl get loggingstack loggingstack-host-cluster -n kommander -o jsonpath='...'
LoggingStackVersionReconciled=True ReconciliationSucceeded
GrafanaLokiV3Deployed=True Succeeded
LoggingStackAppDeploymentsUpdated=True Succeeded
GrafanaLokiV2Removed=True Succeeded

With the LoggingStack green, the third run of nkp upgrade kommander passed the step that had timed out twice, and the management cluster reported platform version v2.18.0.

Round two, workspace scope: stock Loki is broken too (on Nutanix)#

We assumed the pain was self-inflicted: we had customized storage, so we paid the conversion tax. Then the workspace upgrades started migrating the stock Loki instances on our workload clusters (the banner lists every one it will touch, workspace and project scope), and it turned out the problem was bigger than our customization.

Our workspace grafana-loki ran on CSI PersistentVolumes: no S3, no secrets, no overrides - as stock as it gets on Nutanix. The migration replaced it with a v3 whose NKP defaults are broken twice on a Nutanix workload cluster:

$ kubectl get pods -n customera-dev | grep loki-v3
grafana-loki-v3-compactor-0                     0/1   CreateContainerConfigError
grafana-loki-v3-ingester-zone-a-0               0/1   CreateContainerConfigError
(...all eight components...)

$ kubectl describe pod grafana-loki-v3-compactor-0 -n customera-dev | grep Error
  Error: secret "dkp-loki" not found

1. Every v3 component unconditionally mounts a dkp-loki secret that only Rook Ceph's object store auto-creates. No Rook, no secret, no pods.
2. Even with the secret, the rendered v3 config points at http://rook-ceph-rgw-dkp-object-store.<workspace-ns>.svc:80 - a service that does not exist on any Nutanix workload cluster.

Then we checked what the migration had replaced: the v2 stack it retired had been in the exact same state - CreateContainerConfigError on the same missing secret - for 63 days, since the day it was deployed. Its rendered config pointed at the same phantom Rook service. Only the gateway pod (plain nginx) ever ran, which is why a glance at the namespace never screamed. Workspace logging on this cluster never worked, never stored a byte, and nobody noticed for two months.

To be explicit: NKP's workspace logging defaults assume Rook Ceph, which Nutanix's own infrastructure provider does not run - and this predates 2.18. The migration did not break stock workspace logging on Nutanix; it faithfully carried an already-broken deployment into an equally broken v3. Every NKP-on-Nutanix environment that enabled workspace logging with defaults has this problem today, on 2.17 or 2.18 alike. (One consolation: if yours never worked either, there is no data to lose in the migration.)

The fix is the mgmt-cluster pattern at workspace scope: a bucket on Objects, a dkp-loki secret created on the workload cluster (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY keys), and a values override on the grafana-loki-v3 AppDeployment redirecting storage to Objects, including region (previous section).

Project scope: two more Rook assumptions#

Projects get their own Loki, with two more variants of the same problem:

• The project chart mounts a differently-named secret: proj-loki-<project-namespace>, not dkp-loki. Copy your credentials under that exact name into the project namespace on the workload cluster.
• The project app ships a second HelmRelease, project-loki-v3-object-bucket-claims, whose entire job is provisioning that bucket+secret through Rook's ObjectBucketClaim API. On a Nutanix cluster the OBC CRD does not exist, so it fails with resource mapping not found forever - and the app's health check chain waits on it. The escape is the chart's own optional overrides hook: a ConfigMap named project-grafana-loki-v3-overrides (in the project namespace, ON the workload cluster) containing:

dkp:
  project-grafana-loki-v3:
    enabled: false

The OBC HelmRelease then installs an empty release, reports Ready, and the chain drains.

The cleanup phase wants the old stack healthy first#

With both v3 instances green, the LoggingStack entered its cleanup phase (LoggingStackAppDeploymentsUpdated) and stalled, waiting for grafana-loki and project-grafana-loki to become ready. Those are the v2 apps it is supposed to remove.

The cleanup first upgrades the v2 AppDeployments one patch version and waits for them to reconcile before retiring them. Our v2 pods - unblocked by the secrets we had just created - started, dialed the nonexistent Rook endpoint, and crashed (no such host). Helm stalled on the StatefulSets. The migration would have waited forever for an app it was about to delete to become healthy, which on this infrastructure it never was.

The unblock is the same pattern as everything else in this article: give the v2 apps what their charts expect - the secret and the storage override, exactly as done for v3 - so they reconcile once, and the cleanup retires them. The workspace upgrade then completed on the next cycle.

Verified end to end#

Two queries against the v3 gateway. A live one:

$ curl -s "http://grafana-loki-v3-gateway/loki/api/v1/query_range?query={namespace=\"kommander\"}&since=10m&limit=2"
{"status":"success","data":{"resultType":"streams","result":[{"stream":{...cluster-observer...}

And the one that proves storage: a window from 9 to 11 hours ago, older than anything the ingesters hold in memory, so the data must be read back from the nkp-loki bucket on Nutanix Objects:

$ curl -s ".../query_range?query={namespace=\"kommander\"}&start=<T-11h>&end=<T-9h>&limit=1"
{"status":"success","data":{"resultType":"streams","result":[{...}

Ingest, flush to Objects, query back. Working.

Takeaways#

1. A 26/27 platform upgrade timeout has a name - find it in kubectl get nkpcluster ... -o jsonpath='{.status.conditions}', then in the owning operator's logs. The CLI's counter will not tell you.
2. Custom Loki storage overrides WILL block the 2.18 upgrade. Convert your override ConfigMap to the structuredConfig shape above before running nkp upgrade kommander and the migration passes on the first try.
3. The converter only understands the single-URL S3 form (s3: http://<endpoint>/<bucket> under structuredConfig.storage_config.aws) - the working override above is the shape to copy.
4. The operator self-retries every 2 minutes and logs precisely; the requestedAt annotation just skips the wait.
5. Do not panic at the rook-ceph-cluster wait loop on Nutanix-only clusters; it self-releases after 5 minutes.
6. The converter cannot carry an S3 region, and Nutanix Objects requires one. Expect IllegalLocationConstraintException on every flush until you add region back: to the generated grafana-loki-v3-config-overrides for v3, and (if the legacy stack is still running) to your original override, which is safe to restore once the migration reports skipping creation.
7. Take nkp diagnose before upgrading. The support bundle contains every ConfigMap as-it-was; it is how we restored the original override byte-for-byte instead of reconstructing it from memory.
8. There are five separate Rook Ceph assumptions. The workspace secret (dkp-loki), the project secret (proj-loki-<ns>), the storage endpoint, the ObjectBucketClaim chart, and the cleanup phase's demand that v2 be healthy before removal. On Nutanix infrastructure every one of them fails independently. Budget two hours per workspace or cluster with logging enabled, or fix the secrets and overrides before running the workspace upgrade and it collapses to minutes.
9. The cleanup phase needs the v2 apps to reconcile once before it retires them - fix them the same way as v3 (secret + storage override) and the migration finishes on its own.