fix(certgen): bundle previous CA during cert rotation to prevent mTLS disruption

[OliverBailey]

Summary

Fixes #4891 (partial — Rate Limit CA hot-reload addressed in a follow-up PR)

Problem

When certgen --overwrite rotates certificates, ca.crt in each control-plane Secret is replaced atomically with the new CA. Two propagation mechanisms are at play after that write:

• Kubelet volume sync: updates the mounted Secret on disk for each pod, with a default sync period up to 1 minute.
• Envoy SDS reload: Envoy picks up the updated xDS TLS context from the SDS path-based files, which happens near-immediately once the kubelet has synced.

Neither is synchronous. During the convergence window a pod that has picked up a new leaf cert (signed by the new CA) is rejected by a peer that still holds only the old CA in its trust store, causing mTLS authentication failures. This is precisely the incident reproduced on v1.6.1 described in the issue thread.

Fix

When updating an existing Secret that already contains a ca.crt, bundle the outgoing CA together with the incoming CA so that every component trusts both during the transition.

bundleCACerts(newCA, oldCA):

1. Starts the bundle with all certs from newCA (the freshly generated CA).
2. Appends the first non-expired, non-duplicate cert from oldCA — the CA active at the previous rotation.
3. Stops there (break).

Why a maximum of two CAs

Carrying forward only one previous CA keeps the bundle at exactly two entries regardless of rotation frequency.

By the time an operator runs certgen --overwrite a second time, all components will have converged on the certs written during the first rotation (kubelet sync + SDS reload happen within seconds to a minute). The CA from two rotations ago is therefore never needed in practice. Carrying it forward indefinitely would cause unbounded bundle growth for long-lived CAs — the default lifetime is 5 years. The single carry-over is naturally dropped at the rotation after it would have been needed.

Rotation 1: secret stores [CA2, CA1]  ← CA1 carried for convergence window
Rotation 2: secret stores [CA3, CA2]  ← CA1 dropped, CA2 carried
Rotation 3: secret stores [CA4, CA3]  ← always exactly 2

Scope

• ✅ Envoy ↔ Envoy Gateway mTLS: fixed — Envoy SDS reloads in seconds; the bundle covers the overlap window.
• ✅ The envoy-oidc-hmac Secret carries no ca.crt and is unaffected.
• ⏳ Rate Limit CA hot-reload: addressed in a follow-up PR (fix/ratelimit-ca-restart → this branch). Rate Limit does not watch its CA file for changes; that PR triggers a rolling restart of the Rate Limit Deployment after rotation.

Testing

Added TestCreateOrUpdateSecretsBundlesCA and TestBundleCACerts covering:

• Old CA is carried into the updated bundle
• New CA is always first
• Duplicate certs from the old bundle are not re-appended (multi-rotation stability)
• Expired certs from the old bundle are excluded
• Bundle never exceeds two entries across multiple rotations

[netlify]

✅ Deploy Preview for cerulean-figolla-1f9435 canceled.

Name	Link
🔨 Latest commit	9bc2493
🔍 Latest deploy log	https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/69c05d765dd61f00087335b3

[codecov]

Codecov Report

❌ Patch coverage is 94.44444% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.15%. Comparing base (595010a) to head (9bc2493).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/provider/kubernetes/secrets.go	94.44%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8534      +/-   ##
==========================================
+ Coverage   74.14%   74.15%   +0.01%     
==========================================
  Files         242      242              
  Lines       37749    37784      +35     
==========================================
+ Hits        27989    28020      +31     
- Misses       7806     7808       +2     
- Partials     1954     1956       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[arkodg]

• does rotation impact existing connections ?
• also, during the rotation window, a new connection is made, and if either client or server reconciles faster than the other, wont we be in the same situation where the slower peer cannot verify the newer cert ?

[OliverBailey]

Thanks for the questions; good ones @arkodg

Does rotation impact existing connections?

No. TLS/mTLS verification only happens at the handshake. An established connection that completed its handshake before rotation doesn't get re-verified and won't be disrupted. For the Envoy ↔ Envoy Gateway xDS gRPC stream this is a single long-lived connection per Envoy pod, so rotation alone won't break anything in flight.

During the rotation window, won't the slower peer be unable to verify a newer cert?

You're right. The bundle is a targeted improvement, not a complete solution.

What [newCA, oldCA] solves: once a pod's kubelet has synced the updated Secret, its ca.crt now trusts both CAs. So if that pod is verifying an incoming cert from a peer that hasn't synced yet (still presenting an old leaf cert signed by oldCA), verification succeeds.

What it doesn't cover: if the already-updated pod is presenting its new leaf cert (signed by newCA) to a peer whose ca.crt still only contains oldCA, that peer will reject it. The problem is symmetric.

The mitigating factors are:

• The kubelet updates a Secret volume atomically (symlink swap over the whole directory), so for any given pod ca.crt and tls.crt always advance together; no intra-pod partial state.
• The window is bounded: the default kubelet sync period is ~60 s, after which Envoy's SDS reload is near-immediate. So the asymmetric exposure window is on the order of one kubelet sync cycle between any two given pods.
• Existing connections; the vast majority of traffic are unaffected throughout.

A fully race-free approach would require a two-phase rotation: push only the updated ca.crt bundle first, wait for all pods to converge, then push the new leaf certs. That's a significantly larger change to certgen's flow. I think making this meaningfully better is worth landing now. The original incident was a permanent failure that persisted until a manual restart; this fix reduces the exposure to a bounded convergence window. Happy to hear your thoughts on whether the two-phase approach is worth pursuing as a follow-up.

[arkodg]

hey trying to understand the permanent failure case, which client was unable to setup a connection during the rotation change ? is this envoy-ratelimit ? does ratelimit service use updated cert material if its ConfigMap and Secrets have been updated ?

[arkodg]

cc @rajatvig

[rajatvig]

Yes, this might help fix the case when using the Certgen Job construct vs the custom Certificate setup. The core of the problem is that when old CA is renewed but not leafs, the bundle needs to carry both the certs so that the leaf certs can still be validated.

The theory that once the new CA is refreshed, this would work. The other idea I have been trying to wrangle is watching the CA secret for changes and triggering leaf certificate renewals when that happens.

[OliverBailey]

hey trying to understand the permanent failure case, which client was unable to setup a connection during the rotation change ? is this envoy-ratelimit ? does ratelimit service use updated cert material if its ConfigMap and Secrets have been updated ?

@arkodg

Envoy is the client that can't establish the connection — Rate Limit (the server) rejects it because it doesn't hot-reload its CA cert.

The three components behave differently after the kubelet syncs the updated Secret volume:

• Envoy Gateway re-reads TLS material per-connection, so it always uses what's on disk.
• Envoy hot-reloads via SDS path-based references within seconds of the kubelet sync.
• Rate Limit loads its CA once at startup and has no mechanism to watch the mounted secret for changes. The file on disk gets updated by the kubelet but the process keeps verifying against the old CA in memory.

The failure sequence is: kubelet syncs → Envoy SDS-reloads its new leaf cert (signed by the new CA) → Rate Limit still holds the old CA in memory → mTLS handshakes from Envoy to Rate Limit are rejected. The only recovery without intervention is a pod restart.

This PR addresses the Envoy ↔ EG convergence window via the CA bundle. The Rate Limit side is handled in a follow-up PR (fix/ratelimit-ca-restart) which triggers a rolling restart of the Rate Limit Deployment after rotation, so every pod starts fresh with the updated CA bundle.

[OliverBailey]

Yes, this might help fix the case when using the Certgen Job construct vs the custom Certificate setup. The core of the problem is that when old CA is renewed but not leafs, the bundle needs to carry both the certs so that the leaf certs can still be validated.

The theory that once the new CA is refreshed, this would work. The other idea I have been trying to wrangle is watching the CA secret for changes and triggering leaf certificate renewals when that happens.

@rajatvig

Exactly right — the bundle is specifically for the convergence window where the new CA is in place but not all pods have synced yet, so old leaf certs still need to be verifiable.

The CA-watch → leaf renewal idea is a nice proactive complement to this. Rather than tolerating the mismatch window it would eliminate it entirely. It's a bigger change though — needs a controller watching the CA Secret and re-invoking cert generation. Worth tracking as a follow-up issue; happy to open one once these two PRs land.

[arkodg]

@OliverBailey can we fix the issue in ratelimit repo to pick up the newer certs when the files change ?

[OliverBailey]

@arkodg Yeah, that works with me. You want me to make that change, or yourselves to pick it up?

[arkodg]

Would be great if you can drive that change in the rate limit repo