Skip to content

Bootstrap missing models via Engine initContainer (fixes ArgoCD cold-start deadlock for Flux) - #162

Open
therealevanhenry wants to merge 1 commit into
mainfrom
fix/flux-argocd-cold-start-deadlock
Open

therealevanhenry wants to merge 1 commit into
mainfrom
fix/flux-argocd-cold-start-deadlock

Conversation

@therealevanhenry

Copy link
Copy Markdown
Contributor

Summary

  • Add a model-bootstrap initContainer to the Engine deployment that downloads any links in engine.modelManager.models.{add,links} into /models before Engine starts.
  • Resolves a cold-start deadlock seen with Flux STT under ArgoCD: the existing model-management Job carries helm.sh/hook: post-install,post-upgrade, which ArgoCD maps to PostSync. PostSync only runs after the main Sync wave reports Healthy — but on a fresh AWS EFS PVC, Engine pre-allocates GPU resources for Flux at startup, panics with Can't find Flux model with name=..., and the Deployment never goes Healthy. Nova-2/Nova-3 deployments are unaffected because Engine auto-loads those architectures on demand.
  • Bumps chart version to 0.35.2.

Behavior

The new initContainer renders only when:

  • engine.modelManager.volumes.aws.efs.enabled = true, AND
  • At least one entry exists in engine.modelManager.models.add or engine.modelManager.models.links

Existing files are skipped; downloads honor engine.modelManager.volumes.aws.efs.forceDownload. Concurrent replicas on a shared PVC use atomic temp+rename to avoid corrupting in-flight downloads.

The existing post-install model-management Job is unchanged. Under vanilla helm install, both run on cold install (init first, then Job) — idempotent and harmless. Under ArgoCD, Engine can now reach Healthy, Sync completes, and the Job fires as PostSync as it always has.

Render matrix

Verified via helm template across:

Scenario model-bootstrap config-render Job
AWS EFS + models.add 1 0 1
AWS EFS, no models 0 0 0
customVolumeClaim 0 0 0
GCP GPD + models.add 0 0 0
AWS EFS + Flux + billing 1 2 (engine + billing) 1
Voice Agent + AWS EFS + models.add 3 (one per agent engine) 0 1

helm lint clean.

Notes for review

  • Image is alpine (untagged), matching the existing Job. Worth a follow-up to pin / parameterize for airgapped customers; intentionally left in scope-parity with the Job here.
  • Voice Agent renders three model-bootstrap containers that race on the shared EFS PVC. Atomic rename keeps it correct, but log noise is 3× on first install. Acceptable for this fix; happy to add a sentinel-file gate in a follow-up if preferred.
  • Scope is AWS EFS only, mirroring the existing automatic-model-management contract. GCP / customVolumeClaim users continue to pre-populate themselves.

Test plan

  • helm lint ./charts/deepgram-self-hosted — clean
  • helm template with each render-matrix case above — initContainer presence matches table
  • Fresh AWS EFS install with Flux enabled — Engine reaches Healthy without operator intervention
  • Fresh AWS EFS install with Nova-only — no behavior change vs. 0.35.1
  • ArgoCD sync of a Flux deployment from scratch — no PostSync deadlock

The model-management Job carries a `helm.sh/hook: post-install,post-upgrade`
annotation, which ArgoCD maps to PostSync — only running after the main
Sync wave reports Healthy. Nova-2/Nova-3 deployments avoid this because
Engine auto-loads those architectures on demand; Flux pre-allocates GPU
resources at startup and requires its model file to be present in
/models. On a fresh AWS EFS PVC, the Engine pod panics, the Deployment
never goes Healthy, Sync never finishes, and the model download Job
never fires.

Add a model-bootstrap initContainer to the Engine deployment that
downloads any links in engine.modelManager.models.{add,links} into
/models before Engine starts. The container only renders when AWS EFS
is enabled and at least one link is configured. Downloads use atomic
temp+rename so concurrent replicas don't corrupt shared files; existing
files are skipped unless forceDownload is set. The post-install Job is
unchanged and continues to handle ongoing model add/remove operations.

Bump chart version to 0.35.2.
@therealevanhenry
therealevanhenry requested a review from a team as a code owner May 6, 2026 21:37
@connerhughes-dg

Copy link
Copy Markdown
Contributor

No additional Notes for Review from me. This looks good.

# on Sync becoming Healthy. Existing files are skipped; downloads use
# an atomic temp+rename so concurrent replicas don't corrupt files.
- name: model-bootstrap
image: alpine

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the alpine image has been tolerable on the existing model-management Job (post-install hook, gated behind PostSync), but moving it into the Engine pod's cold-start path expands the blast radius:

  1. Untagged → :latest. The same chart version installs differently day-to-day.
  2. Breaks airgapped customers. They mirror Quay via global.pullSecretRef and the standard initContainer via global.initContainer.image. They can't reach docker.io/library/alpine, and there's no pullSecretRef plumbing here. The Job's image: alpine was a latent bug for them (most pre-populate models out-of-band); this PR makes it load-bearing.

I considered reusing global.initContainer.image (ubuntu:22.04 has no wget/curl) and the Engine image itself (10GB pull blocks init on a cold node, killing the parallelism with the Engine image pull that makes alpine viable). Neither works.

Suggested: parameterize the image as engine.modelManager.bootstrap.image, mirroring the shape of global.initContainer.image, with a default alpine:3.19 and a pullSecretRef override falling back to global.pullSecretRef. That keeps the parallelism win, fixes reproducibility, and gives airgapped customers a mirror path. The existing Job could migrate to the same knob in a follow-up.

If that feels like scope creep, at least pin the tag. Untagged latest in the cold-start path of an airgapped-supporting chart is a blocker.

if [ "{{ $.Values.engine.modelManager.volumes.aws.efs.forceDownload }}" = "true" ] || [ ! -f "/models/$FILE_NAME" ]; then
printf "[%s] [INFO] Downloading model file: %s\n" "$(date -u +$TIMESTAMP_FORMAT)" "$FILE_NAME"
TMP=$(mktemp "/models/.${FILE_NAME}.XXXXXX")
if wget -q -O "$TMP" "$link"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR moves the download from PostSync into the Sync path, but the script keeps three flaws from the existing Job that are tolerable in a post-install hook and not tolerable in cold-start:

  1. No timeout. If the link host hangs, the init container hangs forever. Kubernetes has no inherent init timeout, and the chart doesn't set activeDeadlineSeconds. ArgoCD Sync waits on Engine Healthy, which waits on init. We've potentially traded the PostSync deadlock for a Sync hang.
  2. -q swallows all output on failure. The only signal is [ERROR] Failed to download model file: X - no HTTP status, DNS, or TLS detail. It's painful to debug under an outage.
  3. No retry. A transient blip kills the init, the pod restarts, and CrashLoopBackOff stretches a 30-second hiccup into minutes.

All three are cheap to fix:

wget --timeout=60 --tries=3 --waitretry=10 -O "$TMP" "$link" or switch to curl --max-time 300 --retry 3 --retry-delay 10 --retry-all-errors -fsSL --show-error -o "$TMP" "$link", which gives better error output by default.

echo "$ADD_LINKS" | while IFS= read -r link; do
[ -z "$link" ] && continue
FILE_NAME=$(basename "$link")
if [ "{{ $.Values.engine.modelManager.volumes.aws.efs.forceDownload }}" = "true" ] || [ ! -f "/models/$FILE_NAME" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the forceDownload comparison is brittle. A user setting forceDownload: True (which is a valid YAML bool, capital T) renders as True, fails the string compare, and silently doesn't force. Easy to fix here:

[ "{{ ternary "true" "false" $.Values.engine.modelManager.volumes.aws.efs.forceDownload }}" = "true" ]

ternary coerces any truthy value, so True, 1, yes, etc. all collapse correctly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants