Repository navigation
Bootstrap missing models via Engine initContainer (fixes ArgoCD cold-start deadlock for Flux) - #162
therealevanhenry wants to merge 1 commit into
Conversation
The model-management Job carries a `helm.sh/hook: post-install,post-upgrade`
annotation, which ArgoCD maps to PostSync — only running after the main
Sync wave reports Healthy. Nova-2/Nova-3 deployments avoid this because
Engine auto-loads those architectures on demand; Flux pre-allocates GPU
resources at startup and requires its model file to be present in
/models. On a fresh AWS EFS PVC, the Engine pod panics, the Deployment
never goes Healthy, Sync never finishes, and the model download Job
never fires.
Add a model-bootstrap initContainer to the Engine deployment that
downloads any links in engine.modelManager.models.{add,links} into
/models before Engine starts. The container only renders when AWS EFS
is enabled and at least one link is configured. Downloads use atomic
temp+rename so concurrent replicas don't corrupt shared files; existing
files are skipped unless forceDownload is set. The post-install Job is
unchanged and continues to handle ongoing model add/remove operations.
Bump chart version to 0.35.2.
|
No additional Notes for Review from me. This looks good. |
| # on Sync becoming Healthy. Existing files are skipped; downloads use | ||
| # an atomic temp+rename so concurrent replicas don't corrupt files. | ||
| - name: model-bootstrap | ||
| image: alpine |
There was a problem hiding this comment.
Using the alpine image has been tolerable on the existing model-management Job (post-install hook, gated behind PostSync), but moving it into the Engine pod's cold-start path expands the blast radius:
- Untagged →
:latest. The same chart version installs differently day-to-day. - Breaks airgapped customers. They mirror Quay via
global.pullSecretRefand the standard initContainer viaglobal.initContainer.image. They can't reachdocker.io/library/alpine, and there's nopullSecretRefplumbing here. The Job'simage: alpinewas a latent bug for them (most pre-populate models out-of-band); this PR makes it load-bearing.
I considered reusing global.initContainer.image (ubuntu:22.04 has no wget/curl) and the Engine image itself (10GB pull blocks init on a cold node, killing the parallelism with the Engine image pull that makes alpine viable). Neither works.
Suggested: parameterize the image as engine.modelManager.bootstrap.image, mirroring the shape of global.initContainer.image, with a default alpine:3.19 and a pullSecretRef override falling back to global.pullSecretRef. That keeps the parallelism win, fixes reproducibility, and gives airgapped customers a mirror path. The existing Job could migrate to the same knob in a follow-up.
If that feels like scope creep, at least pin the tag. Untagged latest in the cold-start path of an airgapped-supporting chart is a blocker.
| if [ "{{ $.Values.engine.modelManager.volumes.aws.efs.forceDownload }}" = "true" ] || [ ! -f "/models/$FILE_NAME" ]; then | ||
| printf "[%s] [INFO] Downloading model file: %s\n" "$(date -u +$TIMESTAMP_FORMAT)" "$FILE_NAME" | ||
| TMP=$(mktemp "/models/.${FILE_NAME}.XXXXXX") | ||
| if wget -q -O "$TMP" "$link"; then |
There was a problem hiding this comment.
This PR moves the download from PostSync into the Sync path, but the script keeps three flaws from the existing Job that are tolerable in a post-install hook and not tolerable in cold-start:
- No timeout. If the link host hangs, the init container hangs forever. Kubernetes has no inherent init timeout, and the chart doesn't set
activeDeadlineSeconds. ArgoCD Sync waits on Engine Healthy, which waits on init. We've potentially traded the PostSync deadlock for a Sync hang. -qswallows all output on failure. The only signal is[ERROR] Failed to download model file: X- no HTTP status, DNS, or TLS detail. It's painful to debug under an outage.- No retry. A transient blip kills the init, the pod restarts, and CrashLoopBackOff stretches a 30-second hiccup into minutes.
All three are cheap to fix:
wget --timeout=60 --tries=3 --waitretry=10 -O "$TMP" "$link" or switch to curl --max-time 300 --retry 3 --retry-delay 10 --retry-all-errors -fsSL --show-error -o "$TMP" "$link", which gives better error output by default.
| echo "$ADD_LINKS" | while IFS= read -r link; do | ||
| [ -z "$link" ] && continue | ||
| FILE_NAME=$(basename "$link") | ||
| if [ "{{ $.Values.engine.modelManager.volumes.aws.efs.forceDownload }}" = "true" ] || [ ! -f "/models/$FILE_NAME" ]; then |
There was a problem hiding this comment.
Nit: the forceDownload comparison is brittle. A user setting forceDownload: True (which is a valid YAML bool, capital T) renders as True, fails the string compare, and silently doesn't force. Easy to fix here:
[ "{{ ternary "true" "false" $.Values.engine.modelManager.volumes.aws.efs.forceDownload }}" = "true" ]
ternary coerces any truthy value, so True, 1, yes, etc. all collapse correctly.
Summary
model-bootstrapinitContainer to the Engine deployment that downloads any links inengine.modelManager.models.{add,links}into/modelsbefore Engine starts.helm.sh/hook: post-install,post-upgrade, which ArgoCD maps toPostSync. PostSync only runs after the main Sync wave reports Healthy — but on a fresh AWS EFS PVC, Engine pre-allocates GPU resources for Flux at startup, panics withCan't find Flux model with name=..., and the Deployment never goes Healthy. Nova-2/Nova-3 deployments are unaffected because Engine auto-loads those architectures on demand.0.35.2.Behavior
The new initContainer renders only when:
engine.modelManager.volumes.aws.efs.enabled = true, ANDengine.modelManager.models.addorengine.modelManager.models.linksExisting files are skipped; downloads honor
engine.modelManager.volumes.aws.efs.forceDownload. Concurrent replicas on a shared PVC use atomic temp+rename to avoid corrupting in-flight downloads.The existing post-install model-management Job is unchanged. Under vanilla
helm install, both run on cold install (init first, then Job) — idempotent and harmless. Under ArgoCD, Engine can now reach Healthy, Sync completes, and the Job fires as PostSync as it always has.Render matrix
Verified via
helm templateacross:models.addcustomVolumeClaimmodels.addmodels.addhelm lintclean.Notes for review
alpine(untagged), matching the existing Job. Worth a follow-up to pin / parameterize for airgapped customers; intentionally left in scope-parity with the Job here.customVolumeClaimusers continue to pre-populate themselves.Test plan
helm lint ./charts/deepgram-self-hosted— cleanhelm templatewith each render-matrix case above — initContainer presence matches table0.35.1