Skip to content

Add retry-with-backoff for RequestServiceIdentity at startup #126

Open

Description

@rowan-stein

User Request

Gateway crashes (CrashLoopBackOff, exit code 1) when ziti-management isn't ready at startup. This is a latent startup race condition. Add retry logic matching llm-proxy's existing pattern.

Specification

Root Cause

cmd/gateway/main.go calls zitiMgmtClient.RequestServiceIdentity() at startup and uses log.Fatalf on failure. If ziti-management's gRPC server isn't accepting connections yet, the gateway exits immediately and enters CrashLoopBackOff.

Change Required

File: cmd/gateway/main.go

Wrap the RequestServiceIdentity call in a retry loop with exponential backoff:

  • Initial backoff: 1s
  • Max backoff: 15s
  • Multiplier: 2×
  • Total timeout: Controlled by ZITI_ENROLLMENT_TIMEOUT env var (default 2m)
  • Retryable codes: Unavailable, Unknown (connection refused, DNS failure)
  • Non-retryable: All other gRPC codes — fail immediately

Reference agynio/llm-proxy cmd/llm-proxy/main.go which already has retryWithBackoff() implementing this exact pattern (though with 5m default — use 2m).

Add log messages for each retry attempt so startup progress is visible.

What Does NOT Change

  • internal/zitimgmtclient/client.go — remains a thin pass-through
  • No new dependencies needed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions