Run Engine Architecture

Relevant source files

• apps/webapp/app/env.server.ts
• apps/webapp/app/runEngine/concerns/traceEvents.server.ts
• apps/webapp/app/runEngine/services/triggerTask.server.ts
• apps/webapp/app/runEngine/types.ts
• apps/webapp/app/v3/runEngine.server.ts
• apps/webapp/app/v3/services/triggerTaskV1.server.ts
• apps/webapp/app/v3/telemetry/compactMetricExporter.server.ts
• apps/webapp/app/v3/telemetry/loggerExporter.server.ts
• apps/webapp/app/v3/tracer.server.ts
• apps/webapp/test/engine/triggerTask.test.ts
• apps/webapp/test/utils/tracing.ts
• internal-packages/database/prisma/migrations/20250919123905_remove_unnecessary_foreign_key_constraints_task_run_execution_snapshot/migration.sql
• internal-packages/run-engine/src/engine/index.ts
• internal-packages/run-engine/src/engine/locking.ts
• internal-packages/run-engine/src/engine/systems/checkpointSystem.ts
• internal-packages/run-engine/src/engine/systems/delayedRunSystem.ts
• internal-packages/run-engine/src/engine/systems/dequeueSystem.ts
• internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
• internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
• internal-packages/run-engine/src/engine/tests/attemptFailures.test.ts
• internal-packages/run-engine/src/engine/tests/checkpoints.test.ts
• internal-packages/run-engine/src/engine/tests/locking.test.ts
• internal-packages/run-engine/src/engine/tests/waitpoints.test.ts
• internal-packages/run-engine/src/engine/types.ts

Purpose and Scope

This document describes the architecture of the RunEngine class, the core orchestration engine for task execution in Trigger.dev v3/v4. The RunEngine coordinates task lifecycle management through a composition of 11+ specialized systems, integrating with PostgreSQL, Redis, and background workers to provide distributed task execution with checkpointing, retry logic, and real-time observability.

For information about the execution lifecycle and state machine transitions, see Run Lifecycle and State Machine. For queue management details, see Queue Management. For individual system behaviors (checkpoints, waitpoints, batches, etc.), see their respective sections (#4.4-#4.10).

RunEngine Class Overview

The RunEngine class is the central orchestrator located at internal-packages/run-engine/src/engine/index.ts76-1393 It follows a composition-based architecture where specialized systems handle different aspects of task execution. The engine is instantiated as a singleton in the webapp at apps/webapp/app/v3/runEngine.server.ts11-192

Core Responsibilities

• Triggering new task runs
• Dequeuing runs from worker queues using fair selection
• Managing run attempts with retry logic
• Coordinating checkpoints and waitpoints
• Handling batch execution
• Enforcing concurrency limits
• Maintaining execution audit trails

Sources: internal-packages/run-engine/src/engine/index.ts76-384 apps/webapp/app/v3/runEngine.server.ts11-192

System Architecture

Component Composition

Sources: internal-packages/run-engine/src/engine/index.ts76-384 internal-packages/run-engine/src/engine/types.ts23-107

Constructor and Initialization

The RunEngine constructor accepts a RunEngineOptions configuration object and performs the following initialization:

Component	Purpose	Configuration Source
prisma	Primary database client	Required in options
readOnlyPrisma	Read replica for queries	Optional, falls back to prisma
runLockRedis	Redis client for distributed locks	options.runLock.redis with prefix runlock:
runLock	Redlock-based distributed locker	RunLocker with retry configuration
runQueue	Fair queue selection and management	RunQueue with FairQueueSelectionStrategy
worker	Background job processor	Worker with workerCatalog
batchQueue	DRR-based batch processing queue	BatchQueue with quantum and rate limiting
eventBus	Real-time event notifications	EventEmitter<EventBusEvents>

Sources: internal-packages/run-engine/src/engine/index.ts106-384 internal-packages/run-engine/src/engine/types.ts23-107

System Resources Pattern

All specialized systems receive a SystemResources object containing shared dependencies:

This pattern ensures consistent access to core infrastructure across all systems without tight coupling.

Sources: internal-packages/run-engine/src/engine/systems/systems.ts internal-packages/run-engine/src/engine/index.ts269-279

Specialized Systems

System Composition Diagram

Sources: internal-packages/run-engine/src/engine/index.ts281-383

System Initialization Order

Systems are initialized with explicit dependencies, ensuring proper composition:

1. ExecutionSnapshotSystem (lines 281-284): Core state tracking, requires only base resources
2. EnqueueSystem (lines 286-289): Queue management, depends on ExecutionSnapshotSystem
3. CheckpointSystem (lines 291-295): Suspend/resume, depends on ExecutionSnapshotSystem and EnqueueSystem
4. DelayedRunSystem (lines 297-300): Scheduled runs, depends on EnqueueSystem
5. DebounceSystem (lines 302-308): Run deduplication, depends on ExecutionSnapshotSystem and DelayedRunSystem
6. PendingVersionSystem (lines 310-313): Deployment waiting, depends on EnqueueSystem
7. WaitpointSystem (lines 315-319): External dependencies, depends on ExecutionSnapshotSystem and EnqueueSystem
8. TtlSystem (lines 321-324): Expiration, depends on WaitpointSystem
9. BatchSystem (lines 326-329): Parallel execution, depends on WaitpointSystem
10. BatchQueue (lines 331-358): DRR scheduling, independent initialization
11. RunAttemptSystem (lines 360-369): Execution orchestration, depends on ExecutionSnapshotSystem, BatchSystem, WaitpointSystem, and DelayedRunSystem
12. DequeueSystem (lines 377-383): Task selection, depends on ExecutionSnapshotSystem and RunAttemptSystem

Sources: internal-packages/run-engine/src/engine/index.ts281-383

Core Execution Methods

Task Triggering Flow

The trigger() method at internal-packages/run-engine/src/engine/index.ts389-727 handles:

• Debounce checking and claim management
• TaskRun record creation with initial snapshot
• Waitpoint creation for triggerAndWait scenarios
• Delayed run scheduling via DelayedRunSystem
• Immediate run enqueuing via EnqueueSystem
• TTL expiration scheduling via TtlSystem

Key Parameters:

Parameter	Type	Purpose
friendlyId	string	Human-readable run identifier
environment	MinimalAuthenticatedEnvironment	Execution context
taskIdentifier	string	Task to execute
payload	string	Serialized task input
delayUntil	Date?	Delayed execution timestamp
debounce	object?	Debounce configuration
ttl	string?	Time-to-live expiration
maxAttempts	number?	Retry limit
batch	object?	Batch association

Sources: internal-packages/run-engine/src/engine/index.ts389-727 internal-packages/run-engine/src/engine/types.ts117-187

Dequeue and Execution Flow

The dequeue process at internal-packages/run-engine/src/engine/index.ts735-777 and internal-packages/run-engine/src/engine/systems/dequeueSystem.ts105-623 involves:

1. Message Selection: RunQueue.dequeueMessageFromWorkerQueue() uses fair selection strategy
2. Run Locking: Distributed lock via RunLocker prevents concurrent processing
3. Snapshot Validation: Ensures run is in dequeueable state
4. Worker Resolution: Finds matching BackgroundWorker and BackgroundWorkerTask
5. Run Locking: Updates lockedById, lockedToVersionId, lockedQueueId
6. Snapshot Creation: Creates PENDING_EXECUTING snapshot
7. Attempt Start: Transitions to EXECUTING with new attempt number

Sources: internal-packages/run-engine/src/engine/index.ts735-777 internal-packages/run-engine/src/engine/systems/dequeueSystem.ts105-623 internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts294-628

Attempt Completion Flow

The completeRunAttempt() method at internal-packages/run-engine/src/engine/index.ts833-853 delegates to RunAttemptSystem.completeRunAttempt() at internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts630-667 which routes to either:

• attemptSucceeded() (lines 669-824): Updates status to COMPLETED_SUCCESSFULLY, completes associated waitpoint, resumes parent if applicable, triggers batch completion
• attemptFailed() (lines 826-1162): Determines retry outcome, either enqueues retry or finalizes as failed, completes waitpoint with error

Sources: internal-packages/run-engine/src/engine/index.ts833-853 internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts630-1162

Distributed Locking

RunLocker Architecture

The RunLocker class at internal-packages/run-engine/src/engine/locking.ts70-556 provides distributed locking using Redlock with the following features:

Feature	Implementation	Purpose
Redlock algorithm	redlock npm package	Multi-node lock consensus
AsyncLocalStorage	Node.js async_hooks	Lock context tracking
Automatic extension	Background timer	Prevents timeout during long operations
Exponential backoff	Configurable retry policy	Lock acquisition under contention
Signal-based cancellation	RedlockAbortSignal	Graceful lock release
Nested lock detection	Context checking	Prevents deadlocks from re-locking

Retry Configuration:

Lock Key Patterns:

• engine:runlock:{lockType}:{resource1}:{resource2}:... for most operations
• Lock duration: 5 seconds default (configurable via RUN_ENGINE_RUN_LOCK_DURATION)
• Automatic extension threshold: 1 second before expiration

Sources: internal-packages/run-engine/src/engine/locking.ts70-556 apps/webapp/app/env.server.ts594-601

Common Lock Patterns

All critical state transitions acquire a distributed lock on the run ID to prevent race conditions across multiple webapp instances.

Sources: internal-packages/run-engine/src/engine/index.ts internal-packages/run-engine/src/engine/locking.ts196-368

Background Worker Jobs

Worker Catalog

The workerCatalog at internal-packages/run-engine/src/engine/workerCatalog.ts defines background jobs processed by the EngineWorker:

Job Name	Purpose	Scheduling
finishWaitpoint	Complete waitpoint at scheduled time	availableAt = waitpoint completion time
heartbeatSnapshot	Detect stalled snapshots	Periodic based on heartbeat timeouts
repairSnapshot	Recover from stalled states	Delayed after heartbeat detection
expireRun	TTL-based run expiration	availableAt = TTL expiration time
cancelRun	Asynchronous run cancellation	Immediate
queueRunsPendingVersion	Enqueue runs waiting for deployment	After deployment finalization
tryCompleteBatch	Check batch completion	After batch item completion
continueRunIfUnblocked	Resume run after waitpoint completion	50ms delay (debounced)
enqueueDelayedRun	Enqueue delayed run at scheduled time	availableAt = delayUntil timestamp

Job Registration Example:

Sources: internal-packages/run-engine/src/engine/index.ts199-244 internal-packages/run-engine/src/engine/workerCatalog.ts

Event Bus

Real-time Event System

The EventBus at internal-packages/run-engine/src/engine/index.ts90 is a Node.js EventEmitter providing real-time notifications for:

Event	Emitted By	Payload
runCreated	trigger()	{ time, runId }
runLocked	DequeueSystem	{ time, run, organization, project, environment }
runAttemptStarted	RunAttemptSystem.startRunAttempt()	{ time, run, organization, project, environment }
runAttemptCompleted	RunAttemptSystem.attemptSucceeded()	{ time, run, organization, project, environment }
runAttemptFailed	RunAttemptSystem.attemptFailed()	{ time, run, organization, project, environment }
runStatusChanged	Various systems	{ time, run, organization, project, environment }
runDelayRescheduled	DelayedRunSystem	{ time, run, organization, project, environment }
incomingCheckpointDiscarded	CheckpointSystem	{ time, run, checkpoint, snapshot }
cachedRunCompleted	WaitpointSystem	{ time, span, blockedRunId, hasError, cachedRunId }

These events drive:

• Dashboard real-time updates via Server-Sent Events
• Metrics collection
• External observability integrations
• Worker notifications for state changes

Sources: internal-packages/run-engine/src/engine/eventBus.ts internal-packages/run-engine/src/engine/index.ts90

Configuration and Instantiation

Webapp Integration

The RunEngine is instantiated as a singleton in the webapp at apps/webapp/app/v3/runEngine.server.ts11-192:

Key Configuration Sources:

Configuration	Environment Variables	Default
Worker count	RUN_ENGINE_WORKER_COUNT	4
Tasks per worker	RUN_ENGINE_TASKS_PER_WORKER	10
Worker concurrency	RUN_ENGINE_WORKER_CONCURRENCY_LIMIT	10
Poll interval	RUN_ENGINE_WORKER_POLL_INTERVAL	100ms
Lock duration	RUN_ENGINE_RUN_LOCK_DURATION	5000ms
Heartbeat timeouts	RUN_ENGINE_TIMEOUT_*	Various (60s-600s)
Retry warm start	RUN_ENGINE_RETRY_WARM_START_THRESHOLD_MS	30000ms

Sources: apps/webapp/app/v3/runEngine.server.ts11-192 apps/webapp/app/env.server.ts559-619

Redis Partitioning

The RunEngine uses three separate Redis connections to avoid key collisions and enable independent scaling:

Each Redis connection can be configured with separate:

• Host, port, username, password
• TLS settings
• Key prefix for namespace isolation

Sources: apps/webapp/app/v3/runEngine.server.ts84-103 apps/webapp/app/env.server.ts620-684

Service Integration Example

TriggerTaskService Integration

The webapp's RunEngineTriggerTaskService at apps/webapp/app/runEngine/services/triggerTask.server.ts51-417 demonstrates how to integrate with the RunEngine:

This integration demonstrates the concern-based architecture where specialized components handle:

• Validation: Entitlements, max attempts, parent runs
• Queues: Queue name resolution and concurrency limits
• Payloads: Serialization and offloading
• Idempotency: Cache checking and key management
• Tracing: Event tracking and span management

Sources: apps/webapp/app/runEngine/services/triggerTask.server.ts51-417 apps/webapp/app/runEngine/types.ts59-161