Key Concepts

Relevant source files

This page defines the core terminology and fundamental concepts used throughout the Trigger.dev platform. Understanding these concepts is essential for working with any part of the codebase, from task definition to execution orchestration.

For information about the overall system architecture and component interactions, see System Architecture.

Core Entities

Tasks

A task is the fundamental unit of work in Trigger.dev. Tasks are defined using the SDK and represent reusable, retriggerable operations that can be executed asynchronously.

Code Representation:

SDK function: task(), schemaTask(), toolTask() in @trigger.dev/sdk
Database model: BackgroundWorkerTask in @trigger.dev/database
Identifier: taskIdentifier (string, unique per environment)

Key Properties:

id: Unique identifier for the task
exportName: The exported name from the task file
filePath: Location of the task definition
queue: Associated queue configuration
retry: Retry configuration with limit, factor, minTimeout, maxTimeout
machine: Machine preset specification (CPU/memory allocation)

Task Definition State Machine (Deployment):

Sources: packages/trigger-sdk/package.json internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts44-91

Runs

A run (also called task run) is a single execution instance of a task. Each time a task is triggered, a new run is created.

Code Representation:

Database model: TaskRun in @trigger.dev/database
Run engine class: RunEngine in internal-packages/run-engine/src/engine/index.ts76
ID format: run_[ulid] via RunId type

Run Lifecycle States:

Status	Description	Counts Against Concurrency?
`PENDING_VERSION`	Waiting for worker deployment	No
`QUEUED`	Waiting in queue for execution	No
`DELAYED`	Scheduled for future execution	No
`DEQUEUED`	Removed from queue, being sent to worker	Yes
`EXECUTING`	Currently running	Yes
`WAITING`	Paused (waitpoint/checkpoint)	No
`COMPLETED`	Successfully finished	No
`CANCELED`	User-canceled	No
`FAILED`	Failed after retries	No
`CRASHED`	Worker crashed (OOM, etc.)	No
`SYSTEM_FAILURE`	Platform error	No
`EXPIRED`	TTL exceeded	No
`TIMED_OUT`	maxDuration exceeded	No

Key Properties:

id: Unique run identifier
friendlyId: Human-readable ID (e.g., run_abc123)
status: Current state in the state machine
payload: Input data for the run
output: Result data after completion
metadata: User-defined key-value pairs
context: Execution context (attempt, run IDs, environment)
traceId, spanId: OpenTelemetry correlation IDs

Run State Machine:

Boolean Helper Methods:

isQueued(): Returns true for QUEUED, PENDING_VERSION, DELAYED
isExecuting(): Returns true for EXECUTING, DEQUEUED
isWaiting(): Returns true for WAITING
isCompleted(): Returns true for any terminal state
isSuccess(): Returns true for COMPLETED
isFailed(): Returns true for FAILED, CRASHED, SYSTEM_FAILURE, TIMED_OUT
isCanceled(): Returns true for CANCELED

Sources: internal-packages/run-engine/src/engine/index.ts388-523 internal-packages/run-engine/src/engine/statuses.ts packages/trigger-sdk/CHANGELOG.md446-486

Attempts

An attempt is a single try at executing a run. A run may have multiple attempts if retries are configured.

Code Representation:

Database model: TaskRunAttempt in @trigger.dev/database
System: RunAttemptSystem in internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts63

Key Properties:

number: Sequential attempt number (1, 2, 3, ...)
status: Current state (mirroring run status during execution)
startedAt: When this attempt began
completedAt: When this attempt finished
error: Error information if failed
backgroundWorkerId: Worker that executed this attempt

Retry Logic:

Exponential backoff with configurable factor, minTimeout, maxTimeout
Maximum attempts: 25 (defined as MAX_TASK_RUN_ATTEMPTS)
Retry decision made in retryOutcomeFromCompletion() function
Warm start optimization: If previous attempt finished recently (< retryWarmStartThresholdMs), reuse the worker process

Sources: internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts63-673 internal-packages/run-engine/src/engine/consts.js1

Workers

Workers are processes that execute tasks. There are two types:

Development Workers

Created by trigger dev CLI command
Run locally on developer machines
Use DevSupervisor class to manage task execution
Poll for runs via SSE connection to platform

Production Workers

Created during deployment via trigger deploy
Run in containerized environments (Docker images)
Managed by the platform's worker orchestration
Two worker types:
- BackgroundWorker (v3): Older architecture, managed separately
- Worker (v4): Newer Run Engine 2.0 architecture

Code Representation:

Database model: BackgroundWorker in @trigger.dev/database
Supervisor: Supervisor class in apps/coordinator
Task executor: TaskExecutor in apps/supervisor

Worker Registration Flow:

Sources: apps/webapp/app/runEngine/services/triggerTask.server.ts internal-packages/run-engine/src/engine/systems/dequeueSystem.ts88

Environments

An environment represents a deployment target with its own configuration, API keys, and concurrency limits.

Code Representation:

Database model: RuntimeEnvironment in @trigger.dev/database
Type enum: RuntimeEnvironmentType with values DEVELOPMENT, PRODUCTION, STAGING, PREVIEW

Environment Types:

Type	Purpose	Concurrency Default	API Key Prefix
`DEVELOPMENT`	Local dev runs	10	`tr_dev_`
`PRODUCTION`	Production workloads	100	`tr_prod_`
`STAGING`	Pre-production testing	50	`tr_stag_`
`PREVIEW`	Branch previews	25	`tr_prev_`

Key Properties:

id: Unique environment ID
type: Environment type enum
slug: URL-safe identifier
apiKey: Authentication token
pkApiKey: Public API key (for frontend)
organizationId: Parent organization
projectId: Parent project

Concurrency Configuration:

maximumConcurrencyLimit: Environment-wide concurrency cap
concurrencyBurstFactor: Multiplier for burst capacity (default: 2.0x)
Effective burst limit = maximumConcurrencyLimit * concurrencyBurstFactor

Sources: apps/webapp/app/env.server.ts263-267 internal-packages/run-engine/src/engine/index.ts146-161

Organizations and Projects

Organizations are top-level tenant entities that contain projects. Projects group related tasks and environments.

Code Representation:

Database models: Organization, Project in @trigger.dev/database
Hierarchy: Organization → Project → RuntimeEnvironment → TaskRun

Organization:

id: Unique org ID
slug: URL-safe identifier
title: Display name
Members: OrgMember records with roles

Project:

id: Unique project ID
slug: URL-safe identifier (unique within org)
externalRef: External reference ID (e.g., proj_abc123)
organizationId: Parent organization

Organizational Hierarchy:

Sources: apps/webapp/app/models/organization.server.ts apps/webapp/app/models/project.server.ts

Execution Concepts

Lifecycle Hooks

Lifecycle hooks are functions that execute at specific points during task execution, providing extension points for cross-cutting concerns.

Available Hooks (12 total):

Hook	When It Fires	Use Case
`init`	Once per worker process startup	Initialize connections, load config
`onStartAttempt`	Before each attempt's `run()` function	Per-attempt setup, logging
`onStart`	Before first attempt only (deprecated)	Use `onStartAttempt` with `ctx.run.attempt.number === 1`
`middleware`	Around the `run()` function	Wrapping logic, timing
`onSuccess`	After successful completion	Cleanup, notifications
`onFailure`	After failed attempt	Error handling, alerts
`onComplete`	After any completion (success/failure)	Guaranteed cleanup
`cleanup`	After attempt finishes (always runs)	Resource cleanup
`onWait`	When entering WAITING state	Log waitpoint entry
`onResume`	When resuming from WAITING	Log resume, reset state
`onCancel`	When run is canceled	Cancelation handling
`catchError`	When uncaught error occurs	Error transformation

Hook Execution Order:

Global vs Task-Level Hooks:

Task-level: Defined in task() options
Global: Defined using tasks.onStartAttempt(), tasks.onSuccess(), etc.

Error Handling in Hooks:

Errors in onSuccess, onFailure, onComplete do NOT fail the run (logged only)
Errors in middleware, run() DO fail the run
catchError can transform errors before retry logic

Sources: packages/trigger-sdk/CHANGELOG.md38-73 packages/core/CHANGELOG.md98

Execution Snapshots

An execution snapshot captures the state of a run at a point in time. Snapshots enable state recovery, monitoring, and debugging.

Code Representation:

Database model: TaskRunExecutionSnapshot in @trigger.dev/database
System: ExecutionSnapshotSystem in internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts16

Snapshot Types (by status):

Status	Description	Triggers Action
`PENDING`	Initial creation	No
`EXECUTING`	Run started	No
`EXECUTING_WITH_WAITPOINTS`	Has blocking waitpoints	No
`SUSPENDED`	Checkpointed, worker shutdown	Resume when ready
`DEQUEUED`	Removed from queue	No

Key Properties:

id: Unique snapshot ID (SnapshotId format: snap_[ulid])
status: Current execution status
data: Serialized RunExecutionData containing context
heartbeatAt: Last heartbeat timestamp (for stall detection)
completedAfter: Conditions for resuming suspended run

Snapshot Heartbeat Monitoring:

Each snapshot has a heartbeat timeout based on status
Timeouts configured in HeartbeatTimeouts type
If heartbeat not updated within timeout, snapshot is considered "stalled"
Stalled snapshots trigger recovery workflows (repair, cancel, etc.)

Default Heartbeat Timeouts:

Snapshot Lifecycle:

Sources: internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts16-292 internal-packages/run-engine/src/engine/types.ts66-71

Waitpoints

A waitpoint is a synchronization primitive that allows a run to pause execution and wait for external events before continuing.

Code Representation:

Database model: Waitpoint in @trigger.dev/database
System: WaitpointSystem in internal-packages/run-engine/src/engine/systems/waitpointSystem.ts40
ID format: wait_[ulid] via WaitpointId type

Waitpoint Types:

Type	Description	Use Case
`TASK_RUN`	Wait for child task run to complete	`triggerAndWait()`
`TASK_RUN_BATCH`	Wait for batch of runs to complete	`batchTriggerAndWait()`
`WEBHOOK`	Wait for webhook callback	Human-in-the-loop workflows
`TOKEN`	Wait for token completion	Custom external triggers

Waitpoint States:

Key Properties:

id: Unique waitpoint ID
type: Waitpoint type enum
status: Current state (PENDING, COMPLETED, EXPIRED, CANCELED)
taskRunId: Associated run that is waiting
completedAfter: Optional time threshold before completion
output: Result data after completion (stored as IOPacket)

Blocking Behavior:

When a waitpoint is created, the run transitions to EXECUTING_WITH_WAITPOINTS
Run cannot complete until all waitpoints are resolved
Multiple waitpoints can block a single run
Waitpoints are checked via continueRunIfUnblocked() method

Wait Token Flow:

Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts40-584 packages/trigger-sdk/CHANGELOG.md142-143

Checkpoints

A checkpoint is a snapshot of a run's state that allows it to be suspended, shutdown, and later resumed from the same point.

Code Representation:

Database model: TaskRunCheckpoint in @trigger.dev/database
System: CheckpointSystem in internal-packages/run-engine/src/engine/systems/checkpointSystem.ts15
ID format: check_[ulid] via CheckpointId type

Checkpoint Types:

Type	Description	When Created
`MANUAL`	Explicitly requested via SDK	`checkpoint()` API call
`DURATION`	Auto-created by threshold	Task running > `CHECKPOINT_THRESHOLD_IN_MS`

Key Properties:

id: Unique checkpoint ID
type: Checkpoint type enum
reason: Why checkpoint was created
data: Serialized execution state
createdAt: When checkpoint was taken
restoredAt: When checkpoint was resumed (if applicable)

Checkpoint Lifecycle:

Checkpoint Threshold:

Configured via CHECKPOINT_THRESHOLD_IN_MS environment variable (default: 30 seconds)
When a task runs longer than threshold, automatic checkpoint may be created
Prevents long-running tasks from blocking worker resources
Allows horizontal scaling by releasing worker during suspension

Concurrency Release:

When run is checkpointed and enters WAITING state, concurrency slots are released
Both environment-level and queue-level concurrency is freed
Upon resume, concurrency is re-acquired before execution continues

Sources: internal-packages/run-engine/src/engine/systems/checkpointSystem.ts15-226 apps/webapp/app/env.server.ts405 packages/trigger-sdk/CHANGELOG.md252-257

Queues

A queue manages the ordering and concurrency of task execution. Each task is assigned to a queue, which controls how many runs can execute concurrently.

Code Representation:

Database model: TaskQueue in @trigger.dev/database
Queue system: RunQueue in internal-packages/run-engine/src/run-queue/index.ts
Key producer: RunQueueFullKeyProducer in internal-packages/run-engine/src/run-queue/keyProducer.ts

Queue Properties:

name: Queue identifier (must be URL-safe via sanitizeQueueName())
concurrencyLimit: Max concurrent runs for this queue
rateLimit: Optional rate limiting configuration
Environment-level concurrency also applies (both must have capacity)

Queue Types:

Type	Description	Concurrency Scope
Task-specific	One queue per task	Task-level concurrency control
Shared	Multiple tasks share queue	Shared concurrency limit
Default	Fallback queue	Environment default

Fair Queue Selection Strategy:

The system uses a FairQueueSelectionStrategy to fairly distribute execution capacity across queues:

Queue Selection Algorithm:

Identify queues with available capacity (concurrency limit not reached)
Apply bias factors to score each queue:
- Higher weight for queues with larger concurrency limits
- Higher weight for queues with more available capacity
- Random factor to prevent starvation
Select queue with highest score
Dequeue runs from selected queue

Concurrency Enforcement:

Sources: internal-packages/run-engine/src/run-queue/index.ts internal-packages/run-engine/src/run-queue/fairQueueSelectionStrategy.ts internal-packages/run-engine/src/engine/index.ts142-185

Batches

A batch is a coordinated group of task runs triggered together, with shared completion tracking.

Code Representation:

Database model: Batch in @trigger.dev/database
System: BatchSystem in internal-packages/run-engine/src/engine/systems/batchSystem.ts
Queue: BatchQueue using DRR algorithm in internal-packages/run-engine/src/batch-queue/
ID format: batch_[ulid] via BatchId type

Batch Properties:

id: Unique batch ID
friendlyId: Human-readable ID
status: Current state (PENDING, PROCESSING, COMPLETED, FAILED, CANCELED)
items: Array of batch items (runs)
concurrencyLimit: Max concurrent items in this batch

Batch Triggering Methods:

Method	Wait for Completion?	Use Case
`batchTrigger()`	No	Fire-and-forget multiple runs
`batchTriggerAndWait()`	Yes	Wait for all runs to complete

Batch Completion Tracking:

DRR (Deficit Round Robin) Batch Queue:

The batch queue uses a Deficit Round Robin algorithm to fairly schedule batch items across multiple batches:

Each batch has a "quantum" (default: 5 items per round)
Each batch accumulates a "deficit" counter
When deficit >= quantum, dequeue up to quantum items
Prevents large batches from starving small batches

Batch Item States:

State	Description
`PENDING`	Waiting to be triggered
`TRIGGERED`	Run has been created
`COMPLETED`	Run completed successfully
`FAILED`	Run failed
`CANCELED`	Run was canceled

Waitpoint Integration:

When using batchTriggerAndWait():

Batch is created with type=TASK_RUN_BATCH
Waitpoint is created with type=TASK_RUN_BATCH
Parent run enters WAITING state
When all batch items complete, waitpoint is resolved
Parent run resumes execution

Sources: internal-packages/run-engine/src/batch-queue/types.ts internal-packages/run-engine/src/engine/systems/batchSystem.ts internal-packages/run-engine/src/engine/index.ts331-358

Deployments

A deployment represents a versioned release of tasks to a specific environment. Each deployment contains a Docker image with task code and dependencies.

Code Representation:

Database model: WorkerDeployment in @trigger.dev/database
CLI command: trigger deploy in packages/cli-v3/

Deployment Properties:

id: Unique deployment ID
friendlyId: Human-readable ID (e.g., deploy_abc123)
version: Semantic version string
imageReference: Docker image URI
status: Current deployment state
promotedAt: When deployment was promoted to live
worker: Associated BackgroundWorker record

Deployment States:

Deployment Process Flow:

Image Build Methods:

Method	Location	Use Case
Local	Developer machine	Quick deploys, local Docker
Depot	Remote build service	Faster builds, no local Docker needed
Native Build Server	Platform-side	CI/CD pipelines

Version Labels:

Deployments can be labeled with the CURRENT_DEPLOYMENT_LABEL constant (value: "latest") to indicate the active version for task lookups.

Task Version Pinning:

Runs can be locked to specific deployment versions using:

lockedToVersionId: Specific deployment ID
taskVersion: Semantic version string

This ensures runs execute on the exact code version they were triggered with, even after new deployments.

Sources: packages/cli-v3/CHANGELOG.md73-94 internal-packages/run-engine/src/engine/systems/dequeueSystem.ts151-224 apps/webapp/app/env.server.ts290-343

Core Systems

RunEngine

The RunEngine is the central orchestrator for task execution. It coordinates all subsystems and manages the complete lifecycle of runs.

Code Representation:

Class: RunEngine in internal-packages/run-engine/src/engine/index.ts76
Singleton: engine in apps/webapp/app/v3/runEngine.server.ts11

Key Responsibilities:

Run triggering and queueing
Worker dequeue coordination
Attempt lifecycle management
Concurrency enforcement
State persistence
Recovery from failures

Major Subsystems:

RunEngine Initialization:

Event Bus:

RunEngine uses an EventBus (EventEmitter) to coordinate between systems:

Events: runStarted, runCompleted, runFailed, runCanceled, waitpointCreated, waitpointCompleted
Used for cross-system notifications (e.g., batch completion, waitpoint resolution)

Worker Catalog:

The workerCatalog defines background jobs processed by RunEngine:

finishWaitpoint: Complete a waitpoint
heartbeatSnapshot: Check for stalled snapshots
repairSnapshot: Recover from snapshot errors
expireRun: Handle TTL expiration
cancelRun: Cancel a run
queueRunsPendingVersion: Enqueue runs waiting for deployment
tryCompleteBatch: Attempt batch completion
continueRunIfUnblocked: Resume run if waitpoints resolved
enqueueDelayedRun: Trigger delayed run

Sources: internal-packages/run-engine/src/engine/index.ts76-384 internal-packages/run-engine/src/engine/types.ts23-171 apps/webapp/app/v3/runEngine.server.ts11-130

Data Models

Run Context

The run context (TaskRunContext) is the object passed to the task's run() function and lifecycle hooks, providing access to run metadata and utilities.

Code Representation:

Type: TaskRunContext in @trigger.dev/core/v3/schemas
Also exposed as ctx parameter in task functions

Context Properties:

Backwards Compatibility:

The BackwardsCompatibleTaskRunExecution type includes deprecated fields for compatibility:

task.exportName: Now always provided
attempt.id, attempt.backgroundWorkerId, attempt.backgroundWorkerTaskId, attempt.status: Legacy fields
run.context: Deprecated (use ctx instead)
run.durationMs, run.costInCents, run.baseCostInCents: Computed fields

Sources: internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts75-91 packages/core/CHANGELOG.md233-246

Metadata

Metadata is user-defined key-value data associated with runs, enabling runtime state sharing and hierarchical data propagation.

Code Representation:

API: metadata object from @trigger.dev/sdk
Storage: FlushedRunMetadata type, stored as TaskRun.metadata JSON column

Metadata Scopes:

Scope	Method	Description
Current	`metadata.set(key, value)`	Set on current run
Current	`metadata.get(key)`	Get from current run
Current	`metadata.current()`	Get all current run metadata
Parent	`metadata.parent.set(key, value)`	Set on parent run
Parent	`metadata.parent.get(key)`	Get from parent run
Root	`metadata.root.set(key, value)`	Set on root run
Root	`metadata.root.get(key)`	Get from root run

Metadata Flushing:

Metadata operations are batched and flushed periodically for performance:

Flush interval: BATCH_METADATA_OPERATIONS_FLUSH_INTERVAL_MS (default: 1000ms)
Operations are collapsed: Multiple sets to same key are merged
Flush on run completion: Guarantees metadata is saved

Metadata Streaming:

The metadata.stream() API enables real-time metadata updates to frontends:

Creates a readable stream of metadata changes
Integrates with Realtime API
Automatically flushes on stream operations

Sources: packages/core/CHANGELOG.md87-88 apps/webapp/app/env.server.ts555-557

Trace Context

Trace context enables correlation between Trigger.dev spans and external OpenTelemetry traces.

Code Representation:

Type: TriggerTraceContext in @trigger.dev/core/v3
Properties: traceparent, tracestate (W3C Trace Context format)

Trace Propagation:

Trace Context Format:

traceparent: 00-{traceId}-{spanId}-{flags}
tracestate: vendor1=value1,vendor2=value2

External Trace Integration:

The SDK provides otel.withExternalTrace() to propagate trace context when calling external APIs:

This ensures external traces are correlated with Trigger.dev spans in APM tools (Datadog, New Relic, Axiom, etc.).

Sources: packages/trigger-sdk/CHANGELOG.md162-229 packages/core/CHANGELOG.md319-403

Summary: Entity Relationships

Sources: All sections above

Key Concepts

Relevant source files

For information about the overall system architecture and component interactions, see System Architecture.

Core Entities

Tasks

A task is the fundamental unit of work in Trigger.dev. Tasks are defined using the SDK and represent reusable, retriggerable operations that can be executed asynchronously.

Code Representation:

SDK function: task(), schemaTask(), toolTask() in @trigger.dev/sdk
Database model: BackgroundWorkerTask in @trigger.dev/database
Identifier: taskIdentifier (string, unique per environment)

Key Properties:

id: Unique identifier for the task
exportName: The exported name from the task file
filePath: Location of the task definition
queue: Associated queue configuration
retry: Retry configuration with limit, factor, minTimeout, maxTimeout
machine: Machine preset specification (CPU/memory allocation)

Task Definition State Machine (Deployment):

Sources: packages/trigger-sdk/package.json internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts44-91

Runs

A run (also called task run) is a single execution instance of a task. Each time a task is triggered, a new run is created.

Code Representation:

Database model: TaskRun in @trigger.dev/database
Run engine class: RunEngine in internal-packages/run-engine/src/engine/index.ts76
ID format: run_[ulid] via RunId type

Run Lifecycle States:

Status	Description	Counts Against Concurrency?
`PENDING_VERSION`	Waiting for worker deployment	No
`QUEUED`	Waiting in queue for execution	No
`DELAYED`	Scheduled for future execution	No
`DEQUEUED`	Removed from queue, being sent to worker	Yes
`EXECUTING`	Currently running	Yes
`WAITING`	Paused (waitpoint/checkpoint)	No
`COMPLETED`	Successfully finished	No
`CANCELED`	User-canceled	No
`FAILED`	Failed after retries	No
`CRASHED`	Worker crashed (OOM, etc.)	No
`SYSTEM_FAILURE`	Platform error	No
`EXPIRED`	TTL exceeded	No
`TIMED_OUT`	maxDuration exceeded	No

Key Properties:

id: Unique run identifier
friendlyId: Human-readable ID (e.g., run_abc123)
status: Current state in the state machine
payload: Input data for the run
output: Result data after completion
metadata: User-defined key-value pairs
context: Execution context (attempt, run IDs, environment)
traceId, spanId: OpenTelemetry correlation IDs

Run State Machine:

Boolean Helper Methods:

isQueued(): Returns true for QUEUED, PENDING_VERSION, DELAYED
isExecuting(): Returns true for EXECUTING, DEQUEUED
isWaiting(): Returns true for WAITING
isCompleted(): Returns true for any terminal state
isSuccess(): Returns true for COMPLETED
isFailed(): Returns true for FAILED, CRASHED, SYSTEM_FAILURE, TIMED_OUT
isCanceled(): Returns true for CANCELED

Sources: internal-packages/run-engine/src/engine/index.ts388-523 internal-packages/run-engine/src/engine/statuses.ts packages/trigger-sdk/CHANGELOG.md446-486

Attempts

An attempt is a single try at executing a run. A run may have multiple attempts if retries are configured.

Code Representation:

Database model: TaskRunAttempt in @trigger.dev/database
System: RunAttemptSystem in internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts63

Key Properties:

number: Sequential attempt number (1, 2, 3, ...)
status: Current state (mirroring run status during execution)
startedAt: When this attempt began
completedAt: When this attempt finished
error: Error information if failed
backgroundWorkerId: Worker that executed this attempt

Retry Logic:

Exponential backoff with configurable factor, minTimeout, maxTimeout
Maximum attempts: 25 (defined as MAX_TASK_RUN_ATTEMPTS)
Retry decision made in retryOutcomeFromCompletion() function
Warm start optimization: If previous attempt finished recently (< retryWarmStartThresholdMs), reuse the worker process

Sources: internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts63-673 internal-packages/run-engine/src/engine/consts.js1

Workers

Workers are processes that execute tasks. There are two types:

Development Workers

Created by trigger dev CLI command
Run locally on developer machines
Use DevSupervisor class to manage task execution
Poll for runs via SSE connection to platform

Production Workers

Created during deployment via trigger deploy
Run in containerized environments (Docker images)
Managed by the platform's worker orchestration
Two worker types:
- BackgroundWorker (v3): Older architecture, managed separately
- Worker (v4): Newer Run Engine 2.0 architecture

Code Representation:

Database model: BackgroundWorker in @trigger.dev/database
Supervisor: Supervisor class in apps/coordinator
Task executor: TaskExecutor in apps/supervisor

Worker Registration Flow:

Sources: apps/webapp/app/runEngine/services/triggerTask.server.ts internal-packages/run-engine/src/engine/systems/dequeueSystem.ts88

Environments

An environment represents a deployment target with its own configuration, API keys, and concurrency limits.

Code Representation:

Database model: RuntimeEnvironment in @trigger.dev/database
Type enum: RuntimeEnvironmentType with values DEVELOPMENT, PRODUCTION, STAGING, PREVIEW

Environment Types:

Type	Purpose	Concurrency Default	API Key Prefix
`DEVELOPMENT`	Local dev runs	10	`tr_dev_`
`PRODUCTION`	Production workloads	100	`tr_prod_`
`STAGING`	Pre-production testing	50	`tr_stag_`
`PREVIEW`	Branch previews	25	`tr_prev_`

Key Properties:

id: Unique environment ID
type: Environment type enum
slug: URL-safe identifier
apiKey: Authentication token
pkApiKey: Public API key (for frontend)
organizationId: Parent organization
projectId: Parent project

Concurrency Configuration:

maximumConcurrencyLimit: Environment-wide concurrency cap
concurrencyBurstFactor: Multiplier for burst capacity (default: 2.0x)
Effective burst limit = maximumConcurrencyLimit * concurrencyBurstFactor

Sources: apps/webapp/app/env.server.ts263-267 internal-packages/run-engine/src/engine/index.ts146-161

Organizations and Projects

Organizations are top-level tenant entities that contain projects. Projects group related tasks and environments.

Code Representation:

Database models: Organization, Project in @trigger.dev/database
Hierarchy: Organization → Project → RuntimeEnvironment → TaskRun

Organization:

id: Unique org ID
slug: URL-safe identifier
title: Display name
Members: OrgMember records with roles

Project:

id: Unique project ID
slug: URL-safe identifier (unique within org)
externalRef: External reference ID (e.g., proj_abc123)
organizationId: Parent organization

Organizational Hierarchy:

Sources: apps/webapp/app/models/organization.server.ts apps/webapp/app/models/project.server.ts

Execution Concepts

Lifecycle Hooks

Lifecycle hooks are functions that execute at specific points during task execution, providing extension points for cross-cutting concerns.

Available Hooks (12 total):

Hook	When It Fires	Use Case
`init`	Once per worker process startup	Initialize connections, load config
`onStartAttempt`	Before each attempt's `run()` function	Per-attempt setup, logging
`onStart`	Before first attempt only (deprecated)	Use `onStartAttempt` with `ctx.run.attempt.number === 1`
`middleware`	Around the `run()` function	Wrapping logic, timing
`onSuccess`	After successful completion	Cleanup, notifications
`onFailure`	After failed attempt	Error handling, alerts
`onComplete`	After any completion (success/failure)	Guaranteed cleanup
`cleanup`	After attempt finishes (always runs)	Resource cleanup
`onWait`	When entering WAITING state	Log waitpoint entry
`onResume`	When resuming from WAITING	Log resume, reset state
`onCancel`	When run is canceled	Cancelation handling
`catchError`	When uncaught error occurs	Error transformation

Hook Execution Order:

Global vs Task-Level Hooks:

Task-level: Defined in task() options
Global: Defined using tasks.onStartAttempt(), tasks.onSuccess(), etc.

Error Handling in Hooks:

Errors in onSuccess, onFailure, onComplete do NOT fail the run (logged only)
Errors in middleware, run() DO fail the run
catchError can transform errors before retry logic

Sources: packages/trigger-sdk/CHANGELOG.md38-73 packages/core/CHANGELOG.md98

Execution Snapshots

An execution snapshot captures the state of a run at a point in time. Snapshots enable state recovery, monitoring, and debugging.

Code Representation:

Database model: TaskRunExecutionSnapshot in @trigger.dev/database
System: ExecutionSnapshotSystem in internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts16

Snapshot Types (by status):

Status	Description	Triggers Action
`PENDING`	Initial creation	No
`EXECUTING`	Run started	No
`EXECUTING_WITH_WAITPOINTS`	Has blocking waitpoints	No
`SUSPENDED`	Checkpointed, worker shutdown	Resume when ready
`DEQUEUED`	Removed from queue	No

Key Properties:

id: Unique snapshot ID (SnapshotId format: snap_[ulid])
status: Current execution status
data: Serialized RunExecutionData containing context
heartbeatAt: Last heartbeat timestamp (for stall detection)
completedAfter: Conditions for resuming suspended run

Snapshot Heartbeat Monitoring:

Each snapshot has a heartbeat timeout based on status
Timeouts configured in HeartbeatTimeouts type
If heartbeat not updated within timeout, snapshot is considered "stalled"
Stalled snapshots trigger recovery workflows (repair, cancel, etc.)

Default Heartbeat Timeouts:

Snapshot Lifecycle:

Sources: internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts16-292 internal-packages/run-engine/src/engine/types.ts66-71

Waitpoints

A waitpoint is a synchronization primitive that allows a run to pause execution and wait for external events before continuing.

Code Representation:

Database model: Waitpoint in @trigger.dev/database
System: WaitpointSystem in internal-packages/run-engine/src/engine/systems/waitpointSystem.ts40
ID format: wait_[ulid] via WaitpointId type

Waitpoint Types:

Type	Description	Use Case
`TASK_RUN`	Wait for child task run to complete	`triggerAndWait()`
`TASK_RUN_BATCH`	Wait for batch of runs to complete	`batchTriggerAndWait()`
`WEBHOOK`	Wait for webhook callback	Human-in-the-loop workflows
`TOKEN`	Wait for token completion	Custom external triggers

Waitpoint States:

Key Properties:

id: Unique waitpoint ID
type: Waitpoint type enum
status: Current state (PENDING, COMPLETED, EXPIRED, CANCELED)
taskRunId: Associated run that is waiting
completedAfter: Optional time threshold before completion
output: Result data after completion (stored as IOPacket)

Blocking Behavior:

When a waitpoint is created, the run transitions to EXECUTING_WITH_WAITPOINTS
Run cannot complete until all waitpoints are resolved
Multiple waitpoints can block a single run
Waitpoints are checked via continueRunIfUnblocked() method

Wait Token Flow:

Sources: internal-packages/run-engine/src/engine/systems/waitpointSystem.ts40-584 packages/trigger-sdk/CHANGELOG.md142-143

Checkpoints

A checkpoint is a snapshot of a run's state that allows it to be suspended, shutdown, and later resumed from the same point.

Code Representation:

Database model: TaskRunCheckpoint in @trigger.dev/database
System: CheckpointSystem in internal-packages/run-engine/src/engine/systems/checkpointSystem.ts15
ID format: check_[ulid] via CheckpointId type

Checkpoint Types:

Type	Description	When Created
`MANUAL`	Explicitly requested via SDK	`checkpoint()` API call
`DURATION`	Auto-created by threshold	Task running > `CHECKPOINT_THRESHOLD_IN_MS`

Key Properties:

id: Unique checkpoint ID
type: Checkpoint type enum
reason: Why checkpoint was created
data: Serialized execution state
createdAt: When checkpoint was taken
restoredAt: When checkpoint was resumed (if applicable)

Checkpoint Lifecycle:

Checkpoint Threshold:

Configured via CHECKPOINT_THRESHOLD_IN_MS environment variable (default: 30 seconds)
When a task runs longer than threshold, automatic checkpoint may be created
Prevents long-running tasks from blocking worker resources
Allows horizontal scaling by releasing worker during suspension

Concurrency Release:

When run is checkpointed and enters WAITING state, concurrency slots are released
Both environment-level and queue-level concurrency is freed
Upon resume, concurrency is re-acquired before execution continues

Sources: internal-packages/run-engine/src/engine/systems/checkpointSystem.ts15-226 apps/webapp/app/env.server.ts405 packages/trigger-sdk/CHANGELOG.md252-257

Queues

A queue manages the ordering and concurrency of task execution. Each task is assigned to a queue, which controls how many runs can execute concurrently.

Code Representation:

Database model: TaskQueue in @trigger.dev/database
Queue system: RunQueue in internal-packages/run-engine/src/run-queue/index.ts
Key producer: RunQueueFullKeyProducer in internal-packages/run-engine/src/run-queue/keyProducer.ts

Queue Properties:

name: Queue identifier (must be URL-safe via sanitizeQueueName())
concurrencyLimit: Max concurrent runs for this queue
rateLimit: Optional rate limiting configuration
Environment-level concurrency also applies (both must have capacity)

Queue Types:

Type	Description	Concurrency Scope
Task-specific	One queue per task	Task-level concurrency control
Shared	Multiple tasks share queue	Shared concurrency limit
Default	Fallback queue	Environment default

Fair Queue Selection Strategy:

The system uses a FairQueueSelectionStrategy to fairly distribute execution capacity across queues:

Queue Selection Algorithm:

Identify queues with available capacity (concurrency limit not reached)
Apply bias factors to score each queue:
- Higher weight for queues with larger concurrency limits
- Higher weight for queues with more available capacity
- Random factor to prevent starvation
Select queue with highest score
Dequeue runs from selected queue

Concurrency Enforcement:

Sources: internal-packages/run-engine/src/run-queue/index.ts internal-packages/run-engine/src/run-queue/fairQueueSelectionStrategy.ts internal-packages/run-engine/src/engine/index.ts142-185

Batches

A batch is a coordinated group of task runs triggered together, with shared completion tracking.

Code Representation:

Database model: Batch in @trigger.dev/database
System: BatchSystem in internal-packages/run-engine/src/engine/systems/batchSystem.ts
Queue: BatchQueue using DRR algorithm in internal-packages/run-engine/src/batch-queue/
ID format: batch_[ulid] via BatchId type

Batch Properties:

id: Unique batch ID
friendlyId: Human-readable ID
status: Current state (PENDING, PROCESSING, COMPLETED, FAILED, CANCELED)
items: Array of batch items (runs)
concurrencyLimit: Max concurrent items in this batch

Batch Triggering Methods:

Method	Wait for Completion?	Use Case
`batchTrigger()`	No	Fire-and-forget multiple runs
`batchTriggerAndWait()`	Yes	Wait for all runs to complete

Batch Completion Tracking:

DRR (Deficit Round Robin) Batch Queue:

The batch queue uses a Deficit Round Robin algorithm to fairly schedule batch items across multiple batches:

Each batch has a "quantum" (default: 5 items per round)
Each batch accumulates a "deficit" counter
When deficit >= quantum, dequeue up to quantum items
Prevents large batches from starving small batches

Batch Item States:

State	Description
`PENDING`	Waiting to be triggered
`TRIGGERED`	Run has been created
`COMPLETED`	Run completed successfully
`FAILED`	Run failed
`CANCELED`	Run was canceled

Waitpoint Integration:

When using batchTriggerAndWait():

Batch is created with type=TASK_RUN_BATCH
Waitpoint is created with type=TASK_RUN_BATCH
Parent run enters WAITING state
When all batch items complete, waitpoint is resolved
Parent run resumes execution

Sources: internal-packages/run-engine/src/batch-queue/types.ts internal-packages/run-engine/src/engine/systems/batchSystem.ts internal-packages/run-engine/src/engine/index.ts331-358

Deployments

A deployment represents a versioned release of tasks to a specific environment. Each deployment contains a Docker image with task code and dependencies.

Code Representation:

Database model: WorkerDeployment in @trigger.dev/database
CLI command: trigger deploy in packages/cli-v3/

Deployment Properties:

id: Unique deployment ID
friendlyId: Human-readable ID (e.g., deploy_abc123)
version: Semantic version string
imageReference: Docker image URI
status: Current deployment state
promotedAt: When deployment was promoted to live
worker: Associated BackgroundWorker record

Deployment States:

Deployment Process Flow:

Image Build Methods:

Method	Location	Use Case
Local	Developer machine	Quick deploys, local Docker
Depot	Remote build service	Faster builds, no local Docker needed
Native Build Server	Platform-side	CI/CD pipelines

Version Labels:

Deployments can be labeled with the CURRENT_DEPLOYMENT_LABEL constant (value: "latest") to indicate the active version for task lookups.

Task Version Pinning:

Runs can be locked to specific deployment versions using:

lockedToVersionId: Specific deployment ID
taskVersion: Semantic version string

This ensures runs execute on the exact code version they were triggered with, even after new deployments.

Sources: packages/cli-v3/CHANGELOG.md73-94 internal-packages/run-engine/src/engine/systems/dequeueSystem.ts151-224 apps/webapp/app/env.server.ts290-343

Core Systems

RunEngine

The RunEngine is the central orchestrator for task execution. It coordinates all subsystems and manages the complete lifecycle of runs.

Code Representation:

Class: RunEngine in internal-packages/run-engine/src/engine/index.ts76
Singleton: engine in apps/webapp/app/v3/runEngine.server.ts11

Key Responsibilities:

Run triggering and queueing
Worker dequeue coordination
Attempt lifecycle management
Concurrency enforcement
State persistence
Recovery from failures

Major Subsystems:

RunEngine Initialization:

Event Bus:

RunEngine uses an EventBus (EventEmitter) to coordinate between systems:

Events: runStarted, runCompleted, runFailed, runCanceled, waitpointCreated, waitpointCompleted
Used for cross-system notifications (e.g., batch completion, waitpoint resolution)

Worker Catalog:

The workerCatalog defines background jobs processed by RunEngine:

finishWaitpoint: Complete a waitpoint
heartbeatSnapshot: Check for stalled snapshots
repairSnapshot: Recover from snapshot errors
expireRun: Handle TTL expiration
cancelRun: Cancel a run
queueRunsPendingVersion: Enqueue runs waiting for deployment
tryCompleteBatch: Attempt batch completion
continueRunIfUnblocked: Resume run if waitpoints resolved
enqueueDelayedRun: Trigger delayed run

Sources: internal-packages/run-engine/src/engine/index.ts76-384 internal-packages/run-engine/src/engine/types.ts23-171 apps/webapp/app/v3/runEngine.server.ts11-130

Data Models

Run Context

The run context (TaskRunContext) is the object passed to the task's run() function and lifecycle hooks, providing access to run metadata and utilities.

Code Representation:

Type: TaskRunContext in @trigger.dev/core/v3/schemas
Also exposed as ctx parameter in task functions

Context Properties:

Backwards Compatibility:

The BackwardsCompatibleTaskRunExecution type includes deprecated fields for compatibility:

task.exportName: Now always provided
attempt.id, attempt.backgroundWorkerId, attempt.backgroundWorkerTaskId, attempt.status: Legacy fields
run.context: Deprecated (use ctx instead)
run.durationMs, run.costInCents, run.baseCostInCents: Computed fields

Sources: internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts75-91 packages/core/CHANGELOG.md233-246

Metadata

Metadata is user-defined key-value data associated with runs, enabling runtime state sharing and hierarchical data propagation.

Code Representation:

API: metadata object from @trigger.dev/sdk
Storage: FlushedRunMetadata type, stored as TaskRun.metadata JSON column

Metadata Scopes:

Scope	Method	Description
Current	`metadata.set(key, value)`	Set on current run
Current	`metadata.get(key)`	Get from current run
Current	`metadata.current()`	Get all current run metadata
Parent	`metadata.parent.set(key, value)`	Set on parent run
Parent	`metadata.parent.get(key)`	Get from parent run
Root	`metadata.root.set(key, value)`	Set on root run
Root	`metadata.root.get(key)`	Get from root run

Metadata Flushing:

Metadata operations are batched and flushed periodically for performance:

Flush interval: BATCH_METADATA_OPERATIONS_FLUSH_INTERVAL_MS (default: 1000ms)
Operations are collapsed: Multiple sets to same key are merged
Flush on run completion: Guarantees metadata is saved

Metadata Streaming:

The metadata.stream() API enables real-time metadata updates to frontends:

Creates a readable stream of metadata changes
Integrates with Realtime API
Automatically flushes on stream operations

Sources: packages/core/CHANGELOG.md87-88 apps/webapp/app/env.server.ts555-557

Trace Context

Trace context enables correlation between Trigger.dev spans and external OpenTelemetry traces.

Code Representation:

Type: TriggerTraceContext in @trigger.dev/core/v3
Properties: traceparent, tracestate (W3C Trace Context format)

Trace Propagation:

Trace Context Format:

traceparent: 00-{traceId}-{spanId}-{flags}
tracestate: vendor1=value1,vendor2=value2

External Trace Integration:

The SDK provides otel.withExternalTrace() to propagate trace context when calling external APIs:

This ensures external traces are correlated with Trigger.dev spans in APM tools (Datadog, New Relic, Axiom, etc.).

Sources: packages/trigger-sdk/CHANGELOG.md162-229 packages/core/CHANGELOG.md319-403

Summary: Entity Relationships

Sources: All sections above

Key Concepts

Core Entities

Tasks

Runs

Attempts

Workers

Development Workers

Production Workers

Environments

Organizations and Projects

Execution Concepts

Lifecycle Hooks

Execution Snapshots

Waitpoints

Checkpoints

Queues

Batches

Deployments

Core Systems

RunEngine

Data Models

Run Context

Metadata

Trace Context

Summary: Entity Relationships

On this page

Key Concepts

Core Entities

Tasks

Runs

Attempts

Workers

Development Workers

Production Workers

Environments

Organizations and Projects

Execution Concepts

Lifecycle Hooks

Execution Snapshots

Waitpoints

Checkpoints

Queues

Batches

Deployments

Core Systems

RunEngine

Data Models

Run Context

Metadata

Trace Context

Summary: Entity Relationships

On this page